Skip to contentSkip to conversation

crypto: HCTR2 support

RFC PATCH Superseded Nathan Huckleberry 4 participants 19 messages Feb 10, 2022 – Feb 19, 2022

Subsystems (4)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover crypto: HCTR2 support

HCTR2 is a length-preserving encryption mode that is efficient on processors with instructions to accelerate AES and carryless multiplication, e.g. x86 processors with AES-NI and CLMUL, and ARM processors with the ARMv8 Crypto Extensions.

HCTR2 is specified in https://ia.cr/2021/1441 "Length-preserving encryption with HCTR2" which shows that if AES is secure and HCTR2 is instantiated with AES, then HCTR2 is secure. Reference code and test vectors are at https://github.com/google/hctr2.

As a length-preserving encryption mode, HCTR2 is suitable for applications such as storage encryption where ciphertext expansion is not possible, and thus authenticated encryption cannot be used. Currently, such applications usually use XTS, or in some cases Adiantum. XTS has the disadvantage that it is a narrow-block mode: a bitflip will only change 16 bytes in the resulting ciphertext or plaintext. This reveals more information to an attacker than necessary.

HCTR2 is a wide-block mode, so it provides a stronger security property: a bitflip will change the entire message. HCTR2 is somewhat similar to Adiantum, which is also a wide-block mode. However, HCTR2 is designed to take advantage of existing crypto instructions, while Adiantum targets devices without such hardware support. Adiantum is also designed with longer messages in mind, while HCTR2 is designed to be efficient even on short messages.

The first intended use of this mode in the kernel is for the encryption of filenames, where for efficiency reasons encryption must be fully deterministic (only one ciphertext for each plaintext) and the existing CBC solution leaks more information than necessary for filenames with common prefixes.

HCTR2 uses two passes of an ε-almost-∆-universal hash function called POLYVAL and one pass of a block cipher mode called XCTR. POLYVAL is a polynomial hash designed for efficiency on modern processors and was originally specified for use in AES-GCM-SIV (RFC 8452). XCTR mode is a variant of CTR mode that is more efficient on little-endian machines.

This patchset adds HCTR2 to Linux's crypto API, including generic implementations of XCTR and POLYVAL, hardware accelerated implementations of XCTR and POLYVAL for both x86-64 and ARM64, and a templated implementation of HCTR2.

Nathan Huckleberry (7):
crypto: xctr - Add XCTR support
crypto: polyval - Add POLYVAL support
crypto: hctr2 - Add HCTR2 support
crypto: x86/aesni-xctr: Add accelerated implementation of XCTR
crypto: arm64/aes-xctr: Add accelerated implementation of XCTR
crypto: x86/polyval: Add PCLMULQDQ accelerated implementation of
POLYVAL
crypto: arm64/polyval: Add PMULL accelerated implementation of POLYVAL

 arch/arm64/crypto/Kconfig                |   11 +-
 arch/arm64/crypto/Makefile               |    3 +
 arch/arm64/crypto/aes-glue.c             |   72 +-
 arch/arm64/crypto/aes-modes.S            |  130 ++
 arch/arm64/crypto/polyval-ce-core.S      |  405 ++++++
 arch/arm64/crypto/polyval-ce-glue.c      |  365 ++++++
 arch/x86/crypto/Makefile                 |    5 +-
 arch/x86/crypto/aes_xctrby8_avx-x86_64.S |  529 ++++++++
 arch/x86/crypto/aesni-intel_asm.S        |   70 +
 arch/x86/crypto/aesni-intel_glue.c       |   89 ++
 arch/x86/crypto/polyval-clmulni_asm.S    |  414 ++++++
 arch/x86/crypto/polyval-clmulni_glue.c   |  365 ++++++
 crypto/Kconfig                           |   38 +
 crypto/Makefile                          |    3 +
 crypto/hctr2.c                           |  532 ++++++++
 crypto/polyval-generic.c                 |  199 +++
 crypto/tcrypt.c                          |   10 +
 crypto/testmgr.c                         |   20 +
 crypto/testmgr.h                         | 1500 ++++++++++++++++++++++
 crypto/xctr.c                            |  193 +++
 include/crypto/polyval.h                 |   22 +
 21 files changed, 4970 insertions(+), 5 deletions(-)
 create mode 100644 arch/arm64/crypto/polyval-ce-core.S
 create mode 100644 arch/arm64/crypto/polyval-ce-glue.c
 create mode 100644 arch/x86/crypto/aes_xctrby8_avx-x86_64.S
 create mode 100644 arch/x86/crypto/polyval-clmulni_asm.S
 create mode 100644 arch/x86/crypto/polyval-clmulni_glue.c
 create mode 100644 crypto/hctr2.c
 create mode 100644 crypto/polyval-generic.c
 create mode 100644 crypto/xctr.c
 create mode 100644 include/crypto/polyval.h
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#1 lore raw

RFC PATCH v2 1/7 crypto: xctr - Add XCTR support

Add a generic implementation of XCTR mode as a template. XCTR is a blockcipher mode similar to CTR mode. XCTR uses XORs and little-endian addition rather than big-endian arithmetic which has two advantages: It is slightly faster on little-endian CPUs and it is less likely to be implemented incorrect since integer overflows are not possible on practical input sizes. XCTR is used as a component to implement HCTR2.

More information on XCTR mode can be found in the HCTR2 paper:
https://eprint.iacr.org/2021/1441.pdf

Changes since v1:
* Restricted blocksize to 16-bytes
* Removed xctr.h and u32_to_le_block
* Use single crypto_template instead of array

---
 crypto/Kconfig   |   9 +
 crypto/Makefile  |   1 +
 crypto/tcrypt.c  |   1 +
 crypto/testmgr.c |   6 +
 crypto/testmgr.h | 546 +++++++++++++++++++++++++++++++++++++++++++++++
 crypto/xctr.c    | 193 +++++++++++++++++
 6 files changed, 756 insertions(+)
 create mode 100644 crypto/xctr.c
crypto/Kconfig +9 -0
index fa1741bb568f..8543f34fa200 100644
@@ -452,6 +452,15 @@ config CRYPTO_PCBC
 	  PCBC: Propagating Cipher Block Chaining mode
 	  This block cipher algorithm is required for RxRPC.
 
+config CRYPTO_XCTR
+	tristate
+	select CRYPTO_SKCIPHER
+	select CRYPTO_MANAGER
+	help
+	  XCTR: XOR Counter mode. This blockcipher mode is a variant of CTR mode
+	  using XORs and little-endian addition rather than big-endian arithmetic.
+	  XCTR mode is used to implement HCTR2.
+
 config CRYPTO_XTS
 	tristate "XTS support"
 	select CRYPTO_SKCIPHER
crypto/Makefile +1 -0
index d76bff8d0ffd..6b3fe3df1489 100644
@@ -93,6 +93,7 @@ obj-$(CONFIG_CRYPTO_CTS) += cts.o
 obj-$(CONFIG_CRYPTO_LRW) += lrw.o
 obj-$(CONFIG_CRYPTO_XTS) += xts.o
 obj-$(CONFIG_CRYPTO_CTR) += ctr.o
+obj-$(CONFIG_CRYPTO_XCTR) += xctr.o
 obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o
 obj-$(CONFIG_CRYPTO_ADIANTUM) += adiantum.o
 obj-$(CONFIG_CRYPTO_NHPOLY1305) += nhpoly1305.o
crypto/tcrypt.c +1 -0
index 2a808e843de5..b3a23dbf5b14 100644
@@ -1556,6 +1556,7 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
 		ret += tcrypt_test("rfc3686(ctr(aes))");
 		ret += tcrypt_test("ofb(aes)");
 		ret += tcrypt_test("cfb(aes)");
+		ret += tcrypt_test("xctr(aes)");
 		break;
 
 	case 11:
crypto/testmgr.c +6 -0
index 3a5a3e5cb77b..d2b42ff0b04a 100644
@@ -5451,6 +5451,12 @@ static const struct alg_test_desc alg_test_descs[] = {
 		.suite = {
 			.cipher = __VECS(xchacha20_tv_template)
 		},
+	}, {
+		.alg = "xctr(aes)",
+		.test = alg_test_skcipher,
+		.suite = {
+			.cipher = __VECS(aes_xctr_tv_template)
+		}
 	}, {
 		.alg = "xts(aes)",
 		.generic_driver = "xts(ecb(aes-generic))",
crypto/testmgr.h +546 -0

550 lines not on this page · raw message

crypto/xctr.c new file +193 -0

193 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#2 lore raw

Re: [RFC PATCH v2 1/7] crypto: xctr - Add XCTR support

Quoted 10 lines · Nathan Huckleberry

Add a generic implementation of XCTR mode as a template. XCTR is a blockcipher mode similar to CTR mode. XCTR uses XORs and little-endian addition rather than big-endian arithmetic which has two advantages: It is slightly faster on little-endian CPUs and it is less likely to be implemented incorrect since integer overflows are not possible on practical input sizes. XCTR is used as a component to implement HCTR2.

More information on XCTR mode can be found in the HCTR2 paper:
https://eprint.iacr.org/2021/1441.pdf

Quoted 7 lines

Changes since v1:
* Restricted blocksize to 16-bytes
* Removed xctr.h and u32_to_le_block
* Use single crypto_template instead of array
---

Changelog text conventionally goes in the cover letter, not in the individual patches. Having the changelog be above the scissors line ("---") is especially problematic, as it will show up in the git commit message.

Quoted 6 lines
crypto/Kconfig +0 -0
index fa1741bb568f..8543f34fa200 100644
@@ -452,6 +452,15 @@ config CRYPTO_PCBC
 	  PCBC: Propagating Cipher Block Chaining mode

crypto/Kconfig config CRYPTO_PCBC · line 462

Quoted 10 lines
crypto/Kconfig +8 -0
 	  This block cipher algorithm is required for RxRPC.
 
+config CRYPTO_XCTR
+	tristate
+	select CRYPTO_SKCIPHER
+	select CRYPTO_MANAGER
+	help
+	  XCTR: XOR Counter mode. This blockcipher mode is a variant of CTR mode
+	  using XORs and little-endian addition rather than big-endian arithmetic.
+	  XCTR mode is used to implement HCTR2.

Now that this option isn't user-selectable, no one will see this help text. I think it would be best to remove it, and make sure that the comment in crypto/xctr.c fully explains what XCTR is (currently it's a bit inadequate).

Quoted 4 lines
crypto/Kconfig +4 -0
+/*
+ * Test vectors generated using https://github.com/google/hctr2
+ */
+static const struct cipher_testvec aes_xctr_tv_template[] = {

[...]

Quoted 2 lines
crypto/Kconfig +2 -0
+		.klen	= 16,
+		.len	= 255,

[...]

crypto/Kconfig

Quoted 2 lines
crypto/Kconfig +2 -0
+		.klen	= 16,
+		.len	= 255,

I commented on the test vectors before in the context of the HCTR2 ones, but the same comments apply here: the actual test coverage for the number of test vectors included is not great, due to lengths being repeated. It would be better to vary the lengths a bit more, especially the message lengths. What you have here isn't bad, but I think there's some room for improvement.

Quoted 10 lines

+/*
+ * XCTR mode is a blockcipher mode of operation used to implement HCTR2. XCTR is
+ * closely related to the CTR mode of operation; the main difference is that CTR
+ * generates the keystream using E(CTR + IV) whereas XCTR generates the
+ * keystream using E(CTR ^ IV).
+ *
+ * See the HCTR2 paper for more details:
+ * Length-preserving encryption with HCTR2
+ * (https://eprint.iacr.org/2021/1441.pdf)
+ */

The above comment could use a bit more detail, e.g. mentioning endianness as well as the fact that XCTR avoids having to deal with multi-limb integers.

Quoted 6 lines
crypto/Kconfig +6 -0
+static void crypto_xctr_crypt_final(struct skcipher_walk *walk,
+				   struct crypto_cipher *tfm, u32 byte_ctr)
+{
+	unsigned long alignmask = crypto_cipher_alignmask(tfm);
+	u8 tmp[XCTR_BLOCKSIZE + MAX_CIPHER_ALIGNMASK];
+	u8 *keystream = PTR_ALIGN(tmp + 0, alignmask + 1);

crypto/Kconfig

Quoted 10 lines
crypto/Kconfig +10 -0
+	u8 *src = walk->src.virt.addr;
+	u8 *dst = walk->dst.virt.addr;
+	unsigned int nbytes = walk->nbytes;
+	__le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
+
+	crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+	crypto_cipher_encrypt_one(tfm, keystream, walk->iv);
+	crypto_xor_cpy(dst, keystream, src, nbytes);
+	crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+}

When crypto_cipher_encrypt_one() is used instead of ->cia_encrypt directly, the caller doesn't need to align the buffers, sincec crypto_cipher_encrypt_one() handles that.

Quoted 15 lines
crypto/Kconfig +15 -0
+static int crypto_xctr_crypt_segment(struct skcipher_walk *walk,
+				    struct crypto_cipher *tfm, u32 byte_ctr)
+{
+	void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
+		   crypto_cipher_alg(tfm)->cia_encrypt;
+	u8 *src = walk->src.virt.addr;
+	u8 *dst = walk->dst.virt.addr;
+	unsigned int nbytes = walk->nbytes;
+	__le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
+
+	do {
+		/* create keystream */
+		crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+		fn(crypto_cipher_tfm(tfm), dst, walk->iv);
+		crypto_xor(dst, src, XCTR_BLOCKSIZE);

crypto/Kconfig

Quoted 10 lines
crypto/Kconfig +10 -0
+		crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+
+		ctr32++;
+
+		src += XCTR_BLOCKSIZE;
+		dst += XCTR_BLOCKSIZE;
+	} while ((nbytes -= XCTR_BLOCKSIZE) >= XCTR_BLOCKSIZE);
+
+	return nbytes;
+}

This won't work on big endian systems due to the 'ctr32++' on a __le32 variable. I recommend installing 'sparse' and passing C=2 to make, as it will warn about endianness bugs like this.

Either endianness needs to be converted for the increment, or it needs to be converted when doing the XOR. (The latter would work if the XOR is done manually instead of with crypto_xor, which could be a nice optimization separately from fixing the endianness bug.)

crypto/Kconfig

Quoted 4 lines
crypto/Kconfig +4 -0
+	/* Block size must be >= 4 bytes. */
+	err = -EINVAL;
+	if (alg->cra_blocksize != XCTR_BLOCKSIZE)
+		goto out_free_inst;

The comment above is outdated.

- Eric

#3 lore raw

Re: [RFC PATCH v2 1/7] crypto: xctr - Add XCTR support

Quoted 9 lines · Eric Biggers
Quoted 8 lines · Nathan Huckleberry
crypto/Kconfig +0 -0
index fa1741bb568f..8543f34fa200 100644
@@ -452,6 +452,15 @@ config CRYPTO_PCBC
        PCBC: Propagating Cipher Block Chaining mode
        This block cipher algorithm is required for RxRPC.
 
Quoted 12 lines

crypto/Kconfig config CRYPTO_PCBC · line 462

Quoted 8 lines
crypto/Kconfig +8 -0
+config CRYPTO_XCTR
+     tristate
+     select CRYPTO_SKCIPHER
+     select CRYPTO_MANAGER
+     help
+       XCTR: XOR Counter mode. This blockcipher mode is a variant of CTR mode
+       using XORs and little-endian addition rather than big-endian arithmetic.
+       XCTR mode is used to implement HCTR2.

Now that this option isn't user-selectable, no one will see this help text. I think it would be best to remove it, and make sure that the comment in crypto/xctr.c fully explains what XCTR is (currently it's a bit inadequate).

I generally prefer to have a help text in Kconfig even for hidden symbols, and I read those when trying to find my way through code I'm not familiar with. It's probably a good idea to expand the comment in the source file as well, but I would suggest leaving this one in here.

Arnd

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#4 lore raw

RFC PATCH v2 2/7 crypto: polyval - Add POLYVAL support

In reply to Nathan Huckleberry

Add support for POLYVAL, an ε-Δ-universal hash function similar to GHASH. POLYVAL is used as a component to implement HCTR2 mode.

POLYVAL is implemented as an shash algorithm. The implementation is modified from ghash-generic.c.

More information on POLYVAL can be found in the HCTR2 paper:
https://eprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Added comments explaining the GHASH/POLYVAL trick
* Fixed a bug on non-block-multiple messages

 crypto/Kconfig           |   8 ++
 crypto/Makefile          |   1 +
 crypto/polyval-generic.c | 199 +++++++++++++++++++++++++++
 crypto/tcrypt.c          |   4 +
 crypto/testmgr.c         |   6 +
 crypto/testmgr.h         | 284 +++++++++++++++++++++++++++++++++++++++
 include/crypto/polyval.h |  22 +++
 7 files changed, 524 insertions(+)
 create mode 100644 crypto/polyval-generic.c
 create mode 100644 include/crypto/polyval.h
crypto/Kconfig +8 -0
index 8543f34fa200..0c61d03530a6 100644
@@ -760,6 +760,14 @@ config CRYPTO_GHASH
 	  GHASH is the hash function used in GCM (Galois/Counter Mode).
 	  It is not a general-purpose cryptographic hash function.
 
+config CRYPTO_POLYVAL
+	tristate
+	select CRYPTO_GF128MUL
+	select CRYPTO_HASH
+	help
+	  POLYVAL is the hash function used in HCTR2.  It is not a general-purpose
+	  cryptographic hash function.
+
 config CRYPTO_POLY1305
 	tristate "Poly1305 authenticator algorithm"
 	select CRYPTO_HASH
crypto/Makefile +1 -0
index 6b3fe3df1489..561f901a91d4 100644
@@ -169,6 +169,7 @@ UBSAN_SANITIZE_jitterentropy.o = n
 jitterentropy_rng-y := jitterentropy.o jitterentropy-kcapi.o
 obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
 obj-$(CONFIG_CRYPTO_GHASH) += ghash-generic.o
+obj-$(CONFIG_CRYPTO_POLYVAL) += polyval-generic.o
 obj-$(CONFIG_CRYPTO_USER_API) += af_alg.o
 obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o
 obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
crypto/polyval-generic.c new file +199 -0

199 lines not on this page · raw message

crypto/tcrypt.c +4 -0
index b3a23dbf5b14..ced7467bb481 100644
@@ -1730,6 +1730,10 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
 		ret += tcrypt_test("ccm(sm4)");
 		break;
 
+	case 57:
+		ret += tcrypt_test("polyval");
+		break;
+
 	case 100:
 		ret += tcrypt_test("hmac(md5)");
 		break;
crypto/testmgr.c +6 -0
index d2b42ff0b04a..3e54d17fe644 100644
@@ -5245,6 +5245,12 @@ static const struct alg_test_desc alg_test_descs[] = {
 		.suite = {
 			.hash = __VECS(poly1305_tv_template)
 		}
+	}, {
+		.alg = "polyval",
+		.test = alg_test_hash,
+		.suite = {
+			.hash = __VECS(polyval_tv_template)
+		}
 	}, {
 		.alg = "rfc3686(ctr(aes))",
 		.test = alg_test_skcipher,
crypto/testmgr.h +284 -0

288 lines not on this page · raw message

include/crypto/polyval.h new file +22 -0
new file mode 100644
index 000000000000..fd0c6e124b65
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Common values for the Polyval hash algorithm
+ *
+ * Copyright 2021 Google LLC
+ */
+
+#ifndef _CRYPTO_POLYVAL_H
+#define _CRYPTO_POLYVAL_H
+
+#include <linux/types.h>
+#include <linux/crypto.h>
+
+#define POLYVAL_BLOCK_SIZE	16
+#define POLYVAL_DIGEST_SIZE	16
+
+struct polyval_desc_ctx {
+	u8 buffer[POLYVAL_BLOCK_SIZE];
+	u32 bytes;
+};
+
+#endif
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#5 lore raw

Re: [RFC PATCH v2 2/7] crypto: polyval - Add POLYVAL support

crypto/Kconfig config CRYPTO_GHASH · line 769

Quoted 7 lines · Nathan Huckleberry
crypto/Kconfig +7 -0
+config CRYPTO_POLYVAL
+	tristate
+	select CRYPTO_GF128MUL
+	select CRYPTO_HASH
+	help
+	  POLYVAL is the hash function used in HCTR2.  It is not a general-purpose
+	  cryptographic hash function.

As with XCTR: as this option is no longer user-selectable, no one will see this help text. I think it should just be removed.

crypto/polyval-generic.c line 104

Quoted 6 lines
crypto/polyval-generic.c new file +6 -0
+static int polyval_update(struct shash_desc *desc,
+			 const u8 *src, unsigned int srclen)
+{
+	struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+	const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm);
+	u8 *dst = dctx->buffer;

The dst variable doesn't seem to serve a purpose. It would be clearer to just write dctx->buffer directly (or &dctx->buffer[...], etc).

Quoted 6 lines
crypto/polyval-generic.c new file +6 -0
+	u8 *pos;
+	u8 tmp[POLYVAL_BLOCK_SIZE];
+	int n;
+
+	if (dctx->bytes) {
+		n = min(srclen, dctx->bytes);

crypto/polyval-generic.c line 120

Quoted 10 lines
crypto/polyval-generic.c new file +10 -0
+		pos = dst + dctx->bytes - 1;
+
+		dctx->bytes -= n;
+		srclen -= n;
+
+		while (n--)
+			*pos-- ^= *src++;
+
+		if (!dctx->bytes)
+			gf128mul_4k_lle((be128 *)dst, ctx->gf128);

I thought I mentioned this on v1, but the cast to be128 is violating alignment rules. If the alignment to be128 is needed then a union should be used, e.g.:

struct polyval_desc_ctx {
        union {
                u8 buffer[POLYVAL_BLOCK_SIZE];
                be128 buffer128;
        };
        u32 bytes;
};
Quoted 5 lines
crypto/polyval-generic.c new file +5 -0
+static int polyval_final(struct shash_desc *desc, u8 *dst)
+{
+	struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+	const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm);
+	u8 *buf = dctx->buffer;

crypto/polyval-generic.c line 156

Quoted 10 lines
crypto/polyval-generic.c new file +10 -0
+
+	if (dctx->bytes)
+		gf128mul_4k_lle((be128 *)buf, ctx->gf128);
+	dctx->bytes = 0;
+
+	reverse_block(buf);
+	memcpy(dst, buf, POLYVAL_BLOCK_SIZE);
+
+	return 0;
+}

Same issues as polyval_update().

Quoted 19 lines
(quoted diff) +1 -0
+
include/crypto/polyval.h new file +12 -0
new file mode 100644
index 000000000000..fd0c6e124b65
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Common values for the Polyval hash algorithm
+ *
+ * Copyright 2021 Google LLC
+ */
+
+#ifndef _CRYPTO_POLYVAL_H
+#define _CRYPTO_POLYVAL_H
+
+#include <linux/types.h>
+#include <linux/crypto.h>

include/crypto/polyval.h line 22

Quoted 10 lines
include/crypto/polyval.h new file +10 -0
+
+#define POLYVAL_BLOCK_SIZE	16
+#define POLYVAL_DIGEST_SIZE	16
+
+struct polyval_desc_ctx {
+	u8 buffer[POLYVAL_BLOCK_SIZE];
+	u32 bytes;
+};
+
+#endif

As-is, polyval_desc_ctx is only used by crypto/polyval-generic.c, so it shouldn't be in this header. Either it should be moved to polyval-generic.c, or all implementations should be made to use the same struct.

- Eric

#6 lore raw

RFC PATCH v2 3/7 crypto: hctr2 - Add HCTR2 support

In reply to Nathan Huckleberry

Add support for HCTR2 as a template. HCTR2 is a length-preserving encryption mode that is efficient on processors with instructions to accelerate AES and carryless multiplication, e.g. x86 processors with AES-NI and CLMUL, and ARM processors with the ARMv8 Crypto Extensions.

As a length-preserving encryption mode, HCTR2 is suitable for applications such as storage encryption where ciphertext expansion is not possible, and thus authenticated encryption cannot be used. Currently, such applications usually use XTS, or in some cases Adiantum. XTS has the disadvantage that it is a narrow-block mode: a bitflip will only change 16 bytes in the resulting ciphertext or plaintext. This reveals more information to an attacker than necessary.

HCTR2 is a wide-block mode, so it provides a stronger security property:
a bitflip will change the entire message. HCTR2 is somewhat similar to Adiantum, which is also a wide-block mode. However, HCTR2 is designed to take advantage of existing crypto instructions, while Adiantum targets devices without such hardware support. Adiantum is also designed with longer messages in mind, while HCTR2 is designed to be efficient even on short messages.

HCTR2 requires POLYVAL and XCTR as components. More information on HCTR2 can be found here: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Rename streamcipher -> xctr
* Rename hash -> polyval
* Use __le64 instead of u64 for little-endian length
* memzero_explicit in set_key
* Use crypto request length instead of scatterlist length for polyval
* Add comments referencing the paper's pseudocode
* Derive blockcipher name from xctr name
* Pass IV through request context
* Use .generic_driver
* Make tests more comprehensive

 crypto/Kconfig   |  11 +
 crypto/Makefile  |   1 +
 crypto/hctr2.c   | 532 +++++++++++++++++++++++++++++++++++++
 crypto/tcrypt.c  |   5 +
 crypto/testmgr.c |   8 +
 crypto/testmgr.h | 670 +++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 1227 insertions(+)
 create mode 100644 crypto/hctr2.c
crypto/Kconfig +11 -0
index 0c61d03530a6..2a9029f51caf 100644
@@ -524,6 +524,17 @@ config CRYPTO_ADIANTUM
 
 	  If unsure, say N.
 
+config CRYPTO_HCTR2
+	tristate "HCTR2 support"
+	select CRYPTO_XCTR
+	select CRYPTO_POLYVAL
+	select CRYPTO_MANAGER
+	help
+	  HCTR2 is a length-preserving encryption mode for storage encryption that
+	  is efficient on processors with instructions to accelerate AES and
+	  carryless multiplication, e.g. x86 processors with AES-NI and CLMUL, and
+	  ARM processors with the ARMv8 crypto extensions.
+
 config CRYPTO_ESSIV
 	tristate "ESSIV support for block encryption"
 	select CRYPTO_AUTHENC
crypto/Makefile +1 -0
index 561f901a91d4..2dca9dbdede6 100644
@@ -94,6 +94,7 @@ obj-$(CONFIG_CRYPTO_LRW) += lrw.o
 obj-$(CONFIG_CRYPTO_XTS) += xts.o
 obj-$(CONFIG_CRYPTO_CTR) += ctr.o
 obj-$(CONFIG_CRYPTO_XCTR) += xctr.o
+obj-$(CONFIG_CRYPTO_HCTR2) += hctr2.o
 obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o
 obj-$(CONFIG_CRYPTO_ADIANTUM) += adiantum.o
 obj-$(CONFIG_CRYPTO_NHPOLY1305) += nhpoly1305.o
crypto/hctr2.c new file +532 -0

532 lines not on this page · raw message

crypto/tcrypt.c +5 -0
index ced7467bb481..3a5cd6831e65 100644
@@ -2191,6 +2191,11 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
 				   16, 16, aead_speed_template_19, num_mb);
 		break;
 
+	case 226:
+		test_cipher_speed("hctr2(aes)", ENCRYPT, sec, NULL,
+				  0, speed_template_32);
+		break;
+
 	case 300:
 		if (alg) {
 			test_hash_speed(alg, sec, generic_hash_speed_template);
crypto/testmgr.c +8 -0
index 3e54d17fe644..2e92a4a89285 100644
@@ -4991,6 +4991,14 @@ static const struct alg_test_desc alg_test_descs[] = {
 		.suite = {
 			.hash = __VECS(ghash_tv_template)
 		}
+	}, {
+		.alg = "hctr2(aes)",
+		.generic_driver =
+		    "hctr2_base(xctr(aes-generic),polyval-generic)",
+		.test = alg_test_skcipher,
+		.suite = {
+			.cipher = __VECS(aes_hctr2_tv_template)
+		}
 	}, {
 		.alg = "hmac(md5)",
 		.test = alg_test_hash,
crypto/testmgr.h +670 -0

674 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#7 lore raw

Re: [RFC PATCH v2 3/7] crypto: hctr2 - Add HCTR2 support

Quoted 11 lines · Nathan Huckleberry

Changes since v1:
* Rename streamcipher -> xctr
* Rename hash -> polyval
* Use __le64 instead of u64 for little-endian length
* memzero_explicit in set_key
* Use crypto request length instead of scatterlist length for polyval
* Add comments referencing the paper's pseudocode
* Derive blockcipher name from xctr name
* Pass IV through request context
* Use .generic_driver
* Make tests more comprehensive

This is looking much better than v1. A few comments below.

Quoted 16 lines
crypto/hctr2.c new file +16 -0
+static int hctr2_hash_tweak(struct skcipher_request *req)
+{
+	__le64 tweak_length_block[2];
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	const struct hctr2_tfm_ctx *tctx = crypto_skcipher_ctx(tfm);
+	struct hctr2_request_ctx *rctx = skcipher_request_ctx(req);
+	struct shash_desc *hash_desc = &rctx->u.hash_desc;
+	int err;
+
+	memset(tweak_length_block, 0, sizeof(tweak_length_block));
+	if (req->cryptlen % POLYVAL_BLOCK_SIZE == 0)
+		tweak_length_block[0] = cpu_to_le64(TWEAK_SIZE * 8 * 2 + 2);
+	else
+		tweak_length_block[0] = cpu_to_le64(TWEAK_SIZE * 8 * 2 + 3);
+
+	hash_desc->tfm = tctx->polyval;

crypto/hctr2.c line 126

Quoted 10 lines
crypto/hctr2.c new file +10 -0
+	err = crypto_shash_init(hash_desc);
+	if (err)
+		return err;
+
+	err = crypto_shash_update(hash_desc, (u8 *)tweak_length_block,
+				  sizeof(tweak_length_block));
+	if (err)
+		return err;
+	return crypto_shash_update(hash_desc, req->iv, TWEAK_SIZE);
+}

Have you considered taking advantage of the hash function precomputation that is possible? That should improve the performance a bit, especially on short inputs. Per-key, a hash state could be precomputed containing the tweak_length_block, since it will always have the same contents, due to this implementation only supporting a single tweak length. hctr2_setkey() could compute that and export it using crypto_shash_export() into the hctr2_tfm_ctx. hctr2_crypt() could import it using crypto_shash_import().

Similarly, the tweak only needs to be hashed once per request, as the state could be exported after hashing it the first time, then imported for the second hash step. The saved state would need to be part of the hctr2_request_ctx.

Quoted 10 lines
crypto/hctr2.c new file +10 -0
+static int hctr2_finish(struct skcipher_request *req)
+{
+	struct hctr2_request_ctx *rctx = skcipher_request_ctx(req);
+	u8 digest[POLYVAL_DIGEST_SIZE];
+	int err;
+
+	// U = UU ^ H(T || V)
+	err = hctr2_hash_tweak(req);
+	if (err)
+		return err;

crypto/hctr2.c line 185

Quoted 10 lines
crypto/hctr2.c new file +10 -0
+	err = hctr2_hash_message(req, rctx->bulk_part_dst, digest);
+	if (err)
+		return err;
+	crypto_xor(rctx->first_block, digest, BLOCKCIPHER_BLOCK_SIZE);
+
+	// Copy U into dst scatterlist
+	scatterwalk_map_and_copy(rctx->first_block, req->dst,
+				 0, BLOCKCIPHER_BLOCK_SIZE, 1);
+	return 0;
+}

The comments here and in hctr2_crypt() are assuming encryption, but the code also handles decryption. It would be helpful to give the pseudocode for both, e.g.:

	//    U = UU ^ H(T || V)
	// or M = MM ^ H(T || N)
Quoted 14 lines
crypto/hctr2.c new file +14 -0
+static int hctr2_create_base(struct crypto_template *tmpl, struct rtattr **tb)
+{
+	const char *xctr_name;
+	const char *polyval_name;
+	char blockcipher_name[CRYPTO_MAX_ALG_NAME];
+	int len;
+
+	xctr_name = crypto_attr_alg_name(tb[1]);
+	if (IS_ERR(xctr_name))
+		return PTR_ERR(xctr_name);
+
+	if (!strncmp(xctr_name, "xctr(", 5)) {
+		len = strscpy(blockcipher_name, xctr_name + 5,
+			    sizeof(blockcipher_name));

crypto/hctr2.c line 475

Quoted 10 lines
crypto/hctr2.c new file +10 -0
+
+		if (len < 1)
+			return -EINVAL;
+
+		if (blockcipher_name[len - 1] != ')')
+			return -EINVAL;
+
+		blockcipher_name[len - 1] = 0;
+	} else
+		return -EINVAL;

I don't think this is exactly what Herbert and I had in mind. It's close, but the problem with grabbing the block cipher name from the raw string the user passes in is that the string could be an implementation name like "xctr-aes-ni", rather than an algorithm name like "xctr(aes)".

The correct way to do this is to wait to determine the block cipher algorithm until after calling crypto_grab_skcipher(). Then it will be available in the ->cra_name of the skcipher algorithm (that should be xctr).

Quoted 16 lines
crypto/testmgr.h +8 -0
index da3736e51982..a16b631730e9 100644
@@ -33630,4 +33630,674 @@ static const struct hash_testvec polyval_tv_template[] = {
 	},
 };
 
+/*
+ * Test vectors generated using https://github.com/google/hctr2
+ */
+static const struct cipher_testvec aes_hctr2_tv_template[] = {
+		.klen	= 16,
+		.len	= 16,
+	},
+	{

[...]

Quoted 4 lines
crypto/testmgr.h +4 -0
+		.klen	= 16,
+		.len	= 16,
+	},
+	{

[...]

Quoted 4 lines
crypto/testmgr.h +4 -0
+		.klen	= 16,
+		.len	= 17,
+	},
+	{

[...]

Quoted 4 lines
crypto/testmgr.h +4 -0
+		.klen	= 16,
+		.len	= 17,
+	},
+	{

[...]

Quoted 7 lines
crypto/testmgr.h +7 -0
+		.klen	= 24,
+		.len	= 31,
+	},
+	{
+		.klen	= 24,
+		.len	= 31,
+	},

[...]

Quoted 4 lines
crypto/testmgr.h +4 -0
+	{
+		.klen	= 24,
+		.len	= 48,
+	},

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 24,
+		.len	= 48,

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 128,

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 128,

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 255,

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 255,

[...]

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 512,

[...]

crypto/testmgr.h

Quoted 2 lines
crypto/testmgr.h +2 -0
+		.klen	= 32,
+		.len	= 512,

These are better but still could use some improvement. There are two test vectors for each message length, but they also have the same key length. That provides little additional test coverage over having just half the test vectors. It would be better to use different key lengths within each pair. Maybe always use klen=32 for one test vector, and randomly choose klen=16 or klen=24 for the other test vector. The overall point is: we can't test a zillion test vectors, so we should make sure to get as most test coverage as possible with the smaller set that will actually be included.

- Eric

#8 lore raw

RFC PATCH v2 4/7 crypto: x86/aesni-xctr: Add accelerated implementation of XCTR

In reply to Nathan Huckleberry

Add hardware accelerated versions of XCTR for x86-64 CPUs with AESNI support. These implementations are modified versions of the CTR implementations found in aesni-intel_asm.S and aes_ctrby8_avx-x86_64.S.

More information on XCTR can be found in the HCTR2 paper:
Length-preserving encryption with HCTR2:
https://enterprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Changed ctr32 from u32 to __le32
* Removed references to u32_to_le_block

 arch/x86/crypto/Makefile                 |   2 +-
 arch/x86/crypto/aes_xctrby8_avx-x86_64.S | 529 +++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_asm.S        |  70 +++
 arch/x86/crypto/aesni-intel_glue.c       |  89 ++++
 4 files changed, 689 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aes_xctrby8_avx-x86_64.S
arch/x86/crypto/Makefile +1 -1
index 2831685adf6f..ee2df489b0d9 100644
@@ -48,7 +48,7 @@ chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
-aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
+aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o aes_xctrby8_avx-x86_64.o
 
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
arch/x86/crypto/aes_xctrby8_avx-x86_64.S new file +529 -0

529 lines not on this page · raw message

arch/x86/crypto/aesni-intel_asm.S +70 -0

76 lines not on this page · raw message

arch/x86/crypto/aesni-intel_glue.c +89 -0

119 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#9 lore raw

Re: [RFC PATCH v2 4/7] crypto: x86/aesni-xctr: Add accelerated implementation of XCTR

Quoted 19 lines · Nathan Huckleberry
arch/x86/crypto/aes_xctrby8_avx-x86_64.S new file +13 -0
new file mode 100644
index 000000000000..53d70cab9474
@@ -0,0 +1,529 @@
+/* SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause */
+/*
+ * AES XCTR mode by8 optimization with AVX instructions. (x86_64)
+ *
+ * Copyright(c) 2014 Intel Corporation.
+ *
+ * Contact Information:
+ * James Guilford <[email protected]>
+ * Sean Gulley <[email protected]>
+ * Chandramouli Narayanan <[email protected]>
+ */
+/*
+ * Implement AES XCTR mode with AVX instructions. This code is a modified

arch/x86/crypto/aes_xctrby8_avx-x86_64.S line 23

Quoted 10 lines
arch/x86/crypto/aes_xctrby8_avx-x86_64.S new file +10 -0
+ * version of the Linux kernel's AES CTR by8 implementation.
+ *
+ * This is AES128/192/256 XCTR mode optimization implementation. It requires
+ * the support of Intel(R) AESNI and AVX instructions.
+ *
+ * This work was inspired by the AES XCTR mode optimization published
+ * in Intel Optimized IPSEC Cryptographic library.
+ * Additional information on it can be found at:
+ *    https://github.com/intel/intel-ipsec-mb
+ */

So I haven't looked at this closely yet, but one thing I noticed is that this file is pretty long, and it's almost identical to aes_ctrby8_avx-x86_64.S. Perhaps it would make sense to add XCTR support to that file rather than duplicating most of it into this new file? You could add an is_xctr argument to the macros to allow making parts conditional on CTR or XCTR when needed.

- Eric

#10 lore raw

RFC PATCH v2 5/7 crypto: arm64/aes-xctr: Add accelerated implementation of XCTR

In reply to Nathan Huckleberry

Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8 Crypto Extension support. This XCTR implementation is based on the CTR implementation in aes-modes.S.

More information on XCTR can be found in
the HCTR2 paper: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Added STRIDE back to aes-glue.c

 arch/arm64/crypto/Kconfig     |   4 +-
 arch/arm64/crypto/aes-glue.c  |  72 ++++++++++++++++++-
 arch/arm64/crypto/aes-modes.S | 130 ++++++++++++++++++++++++++++++++++
 3 files changed, 202 insertions(+), 4 deletions(-)
arch/arm64/crypto/Kconfig +2 -2
index 2a965aa0188d..897f9a4b5b67 100644
@@ -84,13 +84,13 @@ config CRYPTO_AES_ARM64_CE_CCM
 	select CRYPTO_LIB_AES
 
 config CRYPTO_AES_ARM64_CE_BLK
-	tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions"
+	tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using ARMv8 Crypto Extensions"
 	depends on KERNEL_MODE_NEON
 	select CRYPTO_SKCIPHER
 	select CRYPTO_AES_ARM64_CE
 
 config CRYPTO_AES_ARM64_NEON_BLK
-	tristate "AES in ECB/CBC/CTR/XTS modes using NEON instructions"
+	tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using NEON instructions"
 	depends on KERNEL_MODE_NEON
 	select CRYPTO_SKCIPHER
 	select CRYPTO_LIB_AES
arch/arm64/crypto/aes-glue.c +70 -2

123 lines not on this page · raw message

arch/arm64/crypto/aes-modes.S +130 -0

136 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#11 lore raw

Re: [RFC PATCH v2 5/7] crypto: arm64/aes-xctr: Add accelerated implementation of XCTR

Quoted 9 lines · Nathan Huckleberry

Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8 Crypto Extension support. This XCTR implementation is based on the CTR implementation in aes-modes.S.

More information on XCTR can be found in
the HCTR2 paper: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

Quoted 6 lines

---

Changes since v1:
* Added STRIDE back to aes-glue.c

NAK. Feel free to respond to my comments/questions against v1 if you want to discuss this.

Quoted 302 lines
 arch/arm64/crypto/Kconfig     |   4 +-
 arch/arm64/crypto/aes-glue.c  |  72 ++++++++++++++++++-
 arch/arm64/crypto/aes-modes.S | 130 ++++++++++++++++++++++++++++++++++
 3 files changed, 202 insertions(+), 4 deletions(-)
arch/arm64/crypto/Kconfig +2 -2
index 2a965aa0188d..897f9a4b5b67 100644
@@ -84,13 +84,13 @@ config CRYPTO_AES_ARM64_CE_CCM
        select CRYPTO_LIB_AES
 
 config CRYPTO_AES_ARM64_CE_BLK
-       tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions"
+       tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using ARMv8 Crypto Extensions"
        depends on KERNEL_MODE_NEON
        select CRYPTO_SKCIPHER
        select CRYPTO_AES_ARM64_CE
 
 config CRYPTO_AES_ARM64_NEON_BLK
-       tristate "AES in ECB/CBC/CTR/XTS modes using NEON instructions"
+       tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using NEON instructions"
        depends on KERNEL_MODE_NEON
        select CRYPTO_SKCIPHER
        select CRYPTO_LIB_AES
arch/arm64/crypto/aes-glue.c +70 -2
index 561dd2332571..dd04f3c5b0f1 100644
@@ -24,6 +24,7 @@
 #ifdef USE_V8_CRYPTO_EXTENSIONS
 #define MODE                   "ce"
 #define PRIO                   300
+#define STRIDE                 5
 #define aes_expandkey          ce_aes_expandkey
 #define aes_ecb_encrypt                ce_aes_ecb_encrypt
 #define aes_ecb_decrypt                ce_aes_ecb_decrypt
@@ -34,13 +35,15 @@
 #define aes_essiv_cbc_encrypt  ce_aes_essiv_cbc_encrypt
 #define aes_essiv_cbc_decrypt  ce_aes_essiv_cbc_decrypt
 #define aes_ctr_encrypt                ce_aes_ctr_encrypt
+#define aes_xctr_encrypt       ce_aes_xctr_encrypt
 #define aes_xts_encrypt                ce_aes_xts_encrypt
 #define aes_xts_decrypt                ce_aes_xts_decrypt
 #define aes_mac_update         ce_aes_mac_update
-MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions");
+MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 Crypto Extensions");
 #else
 #define MODE                   "neon"
 #define PRIO                   200
+#define STRIDE                 4
 #define aes_ecb_encrypt                neon_aes_ecb_encrypt
 #define aes_ecb_decrypt                neon_aes_ecb_decrypt
 #define aes_cbc_encrypt                neon_aes_cbc_encrypt
@@ -50,16 +53,18 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions");
 #define aes_essiv_cbc_encrypt  neon_aes_essiv_cbc_encrypt
 #define aes_essiv_cbc_decrypt  neon_aes_essiv_cbc_decrypt
 #define aes_ctr_encrypt                neon_aes_ctr_encrypt
+#define aes_xctr_encrypt       neon_aes_xctr_encrypt
 #define aes_xts_encrypt                neon_aes_xts_encrypt
 #define aes_xts_decrypt                neon_aes_xts_decrypt
 #define aes_mac_update         neon_aes_mac_update
-MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON");
+MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 NEON");
 #endif
 #if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS)
 MODULE_ALIAS_CRYPTO("ecb(aes)");
 MODULE_ALIAS_CRYPTO("cbc(aes)");
 MODULE_ALIAS_CRYPTO("ctr(aes)");
 MODULE_ALIAS_CRYPTO("xts(aes)");
+MODULE_ALIAS_CRYPTO("xctr(aes)");
 #endif
 MODULE_ALIAS_CRYPTO("cts(cbc(aes))");
 MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)");
@@ -89,6 +94,10 @@ asmlinkage void aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[],
 asmlinkage void aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
                                int rounds, int bytes, u8 ctr[]);
 
+asmlinkage void aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
+                                int rounds, int bytes, u8 ctr[], u8 finalbuf[],
+                                int byte_ctr);
+
 asmlinkage void aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[],
                                int rounds, int bytes, u32 const rk2[], u8 iv[],
                                int first);
@@ -442,6 +451,49 @@ static int __maybe_unused essiv_cbc_decrypt(struct skcipher_request *req)
        return err ?: cbc_decrypt_walk(req, &walk);
 }
 
+static int __maybe_unused xctr_encrypt(struct skcipher_request *req)
+{
+       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+       struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm);
+       int err, rounds = 6 + ctx->key_length / 4;
+       struct skcipher_walk walk;
+       unsigned int byte_ctr = 0;
+
+       err = skcipher_walk_virt(&walk, req, false);
+
+       while (walk.nbytes > 0) {
+               const u8 *src = walk.src.virt.addr;
+               unsigned int nbytes = walk.nbytes;
+               u8 *dst = walk.dst.virt.addr;
+               u8 buf[AES_BLOCK_SIZE];
+               unsigned int tail;
+
+               if (unlikely(nbytes < AES_BLOCK_SIZE))
+                       src = memcpy(buf, src, nbytes);
+               else if (nbytes < walk.total)
+                       nbytes &= ~(AES_BLOCK_SIZE - 1);
+
+               kernel_neon_begin();
+               aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes,
+                                                walk.iv, buf, byte_ctr);
+               kernel_neon_end();
+
+               tail = nbytes % (STRIDE * AES_BLOCK_SIZE);
+               if (tail > 0 && tail < AES_BLOCK_SIZE)
+                       /*
+                        * The final partial block could not be returned using
+                        * an overlapping store, so it was passed via buf[]
+                        * instead.
+                        */
+                       memcpy(dst + nbytes - tail, buf, tail);
+               byte_ctr += nbytes;
+
+               err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+       }
+
+       return err;
+}
+
 static int __maybe_unused ctr_encrypt(struct skcipher_request *req)
 {
        struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
@@ -669,6 +721,22 @@ static struct skcipher_alg aes_algs[] = { {
        .setkey         = skcipher_aes_setkey,
        .encrypt        = ctr_encrypt,
        .decrypt        = ctr_encrypt,
+}, {
+       .base = {
+               .cra_name               = "xctr(aes)",
+               .cra_driver_name        = "xctr-aes-" MODE,
+               .cra_priority           = PRIO,
+               .cra_blocksize          = 1,
+               .cra_ctxsize            = sizeof(struct crypto_aes_ctx),
+               .cra_module             = THIS_MODULE,
+       },
+       .min_keysize    = AES_MIN_KEY_SIZE,
+       .max_keysize    = AES_MAX_KEY_SIZE,
+       .ivsize         = AES_BLOCK_SIZE,
+       .chunksize      = AES_BLOCK_SIZE,
+       .setkey         = skcipher_aes_setkey,
+       .encrypt        = xctr_encrypt,
+       .decrypt        = xctr_encrypt,
 }, {
        .base = {
                .cra_name               = "xts(aes)",
arch/arm64/crypto/aes-modes.S +130 -1
index dc35eb0245c5..3b5c7a5c21e4 100644
@@ -479,6 +479,136 @@ ST5(      mov             v3.16b, v4.16b                  )
        b               .Lctrout
 AES_FUNC_END(aes_ctr_encrypt)
 
+       /*
+        * aes_xctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
+        *                 int bytes, u8 const ctr[], u8 finalbuf[], int
+        *                 byte_ctr)
+        */
+
+AES_FUNC_START(aes_xctr_encrypt)
+       stp             x29, x30, [sp, #-16]!
+       mov             x29, sp
+
+       enc_prepare     w3, x2, x12
+       ld1             {vctr.16b}, [x5]
+
+       umov            x12, vctr.d[0]          /* keep ctr in reg */
+       lsr             x7, x7, #4
+       add             x11, x7, #1
+
+.LxctrloopNx:
+       add             w7, w4, #15
+       sub             w4, w4, #MAX_STRIDE << 4
+       lsr             w7, w7, #4
+       mov             w8, #MAX_STRIDE
+       cmp             w7, w8
+       csel            w7, w7, w8, lt
+       add             x11, x11, x7
+
+       mov             v0.16b, vctr.16b
+       mov             v1.16b, vctr.16b
+       mov             v2.16b, vctr.16b
+       mov             v3.16b, vctr.16b
+ST5(   mov             v4.16b, vctr.16b                )
+
+       sub             x7, x11, #MAX_STRIDE
+       eor             x7, x12, x7
+       ins             v0.d[0], x7
+       sub             x7, x11, #MAX_STRIDE - 1
+       sub             x8, x11, #MAX_STRIDE - 2
+       eor             x7, x7, x12
+       sub             x9, x11, #MAX_STRIDE - 3
+       mov             v1.d[0], x7
+       eor             x8, x8, x12
+       eor             x9, x9, x12
+ST5(   sub             x10, x11, #MAX_STRIDE - 4)
+       mov             v2.d[0], x8
+       eor             x10, x10, x12
+       mov             v3.d[0], x9
+ST5(   mov             v4.d[0], x10                    )
+       tbnz            w4, #31, .Lxctrtail
+       ld1             {v5.16b-v7.16b}, [x1], #48
+ST4(   bl              aes_encrypt_block4x             )
+ST5(   bl              aes_encrypt_block5x             )
+       eor             v0.16b, v5.16b, v0.16b
+ST4(   ld1             {v5.16b}, [x1], #16             )
+       eor             v1.16b, v6.16b, v1.16b
+ST5(   ld1             {v5.16b-v6.16b}, [x1], #32      )
+       eor             v2.16b, v7.16b, v2.16b
+       eor             v3.16b, v5.16b, v3.16b
+ST5(   eor             v4.16b, v6.16b, v4.16b          )
+       st1             {v0.16b-v3.16b}, [x0], #64
+ST5(   st1             {v4.16b}, [x0], #16             )
+       cbz             w4, .Lxctrout
+       b               .LxctrloopNx
+
+.Lxctrout:
+       ldp             x29, x30, [sp], #16
+       ret
+
+.Lxctrtail:
+       /* XOR up to MAX_STRIDE * 16 - 1 bytes of in/output with v0 ... v3/v4 */
+       mov             x17, #16
+       ands            x13, x4, #0xf
+       csel            x13, x13, x17, ne
+
+ST5(   cmp             w4, #64 - (MAX_STRIDE << 4))
+ST5(   csel            x14, x17, xzr, gt               )
+       cmp             w4, #48 - (MAX_STRIDE << 4)
+       csel            x15, x17, xzr, gt
+       cmp             w4, #32 - (MAX_STRIDE << 4)
+       csel            x16, x17, xzr, gt
+       cmp             w4, #16 - (MAX_STRIDE << 4)
+       ble             .Lxctrtail1x
+
+ST5(   mov             v4.d[0], x10                    )
+
+       adr_l           x12, .Lcts_permute_table
+       add             x12, x12, x13
+
+ST5(   ld1             {v5.16b}, [x1], x14             )
+       ld1             {v6.16b}, [x1], x15
+       ld1             {v7.16b}, [x1], x16
+
+ST4(   bl              aes_encrypt_block4x             )
+ST5(   bl              aes_encrypt_block5x             )
+
+       ld1             {v8.16b}, [x1], x13
+       ld1             {v9.16b}, [x1]
+       ld1             {v10.16b}, [x12]
+
+ST4(   eor             v6.16b, v6.16b, v0.16b          )
+ST4(   eor             v7.16b, v7.16b, v1.16b          )
+ST4(   tbl             v3.16b, {v3.16b}, v10.16b       )
+ST4(   eor             v8.16b, v8.16b, v2.16b          )
+ST4(   eor             v9.16b, v9.16b, v3.16b          )
+
+ST5(   eor             v5.16b, v5.16b, v0.16b          )
+ST5(   eor             v6.16b, v6.16b, v1.16b          )
+ST5(   tbl             v4.16b, {v4.16b}, v10.16b       )
+ST5(   eor             v7.16b, v7.16b, v2.16b          )
+ST5(   eor             v8.16b, v8.16b, v3.16b          )
+ST5(   eor             v9.16b, v9.16b, v4.16b          )
+
+ST5(   st1             {v5.16b}, [x0], x14             )
+       st1             {v6.16b}, [x0], x15
+       st1             {v7.16b}, [x0], x16
+       add             x13, x13, x0
+       st1             {v9.16b}, [x13]         // overlapping stores
+       st1             {v8.16b}, [x0]
+       b               .Lxctrout
+
+.Lxctrtail1x:
+       // use finalbuf if less than a full block
+       csel            x0, x0, x6, eq
+       ld1             {v5.16b}, [x1]
+ST5(   mov             v3.16b, v4.16b                  )
+       encrypt_block   v3, w3, x2, x8, w7
+       eor             v5.16b, v5.16b, v3.16b
+       st1             {v5.16b}, [x0]
+       b               .Lxctrout
+AES_FUNC_END(aes_xctr_encrypt)
+
 
        /*
         * aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds,
--

2.35.1.265.g69c8d7142f-goog

#12 lore raw

Re: [RFC PATCH v2 5/7] crypto: arm64/aes-xctr: Add accelerated implementation of XCTR

Quoted 14 lines · Ard Biesheuvel
Quoted 12 lines · Nathan Huckleberry

Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8 Crypto Extension support. This XCTR implementation is based on the CTR implementation in aes-modes.S.

More information on XCTR can be found in
the HCTR2 paper: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

---

Quoted 6 lines
Quoted 3 lines

Changes since v1:
* Added STRIDE back to aes-glue.c

NAK. Feel free to respond to my comments/questions against v1 if you want to discuss this.

Oops, I misunderstood the tail block behavior of the CTR implementation and thought it wouldn't work with XCTR mode. I have XCTR mirroring the tail behavior of CTR now. I'll submit it with the v3.

#13 lore raw

Re: [RFC PATCH v2 5/7] crypto: arm64/aes-xctr: Add accelerated implementation of XCTR

Quoted 20 lines · Nathan Huckleberry
Quoted 18 lines · Ard Biesheuvel
Quoted 15 lines · Nathan Huckleberry

Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8 Crypto Extension support. This XCTR implementation is based on the CTR implementation in aes-modes.S.

More information on XCTR can be found in
the HCTR2 paper: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Added STRIDE back to aes-glue.c

Quoted 6 lines
Quoted 2 lines

NAK. Feel free to respond to my comments/questions against v1 if you want to discuss this.

Oops, I misunderstood the tail block behavior of the CTR implementation and thought it wouldn't work with XCTR mode. I have XCTR mirroring the tail behavior of CTR now. I'll submit it with the v3.

Yes, here's my version for reference:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/commit/?h=hctr2&id=5bb01649223080b7cde5740f441eb5e758ec357f

#14 lore raw

RFC PATCH v2 6/7 crypto: x86/polyval: Add PCLMULQDQ accelerated implementation of POLYVAL

In reply to Nathan Huckleberry

Add hardware accelerated version of POLYVAL for x86-64 CPUs with PCLMULQDQ support.

This implementation is accelerated using PCLMULQDQ instructions to perform the finite field computations. For added efficiency, 8 blocks of the message are processed simultaneously by precomputing the first 8 powers of the key.

Schoolbook multiplication is used instead of Karatsuba multiplication because it was found to be slightly faster on x86-64 machines. Montgomery reduction must be used instead of Barrett reduction due to the difference in modulus between POLYVAL's field and other finite fields.

More information on POLYVAL can be found in the HCTR2 paper:
Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf

---

Changes since v1:
* Rename C, D, EF to LO, MI, HI
* Add comments explaining POLYVAL
* Fix bug in handling of non-block-multiple messages
* Wrap shash in ahash (as done in ghash)

 arch/x86/crypto/Makefile               |   3 +
 arch/x86/crypto/polyval-clmulni_asm.S  | 414 +++++++++++++++++++++++++
 arch/x86/crypto/polyval-clmulni_glue.c | 365 ++++++++++++++++++++++
 crypto/Kconfig                         |  10 +
 4 files changed, 792 insertions(+)
 create mode 100644 arch/x86/crypto/polyval-clmulni_asm.S
 create mode 100644 arch/x86/crypto/polyval-clmulni_glue.c
arch/x86/crypto/Makefile +3 -0
index ee2df489b0d9..c0f13801afba 100644
@@ -69,6 +69,9 @@ libblake2s-x86_64-y := blake2s-core.o blake2s-glue.o
 obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 
+obj-$(CONFIG_CRYPTO_POLYVAL_CLMUL_NI) += polyval-clmulni.o
+polyval-clmulni-y := polyval-clmulni_asm.o polyval-clmulni_glue.o
+
 obj-$(CONFIG_CRYPTO_CRC32C_INTEL) += crc32c-intel.o
 crc32c-intel-y := crc32c-intel_glue.o
 crc32c-intel-$(CONFIG_64BIT) += crc32c-pcl-intel-asm_64.o
arch/x86/crypto/polyval-clmulni_asm.S new file +414 -0

414 lines not on this page · raw message

arch/x86/crypto/polyval-clmulni_glue.c new file +365 -0

365 lines not on this page · raw message

crypto/Kconfig +10 -0
index 2a9029f51caf..b539a5bdc45e 100644
@@ -773,12 +773,22 @@ config CRYPTO_GHASH
 
 config CRYPTO_POLYVAL
 	tristate
+	select CRYPTO_CRYPTD
 	select CRYPTO_GF128MUL
 	select CRYPTO_HASH
 	help
 	  POLYVAL is the hash function used in HCTR2.  It is not a general-purpose
 	  cryptographic hash function.
 
+config CRYPTO_POLYVAL_CLMUL_NI
+	tristate "POLYVAL hash function (CLMUL-NI accelerated)"
+	depends on X86 && 64BIT
+	select CRYPTO_POLYVAL
+	help
+	  This is the x86_64 CLMUL-NI accelerated implementation of POLYVAL. It is
+	  used to efficiently implement HCTR2 on x86-64 processors that support
+	  carry-less multiplication instructions.
+
 config CRYPTO_POLY1305
 	tristate "Poly1305 authenticator algorithm"
 	select CRYPTO_HASH
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#15 lore raw

Re: [RFC PATCH v2 6/7] crypto: x86/polyval: Add PCLMULQDQ accelerated implementation of POLYVAL

Hi Huck, sorry for the slow reviews. Some comments on this:

Quoted 6 lines · Nathan Huckleberry
arch/x86/crypto/polyval-clmulni_asm.S new file +0 -0
new file mode 100644
index 000000000000..bec1a2046b18

arch/x86/crypto/polyval-clmulni_asm.S line 8

Quoted 8 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +8 -0
@@ -0,0 +1,414 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Google LLC
+ *
+ * Use of this source code is governed by an MIT-style
+ * license that can be found in the LICENSE file or at
+ * https://opensource.org/licenses/MIT.
+ */

You're intending this file to be GPL-2.0 only, right? (That's what I'd recommend.) In that case, the mention of the MIT license needs to be removed.

arch/x86/crypto/polyval-clmulni_asm.S line 54

Quoted 2 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +2 -0
+Lgstar:
+	.quad 0xc200000000000000, 0xc200000000000000

Local labels need to start with ".L", not just "L" (note the period).

arch/x86/crypto/polyval-clmulni_asm.S line 61

Quoted 4 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +4 -0
+/*
+ * Performs schoolbook1_iteration on two lists of 128-bit polynomials of length
+ * b pointed to by OP1 and OP2.
+ */

How about renaming OP1 and OP2 to MSG and KEY_POWERS, respectively? (Or KEYS instead of KEY_POWERS if you want something less verbose.) The naming and comments make it sound like OP1 and OP2 are interchangable, but they actually aren't. schoolbook1_iteration assumes that OP1 may be unaligned while OP2 is aligned, which implies that OP1 is the message and OP2 is the key powers.

arch/x86/crypto/polyval-clmulni_asm.S line 69

Quoted 8 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +8 -0
+.macro schoolbook1 b
+	.set by, \b
+	.set i, 0
+	.rept (by)
+		schoolbook1_iteration i 0
+		.set i, (i +1)
+	.endr
+.endm

Rename 'b' to 'count' to make it clear what it is?

Also, the line '.set by, \b' is unnecessary, since you could just use the macro argument directly instead. (I mentioned unnecessary .set statements elsewhere on v1. Make sure to go through and address the same issue everywhere.)

I.e., this could be:

.macro schoolbook1 count

        .set i, 0
	.rept \count
		schoolbook1_iteration i 0
		.set i, i + 1
	.endr

.endm

Quoted 9 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +9 -0
+/*
+ * Computes the product of two 128-bit polynomials at the memory locations
+ * specified by (OP1 + 16*i) and (OP2 + 16*i) and XORs the components of the
+ * 256-bit product into LO, MI, HI.
+ *
+ * The multiplication produces four parts:
+ *   LOW: The polynomial given by performing carryless multiplication of the
+ *   bottom 64-bits of each polynomial
+ *   MID1: The polynomial given by performing carryless multiplication of the

arch/x86/crypto/polyval-clmulni_asm.S line 89

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+ *   bottom 64-bits of the first polynomial and the top 64-bits of the second
+ *   MID2: The polynomial given by performing carryless multiplication of the
+ *   bottom 64-bits of the second polynomial and the top 64-bits of the first
+ *   HIGH: The polynomial given by performing carryless multiplication of the
+ *   top 64-bits of each polynomial
+ *
+ * We compute:
+ *  LO ^= LOW
+ *  Mi ^= MID1 ^ MID2
+ *  Hi ^= HIGH

Mi => MI, and Hi => HI?

arch/x86/crypto/polyval-clmulni_asm.S line 96

Quoted 7 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +7 -0
+ *
+ * Later, the 256-bit result can be extracted as:
+ *   [HI_H : HI_L ^ MI_H : LO_H ^ MI_L : LO_L]
+ * This step is done when computing the polynomial reduction for efficiency
+ * reasons.
+ *
+ * If xor_sum == 1 then XOR the value of SUM into m_0.

In comments, please make it clear whether something is an additional behavior or an alternative behavior. In this case it's the former, so:

	If xor_sum == 1, then also XOR ...

arch/x86/crypto/polyval-clmulni_asm.S line 97

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+ * This avoids an extra multication of SUM and h^N.

multication => multiplication

Quoted 5 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +5 -0
+.macro schoolbook1_iteration i xor_sum
+	.set i, \i
+	.set xor_sum, \xor_sum
+	movups (16*i)(OP1), %xmm0
+	.if(i == 0 && xor_sum == 1)

arch/x86/crypto/polyval-clmulni_asm.S line 113

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+		pxor SUM, %xmm0
+	.endif
+	vpclmulqdq $0x01, (16*i)(OP2), %xmm0, %xmm1
+	vpxor %xmm1, MI, MI
+	vpclmulqdq $0x00, (16*i)(OP2), %xmm0, %xmm2
+	vpxor %xmm2, LO, LO
+	vpclmulqdq $0x11, (16*i)(OP2), %xmm0, %xmm3
+	vpxor %xmm3, HI, HI
+	vpclmulqdq $0x10, (16*i)(OP2), %xmm0, %xmm4
+	vpxor %xmm4, MI, MI

Perhaps the above multiplications and XORs should be reordered slightly so that each XOR doesn't depend on the previous instruction? A good ordering might be:

	vpclmulqdq $0x01, (16*\i)(OP2), %xmm0, %xmm1
	vpclmulqdq $0x10, (16*\i)(OP2), %xmm0, %xmm2
	vpclmulqdq $0x00, (16*\i)(OP2), %xmm0, %xmm3
	vpclmulqdq $0x11, (16*\i)(OP2), %xmm0, %xmm4
	vpxor %xmm1, MI, MI
	vpxor %xmm3, LO, LO
	vpxor %xmm4, HI, HI
	vpxor %xmm2, MI, MI

With that, no instruction would depend on either of the previous two instructions.

This might be more important in the ARM64 version than the x86_64 version, as x86_64 CPUs are pretty aggressive about internally reordering instructions. But it's something to consider in both versions.

Likewise in schoolbook1_noload.

Quoted 5 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +5 -0
+/*
+ * Computes the 128-bit reduction of PL, PH. Stores the result in PH.
+ *
+ * This macro computes p(x) mod g(x) where p(x) is in montgomery form and g(x) =
+ * x^128 + x^127 + x^126 + x^121 + 1.

arch/x86/crypto/polyval-clmulni_asm.S line 157

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+ *
+ * The montgomery form of a polynomial p(x) is p(x)x^{128}. Montgomery reduction
+ * works by simultaneously dividing by x^{128} and computing the modular
+ * reduction.
+ *
+ * Suppose we wish to reduce the montgomery form of p(x) = [P_3 : P_2 : P_1 :
+ * P_0] where P_i is a polynomial of degree at most 64 represented as 64-bits.
+ * Thus we would like to compute:
+ *   p(x) / x^{128} mod g(x)
+ *   = (P_3*x^{192} + P_2*x^{128} + P_1*x^{64} + P_0) / x^{128} mod g(x)

This doesn't explain why there is an additional factor of x^128 that must be removed. That's one of the key parts to understanding this.

Quoted 4 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +4 -0
+ * We now focus on dividing P_1*x^{64} + P_0 by x^{128}. We do this by making
+ * P_1*x^{64} + P_0 divisble by x^{128} then bitshifting. To add divisibility,
+ * we consider the polynomials mod x^{128}.
+ *

arch/x86/crypto/polyval-clmulni_asm.S line 175

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+ * Let c(x) = P_1*x^{64} + P_0.
+ *
+ * Now let m(x) = c(x) mod x^{128}
+ * and
+ * Let z(x) = [c(x) + m(x)g(x)] / x^{128}
+ *
+ * First notice that:
+ * c(x) + m(x)g(x) = c(x) mod g(x).
+ * Furthermore, g(x) mod x^{128} = 1, so we have
+ * c(x) + m(x)g(x) = c(x) + c(x) = 0 (mod x^{128}).

I don't understand this explanation. m(x) = c(x), so m(x) seems to have no purpose. Also, g(x) mod x^{128} is not 1.

I stopped reading here due to the above issues. I think this explanation is too long and misses the mark somewhat. I think that something shorter that explains the intuition behind the algorithm used would be much more helpful. E.g.:

"We have a 256-bit polynomial P_3 : P_2 : P_1 : P_0 that is the product of two 128-bit polynomials in Montgomery form. We need to reduce it mod g(x). Also, since polynomials in Montgomery form have an "extra" factor of x^128, this product has two extra factors of x^128. To get it back into Montgomery form, we need to remove one of these factors by dividing by x^128.

To accomplish both of these goals, we add multiples of g(x) that cancel out the low 128 bits P_1 : P_0, leaving just the high 128 bits, which is the answer.

Since the only nonzero term in the low 64 bits of g(x) is the constant term, the multiple of g(x) needed to cancel out P_0 is P_0 itself. The CPU can only do 64x64 bit multiplications, so split P_0 * g(x) into x^128 * P_0 + x^64 * g*(x) * P_0 + P_0, where g*(x) is bits 64-127 of g(x). Adding this to the original polynomial gives P_3 : P_2 + P_0 + T_1 : P_1 + T_0 : 0, where T = T_1 : T_0 = g*(x) * P0. Thus, bits 0-63 got "folded" into bits 64-191.

Repeating this same process on the next 64 bits "folds" bits 64-127 into bits 128-255, giving the answer in bits 128-255."

I think that covers everything except the optimization you've implemented where you avoid a second pshufd, which makes the second "fold" step in your code appear a bit different from the first. But do I understand correctly that the math is still the same in both steps in your version, and the difference between the steps is just where in the registers each part is stored?

arch/x86/crypto/polyval-clmulni_asm.S line 222

Quoted 3 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +3 -0
+/*
+ * Compute schoolbook multiplication for 8 blocks
+ * M_0h^8 + ... + M_7h^1 (no constant term)

Some places say M while other say m. It's probably best to use 'm', since the key is 'h' (both lower-case).

arch/x86/crypto/polyval-clmulni_asm.S line 225

Quoted 3 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +3 -0
+ *
+ * If reduce is set, computes the montgomery reduction of the
+ * previous full_stride call and XORs with the first message block.

computes => also computes

arch/x86/crypto/polyval-clmulni_asm.S line 226

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+ * (M_0 + REDUCE(PL, PH))h^8 + ... + M_7h^1 (no constant term)

"I.e., the first multiplication uses m_0 + REDUCE(PL, PH) instead of m_0."

arch/x86/crypto/polyval-clmulni_asm.S line 379

Quoted 6 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +6 -0
+/*
+ * Perform polynomial evaluation as specified by POLYVAL. If nblocks = k, this
+ * routine multiplies the value stored at accumulator by h^k and XORs the
+ * evaluated polynomial into it.
+ *
+ * Computes h^k * accumulator + h^kM_0 + ... + h^1M_{k-1} (No constant term)

The above explanation is confusing, especially the second sentence. How about:

* Perform polynomial evaluation as specified by POLYVAL. This computes:
*
* h^k * accumulator + h^k * M_0 + ... + h^1 * m_{k-1}
*
* ... where k=nblocks, h is the hash key, and m_i are the message blocks.

Also, perhaps use n instead of k, given that k could be confused with the key?

arch/x86/crypto/polyval-clmulni_asm.S line 382

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+ * rsi - pointer to precomputed key struct

pointer to the precomputed key powers h^8 ... h^1

arch/x86/crypto/polyval-clmulni_asm.S line 384

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+ * rcx - location to XOR with evaluated polynomial

pointer to the current accumulator

arch/x86/crypto/polyval-clmulni_asm.S line 110

Quoted 2 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +2 -0
+ *
+ * void clmul_polyval_update(const u8 *in, const struct polyhash_ctx* ctx,

polyval_ctx, not polyhash_ctx

arch/x86/crypto/polyval-clmulni_asm.S line 112

Quoted 2 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +2 -0
+#define POLYVAL_BLOCK_SIZE	16
+#define POLYVAL_DIGEST_SIZE	16

Why not get the above definitions by including <crypto/polyval.h>?

Or to put it another way, what is the purpose of having that header file if all the POLYVAL implementations aren't going to include it?

arch/x86/crypto/polyval-clmulni_asm.S line 122

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+static int polyval_final(struct shash_desc *desc, u8 *dst)
+{
+	struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+	struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm);
+	u8 *buf = dctx->buffer;
+
+	if (dctx->bytes) {
+		kernel_fpu_begin();
+		clmul_polyval_mul((u8 *)buf,
+			(u8 *)&ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]);

The cast of buf to 'u8 *' is unnecessary.

arch/x86/crypto/polyval-clmulni_asm.S line 123

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+		.cra_driver_name	= "__polyval-pclmulqdqni",

__polyval-clmulni

arch/x86/crypto/polyval-clmulni_asm.S line 124

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+static int __init polyval_pclmulqdqni_mod_init(void)

polyval_clmulni_mod_init

Quoted 12 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +12 -0
+{
+	int err;
+
+	if (!x86_match_cpu(pcmul_cpu_id))
+		return -ENODEV;
+
+	err = crypto_register_shash(&polyval_alg);
+	if (err)
+		goto err_out;
+	err = crypto_register_ahash(&polyval_async_alg);
+	if (err)
+		goto err_shash;

arch/x86/crypto/polyval-clmulni_asm.S line 146

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+
+	return 0;
+
+err_shash:
+	crypto_unregister_shash(&polyval_alg);
+err_out:
+	return err;
+}
+
+static void __exit polyval_pclmulqdqni_mod_exit(void)

polyval_clmulni_mod_exit

arch/x86/crypto/polyval-clmulni_asm.S line 147

Quoted 1 line
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
+MODULE_ALIAS_CRYPTO("polyval");

Add:

MODULE_ALIAS_CRYPTO("polyval-clmulni");

Otherwise this module won't be loaded if someone requests this by implementation name.

Quoted 7 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +1 -0
 config CRYPTO_POLYVAL
 	tristate
+	select CRYPTO_CRYPTD
 	select CRYPTO_GF128MUL
 	select CRYPTO_HASH
 	help
 	  POLYVAL is the hash function used in HCTR2.  It is not a general-purpose

arch/x86/crypto/polyval-clmulni_asm.S

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +8 -0
 	  cryptographic hash function.
 
+config CRYPTO_POLYVAL_CLMUL_NI
+	tristate "POLYVAL hash function (CLMUL-NI accelerated)"
+	depends on X86 && 64BIT
+	select CRYPTO_POLYVAL
+	help
+	  This is the x86_64 CLMUL-NI accelerated implementation of POLYVAL. It is
+	  used to efficiently implement HCTR2 on x86-64 processors that support
+	  carry-less multiplication instructions.

Shouldn't the 'select CRYPTO_CRYPTD' be under CRYPTO_POLYVAL_CLMUL_NI, not CRYPTO_POLYVAL?

- Eric

#16 lore raw

Re: [RFC PATCH v2 6/7] crypto: x86/polyval: Add PCLMULQDQ accelerated implementation of POLYVAL

Quoted 28 lines · Eric Biggers
Quoted 5 lines

+.macro schoolbook1_iteration i xor_sum

arch/x86/crypto/polyval-clmulni_asm.S new file +4 -0
+	.set i, \i
+	.set xor_sum, \xor_sum
+	movups (16*i)(OP1), %xmm0
+	.if(i == 0 && xor_sum == 1)

arch/x86/crypto/polyval-clmulni_asm.S line 113

Quoted 10 lines
arch/x86/crypto/polyval-clmulni_asm.S new file +10 -0
+		pxor SUM, %xmm0
+	.endif
+	vpclmulqdq $0x01, (16*i)(OP2), %xmm0, %xmm1
+	vpxor %xmm1, MI, MI
+	vpclmulqdq $0x00, (16*i)(OP2), %xmm0, %xmm2
+	vpxor %xmm2, LO, LO
+	vpclmulqdq $0x11, (16*i)(OP2), %xmm0, %xmm3
+	vpxor %xmm3, HI, HI
+	vpclmulqdq $0x10, (16*i)(OP2), %xmm0, %xmm4
+	vpxor %xmm4, MI, MI

Perhaps the above multiplications and XORs should be reordered slightly so that each XOR doesn't depend on the previous instruction? A good ordering might be:

	vpclmulqdq $0x01, (16*\i)(OP2), %xmm0, %xmm1
	vpclmulqdq $0x10, (16*\i)(OP2), %xmm0, %xmm2
	vpclmulqdq $0x00, (16*\i)(OP2), %xmm0, %xmm3
	vpclmulqdq $0x11, (16*\i)(OP2), %xmm0, %xmm4
	vpxor %xmm1, MI, MI
	vpxor %xmm3, LO, LO
	vpxor %xmm4, HI, HI
	vpxor %xmm2, MI, MI
Quoted 8 lines

With that, no instruction would depend on either of the previous two instructions.

This might be more important in the ARM64 version than the x86_64 version, as x86_64 CPUs are pretty aggressive about internally reordering instructions. But it's something to consider in both versions.

Likewise in schoolbook1_noload.

Or slightly better:

vpclmulqdq $0x01, (16*\i)(OP2), %xmm0, %xmm2 vpclmulqdq $0x00, (16*\i)(OP2), %xmm0, %xmm1 vpclmulqdq $0x10, (16*\i)(OP2), %xmm0, %xmm3 vpclmulqdq $0x11, (16*\i)(OP2), %xmm0, %xmm4 vpxor %xmm2, MI, MI
vpxor %xmm1, LO, LO
vpxor %xmm4, HI, HI
vpxor %xmm3, MI, MI

- Eric

#17 lore raw

RFC PATCH v2 7/7 crypto: arm64/polyval: Add PMULL accelerated implementation of POLYVAL

In reply to Nathan Huckleberry

Add hardware accelerated version of POLYVAL for ARM64 CPUs with Crypto Extension support.

This implementation is accelerated using PMULL instructions to perform the finite field computations. For added efficiency, 8 blocks of the message are processed simultaneously by precomputing the first 8 powers of the key.

Karatsuba multiplication is used instead of Schoolbook multiplication because it was found to be slightly faster on ARM64 CPUs. Montgomery reduction must be used instead of Barrett reduction due to the difference in modulus between POLYVAL's field and other finite fields.

---

Changes since v1:
* Rename C, D, E to LO, MI, HI
* Add comments explaining POLYVAL
* Fix bug in handling of non-block-multiple messages
* Wrap shash in ahash (as done in ghash)

 arch/arm64/crypto/Kconfig           |   7 +
 arch/arm64/crypto/Makefile          |   3 +
 arch/arm64/crypto/polyval-ce-core.S | 405 ++++++++++++++++++++++++++++
 arch/arm64/crypto/polyval-ce-glue.c | 365 +++++++++++++++++++++++++
 4 files changed, 780 insertions(+)
 create mode 100644 arch/arm64/crypto/polyval-ce-core.S
 create mode 100644 arch/arm64/crypto/polyval-ce-glue.c
arch/arm64/crypto/Kconfig +7 -0
index 897f9a4b5b67..f7fbe8637e5c 100644
@@ -60,6 +60,13 @@ config CRYPTO_GHASH_ARM64_CE
 	select CRYPTO_GF128MUL
 	select CRYPTO_LIB_AES
 
+config CRYPTO_POLYVAL_ARM64_CE
+	tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)"
+	depends on KERNEL_MODE_NEON
+	select CRYPTO_CRYPTD
+	select CRYPTO_HASH
+	select CRYPTO_POLYVAL
+
 config CRYPTO_CRCT10DIF_ARM64_CE
 	tristate "CRCT10DIF digest algorithm using PMULL instructions"
 	depends on KERNEL_MODE_NEON && CRC_T10DIF
arch/arm64/crypto/Makefile +3 -0
index 09a805cc32d7..53f9af962b86 100644
@@ -26,6 +26,9 @@ sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o
 obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o
 ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o
 
+obj-$(CONFIG_CRYPTO_POLYVAL_ARM64_CE) += polyval-ce.o
+polyval-ce-y := polyval-ce-glue.o polyval-ce-core.o
+
 obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o
 crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o
 
arch/arm64/crypto/polyval-ce-core.S new file +405 -0

405 lines not on this page · raw message

arch/arm64/crypto/polyval-ce-glue.c new file +365 -0

365 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#18 lore raw

Re: [RFC PATCH v2 7/7] crypto: arm64/polyval: Add PMULL accelerated implementation of POLYVAL

A lot of the comments I made on the x86_64 version apply to the arm64 version, but a few more comments below:

Quoted 2 lines · Nathan Huckleberry

Add hardware accelerated version of POLYVAL for ARM64 CPUs with Crypto Extension support.

Nit: It's "Crypto Extensions", not "Crypto Extension".

arch/arm64/crypto/Kconfig config CRYPTO_GHASH_ARM64_CE · line 68

Quoted 6 lines
arch/arm64/crypto/Kconfig +6 -0
+config CRYPTO_POLYVAL_ARM64_CE
+	tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)"
+	depends on KERNEL_MODE_NEON
+	select CRYPTO_CRYPTD
+	select CRYPTO_HASH
+	select CRYPTO_POLYVAL

CRYPTO_POLYVAL selects CRYPTO_HASH already, so there's no need to select it here.

Quoted 73 lines

/*
* Handle any extra blocks before
* full_stride loop.
*/
.macro partial_stride

	eor		LO.16b, LO.16b, LO.16b
	eor		MI.16b, MI.16b, MI.16b
	eor		HI.16b, HI.16b, HI.16b
	add		KEY_START, x1, #(NUM_PRECOMPUTE_POWERS << 4)
	sub		KEY_START, KEY_START, PARTIAL_LEFT, lsl #4
	ld1		{v0.16b}, [KEY_START]
	mov		v1.16b, SUM.16b
	karatsuba1 v0 v1
	karatsuba2
	montgomery_reduction
	mov		SUM.16b, PH.16b
	eor		LO.16b, LO.16b, LO.16b
	eor		MI.16b, MI.16b, MI.16b
	eor		HI.16b, HI.16b, HI.16b
	mov		IND, XZR

.LloopPartial:
cmp IND, PARTIAL_LEFT
bge .LloopExitPartial

sub TMP, IND, PARTIAL_LEFT

cmp TMP, #-4
bgt .Lgt4Partial
ld1 {M0.16b, M1.16b, M2.16b, M3.16b}, [x0], #64
// Clobber key registers
ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [KEY_START], #64
karatsuba1 M0 KEY8
karatsuba1 M1 KEY7
karatsuba1 M2 KEY6
karatsuba1 M3 KEY5
add IND, IND, #4
b .LoutPartial

.Lgt4Partial:

	cmp		TMP, #-3
	bgt		.Lgt3Partial
	ld1		{M0.16b, M1.16b, M2.16b}, [x0], #48
	// Clobber key registers
	ld1		{KEY8.16b, KEY7.16b, KEY6.16b}, [KEY_START], #48
	karatsuba1 M0 KEY8
	karatsuba1 M1 KEY7
	karatsuba1 M2 KEY6
	add		IND, IND, #3
	b		.LoutPartial

.Lgt3Partial:

	cmp		TMP, #-2
	bgt		.Lgt2Partial
	ld1		{M0.16b, M1.16b}, [x0], #32
	// Clobber key registers
	ld1		{KEY8.16b, KEY7.16b}, [KEY_START], #32
	karatsuba1 M0 KEY8
	karatsuba1 M1 KEY7
	add		IND, IND, #2
	b		.LoutPartial

.Lgt2Partial:

	ld1		{M0.16b}, [x0], #16
	// Clobber key registers
	ld1		{KEY8.16b}, [KEY_START], #16
	karatsuba1 M0 KEY8
	add		IND, IND, #1
.LoutPartial:
	b .LloopPartial
.LloopExitPartial:
	karatsuba2
	montgomery_reduction
	eor		SUM.16b, SUM.16b, PH.16b
Quoted 1 line

.endm

This sort of logic for handling different message lengths is necessary, but it's partly why I think that testing different message lengths is so important -- there could be bugs that are specific to particular message lengths.

With CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y, we do get test coverage from the fuzz tests that compare implementations to the corresponding generic implementation. But, it's preferable to not rely on that and have good default test vectors too.

It looks like you already do a relatively good job with the message lengths in the polyval test vectors. But it might be worth adding a test vector where the length mod 128 is 112, so that the case where partial_stride processes 7 blocks is tested. Also one where the message length is greater than 128 (or even 256) bytes but isn't a multiple of 128, so that the case where *both* partial_stride and full_stride are executed is tested.

- Eric