crypto: Adiantum support
Subsystems (10)
Eric Biggers
maintainerv3 cover crypto: Adiantum support
Hello,
We've been working to find a way to bring storage encryption to entry-level Android devices like the inexpensive "Android Go" devices sold in developing countries, and some smartwatches. Unfortunately, often these devices still ship with no encryption, since for cost reasons they have to use older CPUs like ARM Cortex-A7; and these CPUs lack the ARMv8 Cryptography Extensions, making AES-XTS much too slow.
We're trying to change this, since we believe encryption is for everyone, not just those who can afford it. And while it's unknown how long CPUs without AES support will be around, there will likely always be a "low end"; and in any case it's immensely valuable to provide a software-optimized cipher that doesn't depend on hardware support. Lack of hardware support should not be an excuse for no encryption.
But after an extensive search (e.g. see [1]) we were unable to find an existing cipher that simultaneously meets the very strict performance requirements on ARM processors, is secure (including having sufficient security parameters as well as sufficient cryptanalysis of any primitive(s) used), is suitable for practical use in dm-crypt and fscrypt, *and* avoids any particularly controversial primitive.
Therefore, we (well, Paul Crowley did the real work) designed a new encryption mode, Adiantum. In essence, Adiantum makes it secure to use the ChaCha stream cipher for disk encryption. Adiantum is specified by our paper here: https://eprint.iacr.org/2018/720.pdf ("Adiantum:
length-preserving encryption for entry-level processors"). Reference code and test vectors are here: https://github.com/google/adiantum.
Most of the high-level concepts of Adiantum are not new; similar existing modes include XCB, HCTR, and HCH. Adiantum and these modes are true wide-block modes (tweakable super-pseudorandom permutations), so they actually provide a stronger notion of security than XTS.
Adiantum is an improved version of our previous algorithm, HPolyC [2]. Like HPolyC, Adiantum uses XChaCha12, two passes of an ?-almost-?-universal (?A?U) hash function, and one AES-256 encryption of a single 16-byte block. On ARM Cortex-A7, on 4096-byte messages Adiantum is about 4x faster than AES-256-XTS (about 5x for decryption), and about 30% faster than Speck128/256-XTS.
Adiantum is a construction, not a primitive. Its security is reducible to that of XChaCha12 and AES-256, subject to a security bound; the proof is in Section 5 of our paper. Therefore, one need not "trust" Adiantum; they only need trust XChaCha12 and AES-256. Note that of these two primitives, AES-256 currently has the lower security margin.
Adiantum is ~20% faster than HPolyC, with no loss of security; in fact, Adiantum's security bound is slightly better than HPolyC's. It does this by choosing a faster ?A?U hash function: it still uses Poly1305's ?A?U hash function, but now a hash function from the "NH" family of hash functions is used to "compress" the message by 32x first. NH is ?AU (as shown in the UMAC paper[3]) but is over twice as fast as Poly1305. Key agility is reduced, but that's acceptable for disk encryption.
NH is also very simple, and it's easy to implement in SIMD assembly, e.g. in ARM NEON. Now, to get good performance only a SIMD implementation of NH is required, not Poly1305. Therefore, Adiantum can be easier to port to new platforms than HPolyC, despite Adiantum's slightly increased complexity. For now this patchset only includes an ARM32 NEON implementation of NH, but as a proof of concept I've also written SSE2, AVX2, and ARM64 NEON implementations of NH; see
https://github.com/google/adiantum/tree/master/benchmark/src.
This patchset adds Adiantum to Linux's crypto API, focusing on generic and ARM32 implementations. Patches 1-9 add support for XChaCha20 and XChaCha12. Patches 10-13 add NHPoly1305 support, needed for Adiantum hashing. Patch 14 adds Adiantum support as a skcipher template.
Patch 15 adds Adiantum support to fscrypt ("file-based encryption"). In fscrypt, Adiantum is used for filenames encryption as well as contents encryption; since Adiantum is a SPRP, it fixes the information leak when filenames share a common prefix. We also take advantage of Adiantum's support for long tweaks to include the per-inode nonce directly in the tweak, which allows providing an option to skip the per-file key derivation, providing even greater performance benefits.
This patchset applies to v4.20-rc1. It can also be found in git at branch "adiantum-v3" of:
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
As before, the XChaCha and Poly1305 changes conflict with the new "Zinc" crypto library. But I don't know when Zinc will be merged, so for now I've continued to base this patchset on upstream. An experimental version of this patchset based on Zinc can be found at branch "adiantum-zinc" of the git repository above.
Again, for more details please read our paper:
Adiantum: length-preserving encryption for entry-level processors
(https://eprint.iacr.org/2018/720.pdf)
References:
[1] https://www.spinics.net/lists/linux-crypto/msg33000.html
[2] https://patchwork.kernel.org/cover/10558059/
[3] https://fastcrypto.org/umac/umac_proc.pdf
Changed since v2:
- Simplify the generic NH implementation.
- Add patches to reduce atomic walks and disabling preemption.
- Split Poly1305 changes into two patches.
- Add tcrypt test mode for Adiantum.
- Make NEON 'chacha_permute' a function rather than a macro.
- Use .base.* style when declaring algorithms.
- Replace BUG_ON() in chacha_permute() with WARN_ON_ONCE().
- Set Adiantum instance {min,max}_keysize correctly in all cases.
- Make the Adiantum template take the nhpoly1305 driver name as optional third argument (useful for testing).
Thanks to Ard Biesheuvel for reviewing the patches.
Changed since v1:
- Replace HPolyC with Adiantum (uses a faster hash function).
- Drop ARM accelerated Poly1305.
- Add fscrypt patch.
Eric Biggers (15):
crypto: chacha20-generic - add HChaCha20 library function
crypto: chacha20-generic - don't unnecessarily use atomic walk
crypto: chacha20-generic - add XChaCha20 support
crypto: chacha20-generic - refactor to allow varying number of rounds
crypto: chacha - add XChaCha12 support
crypto: arm/chacha20 - limit the preemption-disabled section
crypto: arm/chacha20 - add XChaCha20 support
crypto: arm/chacha20 - refactor to allow varying number of rounds
crypto: arm/chacha - add XChaCha12 support
crypto: poly1305 - use structures for key and accumulator
crypto: poly1305 - add Poly1305 core API
crypto: nhpoly1305 - add NHPoly1305 support
crypto: arm/nhpoly1305 - add NEON-accelerated NHPoly1305
crypto: adiantum - add Adiantum support
fscrypt: add Adiantum support
Documentation/filesystems/fscrypt.rst | 187 +-
arch/arm/crypto/Kconfig | 7 +-
arch/arm/crypto/Makefile | 6 +-
...hacha20-neon-core.S => chacha-neon-core.S} | 98 +-
arch/arm/crypto/chacha-neon-glue.c | 201 ++
arch/arm/crypto/chacha20-neon-glue.c | 127 -
arch/arm/crypto/nh-neon-core.S | 116 +
arch/arm/crypto/nhpoly1305-neon-glue.c | 77 +
arch/arm64/crypto/chacha20-neon-glue.c | 40 +-
arch/x86/crypto/chacha20_glue.c | 52 +-
arch/x86/crypto/poly1305_glue.c | 20 +-
crypto/Kconfig | 46 +-
crypto/Makefile | 4 +-
crypto/adiantum.c | 658 ++++
crypto/chacha20_generic.c | 137 -
crypto/chacha20poly1305.c | 10 +-
crypto/chacha_generic.c | 217 ++
crypto/nhpoly1305.c | 254 ++
crypto/poly1305_generic.c | 174 +-
crypto/tcrypt.c | 12 +
crypto/testmgr.c | 30 +
crypto/testmgr.h | 2856 ++++++++++++++++-
drivers/char/random.c | 51 +-
fs/crypto/crypto.c | 35 +-
fs/crypto/fname.c | 22 +-
fs/crypto/fscrypt_private.h | 66 +-
fs/crypto/keyinfo.c | 322 +-
fs/crypto/policy.c | 5 +-
include/crypto/chacha.h | 53 +
include/crypto/chacha20.h | 27 -
include/crypto/nhpoly1305.h | 74 +
include/crypto/poly1305.h | 28 +-
include/uapi/linux/fs.h | 4 +-
lib/Makefile | 2 +-
lib/{chacha20.c => chacha.c} | 59 +-
35 files changed, 5380 insertions(+), 697 deletions(-)
rename arch/arm/crypto/{chacha20-neon-core.S => chacha-neon-core.S} (90%)
create mode 100644 arch/arm/crypto/chacha-neon-glue.c
delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c
create mode 100644 arch/arm/crypto/nh-neon-core.S
create mode 100644 arch/arm/crypto/nhpoly1305-neon-glue.c
create mode 100644 crypto/adiantum.c
delete mode 100644 crypto/chacha20_generic.c
create mode 100644 crypto/chacha_generic.c
create mode 100644 crypto/nhpoly1305.c
create mode 100644 include/crypto/chacha.h
delete mode 100644 include/crypto/chacha20.h
create mode 100644 include/crypto/nhpoly1305.h
rename lib/{chacha20.c => chacha.c} (58%)Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 1/15 crypto: chacha20-generic - add HChaCha20 library function
From: Eric Biggers <[email protected]>
Refactor the unkeyed permutation part of chacha20_block() into its own function, then add hchacha20_block() which is the ChaCha equivalent of HSalsa20 and is an intermediate step towards XChaCha20 (see
https://cr.yp.to/snuffle/xsalsa-20081128.pdf). HChaCha20 skips the final addition of the initial state, and outputs only certain words of the state. It should not be used for streaming directly.
--- include/crypto/chacha20.h | 2 ++ lib/chacha20.c | 50 ++++++++++++++++++++++++++++++++++----- 2 files changed, 46 insertions(+), 6 deletions(-)
include/crypto/chacha20.h +2 -0
@@ -19,6 +19,8 @@ struct chacha20_ctx { };void chacha20_block (u32 *state,u8 *stream); +void hchacha20_block (const u32 *in,u32 *out); +void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize);
lib/chacha20.c +44 -6
72 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 2/15 crypto: chacha20-generic - don't unnecessarily use atomic walk
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
chacha20-generic doesn't use SIMD instructions or otherwise disable preemption, so passing atomic=true to skcipher_walk_virt() is unnecessary.
--- crypto/chacha20_generic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/chacha20_generic.c +1 -1
@@ -81,7 +81,7 @@ int crypto_chacha20_crypt(struct skcipher_request *req)u32 state[16 ];int err; - err =skcipher_walk_virt (&walk, req, true); + err =skcipher_walk_virt (&walk, req, false);crypto_chacha20_init (state, ctx, walk.iv);
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 3/15 crypto: chacha20-generic - add XChaCha20 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add support for the XChaCha20 stream cipher. XChaCha20 is the application of the XSalsa20 construction
(https://cr.yp.to/snuffle/xsalsa-20081128.pdf) to ChaCha20 rather than to Salsa20. XChaCha20 extends ChaCha20's nonce length from 64 bits (or 96 bits, depending on convention) to 192 bits, while provably retaining ChaCha20's security. XChaCha20 uses the ChaCha20 permutation to map the key and first 128 nonce bits to a 256-bit subkey. Then, it does the ChaCha20 stream cipher with the subkey and remaining 64 bits of nonce.
We need XChaCha support in order to add support for the Adiantum encryption mode. Note that to meet our performance requirements, we actually plan to primarily use the variant XChaCha12. But we believe it's wise to first add XChaCha20 as a baseline with a higher security margin, in case there are any situations where it can be used. Supporting both variants is straightforward.
Since XChaCha20's subkey differs for each request, XChaCha20 can't be a template that wraps ChaCha20; that would require re-keying the underlying ChaCha20 for every request, which wouldn't be thread-safe. Instead, we make XChaCha20 its own top-level algorithm which calls the ChaCha20 streaming implementation internally.
Similar to the existing ChaCha20 implementation, we define the IV to be the nonce and stream position concatenated together. This allows users to seek to any position in the stream.
I considered splitting the code into separate chacha20-common, chacha20, and xchacha20 modules, so that chacha20 and xchacha20 could be enabled/disabled independently. However, since nearly all the code is shared anyway, I ultimately decided there would have been little benefit to the added complexity of separate modules.
--- crypto/Kconfig | 14 +- crypto/chacha20_generic.c | 120 +++++--- crypto/testmgr.c | 6 + crypto/testmgr.h | 577 ++++++++++++++++++++++++++++++++++++++ include/crypto/chacha20.h | 14 +- 5 files changed, 689 insertions(+), 42 deletions(-)
crypto/Kconfig +9 -5
@@ -1387,18 +1387,22 @@ config CRYPTO_SALSA20 Bernstein <[email protected]>. See <http://cr.yp.to/snuffle.html> config CRYPTO_CHACHA20 - tristate "ChaCha20 cipher algorithm" + tristate "ChaCha20 stream cipher algorithms" select CRYPTO_BLKCIPHER help - ChaCha20 cipher algorithm, RFC7539. + The ChaCha20 and XChaCha20 stream cipher algorithms. ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the portable C implementation of ChaCha20. - - See also: + This is the portable C implementation of ChaCha20. See also: <http://cr.yp.to/chacha/chacha-20080128.pdf> + XChaCha20 is the application of the XSalsa20 construction to ChaCha20 + rather than to Salsa20. XChaCha20 extends ChaCha20's nonce length + from 64 bits (or 96 bits using the RFC7539 convention) to 192 bits, + while provably retaining ChaCha20's security. See also: + <https://cr.yp.to/snuffle/xsalsa-20081128.pdf> + config CRYPTO_CHACHA20_X86_64 tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)" depends on X86 && 64BIT
crypto/chacha20_generic.c +84 -36
157 lines not on this page · raw message
crypto/testmgr.c +6 -0
@@ -3544,6 +3544,12 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(aes_xcbc128_tv_template) } + }, { + .alg = "xchacha20", + .test = alg_test_skcipher, + .suite = { + .cipher =__VECS (xchacha20_tv_template) + }, }, { .alg = "xts(aes)", .test = alg_test_skcipher,
crypto/testmgr.h +577 -0
583 lines not on this page · raw message
include/crypto/chacha20.h +13 -1
@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ /* -* Common values for the ChaCha20 algorithm +* Common values and helper functions for the ChaCha20 and XChaCha20 algorithms. +* +* XChaCha20 extends ChaCha20's nonce to 192 bits, while provably retaining +* ChaCha20's security. Here they share the same key size, tfm context, and +* setkey function; only their IV size and encrypt/decrypt function differ. */ #ifndef _CRYPTO_CHACHA20_H
@@ -10,10 +14,15 @@#include <linux/types.h> #include <linux/crypto.h> +/* 32-bit stream position, then 96-bit nonce (RFC7539 convention) */ #define CHACHA20_IV_SIZE 16 +#define CHACHA20_KEY_SIZE 32#define CHACHA20_BLOCK_SIZE 64 +/* 192-bit nonce, then 64-bit stream position */ +#define XCHACHA20_IV_SIZE 32 +struct chacha20_ctx {u32 key[8 ]; };
@@ -22,8 +31,11 @@ void chacha20_block(u32 *state, u8 *stream);void hchacha20_block (const u32 *in,u32 *out);void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv); +int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize); +int crypto_chacha20_crypt (struct skcipher_request *req); +int crypto_xchacha20_crypt (struct skcipher_request *req);#endif
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 4/15 crypto: chacha20-generic - refactor to allow varying number of rounds
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
In preparation for adding XChaCha12 support, rename/refactor chacha20-generic to support different numbers of rounds. The justification for needing XChaCha12 support is explained in more detail in the patch "crypto: chacha - add XChaCha12 support".
The only difference between ChaCha{8,12,20} are the number of rounds itself; all other parts of the algorithm are the same. Therefore, remove the "20" from all definitions, structures, functions, files, etc. that will be shared by all ChaCha versions.
Also make ->setkey() store the round count in the chacha_ctx (previously chacha20_ctx). The generic code then passes the round count through to chacha_block(). There will be a ->setkey() function for each explicitly allowed round count; the encrypt/decrypt functions will be the same. I decided not to do it the opposite way (same ->setkey() function for all round counts, with different encrypt/decrypt functions) because that would have required more boilerplate code in architecture-specific implementations of ChaCha and XChaCha.
---
arch/arm/crypto/chacha20-neon-glue.c | 40 +++----
arch/arm64/crypto/chacha20-neon-glue.c | 40 +++----
arch/x86/crypto/chacha20_glue.c | 52 ++++-----
crypto/Makefile | 2 +-
crypto/chacha20poly1305.c | 10 +-
.../{chacha20_generic.c => chacha_generic.c} | 110 ++++++++++--------
drivers/char/random.c | 51 ++++----
include/crypto/chacha.h | 46 ++++++++
include/crypto/chacha20.h | 41 -------
lib/Makefile | 2 +-
lib/{chacha20.c => chacha.c} | 43 ++++---
11 files changed, 227 insertions(+), 210 deletions(-)
rename crypto/{chacha20_generic.c => chacha_generic.c} (57%)
create mode 100644 include/crypto/chacha.h
delete mode 100644 include/crypto/chacha20.h
rename lib/{chacha20.c => chacha.c} (67%)arch/arm/crypto/chacha20-neon-glue.c +20 -20
@@ -19,7 +19,7 @@*/ #include <crypto/algapi.h> -#include <crypto/chacha20.h> +#include <crypto/chacha.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h>
@@ -34,20 +34,20 @@ asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src);static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) { -u8 buf[CHACHA20_BLOCK_SIZE ]; +u8 buf[CHACHA_BLOCK_SIZE ]; -while (bytes >=CHACHA20_BLOCK_SIZE *4 ) { +while (bytes >=CHACHA_BLOCK_SIZE *4 ) {chacha20_4block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE *4 ; - src +=CHACHA20_BLOCK_SIZE *4 ; - dst +=CHACHA20_BLOCK_SIZE *4 ; + bytes -=CHACHA_BLOCK_SIZE *4 ; + src +=CHACHA_BLOCK_SIZE *4 ; + dst +=CHACHA_BLOCK_SIZE *4 ; state[12 ] +=4 ; } -while (bytes >=CHACHA20_BLOCK_SIZE ) { +while (bytes >=CHACHA_BLOCK_SIZE ) {chacha20_block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE ; - src +=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; + bytes -=CHACHA_BLOCK_SIZE ; + src +=CHACHA_BLOCK_SIZE ; + dst +=CHACHA_BLOCK_SIZE ; state[12 ]++; }if (bytes) {
@@ -60,17 +60,17 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,static int chacha20_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err; -if (req->cryptlen <=CHACHA20_BLOCK_SIZE || !may_use_simd ()) -return crypto_chacha20_crypt (req); +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +return crypto_chacha_crypt (req); err =skcipher_walk_virt (&walk, req, true); -crypto_chacha20_init (state, ctx, walk.iv); +crypto_chacha_init (state, ctx, walk.iv);kernel_neon_begin ();while (walk.nbytes >0 ) {
@@ -93,14 +93,14 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-neon", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct chacha_ctx), .base.cra_module = THIS_MODULE, - .min_keysize = CHACHA20_KEY_SIZE, - .max_keysize = CHACHA20_KEY_SIZE, - .ivsize = CHACHA20_IV_SIZE, - .chunksize = CHACHA20_BLOCK_SIZE, - .walksize = 4 * CHACHA20_BLOCK_SIZE, + .min_keysize = CHACHA_KEY_SIZE, + .max_keysize = CHACHA_KEY_SIZE, + .ivsize = CHACHA_IV_SIZE, + .chunksize = CHACHA_BLOCK_SIZE, + .walksize = 4 * CHACHA_BLOCK_SIZE, .setkey = crypto_chacha20_setkey, .encrypt = chacha20_neon, .decrypt = chacha20_neon,
arch/arm64/crypto/chacha20-neon-glue.c +20 -20
@@ -19,7 +19,7 @@*/ #include <crypto/algapi.h> -#include <crypto/chacha20.h> +#include <crypto/chacha.h> #include <crypto/internal/skcipher.h> #include <linux/kernel.h> #include <linux/module.h>
@@ -34,15 +34,15 @@ asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src);static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) { -u8 buf[CHACHA20_BLOCK_SIZE ]; +u8 buf[CHACHA_BLOCK_SIZE ]; -while (bytes >=CHACHA20_BLOCK_SIZE *4 ) { +while (bytes >=CHACHA_BLOCK_SIZE *4 ) {kernel_neon_begin ();chacha20_4block_xor_neon (state, dst, src);kernel_neon_end (); - bytes -=CHACHA20_BLOCK_SIZE *4 ; - src +=CHACHA20_BLOCK_SIZE *4 ; - dst +=CHACHA20_BLOCK_SIZE *4 ; + bytes -=CHACHA_BLOCK_SIZE *4 ; + src +=CHACHA_BLOCK_SIZE *4 ; + dst +=CHACHA_BLOCK_SIZE *4 ; state[12 ] +=4 ; }
@@ -50,11 +50,11 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,return ;kernel_neon_begin (); -while (bytes >=CHACHA20_BLOCK_SIZE ) { +while (bytes >=CHACHA_BLOCK_SIZE ) {chacha20_block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE ; - src +=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; + bytes -=CHACHA_BLOCK_SIZE ; + src +=CHACHA_BLOCK_SIZE ; + dst +=CHACHA_BLOCK_SIZE ; state[12 ]++; }if (bytes) {
@@ -68,17 +68,17 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,static int chacha20_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err; -if (!may_use_simd () || req->cryptlen <=CHACHA20_BLOCK_SIZE ) -return crypto_chacha20_crypt (req); +if (!may_use_simd () || req->cryptlen <=CHACHA_BLOCK_SIZE ) +return crypto_chacha_crypt (req); err =skcipher_walk_virt (&walk, req, false); -crypto_chacha20_init (state, ctx, walk.iv); +crypto_chacha_init (state, ctx, walk.iv);while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;
@@ -99,14 +99,14 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-neon", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct chacha_ctx), .base.cra_module = THIS_MODULE, - .min_keysize = CHACHA20_KEY_SIZE, - .max_keysize = CHACHA20_KEY_SIZE, - .ivsize = CHACHA20_IV_SIZE, - .chunksize = CHACHA20_BLOCK_SIZE, - .walksize = 4 * CHACHA20_BLOCK_SIZE, + .min_keysize = CHACHA_KEY_SIZE, + .max_keysize = CHACHA_KEY_SIZE, + .ivsize = CHACHA_IV_SIZE, + .chunksize = CHACHA_BLOCK_SIZE, + .walksize = 4 * CHACHA_BLOCK_SIZE, .setkey = crypto_chacha20_setkey, .encrypt = chacha20_neon, .decrypt = chacha20_neon,
arch/x86/crypto/chacha20_glue.c +26 -26
104 lines not on this page · raw message
crypto/Makefile +1 -1
@@ -116,7 +116,7 @@ obj-$(CONFIG_CRYPTO_KHAZAD) += khazad.o obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o -obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o +obj-$(CONFIG_CRYPTO_CHACHA20) += chacha_generic.o obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
crypto/chacha20poly1305.c +5 -5
@@ -13,7 +13,7 @@#include <crypto/internal/hash.h> #include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> -#include <crypto/chacha20.h> +#include <crypto/chacha.h> #include <crypto/poly1305.h> #include <linux/err.h> #include <linux/init.h>
@@ -51,7 +51,7 @@ struct poly_req { };struct chacha_req { -u8 iv[CHACHA20_IV_SIZE ]; +u8 iv[CHACHA_IV_SIZE ];struct scatterlist src[1 ];struct skcipher_request req;/* must be last member */ };
@@ -91,7 +91,7 @@ static void chacha_iv(u8 *iv, struct aead_request *req, u32 icb)memcpy (iv, &leicb,sizeof (leicb));memcpy (iv +sizeof (leicb), ctx->salt, ctx->saltlen);memcpy (iv +sizeof (leicb) + ctx->saltlen, req->iv, -CHACHA20_IV_SIZE -sizeof (leicb) - ctx->saltlen); +CHACHA_IV_SIZE -sizeof (leicb) - ctx->saltlen); }static int poly_verify_tag (struct aead_request *req)
@@ -494,7 +494,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,struct chachapoly_ctx *ctx =crypto_aead_ctx (aead);int err; -if (keylen != ctx->saltlen +CHACHA20_KEY_SIZE ) +if (keylen != ctx->saltlen +CHACHA_KEY_SIZE )return -EINVAL ; keylen -= ctx->saltlen;
@@ -639,7 +639,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb, err = -EINVAL ;/* Need 16-byte IV size, including Initial Block Counter value */ -if (crypto_skcipher_alg_ivsize (chacha) !=CHACHA20_IV_SIZE ) +if (crypto_skcipher_alg_ivsize (chacha) !=CHACHA_IV_SIZE ) goto out_drop_chacha;/* Not a stream cipher? */ if (chacha->base.cra_blocksize !=1 )
crypto/chacha_generic.c renamed from crypto/chacha20_generic.c +59 -51
217 lines not on this page · raw message
drivers/char/random.c +25 -26
158 lines not on this page · raw message
include/crypto/chacha.h new file +46 -0
46 lines not on this page · raw message
include/crypto/chacha20.h deleted +0 -41
41 lines not on this page · raw message
lib/Makefile +1 -1
@@ -20,7 +20,7 @@ KCOV_INSTRUMENT_dynamic_debug.o := n lib-y := ctype.o string.o vsprintf.o cmdline.o \ rbtree.o radix-tree.o timerqueue.o xarray.o \ idr.o int_sqrt.o extable.o \ - sha1.o chacha20.o irq_regs.o argv_split.o \ + sha1.o chacha.o irq_regs.o argv_split.o \ flex_proportions.o ratelimit.o show_mem.o \ is_single_threaded.o plist.o decompress.o kobject_uevent.o \ earlycpio.o seq_buf.o siphash.o dec_and_lock.o \
lib/chacha.c renamed from lib/chacha20.c +24 -19
91 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 5/15 crypto: chacha - add XChaCha12 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the generic implementation of ChaCha20 has been refactored to allow varying the number of rounds, add support for XChaCha12, which is the XSalsa construction applied to ChaCha12. ChaCha12 is one of the three ciphers specified by the original ChaCha paper
(https://cr.yp.to/chacha/chacha-20080128.pdf: "ChaCha, a variant of Salsa20"), alongside ChaCha8 and ChaCha20. ChaCha12 is faster than ChaCha20 but has a lower, but still large, security margin.
We need XChaCha12 support so that it can be used in the Adiantum encryption mode, which enables disk/file encryption on low-end mobile devices where AES-XTS is too slow as the CPUs lack AES instructions.
We'd prefer XChaCha20 (the more popular variant), but it's too slow on some of our target devices, so at least in some cases we do need the XChaCha12-based version. In more detail, the problem is that Adiantum is still much slower than we're happy with, and encryption still has a quite noticeable effect on the feel of low-end devices. Users and vendors push back hard against encryption that degrades the user experience, which always risks encryption being disabled entirely. So we need to choose the fastest option that gives us a solid margin of security, and here that's XChaCha12. The best known attack on ChaCha breaks only 7 rounds and has 2^235 time complexity, so ChaCha12's security margin is still better than AES-256's. Much has been learned about cryptanalysis of ARX ciphers since Salsa20 was originally designed in 2005, and it now seems we can be comfortable with a smaller number of rounds. The eSTREAM project also suggests the 12-round version of Salsa20 as providing the best balance among the different variants:
combining very good performance with a "comfortable margin of security".
Note that it would be trivial to add vanilla ChaCha12 in addition to XChaCha12. However, it's unneeded for now and therefore is omitted.
As discussed in the patch that introduced XChaCha20 support, I considered splitting the code into separate chacha-common, chacha20, xchacha20, and xchacha12 modules, so that these algorithms could be enabled/disabled independently. However, since nearly all the code is shared anyway, I ultimately decided there would have been little benefit to the added complexity.
--- crypto/Kconfig | 8 +- crypto/chacha_generic.c | 26 +- crypto/testmgr.c | 6 + crypto/testmgr.h | 578 ++++++++++++++++++++++++++++++++++++++++ include/crypto/chacha.h | 7 + lib/chacha.c | 6 +- 6 files changed, 625 insertions(+), 6 deletions(-)
crypto/Kconfig +6 -2
@@ -1387,10 +1387,10 @@ config CRYPTO_SALSA20 Bernstein <[email protected]>. See <http://cr.yp.to/snuffle.html> config CRYPTO_CHACHA20 - tristate "ChaCha20 stream cipher algorithms" + tristate "ChaCha stream cipher algorithms" select CRYPTO_BLKCIPHER help - The ChaCha20 and XChaCha20 stream cipher algorithms. + The ChaCha20, XChaCha20, and XChaCha12 stream cipher algorithms. ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. Bernstein and further specified in RFC7539 for use in IETF protocols.
@@ -1403,6 +1403,10 @@ config CRYPTO_CHACHA20 while provably retaining ChaCha20's security. See also: <https://cr.yp.to/snuffle/xsalsa-20081128.pdf> + XChaCha12 is XChaCha20 reduced to 12 rounds, with correspondingly + reduced security margin but increased performance. It can be needed + in some performance-sensitive scenarios. + config CRYPTO_CHACHA20_X86_64 tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)" depends on X86 && 64BIT
crypto/chacha_generic.c +25 -1
@@ -1,5 +1,5 @@/* -* ChaCha20 (RFC7539) and XChaCha20 stream cipher algorithms +* ChaCha and XChaCha stream ciphers, including ChaCha20 (RFC7539) * * Copyright (C) 2015 Martin Willi * Copyright (C) 2018 Google LLC
@@ -106,6 +106,13 @@ int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key, }EXPORT_SYMBOL_GPL (crypto_chacha20_setkey); +int crypto_chacha12_setkey (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keysize) +{ +return chacha_setkey (tfm, key, keysize,12 ); +} +EXPORT_SYMBOL_GPL (crypto_chacha12_setkey); +int crypto_chacha_crypt (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);
@@ -168,6 +175,21 @@ static struct skcipher_alg algs[] = { .setkey = crypto_chacha20_setkey, .encrypt = crypto_xchacha_crypt, .decrypt = crypto_xchacha_crypt, + }, { + .base.cra_name = "xchacha12", + .base.cra_driver_name = "xchacha12-generic", + .base.cra_priority = 100, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct chacha_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = CHACHA_KEY_SIZE, + .max_keysize = CHACHA_KEY_SIZE, + .ivsize = XCHACHA_IV_SIZE, + .chunksize = CHACHA_BLOCK_SIZE, + .setkey = crypto_chacha12_setkey, + .encrypt = crypto_xchacha_crypt, + .decrypt = crypto_xchacha_crypt, } };
@@ -191,3 +213,5 @@ MODULE_ALIAS_CRYPTO("chacha20");MODULE_ALIAS_CRYPTO ("chacha20-generic" );MODULE_ALIAS_CRYPTO ("xchacha20" );MODULE_ALIAS_CRYPTO ("xchacha20-generic" ); +MODULE_ALIAS_CRYPTO ("xchacha12" ); +MODULE_ALIAS_CRYPTO ("xchacha12-generic" );
crypto/testmgr.c +6 -0
@@ -3544,6 +3544,12 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(aes_xcbc128_tv_template) } + }, { + .alg = "xchacha12", + .test = alg_test_skcipher, + .suite = { + .cipher =__VECS (xchacha12_tv_template) + }, }, { .alg = "xchacha20", .test = alg_test_skcipher,
crypto/testmgr.h +578 -0
584 lines not on this page · raw message
include/crypto/chacha.h +7 -0
@@ -5,6 +5,11 @@* XChaCha extends ChaCha's nonce to 192 bits, while provably retaining ChaCha's * security. Here they share the same key size, tfm context, and setkey * function; only their IV size and encrypt/decrypt function differ. +* +* The ChaCha paper specifies 20, 12, and 8-round variants. In general, it is +* recommended to use the 20-round variant ChaCha20. However, the other +* variants can be needed in some performance-sensitive scenarios. The generic +* ChaCha code currently allows only the 20 and 12-round variants. */ #ifndef _CRYPTO_CHACHA_H
@@ -39,6 +44,8 @@ void crypto_chacha_init(u32 *state, struct chacha_ctx *ctx, u8 *iv);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize); +int crypto_chacha12_setkey (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keysize);int crypto_chacha_crypt (struct skcipher_request *req);int crypto_xchacha_crypt (struct skcipher_request *req);
lib/chacha.c +3 -3
@@ -21,7 +21,7 @@ static void chacha_permute(u32 *x, int nrounds)int i;/* whitelist the allowed round counts */ -WARN_ON_ONCE (nrounds !=20 ); +WARN_ON_ONCE (nrounds !=20 && nrounds !=12 );for (i =0 ; i < nrounds; i +=2 ) { x[0 ] += x[4 ]; x[12 ] =rol32 (x[12 ] ^ x[0 ],16 );
@@ -70,7 +70,7 @@ static void chacha_permute(u32 *x, int nrounds)* chacha_block - generate one keystream block and increment block counter * @state: input state matrix (16 32-bit words) * @stream: output keystream block (64 bytes) -* @nrounds: number of rounds (currently must be 20) +* @nrounds: number of rounds (20 or 12; 20 is recommended) * * This is the ChaCha core, a function from 64-byte strings to 64-byte strings. * The caller has already converted the endianness of the input. This function
@@ -96,7 +96,7 @@ EXPORT_SYMBOL(chacha_block);* hchacha_block - abbreviated ChaCha core, for XChaCha * @in: input state matrix (16 32-bit words) * @out: output (8 32-bit words) -* @nrounds: number of rounds (currently must be 20) +* @nrounds: number of rounds (20 or 12; 20 is recommended) * * HChaCha is the ChaCha equivalent of HSalsa and is an intermediate step * towards XChaCha (see https://cr.yp.to/snuffle/xsalsa-20081128.pdf). HChaCha
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 6/15 crypto: arm/chacha20 - limit the preemption-disabled section
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
To improve responsivesess, disable preemption for each step of the walk (which is at most PAGE_SIZE) rather than for the entire encryption/decryption operation.
--- arch/arm/crypto/chacha20-neon-glue.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm/crypto/chacha20-neon-glue.c +3 -3
@@ -68,22 +68,22 @@ static int chacha20_neon(struct skcipher_request *req)if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ())return crypto_chacha_crypt (req); - err =skcipher_walk_virt (&walk, req, true); + err =skcipher_walk_virt (&walk, req, false);crypto_chacha_init (state, ctx, walk.iv); -kernel_neon_begin ();while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); +kernel_neon_begin ();chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, nbytes); +kernel_neon_end (); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); } -kernel_neon_end ();return err; }
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 7/15 crypto: arm/chacha20 - add XChaCha20 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add an XChaCha20 implementation that is hooked up to the ARM NEON implementation of ChaCha20. This is needed for use in the Adiantum encryption mode; see the generic code patch,
"crypto: chacha20-generic - add XChaCha20 support", for more details.
We also update the NEON code to support HChaCha20 on one block, so we can use that in XChaCha20 rather than calling the generic HChaCha20. This required factoring the permutation out into its own macro.
--- arch/arm/crypto/Kconfig | 2 +- arch/arm/crypto/chacha20-neon-core.S | 70 ++++++++++++------ arch/arm/crypto/chacha20-neon-glue.c | 103 ++++++++++++++++++++------- 3 files changed, 126 insertions(+), 49 deletions(-)
arch/arm/crypto/Kconfig +1 -1
@@ -117,7 +117,7 @@ config CRYPTO_CRC32_ARM_CE select CRYPTO_HASH config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" + tristate "NEON accelerated ChaCha20 stream cipher algorithms" depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20
arch/arm/crypto/chacha20-neon-core.S +48 -22
90 lines not on this page · raw message
arch/arm/crypto/chacha20-neon-glue.c +77 -26
145 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Re: [RFC PATCH v3 07/15] crypto: arm/chacha20 - add XChaCha20 support
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Add an XChaCha20 implementation that is hooked up to the ARM NEON implementation of ChaCha20. This is needed for use in the Adiantum encryption mode; see the generic code patch,
"crypto: chacha20-generic - add XChaCha20 support", for more details.
Quoted 5 lines
We also update the NEON code to support HChaCha20 on one block, so we can use that in XChaCha20 rather than calling the generic HChaCha20. This required factoring the permutation out into its own macro.
Signed-off-by Eric Biggers <[email protected]>
Quoted 274 lines
--- arch/arm/crypto/Kconfig | 2 +- arch/arm/crypto/chacha20-neon-core.S | 70 ++++++++++++------ arch/arm/crypto/chacha20-neon-glue.c | 103 ++++++++++++++++++++------- 3 files changed, 126 insertions(+), 49 deletions(-)arch/arm/crypto/Kconfig +1
-1@@ -117,7 +117,7 @@ config CRYPTO_CRC32_ARM_CE select CRYPTO_HASH config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" + tristate "NEON accelerated ChaCha20 stream cipher algorithms" depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20arch/arm/crypto/chacha20-neon-core.S +48
-22@@ -52,27 +52,16 @@ .fpu neon .align 5 -ENTRY(chacha20_block_xor_neon) - // r0: Input state matrix, s - // r1: 1 data block output, o - // r2: 1 data block input, i - - // - // This function encrypts one ChaCha20 block by loading the state matrix - // in four NEON registers. It performs matrix operation on four words in - // parallel, but requireds shuffling to rearrange the words after each - // round. - // - - // x0..3 = s0..3 - add ip, r0, #0x20 - vld1.32 {q0-q1}, [r0] - vld1.32 {q2-q3}, [ip] - - vmov q8, q0 - vmov q9, q1 - vmov q10, q2 - vmov q11, q3 +/* + * chacha20_permute - permute one block + * + * Permute one 64-byte block where the state matrix is stored in the four NEON + * registers q0-q3. It performs matrix operations on four words in parallel, + * but requires shuffling to rearrange the words after each round. + * + * Clobbers: r3, ip, q4-q5 + */ +chacha20_permute: adr ip, .Lrol8_table mov r3, #10@@ -142,6 +131,27 @@ ENTRY(chacha20_block_xor_neon) subs r3, r3, #1 bne .Ldoubleround + bx lr +ENDPROC(chacha20_permute) + +ENTRY(chacha20_block_xor_neon) + // r0: Input state matrix, s + // r1: 1 data block output, o + // r2: 1 data block input, i + push {lr} + + // x0..3 = s0..3 + add ip, r0, #0x20 + vld1.32 {q0-q1}, [r0] + vld1.32 {q2-q3}, [ip] + + vmov q8, q0 + vmov q9, q1 + vmov q10, q2 + vmov q11, q3 + + bl chacha20_permute + add ip, r2, #0x20 vld1.8 {q4-q5}, [r2] vld1.8 {q6-q7}, [ip]@@ -166,9 +176,25 @@ ENTRY(chacha20_block_xor_neon) vst1.8 {q0-q1}, [r1] vst1.8 {q2-q3}, [ip] - bx lr + pop {pc} ENDPROC(chacha20_block_xor_neon) +ENTRY(hchacha20_block_neon) + // r0: Input state matrix, s + // r1: output (8 32-bit words) + push {lr} + + vld1.32 {q0-q1}, [r0]! + vld1.32 {q2-q3}, [r0] + + bl chacha20_permute + + vst1.32 {q0}, [r1]! + vst1.32 {q3}, [r1] + + pop {pc} +ENDPROC(hchacha20_block_neon) + .align 4 .Lctrinc: .word 0, 1, 2, 3 .Lrol8_table: .byte 3, 0, 1, 2, 7, 4, 5, 6arch/arm/crypto/chacha20-neon-glue.c +77
-27@@ -1,5 +1,5 @@/* -* ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions +* ChaCha20 (RFC7539) and XChaCha20 stream ciphers, NEON accelerated * * Copyright (C) 2016 Linaro, Ltd. <[email protected]> * @@ -30,6 +30,7 @@ asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); +asmlinkage void hchacha20_block_neon(const u32 *state, u32 *out);static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src,unsigned int bytes)@@ -57,20 +58,16 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, } } -static int chacha20_neon (struct skcipher_request *req) +static int chacha20_neon_stream_xor (struct skcipher_request *req, +struct chacha_ctx *ctx,u8 *iv) { -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err; -if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) -return crypto_chacha_crypt (req); - err =skcipher_walk_virt (&walk, req, false); -crypto_chacha_init (state, ctx, walk.iv); +crypto_chacha_init (state, ctx, iv);while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;@@ -88,22 +85,73 @@ static int chacha20_neon(struct skcipher_request *req)return err; } -static struct skcipher_alg alg = { - .base.cra_name ="chacha20" , - .base.cra_driver_name ="chacha20-neon" , - .base.cra_priority =300 , - .base.cra_blocksize =1 , - .base.cra_ctxsize =sizeof (struct chacha_ctx ), - .base.cra_module =THIS_MODULE , - - .min_keysize =CHACHA_KEY_SIZE , - .max_keysize =CHACHA_KEY_SIZE , - .ivsize =CHACHA_IV_SIZE , - .chunksize =CHACHA_BLOCK_SIZE , - .walksize =4 *CHACHA_BLOCK_SIZE , - .setkey = crypto_chacha20_setkey, - .encrypt = chacha20_neon, - .decrypt = chacha20_neon, +static int chacha20_neon (struct skcipher_request *req) +{ +struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); +struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); + +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +return crypto_chacha_crypt (req); + +return chacha20_neon_stream_xor (req, ctx, req->iv); +} + +static int xchacha20_neon (struct skcipher_request *req) +{ +struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); +struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm); +struct chacha_ctx subctx; +u32 state[16 ]; +u8 real_iv[16 ]; + +if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ()) +return crypto_xchacha_crypt (req); + +crypto_chacha_init (state, ctx, req->iv); + +kernel_neon_begin (); +hchacha20_block_neon (state, subctx.key); +kernel_neon_end (); + +memcpy (&real_iv[0 ], req->iv +24 ,8 ); +memcpy (&real_iv[8 ], req->iv +16 ,8 ); +return chacha20_neon_stream_xor (req, &subctx, real_iv); +} + +static struct skcipher_alg algs[] = { + { + .base.cra_name ="chacha20" , + .base.cra_driver_name ="chacha20-neon" , + .base.cra_priority =300 , + .base.cra_blocksize =1 , + .base.cra_ctxsize =sizeof (struct chacha_ctx ), + .base.cra_module =THIS_MODULE , + + .min_keysize =CHACHA_KEY_SIZE , + .max_keysize =CHACHA_KEY_SIZE , + .ivsize =CHACHA_IV_SIZE , + .chunksize =CHACHA_BLOCK_SIZE , + .walksize =4 *CHACHA_BLOCK_SIZE , + .setkey = crypto_chacha20_setkey, + .encrypt = chacha20_neon, + .decrypt = chacha20_neon, + }, { + .base.cra_name ="xchacha20" , + .base.cra_driver_name ="xchacha20-neon" , + .base.cra_priority =300 , + .base.cra_blocksize =1 , + .base.cra_ctxsize =sizeof (struct chacha_ctx ), + .base.cra_module =THIS_MODULE , + + .min_keysize =CHACHA_KEY_SIZE , + .max_keysize =CHACHA_KEY_SIZE , + .ivsize =XCHACHA_IV_SIZE , + .chunksize =CHACHA_BLOCK_SIZE , + .walksize =4 *CHACHA_BLOCK_SIZE , + .setkey = crypto_chacha20_setkey, + .encrypt = xchacha20_neon, + .decrypt = xchacha20_neon, + } };static int __init chacha20_simd_mod_init (void )@@ -111,12 +159,12 @@ static int __init chacha20_simd_mod_init(void)if (!(elf_hwcap &HWCAP_NEON ))return -ENODEV ; -return crypto_register_skcipher (&alg); +return crypto_register_skciphers (algs,ARRAY_SIZE (algs)); } static void __exit chacha20_simd_mod_fini(void) { -crypto_unregister_skcipher (&alg); +crypto_unregister_skciphers (algs,ARRAY_SIZE (algs)); }module_init (chacha20_simd_mod_init);@@ -125,3 +173,6 @@ module_exit(chacha20_simd_mod_fini);MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" );MODULE_LICENSE ("GPL v2" );MODULE_ALIAS_CRYPTO ("chacha20" ); +MODULE_ALIAS_CRYPTO ("chacha20-neon" ); +MODULE_ALIAS_CRYPTO ("xchacha20" ); +MODULE_ALIAS_CRYPTO ("xchacha20-neon" ); --2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 8/15 crypto: arm/chacha20 - refactor to allow varying number of rounds
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
In preparation for adding XChaCha12 support, rename/refactor the NEON implementation of ChaCha20 to support different numbers of rounds.
---
arch/arm/crypto/Makefile | 4 +-
...hacha20-neon-core.S => chacha-neon-core.S} | 44 ++++++++-------
...hacha20-neon-glue.c => chacha-neon-glue.c} | 56 ++++++++++---------
3 files changed, 56 insertions(+), 48 deletions(-)
rename arch/arm/crypto/{chacha20-neon-core.S => chacha-neon-core.S} (94%)
rename arch/arm/crypto/{chacha20-neon-glue.c => chacha-neon-glue.c} (72%)arch/arm/crypto/Makefile +2 -2
@@ -9,7 +9,7 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o +obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
@@ -52,7 +52,7 @@ aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o +chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o ifdef REGENERATE_ARM_CRYPTO quiet_cmd_perl = PERL $@
arch/arm/crypto/chacha-neon-core.S renamed from arch/arm/crypto/chacha20-neon-core.S +24 -20
136 lines not on this page · raw message
arch/arm/crypto/chacha-neon-glue.c renamed from arch/arm/crypto/chacha20-neon-glue.c +30 -26
138 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Re: [RFC PATCH v3 08/15] crypto: arm/chacha20 - refactor to allow varying number of rounds
Quoted 6 lines · Eric Biggers
From: Eric Biggers <[email protected]>
In preparation for adding XChaCha12 support, rename/refactor the NEON implementation of ChaCha20 to support different numbers of rounds.
Signed-off-by Eric Biggers <[email protected]>
Quoted 340 lines
--- arch/arm/crypto/Makefile | 4 +- ...hacha20-neon-core.S => chacha-neon-core.S} | 44 ++++++++------- ...hacha20-neon-glue.c => chacha-neon-glue.c} | 56 ++++++++++--------- 3 files changed, 56 insertions(+), 48 deletions(-) rename arch/arm/crypto/{chacha20-neon-core.S => chacha-neon-core.S} (94%) rename arch/arm/crypto/{chacha20-neon-glue.c => chacha-neon-glue.c} (72%)arch/arm/crypto/Makefile +2
-2@@ -9,7 +9,7 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o +obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o@@ -52,7 +52,7 @@ aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o +chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o ifdef REGENERATE_ARM_CRYPTO quiet_cmd_perl = PERL $@arch/arm/crypto/chacha-neon-core.S renamed from arch/arm/crypto/chacha20-neon-core.S +24
-20@@ -1,5 +1,5 @@ /* - * ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions + * ChaCha/XChaCha NEON helper functions * * Copyright (C) 2016 Linaro, Ltd. <[email protected]> *@@ -27,9 +27,9 @@ * (d) vtbl.8 + vtbl.8 (multiple of 8 bits rotations only, * needs index vector) * - * ChaCha20 has 16, 12, 8, and 7-bit rotations. For the 12 and 7-bit - * rotations, the only choices are (a) and (b). We use (a) since it takes - * two-thirds the cycles of (b) on both Cortex-A7 and Cortex-A53. + * ChaCha has 16, 12, 8, and 7-bit rotations. For the 12 and 7-bit rotations, + * the only choices are (a) and (b). We use (a) since it takes two-thirds the + * cycles of (b) on both Cortex-A7 and Cortex-A53. * * For the 16-bit rotation, we use vrev32.16 since it's consistently fastest * and doesn't need a temporary register.@@ -53,18 +53,19 @@ .align 5 /* - * chacha20_permute - permute one block + * chacha_permute - permute one block * * Permute one 64-byte block where the state matrix is stored in the four NEON * registers q0-q3. It performs matrix operations on four words in parallel, * but requires shuffling to rearrange the words after each round. * + * The round count is given in r3. + * * Clobbers: r3, ip, q4-q5 */ -chacha20_permute: +chacha_permute: adr ip, .Lrol8_table - mov r3, #10 vld1.8 {d10}, [ip, :64] .Ldoubleround:@@ -128,16 +129,17 @@ chacha20_permute: // x3 = shuffle32(x3, MASK(0, 3, 2, 1)) vext.8 q3, q3, q3, #4 - subs r3, r3, #1 + subs r3, r3, #2 bne .Ldoubleround bx lr -ENDPROC(chacha20_permute) +ENDPROC(chacha_permute) -ENTRY(chacha20_block_xor_neon) +ENTRY(chacha_block_xor_neon) // r0: Input state matrix, s // r1: 1 data block output, o // r2: 1 data block input, i + // r3: nrounds push {lr} // x0..3 = s0..3@@ -150,7 +152,7 @@ ENTRY(chacha20_block_xor_neon) vmov q10, q2 vmov q11, q3 - bl chacha20_permute + bl chacha_permute add ip, r2, #0x20 vld1.8 {q4-q5}, [r2]@@ -177,30 +179,32 @@ ENTRY(chacha20_block_xor_neon) vst1.8 {q2-q3}, [ip] pop {pc} -ENDPROC(chacha20_block_xor_neon) +ENDPROC(chacha_block_xor_neon) -ENTRY(hchacha20_block_neon) +ENTRY(hchacha_block_neon) // r0: Input state matrix, s // r1: output (8 32-bit words) + // r2: nrounds push {lr} vld1.32 {q0-q1}, [r0]! vld1.32 {q2-q3}, [r0] - bl chacha20_permute + mov r3, r2 + bl chacha_permute vst1.32 {q0}, [r1]! vst1.32 {q3}, [r1] pop {pc} -ENDPROC(hchacha20_block_neon) +ENDPROC(hchacha_block_neon) .align 4 .Lctrinc: .word 0, 1, 2, 3 .Lrol8_table: .byte 3, 0, 1, 2, 7, 4, 5, 6 .align 5 -ENTRY(chacha20_4block_xor_neon) +ENTRY(chacha_4block_xor_neon) push {r4-r5} mov r4, sp // preserve the stack pointer sub ip, sp, #0x20 // allocate a 32 byte buffer@@ -210,9 +214,10 @@ ENTRY(chacha20_4block_xor_neon) // r0: Input state matrix, s // r1: 4 data blocks output, o // r2: 4 data blocks input, i + // r3: nrounds // - // This function encrypts four consecutive ChaCha20 blocks by loading + // This function encrypts four consecutive ChaCha blocks by loading // the state matrix in NEON registers four times. The algorithm performs // each operation on the corresponding word of each state matrix, hence // requires no word shuffling. The words are re-interleaved before the@@ -245,7 +250,6 @@ ENTRY(chacha20_4block_xor_neon) vdup.32 q0, d0[0] adr ip, .Lrol8_table - mov r3, #10 b 1f .Ldoubleround4:@@ -443,7 +447,7 @@ ENTRY(chacha20_4block_xor_neon) vsri.u32 q5, q8, #25 vsri.u32 q6, q9, #25 - subs r3, r3, #1 + subs r3, r3, #2 bne .Ldoubleround4 // x0..7[0-3] are in q0-q7, x10..15[0-3] are in q10-q15.@@ -553,4 +557,4 @@ ENTRY(chacha20_4block_xor_neon) pop {r4-r5} bx lr -ENDPROC(chacha20_4block_xor_neon) +ENDPROC(chacha_4block_xor_neon)arch/arm/crypto/chacha-neon-glue.c renamed from arch/arm/crypto/chacha20-neon-glue.c +30
-27@@ -28,24 +28,26 @@#include <asm/neon.h> #include <asm/simd.h> -asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); -asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); -asmlinkage void hchacha20_block_neon(const u32 *state, u32 *out); - -static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src, -unsigned int bytes) +asmlinkage void chacha_block_xor_neon(const u32 *state, u8 *dst, const u8 *src, +int nrounds); +asmlinkage void chacha_4block_xor_neon(const u32 *state, u8 *dst, const u8 *src, +int nrounds); +asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds); + +static void chacha_doneon (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes,int nrounds) {u8 buf[CHACHA_BLOCK_SIZE ];while (bytes >=CHACHA_BLOCK_SIZE *4 ) { -chacha20_4block_xor_neon (state, dst, src); +chacha_4block_xor_neon (state, dst, src, nrounds); bytes -=CHACHA_BLOCK_SIZE *4 ; src +=CHACHA_BLOCK_SIZE *4 ; dst +=CHACHA_BLOCK_SIZE *4 ; state[12 ] +=4 ; }while (bytes >=CHACHA_BLOCK_SIZE ) { -chacha20_block_xor_neon (state, dst, src); +chacha_block_xor_neon (state, dst, src, nrounds); bytes -=CHACHA_BLOCK_SIZE ; src +=CHACHA_BLOCK_SIZE ; dst +=CHACHA_BLOCK_SIZE ;@@ -53,13 +55,13 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, }if (bytes) {memcpy (buf, src, bytes); -chacha20_block_xor_neon (state, buf, buf); +chacha_block_xor_neon (state, buf, buf, nrounds);memcpy (dst, buf, bytes); } } -static int chacha20_neon_stream_xor (struct skcipher_request *req, -struct chacha_ctx *ctx,u8 *iv) +static int chacha_neon_stream_xor (struct skcipher_request *req, +struct chacha_ctx *ctx,u8 *iv) {struct skcipher_walk walk;u32 state[16 ];@@ -76,8 +78,8 @@ static int chacha20_neon_stream_xor(struct skcipher_request *req, nbytes =round_down (nbytes, walk.stride);kernel_neon_begin (); -chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +chacha_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, + nbytes, ctx->nrounds);kernel_neon_end (); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }@@ -85,7 +87,7 @@ static int chacha20_neon_stream_xor(struct skcipher_request *req,return err; } -static int chacha20_neon (struct skcipher_request *req) +static int chacha_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);@@ -93,10 +95,10 @@ static int chacha20_neon(struct skcipher_request *req)if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ())return crypto_chacha_crypt (req); -return chacha20_neon_stream_xor (req, ctx, req->iv); +return chacha_neon_stream_xor (req, ctx, req->iv); } -static int xchacha20_neon (struct skcipher_request *req) +static int xchacha_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req);struct chacha_ctx *ctx =crypto_skcipher_ctx (tfm);@@ -110,12 +112,13 @@ static int xchacha20_neon(struct skcipher_request *req)crypto_chacha_init (state, ctx, req->iv);kernel_neon_begin (); -hchacha20_block_neon (state, subctx.key); +hchacha_block_neon (state, subctx.key, ctx->nrounds);kernel_neon_end (); + subctx.nrounds = ctx->nrounds;memcpy (&real_iv[0 ], req->iv +24 ,8 );memcpy (&real_iv[8 ], req->iv +16 ,8 ); -return chacha20_neon_stream_xor (req, &subctx, real_iv); +return chacha_neon_stream_xor (req, &subctx, real_iv); }static struct skcipher_alg algs[] = {@@ -133,8 +136,8 @@ static struct skcipher_alg algs[] = { .chunksize = CHACHA_BLOCK_SIZE, .walksize = 4 * CHACHA_BLOCK_SIZE, .setkey = crypto_chacha20_setkey, - .encrypt = chacha20_neon, - .decrypt = chacha20_neon, + .encrypt = chacha_neon, + .decrypt = chacha_neon, }, { .base.cra_name = "xchacha20", .base.cra_driver_name = "xchacha20-neon",@@ -149,12 +152,12 @@ static struct skcipher_alg algs[] = { .chunksize = CHACHA_BLOCK_SIZE, .walksize = 4 * CHACHA_BLOCK_SIZE, .setkey = crypto_chacha20_setkey, - .encrypt = xchacha20_neon, - .decrypt = xchacha20_neon, + .encrypt = xchacha_neon, + .decrypt = xchacha_neon, } }; -static int __init chacha20_simd_mod_init (void ) +static int __init chacha_simd_mod_init (void ) {if (!(elf_hwcap &HWCAP_NEON ))return -ENODEV ;@@ -162,14 +165,15 @@ static int __init chacha20_simd_mod_init(void)return crypto_register_skciphers (algs,ARRAY_SIZE (algs)); } -static void __exit chacha20_simd_mod_fini(void) +static void __exit chacha_simd_mod_fini(void) {crypto_unregister_skciphers (algs,ARRAY_SIZE (algs)); } -module_init (chacha20_simd_mod_init); -module_exit (chacha20_simd_mod_fini); +module_init (chacha_simd_mod_init); +module_exit (chacha_simd_mod_fini); +MODULE_DESCRIPTION ("ChaCha and XChaCha stream ciphers (NEON accelerated)" );MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" );MODULE_LICENSE ("GPL v2" );MODULE_ALIAS_CRYPTO ("chacha20" ); --2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 9/15 crypto: arm/chacha - add XChaCha12 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the 32-bit ARM NEON implementation of ChaCha20 and XChaCha20 has been refactored to support varying the number of rounds, add support for XChaCha12. This is identical to XChaCha20 except for the number of rounds, which is 12 instead of 20.
XChaCha12 is faster than XChaCha20 but has a lower security margin, though still greater than AES-256's since the best known attacks make it through only 7 rounds. See the patch "crypto: chacha - add XChaCha12 support" for more details about why we need XChaCha12 support.
--- arch/arm/crypto/Kconfig | 2 +- arch/arm/crypto/chacha-neon-glue.c | 21 ++++++++++++++++++++- 2 files changed, 21 insertions(+), 2 deletions(-)
arch/arm/crypto/Kconfig +1 -1
@@ -117,7 +117,7 @@ config CRYPTO_CRC32_ARM_CE select CRYPTO_HASH config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 stream cipher algorithms" + tristate "NEON accelerated ChaCha stream cipher algorithms" depends on KERNEL_MODE_NEON select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20
arch/arm/crypto/chacha-neon-glue.c +20 -1
@@ -1,5 +1,6 @@/* -* ChaCha20 (RFC7539) and XChaCha20 stream ciphers, NEON accelerated +* ARM NEON accelerated ChaCha and XChaCha stream ciphers, +* including ChaCha20 (RFC7539) * * Copyright (C) 2016 Linaro, Ltd. <[email protected]> *
@@ -154,6 +155,22 @@ static struct skcipher_alg algs[] = { .setkey = crypto_chacha20_setkey, .encrypt = xchacha_neon, .decrypt = xchacha_neon, + }, { + .base.cra_name = "xchacha12", + .base.cra_driver_name = "xchacha12-neon", + .base.cra_priority = 300, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct chacha_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = CHACHA_KEY_SIZE, + .max_keysize = CHACHA_KEY_SIZE, + .ivsize = XCHACHA_IV_SIZE, + .chunksize = CHACHA_BLOCK_SIZE, + .walksize = 4 * CHACHA_BLOCK_SIZE, + .setkey = crypto_chacha12_setkey, + .encrypt = xchacha_neon, + .decrypt = xchacha_neon, } };
@@ -180,3 +197,5 @@ MODULE_ALIAS_CRYPTO("chacha20");MODULE_ALIAS_CRYPTO ("chacha20-neon" );MODULE_ALIAS_CRYPTO ("xchacha20" );MODULE_ALIAS_CRYPTO ("xchacha20-neon" ); +MODULE_ALIAS_CRYPTO ("xchacha12" ); +MODULE_ALIAS_CRYPTO ("xchacha12-neon" );
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 10/15 crypto: poly1305 - use structures for key and accumulator
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
In preparation for exposing a low-level Poly1305 API which implements the ?-almost-?-universal (?A?U) hash function underlying the Poly1305 MAC and supports block-aligned inputs only, create structures poly1305_key and poly1305_state which hold the limbs of the Poly1305 "r" key and accumulator, respectively.
--- arch/x86/crypto/poly1305_glue.c | 20 +++++++------ crypto/poly1305_generic.c | 52 ++++++++++++++++----------------- include/crypto/poly1305.h | 12 ++++++-- 3 files changed, 47 insertions(+), 37 deletions(-)
arch/x86/crypto/poly1305_glue.c +11 -9
@@ -83,35 +83,37 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx,if (poly1305_use_avx2 && srclen >=POLY1305_BLOCK_SIZE *4 ) {if (unlikely (!sctx->wset)) {if (!sctx->uset) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; }memcpy (sctx->u +5 , sctx->u,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +5 , dctx->r); +poly1305_simd_mult (sctx->u +5 , dctx->r.r);memcpy (sctx->u +10 , sctx->u +5 ,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +10 , dctx->r); +poly1305_simd_mult (sctx->u +10 , dctx->r.r); sctx->wset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *4 ); -poly1305_4block_avx2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_4block_avx2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *4 * blocks; srclen -=POLY1305_BLOCK_SIZE *4 * blocks; }#endif if (likely (srclen >=POLY1305_BLOCK_SIZE *2 )) {if (unlikely (!sctx->uset)) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *2 ); -poly1305_2block_sse2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_2block_sse2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *2 * blocks; srclen -=POLY1305_BLOCK_SIZE *2 * blocks; }if (srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_block_sse2 (dctx->h, src, dctx->r,1 ); +poly1305_block_sse2 (dctx->h.h, src, dctx->r.r,1 ); srclen -=POLY1305_BLOCK_SIZE ; }return srclen;
crypto/poly1305_generic.c +26 -26
88 lines not on this page · raw message
include/crypto/poly1305.h +10 -2
@@ -13,13 +13,21 @@#define POLY1305_KEY_SIZE 32#define POLY1305_DIGEST_SIZE 16 +struct poly1305_key { +u32 r[5 ];/* key, base 2^26 */ +}; + +struct poly1305_state { +u32 h[5 ];/* accumulator, base 2^26 */ +}; +struct poly1305_desc_ctx {/* key */ -u32 r[5 ]; +struct poly1305_key r;/* finalize key */ u32 s[4 ];/* accumulator */ -u32 h[5 ]; +struct poly1305_state h;/* partial buffer */ u8 buf[POLY1305_BLOCK_SIZE ];/* bytes used in partial buffer */
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Re: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
Quoted 169 lines · Eric Biggers
From: Eric Biggers <[email protected]>
In preparation for exposing a low-level Poly1305 API which implements the ?-almost-?-universal (?A?U) hash function underlying the Poly1305 MAC and supports block-aligned inputs only, create structures
poly1305_keyandpoly1305_statewhich hold the limbs of the Poly1305 "r" key and accumulator, respectively.Signed-off-by Eric Biggers <[email protected]>--- arch/x86/crypto/poly1305_glue.c | 20 +++++++------ crypto/poly1305_generic.c | 52 ++++++++++++++++----------------- include/crypto/poly1305.h | 12 ++++++-- 3 files changed, 47 insertions(+), 37 deletions(-)arch/x86/crypto/poly1305_glue.c +11
-9@@ -83,35 +83,37 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx,if (poly1305_use_avx2 && srclen >=POLY1305_BLOCK_SIZE *4 ) {if (unlikely (!sctx->wset)) {if (!sctx->uset) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; }memcpy (sctx->u +5 , sctx->u,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +5 , dctx->r); +poly1305_simd_mult (sctx->u +5 , dctx->r.r);memcpy (sctx->u +10 , sctx->u +5 ,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +10 , dctx->r); +poly1305_simd_mult (sctx->u +10 , dctx->r.r); sctx->wset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *4 ); -poly1305_4block_avx2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_4block_avx2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *4 * blocks; srclen -=POLY1305_BLOCK_SIZE *4 * blocks; }#endif if (likely (srclen >=POLY1305_BLOCK_SIZE *2 )) {if (unlikely (!sctx->uset)) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *2 ); -poly1305_2block_sse2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_2block_sse2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *2 * blocks; srclen -=POLY1305_BLOCK_SIZE *2 * blocks; }if (srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_block_sse2 (dctx->h, src, dctx->r,1 ); +poly1305_block_sse2 (dctx->h.h, src, dctx->r.r,1 ); srclen -=POLY1305_BLOCK_SIZE ; }return srclen;crypto/poly1305_generic.c +26
-26@@ -38,7 +38,7 @@ int crypto_poly1305_init(struct shash_desc *desc) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); -memset (dctx->h,0 ,sizeof (dctx->h)); +memset (dctx->h.h,0 ,sizeof (dctx->h.h)); dctx->buflen =0 ; dctx->rset = false; dctx->sset = false;@@ -50,11 +50,11 @@ EXPORT_SYMBOL_GPL(crypto_poly1305_init);static void poly1305_setrkey (struct poly1305_desc_ctx *dctx,const u8 *key) {/* r &= 0xffffffc0ffffffc0ffffffc0fffffff */ - dctx->r[0 ] = (get_unaligned_le32 (key +0 ) >>0 ) &0x3ffffff ; - dctx->r[1 ] = (get_unaligned_le32 (key +3 ) >>2 ) &0x3ffff03 ; - dctx->r[2 ] = (get_unaligned_le32 (key +6 ) >>4 ) &0x3ffc0ff ; - dctx->r[3 ] = (get_unaligned_le32 (key +9 ) >>6 ) &0x3f03fff ; - dctx->r[4 ] = (get_unaligned_le32 (key +12 ) >>8 ) &0x00fffff ; + dctx->r.r[0 ] = (get_unaligned_le32 (key +0 ) >>0 ) &0x3ffffff ; + dctx->r.r[1 ] = (get_unaligned_le32 (key +3 ) >>2 ) &0x3ffff03 ; + dctx->r.r[2 ] = (get_unaligned_le32 (key +6 ) >>4 ) &0x3ffc0ff ; + dctx->r.r[3 ] = (get_unaligned_le32 (key +9 ) >>6 ) &0x3f03fff ; + dctx->r.r[4 ] = (get_unaligned_le32 (key +12 ) >>8 ) &0x00fffff ; }static void poly1305_setskey (struct poly1305_desc_ctx *dctx,const u8 *key)@@ -107,22 +107,22 @@ static unsigned int poly1305_blocks(struct poly1305_desc_ctx *dctx, srclen = datalen; } - r0 = dctx->r[0 ]; - r1 = dctx->r[1 ]; - r2 = dctx->r[2 ]; - r3 = dctx->r[3 ]; - r4 = dctx->r[4 ]; + r0 = dctx->r.r[0 ]; + r1 = dctx->r.r[1 ]; + r2 = dctx->r.r[2 ]; + r3 = dctx->r.r[3 ]; + r4 = dctx->r.r[4 ]; s1 = r1 *5 ; s2 = r2 *5 ; s3 = r3 *5 ; s4 = r4 *5 ; - h0 = dctx->h[0 ]; - h1 = dctx->h[1 ]; - h2 = dctx->h[2 ]; - h3 = dctx->h[3 ]; - h4 = dctx->h[4 ]; + h0 = dctx->h.h[0 ]; + h1 = dctx->h.h[1 ]; + h2 = dctx->h.h[2 ]; + h3 = dctx->h.h[3 ]; + h4 = dctx->h.h[4 ];while (likely (srclen >=POLY1305_BLOCK_SIZE )) {@@ -157,11 +157,11 @@ static unsigned int poly1305_blocks(struct poly1305_desc_ctx *dctx, srclen -=POLY1305_BLOCK_SIZE ; } - dctx->h[0 ] = h0; - dctx->h[1 ] = h1; - dctx->h[2 ] = h2; - dctx->h[3 ] = h3; - dctx->h[4 ] = h4; + dctx->h.h[0 ] = h0; + dctx->h.h[1 ] = h1; + dctx->h.h[2 ] = h2; + dctx->h.h[3 ] = h3; + dctx->h.h[4 ] = h4;return srclen; }@@ -220,11 +220,11 @@ int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) }/* fully carry h */ - h0 = dctx->h[0 ]; - h1 = dctx->h[1 ]; - h2 = dctx->h[2 ]; - h3 = dctx->h[3 ]; - h4 = dctx->h[4 ]; + h0 = dctx->h.h[0 ]; + h1 = dctx->h.h[1 ]; + h2 = dctx->h.h[2 ]; + h3 = dctx->h.h[3 ]; + h4 = dctx->h.h[4 ]; h2 += (h1 >>26 ); h1 = h1 &0x3ffffff ; h3 += (h2 >>26 ); h2 = h2 &0x3ffffff ;include/crypto/poly1305.h +0
-0@@ -13,13 +13,21 @@#define POLY1305_KEY_SIZE 32
include/crypto/poly1305.h line 23
Quoted 10 lines
include/crypto/poly1305.h +8
-0#define POLY1305_DIGEST_SIZE 16 +struct poly1305_key { +u32 r[5 ];/* key, base 2^26 */ +}; + +struct poly1305_state { +u32 h[5 ];/* accumulator, base 2^26 */ +}; +
Sorry to bikeshed, but wouldn't it make more sense to have single definition
struct poly1305_val {
u32 v[5]; /* base 2^26 */
};and have 'key' and 'accum[ulator]' fields of type struct poly1305_val in the struct below?
Quoted 15 lines
include/crypto/poly1305.h +2
-3struct poly1305_desc_ctx {/* key */ -u32 r[5 ]; +struct poly1305_key r;/* finalize key */ u32 s[4 ];/* accumulator */ -u32 h[5 ]; +struct poly1305_state h;/* partial buffer */ u8 buf[POLY1305_BLOCK_SIZE ];/* bytes used in partial buffer */ --2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRe: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
Hi Ard,
Quoted 183 lines · Ard Biesheuvel
Quoted 169 lines · Eric Biggers
From: Eric Biggers <[email protected]>
In preparation for exposing a low-level Poly1305 API which implements the ?-almost-?-universal (?A?U) hash function underlying the Poly1305 MAC and supports block-aligned inputs only, create structures
poly1305_keyandpoly1305_statewhich hold the limbs of the Poly1305 "r" key and accumulator, respectively.Signed-off-by Eric Biggers <[email protected]>--- arch/x86/crypto/poly1305_glue.c | 20 +++++++------ crypto/poly1305_generic.c | 52 ++++++++++++++++----------------- include/crypto/poly1305.h | 12 ++++++-- 3 files changed, 47 insertions(+), 37 deletions(-)arch/x86/crypto/poly1305_glue.c +11
-9@@ -83,35 +83,37 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx,if (poly1305_use_avx2 && srclen >=POLY1305_BLOCK_SIZE *4 ) {if (unlikely (!sctx->wset)) {if (!sctx->uset) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; }memcpy (sctx->u +5 , sctx->u,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +5 , dctx->r); +poly1305_simd_mult (sctx->u +5 , dctx->r.r);memcpy (sctx->u +10 , sctx->u +5 ,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u +10 , dctx->r); +poly1305_simd_mult (sctx->u +10 , dctx->r.r); sctx->wset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *4 ); -poly1305_4block_avx2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_4block_avx2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *4 * blocks; srclen -=POLY1305_BLOCK_SIZE *4 * blocks; }#endif if (likely (srclen >=POLY1305_BLOCK_SIZE *2 )) {if (unlikely (!sctx->uset)) { -memcpy (sctx->u, dctx->r,sizeof (sctx->u)); -poly1305_simd_mult (sctx->u, dctx->r); +memcpy (sctx->u, dctx->r.r,sizeof (sctx->u)); +poly1305_simd_mult (sctx->u, dctx->r.r); sctx->uset = true; } blocks = srclen / (POLY1305_BLOCK_SIZE *2 ); -poly1305_2block_sse2 (dctx->h, src, dctx->r, blocks, sctx->u); +poly1305_2block_sse2 (dctx->h.h, src, dctx->r.r, blocks, + sctx->u); src +=POLY1305_BLOCK_SIZE *2 * blocks; srclen -=POLY1305_BLOCK_SIZE *2 * blocks; }if (srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_block_sse2 (dctx->h, src, dctx->r,1 ); +poly1305_block_sse2 (dctx->h.h, src, dctx->r.r,1 ); srclen -=POLY1305_BLOCK_SIZE ; }return srclen;crypto/poly1305_generic.c +26
-26@@ -38,7 +38,7 @@ int crypto_poly1305_init(struct shash_desc *desc) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); -memset (dctx->h,0 ,sizeof (dctx->h)); +memset (dctx->h.h,0 ,sizeof (dctx->h.h)); dctx->buflen =0 ; dctx->rset = false; dctx->sset = false;@@ -50,11 +50,11 @@ EXPORT_SYMBOL_GPL(crypto_poly1305_init);static void poly1305_setrkey (struct poly1305_desc_ctx *dctx,const u8 *key) {/* r &= 0xffffffc0ffffffc0ffffffc0fffffff */ - dctx->r[0 ] = (get_unaligned_le32 (key +0 ) >>0 ) &0x3ffffff ; - dctx->r[1 ] = (get_unaligned_le32 (key +3 ) >>2 ) &0x3ffff03 ; - dctx->r[2 ] = (get_unaligned_le32 (key +6 ) >>4 ) &0x3ffc0ff ; - dctx->r[3 ] = (get_unaligned_le32 (key +9 ) >>6 ) &0x3f03fff ; - dctx->r[4 ] = (get_unaligned_le32 (key +12 ) >>8 ) &0x00fffff ; + dctx->r.r[0 ] = (get_unaligned_le32 (key +0 ) >>0 ) &0x3ffffff ; + dctx->r.r[1 ] = (get_unaligned_le32 (key +3 ) >>2 ) &0x3ffff03 ; + dctx->r.r[2 ] = (get_unaligned_le32 (key +6 ) >>4 ) &0x3ffc0ff ; + dctx->r.r[3 ] = (get_unaligned_le32 (key +9 ) >>6 ) &0x3f03fff ; + dctx->r.r[4 ] = (get_unaligned_le32 (key +12 ) >>8 ) &0x00fffff ; }static void poly1305_setskey (struct poly1305_desc_ctx *dctx,const u8 *key)@@ -107,22 +107,22 @@ static unsigned int poly1305_blocks(struct poly1305_desc_ctx *dctx, srclen = datalen; } - r0 = dctx->r[0 ]; - r1 = dctx->r[1 ]; - r2 = dctx->r[2 ]; - r3 = dctx->r[3 ]; - r4 = dctx->r[4 ]; + r0 = dctx->r.r[0 ]; + r1 = dctx->r.r[1 ]; + r2 = dctx->r.r[2 ]; + r3 = dctx->r.r[3 ]; + r4 = dctx->r.r[4 ]; s1 = r1 *5 ; s2 = r2 *5 ; s3 = r3 *5 ; s4 = r4 *5 ; - h0 = dctx->h[0 ]; - h1 = dctx->h[1 ]; - h2 = dctx->h[2 ]; - h3 = dctx->h[3 ]; - h4 = dctx->h[4 ]; + h0 = dctx->h.h[0 ]; + h1 = dctx->h.h[1 ]; + h2 = dctx->h.h[2 ]; + h3 = dctx->h.h[3 ]; + h4 = dctx->h.h[4 ];while (likely (srclen >=POLY1305_BLOCK_SIZE )) {@@ -157,11 +157,11 @@ static unsigned int poly1305_blocks(struct poly1305_desc_ctx *dctx, srclen -=POLY1305_BLOCK_SIZE ; } - dctx->h[0 ] = h0; - dctx->h[1 ] = h1; - dctx->h[2 ] = h2; - dctx->h[3 ] = h3; - dctx->h[4 ] = h4; + dctx->h.h[0 ] = h0; + dctx->h.h[1 ] = h1; + dctx->h.h[2 ] = h2; + dctx->h.h[3 ] = h3; + dctx->h.h[4 ] = h4;return srclen; }@@ -220,11 +220,11 @@ int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) }/* fully carry h */ - h0 = dctx->h[0 ]; - h1 = dctx->h[1 ]; - h2 = dctx->h[2 ]; - h3 = dctx->h[3 ]; - h4 = dctx->h[4 ]; + h0 = dctx->h.h[0 ]; + h1 = dctx->h.h[1 ]; + h2 = dctx->h.h[2 ]; + h3 = dctx->h.h[3 ]; + h4 = dctx->h.h[4 ]; h2 += (h1 >>26 ); h1 = h1 &0x3ffffff ; h3 += (h2 >>26 ); h2 = h2 &0x3ffffff ;include/crypto/poly1305.h +0
-0@@ -13,13 +13,21 @@#define POLY1305_KEY_SIZE 32include/crypto/poly1305.h line 23
Quoted 10 lines
include/crypto/poly1305.h +8
-0#define POLY1305_DIGEST_SIZE 16 +struct poly1305_key { +u32 r[5 ];/* key, base 2^26 */ +}; + +struct poly1305_state { +u32 h[5 ];/* accumulator, base 2^26 */ +}; +Sorry to bikeshed, but wouldn't it make more sense to have single definition
Quoted 7 lines
struct poly1305_val { u32 v[5]; /* base 2^26 */ };and have 'key' and 'accum[ulator]' fields of type struct
poly1305_valin the struct below?
I prefer separate types so that the type system enforces that a key is never accidentally used as an accumulator, and vice versa. Then the poly1305_core_* functions will be harder to misuse, and the Poly1305 MAC implementations harder to get wrong.
The key also has certain bits clear whereas the accumulator does not. In the future, the Poly1305 C implementation might use base 2^32 and take advantage of this. In that case, the two inputs to each multiplication won't be interchangeable, so using the same type for both would be extra confusing.
Having a poly1305_val nested inside poly1305_key and poly1305_state would work, but seemed excessive.
Quoted 5 lines
Quoted 5 lines
include/crypto/poly1305.h +1
-1struct poly1305_desc_ctx {/* key */ -u32 r[5 ]; +struct poly1305_key r;/* finalize key */
Quoted 8 lines
include/crypto/poly1305.h
Quoted 8 lines
include/crypto/poly1305.h +1
-2u32 s[4 ];/* accumulator */ -u32 h[5 ]; +struct poly1305_state h;/* partial buffer */ u8 buf[POLY1305_BLOCK_SIZE ];/* bytes used in partial buffer */ --
Thanks,
- Eric
Herbert Xu
maintainerRe: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
Hi Eric:
Quoted 6 lines · Eric Biggers
I prefer separate types so that the type system enforces that a key is never accidentally used as an accumulator, and vice versa. Then the poly1305_core_* functions will be harder to misuse, and the Poly1305 MAC implementations harder to get wrong.
Quoted 7 lines
The key also has certain bits clear whereas the accumulator does not. In the future, the Poly1305 C implementation might use base 2^32 and take advantage of this. In that case, the two inputs to each multiplication won't be interchangeable, so using the same type for both would be extra confusing.
Having a
poly1305_valnested insidepoly1305_keyandpoly1305_statewould work, but seemed excessive.
So it looks like there are no more unresolved issues with this patch series. Please let me know when you want it to go in.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerRe: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
Hi Herbert,
Quoted 14 lines · Herbert Xu
Hi Eric:
Quoted 11 lines · Eric Biggers
I prefer separate types so that the type system enforces that a key is never accidentally used as an accumulator, and vice versa. Then the poly1305_core_* functions will be harder to misuse, and the Poly1305 MAC implementations harder to get wrong.
The key also has certain bits clear whereas the accumulator does not. In the future, the Poly1305 C implementation might use base 2^32 and take advantage of this. In that case, the two inputs to each multiplication won't be interchangeable, so using the same type for both would be extra confusing.
Quoted 6 lines
Quoted 2 lines
Having a
poly1305_valnested insidepoly1305_keyandpoly1305_statewould work, but seemed excessive.So it looks like there are no more unresolved issues with this patch series. Please let me know when you want it to go in.
I believe it's ready to go in. I'll need to rebase it onto cryptodev, to resolve some small conflicts with the recent ChaCha20-related changes. Also I'll probably omit the fscrypt patch (patch 15/15) so that that patch can be taken through the fscrypt tree instead.
Do you prefer that this be merged before or after Zinc? It seems it may still be a while before the community is satisfied with Zinc (and Wireguard which is in the same patchset), and I don't want this blocked unnecessarily... So on my part I'd prefer to just have this merged as-is.
Of course, it's always possible to change the xchacha12 and xchacha20 implementations later, whether that's to use "Zinc" or otherwise. And NHPoly1305 and Adiantum itself will be the same either way, except that the Poly1305 helpers may change slightly.
What do you think?
Thanks!
- Eric
Re: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
Quoted 35 lines · Eric Biggers
Hi Herbert,
Quoted 14 lines · Herbert Xu
Hi Eric:
Quoted 11 lines · Eric Biggers
I prefer separate types so that the type system enforces that a key is never accidentally used as an accumulator, and vice versa. Then the poly1305_core_* functions will be harder to misuse, and the Poly1305 MAC implementations harder to get wrong.
The key also has certain bits clear whereas the accumulator does not. In the future, the Poly1305 C implementation might use base 2^32 and take advantage of this. In that case, the two inputs to each multiplication won't be interchangeable, so using the same type for both would be extra confusing.
Quoted 6 lines
Quoted 2 lines
Having a
poly1305_valnested insidepoly1305_keyandpoly1305_statewould work, but seemed excessive.So it looks like there are no more unresolved issues with this patch series. Please let me know when you want it to go in.
I believe it's ready to go in. I'll need to rebase it onto cryptodev, to resolve some small conflicts with the recent ChaCha20-related changes. Also I'll probably omit the fscrypt patch (patch 15/15) so that that patch can be taken through the fscrypt tree instead.
Do you prefer that this be merged before or after Zinc? It seems it may still be a while before the community is satisfied with Zinc (and Wireguard which is in the same patchset), and I don't want this blocked unnecessarily... So on my part I'd prefer to just have this merged as-is.
Quoted 7 lines
Of course, it's always possible to change the xchacha12 and xchacha20 implementations later, whether that's to use "Zinc" or otherwise. And NHPoly1305 and Adiantum itself will be the same either way, except that the Poly1305 helpers may change slightly.
What do you think?
I think this is ready to go in. Most of Zinc itself will not be blocked by this, only the changes to the crypto API ChaCha20 implementation will have to wait until the Zinc version gains support for the reduced round variant, but that will not block WireGuard.
Jason A. Donenfeld
maintainerRe: [RFC PATCH v3 10/15] crypto: poly1305 - use structures for key and accumulator
In reply to Eric BiggersHi Eric,
Quoted 4 lines · Eric Biggers
Do you prefer that this be merged before or after Zinc? It seems it may still be a while before the community is satisfied with Zinc (and Wireguard which is in the same patchset), and I don't want this blocked unnecessarily... So on my part I'd prefer to just have this merged as-is.
Personally I'd prefer this be merged after Zinc, since there's work to be done on adjusting the 20->12 in chacha20. That's not really much of a reason though. But maybe we can just sidestep the ordering concern all together:
What I suspect we should do is make the initial Zinc merge _without_ those top two patches that replace the crypto api's implementations with Zinc, and defer those until after the initial merges. Those commits are already written -- so there's no chance it won't happen due to laziness or something -- and I think the general merge will go a bit more smoothly if we wait until after. (Somebody suggested we do it this way at Plumbers, and was actually a bit surprised I had already ported the crypto API stuff to Zinc.) This way, Adiantum and Zinc aren't potentially co-dependent in their initial merges and we can work on the details carefully with each other after both have landed. I figure this might make things a little bit less stressful for both of us. How would you feel about doing it that?
Jason
Herbert Xu
maintainerRFC PATCH 1/1 zinc chacha20 generic implementation using crypto API code
Quoted 5 lines · Jason A. Donenfeld
Personally I'd prefer this be merged after Zinc, since there's work to be done on adjusting the 20->12 in chacha20. That's not really much of a reason though. But maybe we can just sidestep the ordering concern all together:
In response to Martin's patch-set which I merged last week, I think here is quick way out for the zinc interface.
Going through the past zinc discussions it would appear that everybody is quite happy with the zinc interface per se. The most contentious areas are in fact the algorithm implementations under zinc, as well as the conversion of the crypto API algorithms over to using the zinc interface (e.g., you can no longer access specific implementations).
My proposal is to merge the zinc interface as is, but to invert how we place the algorithm implementations. IOW the implementations should stay where they are now, with in the crypto API. However, we will provide direct access to them for zinc without going through the crypto API. IOW we'll just export the functions directly.
Here is a proof of concept patch to do it for chacha20-generic. It basically replaces patch 3 in the October zinc patch series. Actually this patch also exports the arm/x86 chacha20 functions too so they are ready for use by zinc.
If everyone is happy with this then we can immediately add the zinc interface and expose the existing crypto API algorithms through it. This would allow wireguard to be merged right away.
In parallel, the discussions over replacing the implementations underneath can carry on without stalling the whole project.
PS This patch is totally untested :)
Thanks,
arch/arm/crypto/chacha20-neon-glue.c +6 -5
@@ -31,7 +31,7 @@ asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); -static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_doneon (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {u8 buf[CHACHA20_BLOCK_SIZE ];
@@ -56,11 +56,12 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,memcpy (dst, buf, bytes); } } +EXPORT_SYMBOL_GPL (crypto_chacha20_doneon);static int chacha20_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err;
@@ -79,8 +80,8 @@ static int chacha20_neon(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_doneon (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }kernel_neon_end ();
@@ -93,7 +94,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-neon", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
arch/x86/crypto/chacha20_glue.c +6 -5
@@ -39,7 +39,7 @@ static unsigned int chacha20_advance(unsigned int len, unsigned int maxblocks)return round_up (len,CHACHA20_BLOCK_SIZE ) /CHACHA20_BLOCK_SIZE ; } -static void chacha20_dosimd (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_dosimd (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {#ifdef CONFIG_AS_AVX2
@@ -85,11 +85,12 @@ static void chacha20_dosimd(u32 *state, u8 *dst, const u8 *src, state[12 ]++; } } +EXPORT_SYMBOL_GPL (crypto_chacha20_dosimd);static int chacha20_simd (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);u32 *state, state_buf[16 +2 ]__aligned(8) ;struct skcipher_walk walk;int err;
@@ -112,8 +113,8 @@ static int chacha20_simd(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_dosimd (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_dosimd (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }
@@ -128,7 +129,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-simd", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
crypto/chacha20_generic.c +8 -7
@@ -15,7 +15,7 @@#include <crypto/internal/skcipher.h> #include <linux/module.h> -static void chacha20_docrypt (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_generic (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {/* aligned to potentially speed up crypto_xor() */
@@ -35,8 +35,9 @@ static void chacha20_docrypt(u32 *state, u8 *dst, const u8 *src,crypto_xor (dst, stream, bytes); } } +EXPORT_SYMBOL_GPL (crypto_chacha20_generic); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv) +void crypto_chacha20_init (u32 *state,struct crypto_chacha20_ctx *ctx,u8 *iv) { state[0 ] =0x61707865 ;/* "expa" */ state[1 ] =0x3320646e ;/* "nd 3" */
@@ -60,7 +61,7 @@ EXPORT_SYMBOL_GPL(crypto_chacha20_init);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize) { -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);int i;if (keysize !=CHACHA20_KEY_SIZE )
@@ -76,7 +77,7 @@ EXPORT_SYMBOL_GPL(crypto_chacha20_setkey);int crypto_chacha20_crypt (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err;
@@ -91,8 +92,8 @@ int crypto_chacha20_crypt(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_docrypt (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_generic (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }
@@ -105,7 +106,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-generic", .base.cra_priority = 100, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
include/crypto/chacha20.h +8 -2
@@ -15,14 +15,20 @@#define CHACHA20_BLOCK_SIZE 64#define CHACHAPOLY_IV_SIZE 12 -struct chacha20_ctx { +struct crypto_chacha20_ctx {u32 key[8 ]; };void chacha20_block (u32 *state,u8 *stream); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv); +void crypto_chacha20_generic (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes); +void crypto_chacha20_init (u32 *state,struct crypto_chacha20_ctx *ctx,u8 *iv);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize);int crypto_chacha20_crypt (struct skcipher_request *req); +void crypto_chacha20_dosimd (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes); +void crypto_chacha20_doneon (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes);#endif
include/zinc/chacha20.h new file +70 -0
70 lines not on this page · raw message
lib/zinc/Kconfig +4 -0
@@ -1,3 +1,7 @@ +config ZINC_CHACHA20 + tristate + select CRYPTO_CHACHA20 + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" help
lib/zinc/Makefile +3 -0
@@ -1,3 +1,6 @@ ccflags-y := -O2 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc_chacha20-y := chacha20/chacha20.o +obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +108 -0
108 lines not on this page · raw message
lib/zinc/selftest/chacha20.c new file +2698 -0
2,657 lines not on this page · raw message
lib/zinc/selftest/run.h new file +48 -0
48 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Hi Herbert,
Quoted 5 lines · Herbert Xu
My proposal is to merge the zinc interface as is, but to invert how we place the algorithm implementations. IOW the implementations should stay where they are now, with in the crypto API. However, we will provide direct access to them for zinc without going through the crypto API. IOW we'll just export the functions directly.
Sorry, but there isn't a chance I'll agree to this. The whole idea is to have a clean place to focus on synchronous software implementations, and not keep it tangled up with the asynchronous API. If WireGuard and Zinc go in, it's going to be done right, not in a way that introduces more problems and organizational complexity. Avoiding the solution proposed here is kind of the point. And given that cryptographic code is so security sensitive, I want to make sure this is done right, as opposed to rushing it with some half-baked concoction that will be maybe possibly be replaced "later". From talking to folks at Plumbers, I believe most of the remaining issues are taken care of by the upcoming v9, and that you're overstating the status of discussions regarding that.
I think the remaining issue right now is how to order my series and Eric's series. If Eric's goes in first, it means that I can either a) spend some time developing Zinc further _now_ to support chacha12 and keep the top two conversion patches, or b) just drop the top two conversion patches and work that out carefully with Eric after. I think (b) would be better, but I'm happy to do (a) if you disagree. And as I mentioned in the last email, I'd prefer Eric's work to go in after Zinc, but I don't think the reasoning for that is particularly strong, so it seems fine to merge Eric's work first.
I'll post v9 pretty soon and you can see how things are shaping up. Hopefully before then Eric/Ard/you can provide some feedback on whether you'd prefer (a) or (b) above.
Regards,
Jason
Herbert Xu
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Hi Jason:
Quoted 4 lines · Jason A. Donenfeld
Sorry, but there isn't a chance I'll agree to this. The whole idea is to have a clean place to focus on synchronous software implementations, and not keep it tangled up with the asynchronous API.
There is nothing asynchronous in the relevant crypto code at all. The underlying CPU-based software implementations are all completely synchronous, including the architecture-specific ones such as x86/arm.
So I don't understand why you are rejecting it.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
In reply to Jason A. DonenfeldQuoted 9 lines · Jason A. Donenfeld
Hi Herbert,
Quoted 5 lines · Herbert Xu
My proposal is to merge the zinc interface as is, but to invert how we place the algorithm implementations. IOW the implementations should stay where they are now, with in the crypto API. However, we will provide direct access to them for zinc without going through the crypto API. IOW we'll just export the functions directly.
Quoted 12 lines
Sorry, but there isn't a chance I'll agree to this. The whole idea is to have a clean place to focus on synchronous software implementations, and not keep it tangled up with the asynchronous API. If WireGuard and Zinc go in, it's going to be done right, not in a way that introduces more problems and organizational complexity. Avoiding the solution proposed here is kind of the point. And given that cryptographic code is so security sensitive, I want to make sure this is done right, as opposed to rushing it with some half-baked concoction that will be maybe possibly be replaced "later". From talking to folks at Plumbers, I believe most of the remaining issues are taken care of by the upcoming v9, and that you're overstating the status of discussions regarding that.
Will v9 include a documentation file for Zinc in Documentation/crypto/? That's been suggested several times.
Quoted 11 lines
I think the remaining issue right now is how to order my series and Eric's series. If Eric's goes in first, it means that I can either a) spend some time developing Zinc further _now_ to support chacha12 and keep the top two conversion patches, or b) just drop the top two conversion patches and work that out carefully with Eric after. I think (b) would be better, but I'm happy to do (a) if you disagree. And as I mentioned in the last email, I'd prefer Eric's work to go in after Zinc, but I don't think the reasoning for that is particularly strong, so it seems fine to merge Eric's work first.
Quoted 4 lines
I'll post v9 pretty soon and you can see how things are shaping up. Hopefully before then Eric/Ard/you can provide some feedback on whether you'd prefer (a) or (b) above.
I'd still prefer to see the conversion patches included. Skipping them would be kicking the can down the road and avoiding issues that will need to be addressed anyway. Like you, I don't want a "half-baked concoction that will be maybe possibly be replaced 'later'" :-)
Either way though, it would make things much easier if you at least named the files, structures, constants, etc. "ChaCha" rather than "ChaCha20" from the start where appropriate. For an example, see the commit "crypto: chacha - prepare for supporting non-20-round variants" on my "adiantum-zinc" branch:
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/commit/?h=adiantum-zinc&id=754af8d7d39f31238114426e39786c84d7cc0f98
Then the actual introduction of the 12-round variant is much less noisy.
- Eric
Jason A. Donenfeld
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Hi Eric,
Quoted 2 lines · Eric Biggers
Will v9 include a documentation file for Zinc in
Documentation/crypto/? That's been suggested several times.
I had started writing that there, but then thought that the requested information could go in the commit message instead. But I'm guessing you're asking again now because you poked into the repo and didn't find the Documentation/, so presumably you still want it. I can reorganize the presentation of that to be more suitable for Documentation/, and I'll have that for v9.
Quoted 4 lines
I'd still prefer to see the conversion patches included. Skipping them would be kicking the can down the road and avoiding issues that will need to be addressed anyway. Like you, I don't want a "half-baked concoction that will be maybe possibly be replaced 'later'" :-)
Okay, fair enough. Will do.
Quoted 6 lines
Either way though, it would make things much easier if you at least named the files, structures, constants, etc. "ChaCha" rather than "ChaCha20" from the start where appropriate. For an example, see the commit "crypto: chacha - prepare for supporting non-20-round variants" on my "adiantum-zinc" branch:
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/commit/?h=adiantum-zinc&id=754af8d7d39f31238114426e39786c84d7cc0f98
Then the actual introduction of the 12-round variant is much less noisy.
That's a good idea. I'll do it like that. I'll likely order it as what we have now (renamed to omit the 20), and then put the 12 stuff on top of that, so it's easier to see what's changed in the process. I noticed in that branch, you didn't port the assembly to support fewer rounds. Shall I follow suite, and then expect patches from you later doing that? Or were you expecting me to also port the architecture implementations to chacha12 as well?
Regards,
Jason
Eric Biggers
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Quoted 6 lines · Jason A. Donenfeld
Hi Eric,
Quoted 2 lines · Eric Biggers
Will v9 include a documentation file for Zinc in
Documentation/crypto/? That's been suggested several times.
Quoted 7 lines
I had started writing that there, but then thought that the requested information could go in the commit message instead. But I'm guessing you're asking again now because you poked into the repo and didn't find the
Documentation/, so presumably you still want it. I can reorganize the presentation of that to be more suitable forDocumentation/, and I'll have that for v9.
It's much better to have the documentation in a permanent location.
Quoted 14 lines
Quoted 4 lines
I'd still prefer to see the conversion patches included. Skipping them would be kicking the can down the road and avoiding issues that will need to be addressed anyway. Like you, I don't want a "half-baked concoction that will be maybe possibly be replaced 'later'" :-)
Okay, fair enough. Will do.
Quoted 6 lines
Either way though, it would make things much easier if you at least named the files, structures, constants, etc. "ChaCha" rather than "ChaCha20" from the start where appropriate. For an example, see the commit "crypto: chacha - prepare for supporting non-20-round variants" on my "adiantum-zinc" branch:
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/commit/?h=adiantum-zinc&id=754af8d7d39f31238114426e39786c84d7cc0f98
Then the actual introduction of the 12-round variant is much less noisy.
Quoted 8 lines
That's a good idea. I'll do it like that. I'll likely order it as what we have now (renamed to omit the 20), and then put the 12 stuff on top of that, so it's easier to see what's changed in the process. I noticed in that branch, you didn't port the assembly to support fewer rounds. Shall I follow suite, and then expect patches from you later doing that? Or were you expecting me to also port the architecture implementations to chacha12 as well?
I actually did add ChaCha12 support to most of the Zinc assembly in "[WIP] crypto: assembly support for ChaCha12"
(https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/commit/?h=adiantum-zinc&id=0a7787a515a977e11b680f1752b430ca1744e399).
But I skipped AVX-512 and MIPS since I didn't have a way to test those yet, and I haven't ported the changes to your new perl scripts yet.
- Eric
Jason A. Donenfeld
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Hi Eric,
Quoted 1 line · Eric Biggers
It's much better to have the documentation in a permanent location.
Agreed.
Quoted 5 lines
I actually did add ChaCha12 support to most of the Zinc assembly in "[WIP] crypto: assembly support for ChaCha12"
(https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/commit/?h=adiantum-zinc&id=0a7787a515a977e11b680f1752b430ca1744e399).
But I skipped AVX-512 and MIPS since I didn't have a way to test those yet, and I haven't ported the changes to your new perl scripts yet.
Oh, great. If you're playing with this in the context of the WireGuard repo, you can run ARCH=mips make test-qemu -j$(nproc) (which is what's running on https://www.wireguard.com/build-status/ ). For AVX-512 testing, I've got a box I'm happy to give you access to, if you want to send an SSH key and your employer allows for that kind of thing etc. I can have a stab at all of this too if you like.
Jason
Herbert Xu
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
In reply to Eric BiggersQuoted 16 lines · Eric Biggers
Quoted 11 lines
I think the remaining issue right now is how to order my series and Eric's series. If Eric's goes in first, it means that I can either a) spend some time developing Zinc further _now_ to support chacha12 and keep the top two conversion patches, or b) just drop the top two conversion patches and work that out carefully with Eric after. I think (b) would be better, but I'm happy to do (a) if you disagree. And as I mentioned in the last email, I'd prefer Eric's work to go in after Zinc, but I don't think the reasoning for that is particularly strong, so it seems fine to merge Eric's work first.
Quoted 4 lines
I'll post v9 pretty soon and you can see how things are shaping up. Hopefully before then Eric/Ard/you can provide some feedback on whether you'd prefer (a) or (b) above.
Quoted 4 lines
I'd still prefer to see the conversion patches included. Skipping them would be kicking the can down the road and avoiding issues that will need to be addressed anyway. Like you, I don't want a "half-baked concoction that will be maybe possibly be replaced 'later'" :-)
Are you guys talking about the conversion patches to eliminate the two copies of chacha code that would otherwise exist in crypto as well as in zinc?
If so I think that's not negotiable. Having two copies of the same code is just not acceptable.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerRe: [RFC PATCH] zinc chacha20 generic implementation using crypto API code
Hi Herbert,
Quoted 11 lines · Herbert Xu
Quoted 4 lines
I'd still prefer to see the conversion patches included. Skipping them would be kicking the can down the road and avoiding issues that will need to be addressed anyway. Like you, I don't want a "half-baked concoction that will be maybe possibly be replaced 'later'" :-)
Are you guys talking about the conversion patches to eliminate the two copies of chacha code that would otherwise exist in crypto as well as in zinc?
If so I think that's not negotiable. Having two copies of the same code is just not acceptable.
Yes, that's the topic. Eric already expressed his preference to keep them, and I agreed, so the plan is to keep them.
Jason
Herbert Xu
maintainerv2 cover Exporting existing crypto API code through zinc
In reply to Herbert XuQuoted 22 lines · Herbert Xu
In response to Martin's patch-set which I merged last week, I think here is quick way out for the zinc interface.
Going through the past zinc discussions it would appear that everybody is quite happy with the zinc interface per se. The most contentious areas are in fact the algorithm implementations under zinc, as well as the conversion of the crypto API algorithms over to using the zinc interface (e.g., you can no longer access specific implementations).
My proposal is to merge the zinc interface as is, but to invert how we place the algorithm implementations. IOW the implementations should stay where they are now, with in the crypto API. However, we will provide direct access to them for zinc without going through the crypto API. IOW we'll just export the functions directly.
Here is a proof of concept patch to do it for chacha20-generic. It basically replaces patch 3 in the October zinc patch series. Actually this patch also exports the arm/x86 chacha20 functions too so they are ready for use by zinc.
Quoted 8 lines
If everyone is happy with this then we can immediately add the zinc interface and expose the existing crypto API algorithms through it. This would allow wireguard to be merged right away.
In parallel, the discussions over replacing the implementations underneath can carry on without stalling the whole project.
PS This patch is totally untested :)
Here is an updated version demonstrating how we could access the accelerated versions of chacha20. It also includes a final patch to deposit the new zinc version of x86-64 chacha20 into arch/x86/crypto where it can be used by both the crypto API as well as zinc.
The key differences between this and the last zinc patch series:
1. The crypto API algorithms remain individually accessible, this is crucial as these algorithm names are exported to user-space so changing the names to foo-zinc is not going to work.
2. The arch-specific algorithm code lives in their own module rather than in a monolithic module.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu
maintainerPATCH v2 1/4 crypto: chacha20 - Export chacha20 functions without crypto API
This patch exports the raw chacha20 functions, including the generic as well as x86/arm accelerated versions. This allows them to be used without going through the crypto API.
This patch also renames struct chacha20_ctx to crypto_chacha20_ctx to avoid naming conflicts with zinc.
In order to ensure that zinc can link to the requisite functions, this function removes the failure mode from the x86/arm accelerated glue code so that the modules will always load, even if the hardware is not available. In that case, the crypto API functions would not be registered.
---
arch/arm/crypto/chacha20-neon-glue.c | 16 ++++++++++------ arch/x86/crypto/chacha20_glue.c | 16 ++++++++++------ crypto/chacha20_generic.c | 15 ++++++++------- include/crypto/chacha20.h | 10 ++++++++-- 4 files changed, 36 insertions(+), 21 deletions(-)
arch/arm/crypto/chacha20-neon-glue.c +10 -6
@@ -31,7 +31,7 @@ asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); -static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_doneon (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {u8 buf[CHACHA20_BLOCK_SIZE ];
@@ -56,11 +56,12 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,memcpy (dst, buf, bytes); } } +EXPORT_SYMBOL_GPL (crypto_chacha20_doneon);static int chacha20_neon (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err;
@@ -79,8 +80,8 @@ static int chacha20_neon(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_doneon (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }kernel_neon_end ();
@@ -93,7 +94,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-neon", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
@@ -109,13 +110,16 @@ static struct skcipher_alg alg = {static int __init chacha20_simd_mod_init (void ) {if (!(elf_hwcap &HWCAP_NEON )) -return -ENODEV ; +return 0 ;return crypto_register_skcipher (&alg); } static void __exit chacha20_simd_mod_fini(void) { +if (!(elf_hwcap &HWCAP_NEON )) +return ; +crypto_unregister_skcipher (&alg); }
arch/x86/crypto/chacha20_glue.c +10 -6
@@ -39,7 +39,7 @@ static unsigned int chacha20_advance(unsigned int len, unsigned int maxblocks)return round_up (len,CHACHA20_BLOCK_SIZE ) /CHACHA20_BLOCK_SIZE ; } -static void chacha20_dosimd (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_dosimd (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {#ifdef CONFIG_AS_AVX2
@@ -85,11 +85,12 @@ static void chacha20_dosimd(u32 *state, u8 *dst, const u8 *src, state[12 ]++; } } +EXPORT_SYMBOL_GPL (crypto_chacha20_dosimd);static int chacha20_simd (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);u32 *state, state_buf[16 +2 ]__aligned(8) ;struct skcipher_walk walk;int err;
@@ -112,8 +113,8 @@ static int chacha20_simd(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_dosimd (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_dosimd (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }
@@ -128,7 +129,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-simd", .base.cra_priority = 300, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
@@ -143,7 +144,7 @@ static struct skcipher_alg alg = {static int __init chacha20_simd_mod_init (void ) {if (!boot_cpu_has (X86_FEATURE_SSSE3 )) -return -ENODEV ; +return 0 ;#ifdef CONFIG_AS_AVX2 chacha20_use_avx2 = boot_cpu_has(X86_FEATURE_AVX) &&
@@ -155,6 +156,9 @@ static int __init chacha20_simd_mod_init(void) static void __exit chacha20_simd_mod_fini(void) { +if (!boot_cpu_has (X86_FEATURE_SSSE3 )) +return ; +crypto_unregister_skcipher (&alg); }
crypto/chacha20_generic.c +8 -7
@@ -15,7 +15,7 @@#include <crypto/internal/skcipher.h> #include <linux/module.h> -static void chacha20_docrypt (u32 *state,u8 *dst,const u8 *src, +void crypto_chacha20_generic (u32 *state,u8 *dst,const u8 *src,unsigned int bytes) {/* aligned to potentially speed up crypto_xor() */
@@ -35,8 +35,9 @@ static void chacha20_docrypt(u32 *state, u8 *dst, const u8 *src,crypto_xor (dst, stream, bytes); } } +EXPORT_SYMBOL_GPL (crypto_chacha20_generic); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv) +void crypto_chacha20_init (u32 *state,struct crypto_chacha20_ctx *ctx,u8 *iv) { state[0 ] =0x61707865 ;/* "expa" */ state[1 ] =0x3320646e ;/* "nd 3" */
@@ -60,7 +61,7 @@ EXPORT_SYMBOL_GPL(crypto_chacha20_init);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize) { -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);int i;if (keysize !=CHACHA20_KEY_SIZE )
@@ -76,7 +77,7 @@ EXPORT_SYMBOL_GPL(crypto_chacha20_setkey);int crypto_chacha20_crypt (struct skcipher_request *req) {struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); +struct crypto_chacha20_ctx *ctx =crypto_skcipher_ctx (tfm);struct skcipher_walk walk;u32 state[16 ];int err;
@@ -91,8 +92,8 @@ int crypto_chacha20_crypt(struct skcipher_request *req)if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); -chacha20_docrypt (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); +crypto_chacha20_generic (state, walk.dst.virt.addr, + walk.src.virt.addr, nbytes); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); }
@@ -105,7 +106,7 @@ static struct skcipher_alg alg = { .base.cra_driver_name = "chacha20-generic", .base.cra_priority = 100, .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct chacha20_ctx), + .base.cra_ctxsize = sizeof(struct crypto_chacha20_ctx), .base.cra_module = THIS_MODULE, .min_keysize = CHACHA20_KEY_SIZE,
include/crypto/chacha20.h +8 -2
@@ -15,14 +15,20 @@#define CHACHA20_BLOCK_SIZE 64#define CHACHAPOLY_IV_SIZE 12 -struct chacha20_ctx { +struct crypto_chacha20_ctx {u32 key[8 ]; };void chacha20_block (u32 *state,u8 *stream); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv); +void crypto_chacha20_generic (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes); +void crypto_chacha20_init (u32 *state,struct crypto_chacha20_ctx *ctx,u8 *iv);int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key,unsigned int keysize);int crypto_chacha20_crypt (struct skcipher_request *req); +void crypto_chacha20_dosimd (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes); +void crypto_chacha20_doneon (u32 *state,u8 *dst,const u8 *src, +unsigned int bytes);#endif
Herbert Xu
maintainerPATCH v2 2/4 zinc: ChaCha20 generic C implementation and selftest
In reply to Herbert XuFrom: Jason A. Donenfeld <[email protected]>
This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
---
include/zinc/chacha20.h | 70 + lib/zinc/Kconfig | 4 lib/zinc/Makefile | 3 lib/zinc/chacha20/chacha20.c | 108 + lib/zinc/selftest/chacha20.c | 2698 +++++++++++++++++++++++++++++++++++++++++++ lib/zinc/selftest/run.h | 48 6 files changed, 2931 insertions(+)
include/zinc/chacha20.h new file +70 -0
@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20_H +#define _ZINC_CHACHA20_H + +#include <asm/unaligned.h> +#include <linux/simd.h> +#include <linux/kernel.h> +#include <linux/types.h> + +enum { +CHACHA20_NONCE_SIZE =16 , +CHACHA20_KEY_SIZE =32 , +CHACHA20_KEY_WORDS =CHACHA20_KEY_SIZE /sizeof (u32), +CHACHA20_BLOCK_SIZE =64 , +CHACHA20_BLOCK_WORDS =CHACHA20_BLOCK_SIZE /sizeof (u32), +HCHACHA20_NONCE_SIZE =CHACHA20_NONCE_SIZE , +HCHACHA20_KEY_SIZE =CHACHA20_KEY_SIZE +}; + +enum {/* expand 32-byte k */ +CHACHA20_CONSTANT_EXPA =0x61707865U , +CHACHA20_CONSTANT_ND_3 =0x3320646eU , +CHACHA20_CONSTANT_2_BY =0x79622d32U , +CHACHA20_CONSTANT_TE_K =0x6b206574U +}; + +struct chacha20_ctx { +union { +u32 state[16 ]; +struct { +u32 constant[4 ]; +u32 key[8 ]; +u32 counter[4 ]; + }; + }; +}; + +static inline void chacha20_init (struct chacha20_ctx *ctx, +const u8 key[CHACHA20_KEY_SIZE ], +const u64 nonce) +{ + ctx->constant[0 ] =CHACHA20_CONSTANT_EXPA ; + ctx->constant[1 ] =CHACHA20_CONSTANT_ND_3 ; + ctx->constant[2 ] =CHACHA20_CONSTANT_2_BY ; + ctx->constant[3 ] =CHACHA20_CONSTANT_TE_K ; + ctx->key[0 ] =get_unaligned_le32 (key +0 ); + ctx->key[1 ] =get_unaligned_le32 (key +4 ); + ctx->key[2 ] =get_unaligned_le32 (key +8 ); + ctx->key[3 ] =get_unaligned_le32 (key +12 ); + ctx->key[4 ] =get_unaligned_le32 (key +16 ); + ctx->key[5 ] =get_unaligned_le32 (key +20 ); + ctx->key[6 ] =get_unaligned_le32 (key +24 ); + ctx->key[7 ] =get_unaligned_le32 (key +28 ); + ctx->counter[0 ] =0 ; + ctx->counter[1 ] =0 ; + ctx->counter[2 ] = nonce &U32_MAX ; + ctx->counter[3 ] = nonce >>32 ; +} +void chacha20 (struct chacha20_ctx *ctx,u8 *dst,const u8 *src,u32 len, +simd_context_t *simd_context); + +void hchacha20 (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t *simd_context); + +#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
@@ -1,3 +1,7 @@ +config ZINC_CHACHA20 + tristate + select CRYPTO_CHACHA20 + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" help
lib/zinc/Makefile +3 -0
@@ -1,3 +1,6 @@ ccflags-y := -O2 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc_chacha20-y := chacha20/chacha20.o +obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +108 -0
108 lines not on this page · raw message
lib/zinc/selftest/chacha20.c new file +2698 -0
2,698 lines not on this page · raw message
lib/zinc/selftest/run.h new file +48 -0
48 lines not on this page · raw message
Herbert Xu
maintainerPATCH v2 3/4 zinc: Add x86 accelerated ChaCha20
In reply to Herbert XuThis patch exposes the crypto API x86 chacha20 implementation through zinc.
---
lib/zinc/Kconfig | 1 lib/zinc/chacha20/chacha20-x86_64-glue.c | 55 +++++++++++++++++++++++++++++++ lib/zinc/chacha20/chacha20.c | 4 ++ 3 files changed, 60 insertions(+)
lib/zinc/Kconfig +1 -0
@@ -1,6 +1,7 @@ config ZINC_CHACHA20 tristate select CRYPTO_CHACHA20 + select CRYPTO_CHACHA20_X86_64 if ZINC_ARCH_X86_64 config ZINC_SELFTEST bool "Zinc cryptography library self-tests"
lib/zinc/chacha20/chacha20-x86_64-glue.c new file +55 -0
55 lines not on this page · raw message
lib/zinc/chacha20/chacha20.c +4 -0
@@ -17,6 +17,9 @@#include <crypto/algapi.h> // For crypto_xor_cpy. #include <crypto/chacha20.h> +#if defined(CONFIG_ZINC_ARCH_X86_64 ) +#include "chacha20-x86_64-glue.c" +#else static bool *const chacha20_nobs[]__initconst = { };static void __init chacha20_fpu_init (void ) {
@@ -34,6 +37,7 @@ static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS], {return false; } +#endif void chacha20 (struct chacha20_ctx *ctx,u8 *dst,const u8 *src,u32 len,simd_context_t *simd_context)
Herbert Xu
maintainerPATCH v2 4/4 zinc: ChaCha20 x86_64 implementation
In reply to Herbert XuFrom: Jason A. Donenfeld <[email protected]>
This ports SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.
- When aligning the stack, the original code was saving %rsp to %r9.
To keep objtool happy, we use instead the DRAP idiom to save %rsp
to %r10:
leaq 8(%rsp),%r10
... code here ...
leaq -8(%r10),%rsp - The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.
- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
Cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"):
size old new
---- ---- ----
0 62 52
16 414 376
32 410 400
48 414 422
64 362 356
80 714 666
96 714 700
112 712 718
128 692 646
144 1042 674
160 1042 694
176 1042 726
192 1018 650
208 1366 686
224 1366 696
240 1366 722
256 640 656
272 988 1246
288 988 1276
304 992 1296
320 972 1222
336 1318 1256
352 1318 1276
368 1316 1294
384 1294 1218
400 1642 1258
416 1642 1282
432 1642 1302
448 1628 1224
464 1970 1258
480 1970 1280
496 1970 1300
512 656 676
528 1010 1290
544 1010 1306
560 1010 1332
576 986 1254
592 1340 1284
608 1334 1310
624 1340 1334
640 1314 1254
656 1664 1282
672 1674 1306
688 1662 1336
704 1638 1250
720 1992 1292
736 1994 1308
752 1988 1334
768 1252 1254
784 1596 1290
800 1596 1314
816 1596 1330
832 1576 1256
848 1922 1286
864 1922 1314
880 1926 1338
896 1898 1258
912 2248 1288
928 2248 1320
944 2248 1338
960 2226 1268
976 2574 1288
992 2576 1312
1008 2574 1340
Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:
size old new
---- ---- ----
0 64 54
16 386 372
32 388 396
48 388 420
64 366 350
80 708 666
96 708 692
112 706 736
128 692 648
144 1036 682
160 1036 708
176 1036 730
192 1016 658
208 1360 684
224 1362 708
240 1360 732
256 644 500
272 990 526
288 988 556
304 988 576
320 972 500
336 1314 532
352 1316 558
368 1318 578
384 1308 506
400 1644 532
416 1644 556
432 1644 594
448 1624 508
464 1970 534
480 1970 556
496 1968 582
512 660 624
528 1016 682
544 1016 702
560 1018 728
576 998 654
592 1344 680
608 1344 708
624 1344 730
640 1326 654
656 1670 686
672 1670 708
688 1670 732
704 1652 658
720 1998 682
736 1998 710
752 1996 734
768 1256 662
784 1606 688
800 1606 714
816 1606 736
832 1584 660
848 1948 688
864 1950 714
880 1948 736
896 1912 688
912 2258 718
928 2258 744
944 2256 768
960 2238 692
976 2584 718
992 2584 744
1008 2584 770
---
arch/x86/crypto/Makefile | 3 arch/x86/crypto/chacha20-avx2-x86_64.S | 1026 ------------ arch/x86/crypto/chacha20-ssse3-x86_64.S | 761 -------- arch/x86/crypto/chacha20-zinc-x86_64.S | 2632 +++++++++++++++++++++++++++++++ arch/x86/crypto/chacha20_glue.c | 119 - include/crypto/chacha20.h | 4 lib/zinc/chacha20/chacha20-x86_64-glue.c | 3 7 files changed, 2692 insertions(+), 1856 deletions(-)
arch/x86/crypto/Makefile +1 -2
@@ -74,7 +74,7 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o -chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o +chacha20-x86_64-y := chacha20-zinc-x86_64.o chacha20_glue.o serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -97,7 +97,6 @@ endif ifeq ($(avx2_supported),yes) camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o - chacha20-x86_64-y += chacha20-avx2-x86_64.o serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
arch/x86/crypto/chacha20-avx2-x86_64.S deleted +0 -1026
1,026 lines not on this page · raw message
arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0 -761
761 lines not on this page · raw message
arch/x86/crypto/chacha20-zinc-x86_64.S new file +2632 -0
1,198 lines not on this page · raw message
arch/x86/crypto/chacha20_glue.c +53 -66
119 lines not on this page · raw message
lib/zinc/chacha20/chacha20-x86_64-glue.c +2 -1
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
In reply to Herbert XuQuoted 39 lines · Herbert Xu
Quoted 22 lines · Herbert Xu
In response to Martin's patch-set which I merged last week, I think here is quick way out for the zinc interface.
Going through the past zinc discussions it would appear that everybody is quite happy with the zinc interface per se. The most contentious areas are in fact the algorithm implementations under zinc, as well as the conversion of the crypto API algorithms over to using the zinc interface (e.g., you can no longer access specific implementations).
My proposal is to merge the zinc interface as is, but to invert how we place the algorithm implementations. IOW the implementations should stay where they are now, with in the crypto API. However, we will provide direct access to them for zinc without going through the crypto API. IOW we'll just export the functions directly.
Here is a proof of concept patch to do it for chacha20-generic. It basically replaces patch 3 in the October zinc patch series. Actually this patch also exports the arm/x86 chacha20 functions too so they are ready for use by zinc.
Quoted 8 lines
If everyone is happy with this then we can immediately add the zinc interface and expose the existing crypto API algorithms through it. This would allow wireguard to be merged right away.
In parallel, the discussions over replacing the implementations underneath can carry on without stalling the whole project.
PS This patch is totally untested :)
Here is an updated version demonstrating how we could access the accelerated versions of chacha20. It also includes a final patch to deposit the new zinc version of x86-64 chacha20 into
arch/x86/cryptowhere it can be used by both the crypto API as well as zinc.
Quoted 6 lines
The key differences between this and the last zinc patch series:
1. The crypto API algorithms remain individually accessible, this is crucial as these algorithm names are exported to user-space so changing the names to foo-zinc is not going to work.
Are you saying user space may use names like "ctr-aes-neon" directly rather than "ctr(aes)" for any implementation of the mode?
If so, that is highly unfortunate, since it means we'd be breaking user space by wrapping a crypto library function with its own arch specific specializations into a generic crypto API wrapper.
Note that I think that using AF_ALG to access software crypto routines (as opposed to accelerators) is rather pointless to begin with, but if it permits that today than we're stuck with it.
Quoted 3 lines
2. The arch-specific algorithm code lives in their own module rather than in a monolithic module.
This is one of the remaining issues I have with Zinc. However, modulo the above concerns, I think it does make sense to have a separate function API for synchronous software routines below the current crypto API. What I don't want is a separate Zinc kingdom with a different name, different maintainers and going through a different tree, and with its own approach to test vectors, arch specific code, etc etc
Herbert Xu
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
Hi Ard:
Quoted 7 lines · Ard Biesheuvel
Quoted 4 lines
1. The crypto API algorithms remain individually accessible, this is crucial as these algorithm names are exported to user-space so changing the names to foo-zinc is not going to work.
Are you saying user space may use names like "ctr-aes-neon" directly rather than "ctr(aes)" for any implementation of the mode?
Yes. In fact it's used for FIPS certification testing.
Quoted 3 lines
If so, that is highly unfortunate, since it means we'd be breaking user space by wrapping a crypto library function with its own arch specific specializations into a generic crypto API wrapper.
You can never break things by introducing new algorithms. The problem is only when you remove existing ones.
Quoted 3 lines
Note that I think that using
AF_ALGto access software crypto routines (as opposed to accelerators) is rather pointless to begin with, but if it permits that today than we're stuck with it.
Sure, nobody sane should be doing it. But when it comes to government certification... :)
Quoted 10 lines
Quoted 2 lines
2. The arch-specific algorithm code lives in their own module rather than in a monolithic module.
This is one of the remaining issues I have with Zinc. However, modulo the above concerns, I think it does make sense to have a separate function API for synchronous software routines below the current crypto API. What I don't want is a separate Zinc kingdom with a different name, different maintainers and going through a different tree, and with its own approach to test vectors, arch specific code, etc etc
Even without the issue of replacing chacha20-generic with chacha20-zinc which breaks point 1 above, we simply don't want or need to go through zinc's run-time implementation choice for the crypto API algorithms. They've already paid for the indirect function call so why make them go through yet another run-time branch?
If the optics of having the code in crypto is the issue, we could move the actual algorithm code into a third location, perhaps arch/x86/crypto/lib or arch/x86/lib/crypto. Then both zinc and the crypto API can sit on top of that.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
Quoted 3 lines · Herbert Xu
Yes. In fact it's used for FIPS certification testing. Sure, nobody sane should be doing it. But when it comes to government certification... :)
The kernel does not aim toward any FIPS certification, and we're not going to start bloating our designs to fulfill this. It's never been a goal. Maybe ask Ted to add a FIPS mode to random.c and see what happens... When you start arguing "because FIPS!" as your justification, you really hit a head scratcher.
Quoted 3 lines
They've already paid for the indirect
function call so why make them go through yet another run-time branch?
The indirect function call in the crypto API is the performance hit. The branch in Zinc is not, as the predictor does the correct thing every single time. I'm not able to distinguish between the two looking at the performance measurements between it being there and the branch being commented out.
Give me a few more days to finish v9's latest required changes for chacha12, and then I'll submit a revision that I think should address the remaining technical objections raised over the last several months we've been discussing this.
Regards,
Jason
Theodore Y. Ts'o
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
Quoted 10 lines · Jason A. Donenfeld
Quoted 3 lines · Herbert Xu
Yes. In fact it's used for FIPS certification testing. Sure, nobody sane should be doing it. But when it comes to government certification... :)
The kernel does not aim toward any FIPS certification, and we're not going to start bloating our designs to fulfill this. It's never been a goal. Maybe ask Ted to add a FIPS mode to
random.cand see what happens... When you start arguing "because FIPS!" as your justification, you really hit a head scratcher.
There are crazy people who go for FIPS certification for the kernel. That's why crypto/drbg.c exists. There is a crazy fips mode in drivers/char/random.c which causes the kernel to panic with a 1 in 2**80 probability each time _extract_entropy is called. It's not the default, and I have no idea if any one uses it, or if it's like the NIST OSI mandate, which forced everyone to buy an OSI networking stack --- and then put it on the shelf and use TCP/IP instead.
Press release from March 2018:
- Ted
Herbert Xu
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
In reply to Jason A. DonenfeldQuoted 10 lines · Jason A. Donenfeld
Quoted 3 lines · Herbert Xu
Yes. In fact it's used for FIPS certification testing. Sure, nobody sane should be doing it. But when it comes to government certification... :)
The kernel does not aim toward any FIPS certification, and we're not going to start bloating our designs to fulfill this. It's never been a goal. Maybe ask Ted to add a FIPS mode to
random.cand see what happens... When you start arguing "because FIPS!" as your justification, you really hit a head scratcher.
FIPS is not the issue. The issue is that we need to keep the existing user-space ABI. We can't break that.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
In reply to Herbert XuHi Herbert,
Quoted 5 lines · Herbert Xu
Here is an updated version demonstrating how we could access the accelerated versions of chacha20. It also includes a final patch to deposit the new zinc version of x86-64 chacha20 into
arch/x86/cryptowhere it can be used by both the crypto API as well as zinc.
N'ack. As I mentioned in the last email, this really isn't okay, and mostly defeats the purpose of Zinc in the first place, adding complexity in a discombobulated half-baked way. Your proposal seems to be the worst of all worlds. Having talked to lots of people about the design of Zinc at this point to very positive reception, I'm going to move forward with submitting v9, once I rebase with the required changes for Eric's latest merge.
Jason
Herbert Xu
maintainerRe: [RFC PATCH v2 0/4] Exporting existing crypto API code through zinc
Quoted 8 lines · Jason A. Donenfeld
N'ack. As I mentioned in the last email, this really isn't okay, and mostly defeats the purpose of Zinc in the first place, adding complexity in a discombobulated half-baked way. Your proposal seems to be the worst of all worlds. Having talked to lots of people about the design of Zinc at this point to very positive reception, I'm going to move forward with submitting v9, once I rebase with the required changes for Eric's latest merge.
Do you have any actual technical reasons for rejecting this apart from your apparent hatred of the existing crypto API?
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerRFC PATCH v3 11/15 crypto: poly1305 - add Poly1305 core API
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Expose a low-level Poly1305 API which implements the ?-almost-?-universal (?A?U) hash function underlying the Poly1305 MAC and supports block-aligned inputs only.
This is needed for Adiantum hashing, which builds an ?A?U hash function from NH and a polynomial evaluation in GF(2^{130}-5); this polynomial evaluation is identical to the one the Poly1305 MAC does. However, the crypto_shash Poly1305 API isn't very appropriate for this because its calling convention assumes it is used as a MAC, with a 32-byte "one-time key" provided for every digest.
But by design, in Adiantum hashing the performance of the polynomial evaluation isn't nearly as critical as NH. So it suffices to just have some C helper functions. Thus, this patch adds such functions.
--- crypto/poly1305_generic.c | 174 ++++++++++++++++++++++---------------- include/crypto/poly1305.h | 16 ++++ 2 files changed, 115 insertions(+), 75 deletions(-)
crypto/poly1305_generic.c +99 -75
@@ -38,7 +38,7 @@ int crypto_poly1305_init(struct shash_desc *desc) {struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); -memset (dctx->h.h,0 ,sizeof (dctx->h.h)); +poly1305_core_init (&dctx->h); dctx->buflen =0 ; dctx->rset = false; dctx->sset = false;
@@ -47,23 +47,16 @@ int crypto_poly1305_init(struct shash_desc *desc) }EXPORT_SYMBOL_GPL (crypto_poly1305_init); -static void poly1305_setrkey (struct poly1305_desc_ctx *dctx,const u8 *key) +void poly1305_core_setkey (struct poly1305_key *key,const u8 *raw_key) {/* r &= 0xffffffc0ffffffc0ffffffc0fffffff */ - dctx->r.r[0 ] = (get_unaligned_le32 (key +0 ) >>0 ) &0x3ffffff ; - dctx->r.r[1 ] = (get_unaligned_le32 (key +3 ) >>2 ) &0x3ffff03 ; - dctx->r.r[2 ] = (get_unaligned_le32 (key +6 ) >>4 ) &0x3ffc0ff ; - dctx->r.r[3 ] = (get_unaligned_le32 (key +9 ) >>6 ) &0x3f03fff ; - dctx->r.r[4 ] = (get_unaligned_le32 (key +12 ) >>8 ) &0x00fffff ; -} - -static void poly1305_setskey (struct poly1305_desc_ctx *dctx,const u8 *key) -{ - dctx->s[0 ] =get_unaligned_le32 (key +0 ); - dctx->s[1 ] =get_unaligned_le32 (key +4 ); - dctx->s[2 ] =get_unaligned_le32 (key +8 ); - dctx->s[3 ] =get_unaligned_le32 (key +12 ); + key->r[0 ] = (get_unaligned_le32 (raw_key +0 ) >>0 ) &0x3ffffff ; + key->r[1 ] = (get_unaligned_le32 (raw_key +3 ) >>2 ) &0x3ffff03 ; + key->r[2 ] = (get_unaligned_le32 (raw_key +6 ) >>4 ) &0x3ffc0ff ; + key->r[3 ] = (get_unaligned_le32 (raw_key +9 ) >>6 ) &0x3f03fff ; + key->r[4 ] = (get_unaligned_le32 (raw_key +12 ) >>8 ) &0x00fffff ; } +EXPORT_SYMBOL_GPL (poly1305_core_setkey);/* * Poly1305 requires a unique key for each tag, which implies that we can't set
@@ -75,13 +68,16 @@ unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx, {if (!dctx->sset) {if (!dctx->rset && srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_setrkey (dctx, src); +poly1305_core_setkey (&dctx->r, src); src +=POLY1305_BLOCK_SIZE ; srclen -=POLY1305_BLOCK_SIZE ; dctx->rset = true; }if (srclen >=POLY1305_BLOCK_SIZE ) { -poly1305_setskey (dctx, src); + dctx->s[0 ] =get_unaligned_le32 (src +0 ); + dctx->s[1 ] =get_unaligned_le32 (src +4 ); + dctx->s[2 ] =get_unaligned_le32 (src +8 ); + dctx->s[3 ] =get_unaligned_le32 (src +12 ); src +=POLY1305_BLOCK_SIZE ; srclen -=POLY1305_BLOCK_SIZE ; dctx->sset = true;
@@ -91,41 +87,37 @@ unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx, }EXPORT_SYMBOL_GPL (crypto_poly1305_setdesckey); -static unsigned int poly1305_blocks (struct poly1305_desc_ctx *dctx, -const u8 *src,unsigned int srclen, -u32 hibit) +static void poly1305_blocks_internal (struct poly1305_state *state, +const struct poly1305_key *key, +const void *src,unsigned int nblocks, +u32 hibit) {u32 r0, r1, r2, r3, r4;u32 s1, s2, s3, s4;u32 h0, h1, h2, h3, h4;u64 d0, d1, d2, d3, d4; -unsigned int datalen; -if (unlikely (!dctx->sset)) { - datalen =crypto_poly1305_setdesckey (dctx, src, srclen); - src += srclen - datalen; - srclen = datalen; - } +if (!nblocks) +return ; - r0 = dctx->r.r[0 ]; - r1 = dctx->r.r[1 ]; - r2 = dctx->r.r[2 ]; - r3 = dctx->r.r[3 ]; - r4 = dctx->r.r[4 ]; + r0 = key->r[0 ]; + r1 = key->r[1 ]; + r2 = key->r[2 ]; + r3 = key->r[3 ]; + r4 = key->r[4 ]; s1 = r1 *5 ; s2 = r2 *5 ; s3 = r3 *5 ; s4 = r4 *5 ; - h0 = dctx->h.h[0 ]; - h1 = dctx->h.h[1 ]; - h2 = dctx->h.h[2 ]; - h3 = dctx->h.h[3 ]; - h4 = dctx->h.h[4 ]; - -while (likely (srclen >=POLY1305_BLOCK_SIZE )) { + h0 = state->h[0 ]; + h1 = state->h[1 ]; + h2 = state->h[2 ]; + h3 = state->h[3 ]; + h4 = state->h[4 ]; + do { /* h += m[i] */ h0 += (get_unaligned_le32(src + 0) >> 0) & 0x3ffffff; h1 += (get_unaligned_le32(src + 3) >> 2) & 0x3ffffff;
@@ -154,16 +146,36 @@ static unsigned int poly1305_blocks(struct poly1305_desc_ctx *dctx, h1 += h0 >>26 ; h0 = h0 &0x3ffffff ; src +=POLY1305_BLOCK_SIZE ; - srclen -=POLY1305_BLOCK_SIZE ; - } + }while (--nblocks); - dctx->h.h[0 ] = h0; - dctx->h.h[1 ] = h1; - dctx->h.h[2 ] = h2; - dctx->h.h[3 ] = h3; - dctx->h.h[4 ] = h4; + state->h[0 ] = h0; + state->h[1 ] = h1; + state->h[2 ] = h2; + state->h[3 ] = h3; + state->h[4 ] = h4; +} -return srclen; +void poly1305_core_blocks (struct poly1305_state *state, +const struct poly1305_key *key, +const void *src,unsigned int nblocks) +{ +poly1305_blocks_internal (state, key, src, nblocks,1 <<24 ); +} +EXPORT_SYMBOL_GPL (poly1305_core_blocks); + +static void poly1305_blocks (struct poly1305_desc_ctx *dctx, +const u8 *src,unsigned int srclen,u32 hibit) +{ +unsigned int datalen; + +if (unlikely (!dctx->sset)) { + datalen =crypto_poly1305_setdesckey (dctx, src, srclen); + src += srclen - datalen; + srclen = datalen; + } + +poly1305_blocks_internal (&dctx->h, &dctx->r, + src, srclen /POLY1305_BLOCK_SIZE , hibit); } int crypto_poly1305_update(struct shash_desc *desc,
@@ -187,9 +199,9 @@ int crypto_poly1305_update(struct shash_desc *desc, }if (likely (srclen >=POLY1305_BLOCK_SIZE )) { - bytes =poly1305_blocks (dctx, src, srclen,1 <<24 ); - src += srclen - bytes; - srclen = bytes; +poly1305_blocks (dctx, src, srclen,1 <<24 ); + src += srclen - (srclen %POLY1305_BLOCK_SIZE ); + srclen %=POLY1305_BLOCK_SIZE ; }if (unlikely (srclen)) {
@@ -201,30 +213,18 @@ int crypto_poly1305_update(struct shash_desc *desc, }EXPORT_SYMBOL_GPL (crypto_poly1305_update); -int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) +void poly1305_core_emit (const struct poly1305_state *state,void *dst) { -struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc);u32 h0, h1, h2, h3, h4;u32 g0, g1, g2, g3, g4;u32 mask; -u64 f =0 ; - -if (unlikely (!dctx->sset)) -return -ENOKEY ; - -if (unlikely (dctx->buflen)) { - dctx->buf[dctx->buflen++] =1 ; -memset (dctx->buf + dctx->buflen,0 , -POLY1305_BLOCK_SIZE - dctx->buflen); -poly1305_blocks (dctx, dctx->buf,POLY1305_BLOCK_SIZE ,0 ); - }/* fully carry h */ - h0 = dctx->h.h[0 ]; - h1 = dctx->h.h[1 ]; - h2 = dctx->h.h[2 ]; - h3 = dctx->h.h[3 ]; - h4 = dctx->h.h[4 ]; + h0 = state->h[0 ]; + h1 = state->h[1 ]; + h2 = state->h[2 ]; + h3 = state->h[3 ]; + h4 = state->h[4 ]; h2 += (h1 >>26 ); h1 = h1 &0x3ffffff ; h3 += (h2 >>26 ); h2 = h2 &0x3ffffff ;
@@ -254,16 +254,40 @@ int crypto_poly1305_final(struct shash_desc *desc, u8 *dst) h4 = (h4 & mask) | g4;/* h = h % (2^128) */ - h0 = (h0 >>0 ) | (h1 <<26 ); - h1 = (h1 >>6 ) | (h2 <<20 ); - h2 = (h2 >>12 ) | (h3 <<14 ); - h3 = (h3 >>18 ) | (h4 <<8 ); +put_unaligned_le32 ((h0 >>0 ) | (h1 <<26 ), dst +0 ); +put_unaligned_le32 ((h1 >>6 ) | (h2 <<20 ), dst +4 ); +put_unaligned_le32 ((h2 >>12 ) | (h3 <<14 ), dst +8 ); +put_unaligned_le32 ((h3 >>18 ) | (h4 <<8 ), dst +12 ); +} +EXPORT_SYMBOL_GPL (poly1305_core_emit); + +int crypto_poly1305_final (struct shash_desc *desc,u8 *dst) +{ +struct poly1305_desc_ctx *dctx =shash_desc_ctx (desc); +__le32 digest[4 ]; +u64 f =0 ; + +if (unlikely (!dctx->sset)) +return -ENOKEY ; + +if (unlikely (dctx->buflen)) { + dctx->buf[dctx->buflen++] =1 ; +memset (dctx->buf + dctx->buflen,0 , +POLY1305_BLOCK_SIZE - dctx->buflen); +poly1305_blocks (dctx, dctx->buf,POLY1305_BLOCK_SIZE ,0 ); + } + +poly1305_core_emit (&dctx->h, digest);/* mac = (h + s) % (2^128) */ - f = (f >>32 ) + h0 + dctx->s[0 ];put_unaligned_le32 (f, dst +0 ); - f = (f >>32 ) + h1 + dctx->s[1 ];put_unaligned_le32 (f, dst +4 ); - f = (f >>32 ) + h2 + dctx->s[2 ];put_unaligned_le32 (f, dst +8 ); - f = (f >>32 ) + h3 + dctx->s[3 ];put_unaligned_le32 (f, dst +12 ); + f = (f >>32 ) +le32_to_cpu (digest[0 ]) + dctx->s[0 ]; +put_unaligned_le32 (f, dst +0 ); + f = (f >>32 ) +le32_to_cpu (digest[1 ]) + dctx->s[1 ]; +put_unaligned_le32 (f, dst +4 ); + f = (f >>32 ) +le32_to_cpu (digest[2 ]) + dctx->s[2 ]; +put_unaligned_le32 (f, dst +8 ); + f = (f >>32 ) +le32_to_cpu (digest[3 ]) + dctx->s[3 ]; +put_unaligned_le32 (f, dst +12 );return 0 ; }
include/crypto/poly1305.h +16 -0
@@ -38,6 +38,22 @@ struct poly1305_desc_ctx {bool sset; }; +/* +* Poly1305 core functions. These implement the ?-almost-?-universal hash +* function underlying the Poly1305 MAC, i.e. they don't add an encrypted nonce +* ("s key") at the end. They also only support block-aligned inputs. +*/ +void poly1305_core_setkey (struct poly1305_key *key,const u8 *raw_key); +static inline void poly1305_core_init (struct poly1305_state *state) +{ +memset (state->h,0 ,sizeof (state->h)); +} +void poly1305_core_blocks (struct poly1305_state *state, +const struct poly1305_key *key, +const void *src,unsigned int nblocks); +void poly1305_core_emit (const struct poly1305_state *state,void *dst); + +/* Crypto API helper functions for the Poly1305 MAC */ int crypto_poly1305_init (struct shash_desc *desc);unsigned int crypto_poly1305_setdesckey (struct poly1305_desc_ctx *dctx,const u8 *src,unsigned int srclen);
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 12/15 crypto: nhpoly1305 - add NHPoly1305 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add a generic implementation of NHPoly1305, an ?-almost-?-universal hash function used in the Adiantum encryption mode.
CONFIG_NHPOLY1305 is not selectable by itself since there won't be any real reason to enable it without also enabling Adiantum support.
--- crypto/Kconfig | 5 + crypto/Makefile | 1 + crypto/nhpoly1305.c | 254 +++++++ crypto/testmgr.c | 6 + crypto/testmgr.h | 1240 ++++++++++++++++++++++++++++++++++- include/crypto/nhpoly1305.h | 74 +++ 6 files changed, 1576 insertions(+), 4 deletions(-) create mode 100644 crypto/nhpoly1305.c create mode 100644 include/crypto/nhpoly1305.h
crypto/Kconfig +5 -0
@@ -493,6 +493,11 @@ config CRYPTO_KEYWRAP Support for key wrapping (NIST SP800-38F / RFC3394) without padding. +config CRYPTO_NHPOLY1305 + tristate + select CRYPTO_HASH + select CRYPTO_POLY1305 + comment "Hash modes" config CRYPTO_CMAC
crypto/Makefile +1 -0
@@ -84,6 +84,7 @@ obj-$(CONFIG_CRYPTO_LRW) += lrw.o obj-$(CONFIG_CRYPTO_XTS) += xts.o obj-$(CONFIG_CRYPTO_CTR) += ctr.o obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o +obj-$(CONFIG_CRYPTO_NHPOLY1305) += nhpoly1305.o obj-$(CONFIG_CRYPTO_GCM) += gcm.o obj-$(CONFIG_CRYPTO_CCM) += ccm.o obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o
crypto/nhpoly1305.c new file +254 -0
254 lines not on this page · raw message
crypto/testmgr.c +6 -0
@@ -3291,6 +3291,12 @@ static const struct alg_test_desc alg_test_descs[] = { .dec = __VECS(morus640_dec_tv_template), } } + }, { + .alg = "nhpoly1305", + .test = alg_test_hash, + .suite = { + .hash = __VECS(nhpoly1305_tv_template) + } }, { .alg = "ofb(aes)", .test = alg_test_skcipher,
crypto/testmgr.h +1236 -4
1,259 lines not on this page · raw message
include/crypto/nhpoly1305.h new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 13/15 crypto: arm/nhpoly1305 - add NEON-accelerated NHPoly1305
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add an ARM NEON implementation of NHPoly1305, an ?-almost-?-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually NEON-accelerated; the Poly1305 part is less performance-critical so is just implemented in C.
--- arch/arm/crypto/Kconfig | 5 ++ arch/arm/crypto/Makefile | 2 + arch/arm/crypto/nh-neon-core.S | 116 +++++++++++++++++++++++++ arch/arm/crypto/nhpoly1305-neon-glue.c | 77 ++++++++++++++++ 4 files changed, 200 insertions(+) create mode 100644 arch/arm/crypto/nh-neon-core.S create mode 100644 arch/arm/crypto/nhpoly1305-neon-glue.c
arch/arm/crypto/Kconfig +5 -0
@@ -122,4 +122,9 @@ config CRYPTO_CHACHA20_NEON select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20 +config CRYPTO_NHPOLY1305_NEON + tristate "NEON accelerated NHPoly1305 hash function (for Adiantum)" + depends on KERNEL_MODE_NEON + select CRYPTO_NHPOLY1305 + endif
arch/arm/crypto/Makefile +2 -0
@@ -10,6 +10,7 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha-neon.o +obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
@@ -53,6 +54,7 @@ ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o +nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o ifdef REGENERATE_ARM_CRYPTO quiet_cmd_perl = PERL $@
arch/arm/crypto/nh-neon-core.S new file +116 -0
116 lines not on this page · raw message
arch/arm/crypto/nhpoly1305-neon-glue.c new file +77 -0
77 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 14/15 crypto: adiantum - add Adiantum support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add support for the Adiantum encryption mode. Adiantum was designed by Paul Crowley and is specified by our paper:
Adiantum: length-preserving encryption for entry-level processors
(https://eprint.iacr.org/2018/720.pdf)
See our paper for full details; this patch only provides an overview.
Adiantum is a tweakable, length-preserving encryption mode designed for fast and secure disk encryption, especially on CPUs without dedicated crypto instructions. Adiantum encrypts each sector using the XChaCha12 stream cipher, two passes of an ?-almost-?-universal (?A?U) hash function, and an invocation of the AES-256 block cipher on a single 16-byte block. On CPUs without AES instructions, Adiantum is much faster than AES-XTS; for example, on ARM Cortex-A7, on 4096-byte sectors Adiantum encryption is about 4 times faster than AES-256-XTS encryption, and decryption about 5 times faster.
Adiantum is a specialization of the more general HBSH construction. Our earlier proposal, HPolyC, was also a HBSH specialization, but it used a different ?A?U hash function, one based on Poly1305 only. Adiantum's ?A?U hash function, which is based primarily on the "NH" hash function like that used in UMAC (RFC4418), is about twice as fast as HPolyC's; consequently, Adiantum is about 20% faster than HPolyC.
This speed comes with no loss of security: Adiantum is provably just as secure as HPolyC, in fact slightly *more* secure. Like HPolyC, Adiantum's security is reducible to that of XChaCha12 and AES-256, subject to a security bound. XChaCha12 itself has a security reduction to ChaCha12. Therefore, one need not "trust" Adiantum; one need only trust ChaCha12 and AES-256. Note that the ?A?U hash function is only used for its proven combinatorical properties so cannot be "broken".
Adiantum is also a true wide-block encryption mode, so flipping any plaintext bit in the sector scrambles the entire ciphertext, and vice versa. No other such mode is available in the kernel currently; doing the same with XTS scrambles only 16 bytes. Adiantum also supports arbitrary-length tweaks and naturally supports any length input >= 16 bytes without needing "ciphertext stealing".
For the stream cipher, Adiantum uses XChaCha12 rather than XChaCha20 in order to make encryption feasible on the widest range of devices. Although the 20-round variant is quite popular, the best known attacks on ChaCha are on only 7 rounds, so ChaCha12 still has a substantial security margin; in fact, larger than AES-256's. 12-round Salsa20 is also the eSTREAM recommendation. For the block cipher, Adiantum uses AES-256, despite it having a lower security margin than XChaCha12 and needing table lookups, due to AES's extensive adoption and analysis making it the obvious first choice. Nevertheless, for flexibility this patch also permits the "adiantum" template to be instantiated with XChaCha20 and/or with an alternate block cipher.
We need Adiantum support in the kernel for use in dm-crypt and fscrypt, where currently the only other suitable options are block cipher modes such as AES-XTS. A big problem with this is that many low-end mobile devices (e.g. Android Go phones sold primarily in developing countries, as well as some smartwatches) still have CPUs that lack AES instructions, e.g. ARM Cortex-A7. Sadly, AES-XTS encryption is much too slow to be viable on these devices. We did find that some "lightweight" block ciphers are fast enough, but these suffer from problems such as not having much cryptanalysis or being too controversial.
The ChaCha stream cipher has excellent performance but is insecure to use directly for disk encryption, since each sector's IV is reused each time it is overwritten. Even restricting the threat model to offline attacks only isn't enough, since modern flash storage devices don't guarantee that "overwrites" are really overwrites, due to wear-leveling. Adiantum avoids this problem by constructing a
"tweakable super-pseudorandom permutation"; this is the strongest possible security model for length-preserving encryption.
Of course, storing random nonces along with the ciphertext would be the ideal solution. But doing that with existing hardware and filesystems runs into major practical problems; in most cases it would require data journaling (like dm-integrity) which severely degrades performance. Thus, for now length-preserving encryption is still needed.
--- crypto/Kconfig | 23 ++ crypto/Makefile | 1 + crypto/adiantum.c | 658 ++++++++++++++++++++++++++++++++++++++++++++++ crypto/tcrypt.c | 12 + crypto/testmgr.c | 12 + crypto/testmgr.h | 461 ++++++++++++++++++++++++++++++++ 6 files changed, 1167 insertions(+) create mode 100644 crypto/adiantum.c
crypto/Kconfig +23 -0
@@ -498,6 +498,29 @@ config CRYPTO_NHPOLY1305 select CRYPTO_HASH select CRYPTO_POLY1305 +config CRYPTO_ADIANTUM + tristate "Adiantum support" + select CRYPTO_CHACHA20 + select CRYPTO_POLY1305 + select CRYPTO_NHPOLY1305 + help + Adiantum is a tweakable, length-preserving encryption mode + designed for fast and secure disk encryption, especially on + CPUs without dedicated crypto instructions. It encrypts + each sector using the XChaCha12 stream cipher, two passes of + an ?-almost-?-universal hash function, and an invocation of + the AES-256 block cipher on a single 16-byte block. On CPUs + without AES instructions, Adiantum is much faster than + AES-XTS. + + Adiantum's security is provably reducible to that of its + underlying stream and block ciphers, subject to a security + bound. Unlike XTS, Adiantum is a true wide-block encryption + mode, so it actually provides an even stronger notion of + security than XTS, subject to the security bound. + + If unsure, say N. + comment "Hash modes" config CRYPTO_CMAC
crypto/Makefile +1 -0
@@ -84,6 +84,7 @@ obj-$(CONFIG_CRYPTO_LRW) += lrw.o obj-$(CONFIG_CRYPTO_XTS) += xts.o obj-$(CONFIG_CRYPTO_CTR) += ctr.o obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o +obj-$(CONFIG_CRYPTO_ADIANTUM) += adiantum.o obj-$(CONFIG_CRYPTO_NHPOLY1305) += nhpoly1305.o obj-$(CONFIG_CRYPTO_GCM) += gcm.o obj-$(CONFIG_CRYPTO_CCM) += ccm.o
crypto/adiantum.c new file +658 -0
658 lines not on this page · raw message
crypto/tcrypt.c +12 -0
@@ -2297,6 +2297,18 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)test_cipher_speed ("ctr(sm4)" ,DECRYPT , sec,NULL ,0 , speed_template_16);break ; + +case 219 : +test_cipher_speed ("adiantum(xchacha12,aes)" ,ENCRYPT , sec,NULL , +0 , speed_template_32); +test_cipher_speed ("adiantum(xchacha12,aes)" ,DECRYPT , sec,NULL , +0 , speed_template_32); +test_cipher_speed ("adiantum(xchacha20,aes)" ,ENCRYPT , sec,NULL , +0 , speed_template_32); +test_cipher_speed ("adiantum(xchacha20,aes)" ,DECRYPT , sec,NULL , +0 , speed_template_32); +break ; +case 300 :if (alg) {test_hash_speed (alg, sec, generic_hash_speed_template);
crypto/testmgr.c +12 -0
@@ -2404,6 +2404,18 @@ static int alg_test_null(const struct alg_test_desc *desc,/* Please keep this list sorted by algorithm name. */ static const struct alg_test_desc alg_test_descs[] = { { + .alg ="adiantum(xchacha12,aes)" , + .test = alg_test_skcipher, + .suite = { + .cipher =__VECS (adiantum_xchacha12_aes_tv_template) + }, + }, { + .alg ="adiantum(xchacha20,aes)" , + .test = alg_test_skcipher, + .suite = { + .cipher =__VECS (adiantum_xchacha20_aes_tv_template) + }, + }, { .alg ="aegis128" , .test = alg_test_aead, .suite = {
crypto/testmgr.h +461 -0
467 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Eric Biggers
maintainerRFC PATCH v3 15/15 fscrypt: add Adiantum support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add support for the Adiantum encryption mode to fscrypt. Adiantum is a tweakable, length-preserving encryption mode with security provably reducible to that of XChaCha12 and AES-256, subject to a security bound. It's also a true wide-block mode, unlike XTS. See the paper "Adiantum: length-preserving encryption for entry-level processors"
(https://eprint.iacr.org/2018/720.pdf) for full details; also see the crypto API patch which added Adiantum.
On sufficiently long inputs, Adiantum's performance-critical parts are XChaCha12 and the NH hash function. These algorithms are fast even on processors without dedicated crypto instructions. Adiantum makes it feasible to enable storage encryption on low-end mobile devices that lack AES instructions; currently such devices are unencrypted. On ARM Cortex-A7, on 4096-byte messages Adiantum encryption is about 4 times faster than AES-256-XTS encryption; decryption is about 5 times faster.
Adiantum is also suitable to replace CTS-CBC for fscrypt's filenames encryption, fixing the information leakage in encrypted filenames when two filenames in a directory share a common prefix.
Adiantum accepts long IVs, so in fscrypt we include the 16-byte per-inode nonce in the IVs too, and we allow userspace to choose to use the master key directly rather than deriving per-file keys. This is especially desirable because each Adiantum tfm (crypto_skcipher) uses more memory than an AES-XTS tfm, since Adiantum has more sub-tfms as well as a long subkey for NH.
--- Documentation/filesystems/fscrypt.rst | 187 ++++++++------- fs/crypto/crypto.c | 35 +-- fs/crypto/fname.c | 22 +- fs/crypto/fscrypt_private.h | 66 +++++- fs/crypto/keyinfo.c | 322 ++++++++++++++++++++------ fs/crypto/policy.c | 5 +- include/uapi/linux/fs.h | 4 +- 7 files changed, 454 insertions(+), 187 deletions(-)
Documentation/filesystems/fscrypt.rst +109 -78
@@ -132,47 +132,31 @@ designed for this purpose be used, such as scrypt, PBKDF2, or Argon2. Per-file keys ------------- -Master keys are not used to encrypt file contents or names directly. -Instead, a unique key is derived for each encrypted file, including -each regular file, directory, and symbolic link. This has several -advantages: - -- In cryptosystems, the same key material should never be used for - different purposes. Using the master key as both an XTS key for - contents encryption and as a CTS-CBC key for filenames encryption - would violate this rule. -- Per-file keys simplify the choice of IVs (Initialization Vectors) - for contents encryption. Without per-file keys, to ensure IV - uniqueness both the inode and logical block number would need to be - encoded in the IVs. This would make it impossible to renumber - inodes, which e.g. ``resize2fs`` can do when resizing an ext4 - filesystem. With per-file keys, it is sufficient to encode just the - logical block number in the IVs. -- Per-file keys strengthen the encryption of filenames, where IVs are - reused out of necessity. With a unique key per directory, IV reuse - is limited to within a single directory. -- Per-file keys allow individual files to be securely erased simply by - securely erasing their keys. (Not yet implemented.) - -A KDF (Key Derivation Function) is used to derive per-file keys from -the master key. This is done instead of wrapping a randomly-generated -key for each file because it reduces the size of the encryption xattr, -which for some filesystems makes the xattr more likely to fit in-line -in the filesystem's inode table. With a KDF, only a 16-byte nonce is -required --- long enough to make key reuse extremely unlikely. A -wrapped key, on the other hand, would need to be up to 64 bytes --- -the length of an AES-256-XTS key. Furthermore, currently there is no -requirement to support unlocking a file with multiple alternative -master keys or to support rotating master keys. Instead, the master -keys may be wrapped in userspace, e.g. as done by the `fscrypt -<https://github.com/google/fscrypt>`_ tool. - -The current KDF encrypts the master key using the 16-byte nonce as an -AES-128-ECB key. The output is used as the derived key. If the -output is longer than needed, then it is truncated to the needed -length. Truncation is the norm for directories and symlinks, since -those use the CTS-CBC encryption mode which requires a key half as -long as that required by the XTS encryption mode. +Since each master key can protect many files, it is necessary to +"tweak" the encryption of each file, so that the same plaintext in two +files doesn't map to the same ciphertext, or vice versa. + +In most cases, fscrypt solves this problem by deriving per-file keys. +When a new encrypted inode (regular file, directory, or symlink) is +created, fscrypt generates a 16-byte nonce uniformly at random and +stores it in the inode's encryption xattr. Then, a KDF (Key +Derivation Function) is used to derive an inode-specific encryption +key from the master key and nonce. + +The Adiantum encryption mode (see `Encryption modes and usage`_) is +special, since it accepts long IVs and is suited for both contents and +filenames encryption. For it, fscrypt includes the 16-byte nonce in +the IVs, and users can choose to use the master key for Adiantum +encryption directly, for added efficiency. In this configuration, no +per-file keys are derived. However, when doing this, users must take +care not to reuse the same master key for any other modes. + +Below, the KDF and design considerations are described in more detail. + +The current KDF works by encrypting the master key with AES-128-ECB, +using the 16-byte nonce as the AES key. The output is used as the +derived key. If the output is longer than needed, then it is +truncated to the needed length. Note: this KDF meets the primary security requirement, which is to produce unique derived keys that preserve the entropy of the master
@@ -181,6 +165,28 @@ However, it is nonstandard and has some problems such as being reversible, so it is generally considered to be a mistake! It may be replaced with HKDF or another more standard KDF in the future. +Key derivation was chosen over key wrapping because wrapped keys would +require larger xattrs which would be less likely to fit in-line in the +filesystem's inode table, and there didn't appear to be any +significant advantages to key wrapping. In particular, currently +there is no requirement to support unlocking a file with multiple +alternative master keys or to support rotating master keys. Instead, +the master keys may be wrapped in userspace, e.g. as done by the +`fscrypt <https://github.com/google/fscrypt>`_ tool. + +Including the inode number in the IVs was considered. However, it was +rejected as it would have prevented ext4 filesystems from being +resized, and by itself still wouldn't have been sufficient to prevent +the same key from being directly reused for both XTS and CTS-CBC. + +Including the per-inode nonce in the IVs would allow filesystem +resizing, but it wouldn't allow key reuse between two different modes, +nor would it be compatible with XTS and CTS-CBC since a 16-byte IV +isn't long enough to contain both a collision-resistant nonce and a +block offset. However, this method works well for Adiantum, which +accepts longer IVs and is suited for both contents and filenames +encryption. + Encryption modes and usage ==========================
@@ -191,54 +197,76 @@ Currently, the following pairs of encryption modes are supported: - AES-256-XTS for contents and AES-256-CTS-CBC for filenames - AES-128-CBC for contents and AES-128-CTS-CBC for filenames +- Adiantum for both contents and filenames + +If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair. -It is strongly recommended to use AES-256-XTS for contents encryption. AES-128-CBC was added only for low-powered embedded devices with crypto accelerators such as CAAM or CESA that do not support XTS. +Adiantum is a (primarily) stream cipher-based mode that was designed +primarily for CPUs that don't support AES instructions. Though +Adiantum actually provides a stronger formal notion of security than +XTS and CTS-CBC (since it's a true wide-block mode), it's also newer +and depends on the security of XChaCha12 in addition to AES-256. +Adiantum support is only available if it has been enabled in the +crypto API via CONFIG_CRYPTO_ADIANTUM. Also, on ARM platforms, to get +good performance CONFIG_CRYPTO_CHACHA20_NEON and +CONFIG_CRYPTO_NHPOLY1305_NEON must be enabled. + New encryption modes can be added relatively easily, without changes to individual filesystems. However, authenticated encryption (AE) modes are not currently supported because of the difficulty of dealing with ciphertext expansion. +Contents encryption +------------------- + For file contents, each filesystem block is encrypted independently. Currently, only the case where the filesystem block size is equal to -the system's page size (usually 4096 bytes) is supported. With the -XTS mode of operation (recommended), the logical block number within -the file is used as the IV. With the CBC mode of operation (not -recommended), ESSIV is used; specifically, the IV for CBC is the -logical block number encrypted with AES-256, where the AES-256 key is -the SHA-256 hash of the inode's data encryption key. - -For filenames, the full filename is encrypted at once. Because of the -requirements to retain support for efficient directory lookups and -filenames of up to 255 bytes, a constant initialization vector (IV) is -used. However, each encrypted directory uses a unique key, which -limits IV reuse to within a single directory. Note that IV reuse in -the context of CTS-CBC encryption means that when the original -filenames share a common prefix at least as long as the cipher block -size (16 bytes for AES), the corresponding encrypted filenames will -also share a common prefix. This is undesirable; it may be fixed in -the future by switching to an encryption mode that is a strong -pseudorandom permutation on arbitrary-length messages, e.g. the HEH -(Hash-Encrypt-Hash) mode. - -Since filenames are encrypted with the CTS-CBC mode of operation, the -plaintext and ciphertext filenames need not be multiples of the AES -block size, i.e. 16 bytes. However, the minimum size that can be -encrypted is 16 bytes, so shorter filenames are NUL-padded to 16 bytes -before being encrypted. In addition, to reduce leakage of filename -lengths via their ciphertexts, all filenames are NUL-padded to the -next 4, 8, 16, or 32-byte boundary (configurable). 32 is recommended -since this provides the best confidentiality, at the cost of making -directory entries consume slightly more space. Note that since NUL -(``\0``) is not otherwise a valid character in filenames, the padding -will never produce duplicate plaintexts. +the system's page size (usually 4096 bytes) is supported. IVs are +chosen as follows, depending on the contents encryption mode: + +- XTS: the logical block number within the file. +- CBC: ESSIV, specifically the logical block number within the file + encrypted with AES-256, where the AES-256 key is the SHA-256 hash of + the inode's data encryption key. +- Adiantum: the logical block number within the file concatenated with + the per-file nonce. (As noted earlier, including the nonce makes it + safe to use the same key for many files.) + +Filenames encryption +-------------------- + +For filenames, each full filename is encrypted at once. Because of +the requirements to retain support for efficient directory lookups and +filenames of up to 255 bytes, the same IV is used for every filename +in a directory. + +However, each encrypted directory still uses a unique key; or +alternatively has the inode's nonce included in the IV (for Adiantum). +Thus, IV reuse is limited to within a single directory. + +With CTS-CBC, the IV reuse means that when the plaintext filenames +share a common prefix at least as long as the cipher block size (16 +bytes for AES), the corresponding encrypted filenames will also share +a common prefix. This is undesirable. Adiantum does not have this +weakness, as it is a super-pseudorandom permutation. + +All supported filenames encryption modes accept any plaintext length +>= 16 bytes; cipher block alignment is not required. However, +filenames shorter than 16 bytes are NUL-padded to 16 bytes before +being encrypted. In addition, to reduce leakage of filename lengths +via their ciphertexts, all filenames are NUL-padded to the next 4, 8, +16, or 32-byte boundary (configurable). 32 is recommended since this +provides the best confidentiality, at the cost of making directory +entries consume slightly more space. Note that since NUL (``\0``) is +not otherwise a valid character in filenames, the padding will never +produce duplicate plaintexts. Symbolic link targets are considered a type of filename and are -encrypted in the same way as filenames in directory entries. Each -symlink also uses a unique key; hence, the hardcoded IV is not a -problem for symlinks. +encrypted in the same way as filenames in directory entries, except +that IV reuse is not a problem as each symlink has its own inode. User API ========
@@ -272,9 +300,12 @@ This structure must be initialized as follows: and FS_ENCRYPTION_MODE_AES_256_CTS (4) for ``filenames_encryption_mode``. -- ``flags`` must be set to a value from ``<linux/fs.h>`` which +- ``flags`` must contain a value from ``<linux/fs.h>`` which identifies the amount of NUL-padding to use when encrypting - filenames. If unsure, use FS_POLICY_FLAGS_PAD_32 (0x3). + filenames. In addition, if the chosen encryption modes are both + FS_ENCRYPTION_MODE_ADIANTUM, this can contain FS_POLICY_FLAGS_DIRECT + to specify that the master key should be used directly, without key + derivation. If unsure, use FS_POLICY_FLAGS_PAD_32 (0x3). - ``master_key_descriptor`` specifies how to find the master key in the keyring; see `Adding keys`_. It is up to userspace to choose a
fs/crypto/crypto.c +22 -13
@@ -133,15 +133,32 @@ struct fscrypt_ctx *fscrypt_get_ctx(const struct inode *inode, gfp_t gfp_flags) }EXPORT_SYMBOL (fscrypt_get_ctx); +void fscrypt_prepare_iv (union fscrypt_iv *iv,u64 lblk_num, +const struct fscrypt_info *ci) +{ +if (ci->ci_mode->uses_long_ivs) { +/* Using lblk_num and nonce as tweak */ + iv->long_iv.index =cpu_to_le64 (lblk_num); +memcpy (iv->long_iv.nonce, ci->ci_nonce, +FS_KEY_DERIVATION_NONCE_SIZE ); + iv->long_iv.unused =0 ; + }else { +/* Using only lblk_num as tweak (key was derived from nonce) */ + iv->short_iv.index =cpu_to_le64 (lblk_num); + iv->short_iv.unused =0 ; +if (ci->ci_essiv_tfm !=NULL ) { +crypto_cipher_encrypt_one (ci->ci_essiv_tfm, (u8 *)iv, + (const u8 *)iv); + } + } +} +int fscrypt_do_page_crypto (const struct inode *inode,fscrypt_direction_t rw,u64 lblk_num,struct page *src_page,struct page *dest_page,unsigned int len,unsigned int offs,gfp_t gfp_flags) { -struct { -__le64 index; -u8 padding[FS_IV_SIZE -sizeof (__le64)]; - } iv; +union fscrypt_iv iv;struct skcipher_request *req =NULL ;DECLARE_CRYPTO_WAIT (wait);struct scatterlist dst, src;
@@ -151,15 +168,7 @@ int fscrypt_do_page_crypto(const struct inode *inode, fscrypt_direction_t rw,BUG_ON (len ==0 ); -BUILD_BUG_ON (sizeof (iv) !=FS_IV_SIZE ); -BUILD_BUG_ON (AES_BLOCK_SIZE !=FS_IV_SIZE ); - iv.index =cpu_to_le64 (lblk_num); -memset (iv.padding,0 ,sizeof (iv.padding)); - -if (ci->ci_essiv_tfm !=NULL ) { -crypto_cipher_encrypt_one (ci->ci_essiv_tfm, (u8 *)&iv, - (u8 *)&iv); - } +fscrypt_prepare_iv (&iv, lblk_num, ci); req =skcipher_request_alloc (tfm, gfp_flags);if (!req)
fs/crypto/fname.c +12 -10
@@ -40,10 +40,11 @@ int fname_encrypt(struct inode *inode, const struct qstr *iname, {struct skcipher_request *req =NULL ;DECLARE_CRYPTO_WAIT (wait); -struct crypto_skcipher *tfm = inode->i_crypt_info->ci_ctfm; -int res =0 ; -char iv[FS_CRYPTO_BLOCK_SIZE ]; +struct fscrypt_info *ci = inode->i_crypt_info; +struct crypto_skcipher *tfm = ci->ci_ctfm; +union fscrypt_iv iv;struct scatterlist sg; +int res;/* * Copy the filename to the output buffer for encrypting in-place and
@@ -55,7 +56,7 @@ int fname_encrypt(struct inode *inode, const struct qstr *iname,memset (out + iname->len,0 , olen - iname->len);/* Initialize the IV */ -memset (iv,0 ,FS_CRYPTO_BLOCK_SIZE ); +fscrypt_prepare_iv (&iv,0 , ci);/* Set up the encryption request */ req =skcipher_request_alloc (tfm,GFP_NOFS );
@@ -65,7 +66,7 @@ int fname_encrypt(struct inode *inode, const struct qstr *iname,CRYPTO_TFM_REQ_MAY_BACKLOG |CRYPTO_TFM_REQ_MAY_SLEEP , crypto_req_done, &wait);sg_init_one (&sg, out, olen); -skcipher_request_set_crypt (req, &sg, &sg, olen, iv); +skcipher_request_set_crypt (req, &sg, &sg, olen, &iv);/* Do the encryption */ res =crypto_wait_req (crypto_skcipher_encrypt (req), &wait);
@@ -94,9 +95,10 @@ static int fname_decrypt(struct inode *inode,struct skcipher_request *req =NULL ;DECLARE_CRYPTO_WAIT (wait);struct scatterlist src_sg, dst_sg; -struct crypto_skcipher *tfm = inode->i_crypt_info->ci_ctfm; -int res =0 ; -char iv[FS_CRYPTO_BLOCK_SIZE ]; +struct fscrypt_info *ci = inode->i_crypt_info; +struct crypto_skcipher *tfm = ci->ci_ctfm; +union fscrypt_iv iv; +int res;/* Allocate request */ req =skcipher_request_alloc (tfm,GFP_NOFS );
@@ -107,12 +109,12 @@ static int fname_decrypt(struct inode *inode, crypto_req_done, &wait);/* Initialize IV */ -memset (iv,0 ,FS_CRYPTO_BLOCK_SIZE ); +fscrypt_prepare_iv (&iv,0 , ci);/* Create decryption request */ sg_init_one (&src_sg, iname->name, iname->len);sg_init_one (&dst_sg, oname->name, oname->len); -skcipher_request_set_crypt (req, &src_sg, &dst_sg, iname->len, iv); +skcipher_request_set_crypt (req, &src_sg, &dst_sg, iname->len, &iv); res =crypto_wait_req (crypto_skcipher_decrypt (req), &wait);skcipher_request_free (req);if (res <0 ) {
fs/crypto/fscrypt_private.h +60 -6
101 lines not on this page · raw message
fs/crypto/keyinfo.c +245 -77
404 lines not on this page · raw message
fs/crypto/policy.c +3 -2
@@ -199,7 +199,8 @@ int fscrypt_has_permitted_context(struct inode *parent, struct inode *child) child_ci = child->i_crypt_info;if (parent_ci && child_ci) { -return memcmp (parent_ci->ci_master_key, child_ci->ci_master_key, +return memcmp (parent_ci->ci_master_key_descriptor, + child_ci->ci_master_key_descriptor,FS_KEY_DESCRIPTOR_SIZE ) ==0 && (parent_ci->ci_data_mode == child_ci->ci_data_mode) && (parent_ci->ci_filename_mode ==
@@ -254,7 +255,7 @@ int fscrypt_inherit_context(struct inode *parent, struct inode *child, ctx.contents_encryption_mode = ci->ci_data_mode; ctx.filenames_encryption_mode = ci->ci_filename_mode; ctx.flags = ci->ci_flags; -memcpy (ctx.master_key_descriptor, ci->ci_master_key, +memcpy (ctx.master_key_descriptor, ci->ci_master_key_descriptor,FS_KEY_DESCRIPTOR_SIZE );get_random_bytes (ctx.nonce,FS_KEY_DERIVATION_NONCE_SIZE );BUILD_BUG_ON (sizeof (ctx) !=FSCRYPT_SET_CONTEXT_MAX_SIZE );
include/uapi/linux/fs.h +3 -1
@@ -269,7 +269,8 @@ struct fsxattr {#define FS_POLICY_FLAGS_PAD_16 0x02#define FS_POLICY_FLAGS_PAD_32 0x03#define FS_POLICY_FLAGS_PAD_MASK 0x03 -#define FS_POLICY_FLAGS_VALID 0x03 +#define FS_POLICY_FLAGS_DIRECT 0x04/* use master key directly */ +#define FS_POLICY_FLAGS_VALID 0x07/* Encryption algorithms */ #define FS_ENCRYPTION_MODE_INVALID 0
@@ -281,6 +282,7 @@ struct fsxattr {#define FS_ENCRYPTION_MODE_AES_128_CTS 6#define FS_ENCRYPTION_MODE_SPECK128_256_XTS 7/* Removed, do not use. */ #define FS_ENCRYPTION_MODE_SPECK128_256_CTS 8/* Removed, do not use. */ +#define FS_ENCRYPTION_MODE_ADIANTUM 9struct fscrypt_policy {__u8 version;
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Hi Eric,
Quoted 5 lines
This patchset adds Adiantum to Linux's crypto API, focusing on generic and ARM32 implementations. Patches 1-9 add support for XChaCha20 and XChaCha12. Patches 10-13 add NHPoly1305 support, needed for Adiantum hashing. Patch 14 adds Adiantum support as a skcipher template.
Nice work. I did a quick review only, but you may add my
for patches 1-5, 10 and 11.
Thanks,
Martin