WireGuard: Secure Network Tunnel
Subsystems (30)
Jason A. Donenfeld
maintainerv4 cover WireGuard: Secure Network Tunnel
Changes v3->v4:
- Remove mistaken double 07/17 patch.
- Fix whitespace issues in blake2s assembly.
- It's not possible to put compound literals into __initconst, so we now instead just use boring fixed size struct members.
- Move away from makefile ifdef maze and instead prefer kconfig values, which also makes the design a bit more modular too, which could help in the future.
- Port old crypto API implementations (ChaCha20 and Poly1305) to Zinc.
- Port security/keys/big_key to Zinc as second example of a good usage of Zinc.
- Document precisely what is different between the kernel code and CRYPTOGAMS code when the CRYPTOGAMS code is used.
- Move changelog to top of 00/20 message so that people can actually find it.
-----------------------------------------------------------
This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:
* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard
-----------------------------------------------------------
WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.
Ample information, including documentation, installation instructions, and project details, is available at:
* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf
As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:
* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/
The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:
* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf
The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:
* https://www.wireguard.com/formal-verification/
This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.
It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.
Enjoy,
Jason
Jason A. Donenfeld
maintainerPATCH v4 1/20 asm: simd context helper API
Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:
simd_context_t simd_context = simd_get();
while ((item = get_item_from_queue()) != NULL) {
encrypt_item(item, simd_context);
simd_context = simd_relax(simd_context);
}
simd_put(simd_context);The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.
--- arch/alpha/include/asm/Kbuild | 5 ++-- arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/simd.h | 42 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/simd.h | 37 +++++++++++++++++++++----- arch/c6x/include/asm/Kbuild | 3 ++- arch/h8300/include/asm/Kbuild | 3 ++- arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nds32/include/asm/Kbuild | 7 ++--- arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 7 ++--- arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 3 ++- arch/riscv/include/asm/Kbuild | 3 ++- arch/s390/include/asm/Kbuild | 3 ++- arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 3 ++- arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/simd.h | 30 ++++++++++++++++++++- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/simd.h | 15 +++++++++++ include/linux/simd.h | 28 ++++++++++++++++++++ 26 files changed, 180 insertions(+), 21 deletions(-) create mode 100644 arch/arm/include/asm/simd.h create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +3 -2
@@ -2,14 +2,15 @@ generic-y += compat.h +generic-y += current.h generic-y += exec.h generic-y += export.h generic-y += fb.h generic-y += irq_work.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += trace_clock.h -generic-y += current.h -generic-y += kprobes.h
arch/arc/include/asm/Kbuild +1 -0
@@ -22,6 +22,7 @@ generic-y += parport.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/arm/include/asm/simd.h new file +42 -0
42 lines not on this page · raw message
arch/arm64/include/asm/simd.h +31 -6
@@ -1,11 +1,10 @@ -/* -* Copyright (C) 2017 Linaro Ltd. <[email protected]> +/* SPDX-License-Identifier: GPL-2.0 * -* This program is free software; you can redistribute it and/or modify it -* under the terms of the GNU General Public License version 2 as published -* by the Free Software Foundation. +* Copyright (C) 2017 Linaro Ltd. <[email protected]> +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ +#include <linux/simd.h> #ifndef __ASM_SIMD_H#define __ASM_SIMD_H
@@ -16,6 +15,8 @@#include <linux/types.h> #ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> +#include <asm/simd.h> DECLARE_PER_CPU (bool, kernel_neon_busy);
@@ -40,12 +41,36 @@ static __must_check inline bool may_use_simd(void) !this_cpu_read (kernel_neon_busy); } +static inline simd_context_t simd_get (void ) +{ +bool have_simd =may_use_simd (); +if (have_simd) +kernel_neon_begin (); +return have_simd ?HAVE_FULL_SIMD :HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t prior_context) +{ +if (prior_context !=HAVE_NO_SIMD ) +kernel_neon_end (); +} +#else /* ! CONFIG_KERNEL_MODE_NEON */ -static __must_check inline bool may_use_simd (void ) { +static __must_check inline bool may_use_simd (void ) +{return false; } +static inline simd_context_t simd_get (void ) +{ +return HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t prior_context) +{ +} +#endif /* ! CONFIG_KERNEL_MODE_NEON */ #endif
arch/c6x/include/asm/Kbuild +2 -1
@@ -5,8 +5,8 @@ generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h generic-y += preempt.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += tlbflush.h generic-y += topology.h generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
@@ -8,8 +8,8 @@ generic-y += current.h generic-y += delay.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += spinlock.h generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
@@ -29,6 +29,7 @@ generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += topology.h generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
@@ -4,6 +4,7 @@ generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h +generic-y += simd.h generic-y += trace_clock.h generic-y += vtime.h generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h generic-y += percpu.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
@@ -25,6 +25,7 @@ generic-y += parport.h generic-y += percpu.h generic-y += preempt.h generic-y += serial.h +generic-y += simd.h generic-y += syscalls.h generic-y += topology.h generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
@@ -16,6 +16,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
@@ -7,14 +7,14 @@ generic-y += bug.h generic-y += bugs.h generic-y += checksum.h generic-y += clkdev.h -generic-y += cmpxchg.h generic-y += cmpxchg-local.h +generic-y += cmpxchg.h generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h generic-y += segment.h generic-y += serial.h generic-y += shmbuf.h +generic-y += simd.h generic-y += sizes.h generic-y += stat.h generic-y += switch_to.h generic-y += timex.h generic-y += topology.h generic-y += trace_clock.h -generic-y += xor.h generic-y += unaligned.h generic-y += user.h generic-y += vga.h generic-y += word-at-a-time.h +generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
@@ -33,6 +33,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
@@ -28,12 +28,13 @@ generic-y += module.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += qspinlock_types.h -generic-y += qspinlock.h -generic-y += qrwlock_types.h generic-y += qrwlock.h +generic-y += qrwlock_types.h +generic-y += qspinlock.h +generic-y += qspinlock_types.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += string.h generic-y += switch_to.h generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
@@ -17,6 +17,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += seccomp.h generic-y += segment.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
@@ -4,7 +4,8 @@ generic-y += irq_regs.h generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h +generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += vtime.h -generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
@@ -5,9 +5,9 @@ generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-contiguous.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h generic-y += shmbuf.h generic-y += shmparam.h generic-y += signal.h +generic-y += simd.h generic-y += socket.h generic-y += sockios.h generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h generic-y += asm-offsets.h generic-y += cacheflush.h generic-y += device.h +generic-y += div64.h generic-y += dma-contiguous.h generic-y += dma-mapping.h -generic-y += div64.h generic-y += emergency-restart.h generic-y += export.h generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
@@ -16,6 +16,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += trace_clock.h generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
@@ -17,5 +17,6 @@ generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
@@ -16,15 +16,16 @@ generic-y += io.h generic-y += irq_regs.h generic-y += irq_work.h generic-y += kdebug.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += param.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += switch_to.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.h generic-y += xor.h -generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
@@ -27,6 +27,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += syscalls.h generic-y += topology.h
arch/x86/include/asm/simd.h +29 -1
@@ -1,4 +1,11 @@ -/* SPDX-License-Identifier: GPL-2.0 */ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H#include <asm/fpu/api.h>
@@ -10,3 +17,24 @@ static __must_check inline bool may_use_simd(void) {return irq_fpu_usable (); } + +static inline simd_context_t simd_get (void ) +{ +bool have_simd = false; +#if !defined(CONFIG_UML ) + have_simd =may_use_simd (); +if (have_simd) +kernel_fpu_begin (); +#endif +return have_simd ?HAVE_FULL_SIMD :HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t prior_context) +{ +#if !defined(CONFIG_UML ) +if (prior_context !=HAVE_NO_SIMD ) +kernel_fpu_end (); +#endif +} + +#endif /* _ASM_SIMD_H */
arch/xtensa/include/asm/Kbuild +1 -0
@@ -24,6 +24,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += sections.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.h
include/asm-generic/simd.h +15 -0
@@ -1,5 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H +#include <linux/hardirq.h> /*
@@ -13,3 +17,14 @@ static __must_check inline bool may_use_simd(void) {return !in_interrupt (); } + +static inline simd_context_t simd_get (void ) +{ +return HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t prior_context) +{ +} + +#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _SIMD_H +#define _SIMD_H + +typedef enum { +HAVE_NO_SIMD , +HAVE_FULL_SIMD +}simd_context_t ; + +#include <linux/sched.h> +#include <asm/simd.h> + +static inline simd_context_t simd_relax (simd_context_t prior_context) +{ +#ifdef CONFIG_PREEMPT +if (prior_context !=HAVE_NO_SIMD &&need_resched ()) { +simd_put (prior_context); +return simd_get (); + } +#endif +return prior_context; +} + +#endif /* _SIMD_H */
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 2/20 zinc: introduce minimal cryptography library
In reply to Jason A. DonenfeldZinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.
Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:
* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.
* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.
* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.
This last observation has seen itself play out several times over and over again within the kernel:
* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.
* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.
* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.
Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.
It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.
All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].
Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.
SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.
More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.
Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.
Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.
Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.
As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.
[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof
--- MAINTAINERS | 8 ++++++++ lib/Kconfig | 2 ++ lib/Makefile | 2 ++ lib/zinc/Kconfig | 46 ++++++++++++++++++++++++++++++++++++++++++++++ lib/zinc/Makefile | 8 ++++++++ lib/zinc/main.c | 31 +++++++++++++++++++++++++++++++ 6 files changed, 97 insertions(+) create mode 100644 lib/zinc/Kconfig create mode 100644 lib/zinc/Makefile create mode 100644 lib/zinc/main.c
MAINTAINERS +8 -0
@@ -16160,6 +16160,14 @@ Q: https://patchwork.linuxtv.org/project/linux-media/list/ S: Maintained F: drivers/media/dvb-frontends/zd1301_demod* +ZINC CRYPTOGRAPHY LIBRARY +M: Jason A. Donenfeld <[email protected]> +M: Samuel Neves <[email protected]> +S: Maintained +F: lib/zinc/ +F: include/zinc/ +L: [email protected] + ZPOOL COMPRESSED PAGE STORAGE API M: Dan Streetman <[email protected]> L: [email protected]
lib/Kconfig +2 -0
@@ -485,6 +485,8 @@ config GLOB_SELFTEST module load) by a small amount, so you're welcome to play with it, but you probably don't need it. +source "lib/zinc/Kconfig" + # # Netlink attribute parsing support is select'ed if needed #
lib/Makefile +2 -0
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o obj-$(CONFIG_ASN1) += asn1_decoder.o +obj-$(CONFIG_ZINC) += zinc/ + obj-$(CONFIG_FONT_SUPPORT) += fonts/ obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +46 -0
46 lines not on this page · raw message
lib/zinc/Makefile new file +8 -0
@@ -0,0 +1,8 @@ +ccflags-y := -O3 +ccflags-y += -Wframe-larger-than=8192 +ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt' +ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc-y += main.o + +obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/main.c new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <linux/init.h> +#include <linux/module.h> + +#ifdef DEBUG +#define selftest (which) do { \ + if (!which ## _selftest()) \ + return -ENOTRECOVERABLE; \ +} while (0) +#else +#define selftest (which) +#endif + +static int __init mod_init (void ) +{ +return 0 ; +} + +static void __exit mod_exit(void) +{ +} + +module_init (mod_init); +module_exit (mod_exit); +MODULE_LICENSE ("GPL v2" ); +MODULE_DESCRIPTION ("Zinc cryptography library" ); +MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 3/20 zinc: ChaCha20 generic C implementation
In reply to Jason A. DonenfeldThis implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
--- include/zinc/chacha20.h | 54 +++++++++++ lib/zinc/Kconfig | 5 ++ lib/zinc/Makefile | 4 + lib/zinc/chacha20/chacha20.c | 168 +++++++++++++++++++++++++++++++++++ lib/zinc/main.c | 5 ++ 5 files changed, 236 insertions(+) create mode 100644 include/zinc/chacha20.h create mode 100644 lib/zinc/chacha20/chacha20.c
include/zinc/chacha20.h new file +54 -0
@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20_H +#define _ZINC_CHACHA20_H + +#include <asm/unaligned.h> +#include <linux/simd.h> +#include <linux/kernel.h> +#include <linux/types.h> + +enum { +CHACHA20_IV_SIZE =16 , +CHACHA20_KEY_SIZE =32 , +CHACHA20_BLOCK_SIZE =64 , +CHACHA20_BLOCK_WORDS =CHACHA20_BLOCK_SIZE /sizeof (u32), +HCHACHA20_KEY_SIZE =32 , +HCHACHA20_NONCE_SIZE =16 +}; + +struct chacha20_ctx { +u32 key[8 ]; +u32 counter[4 ]; +}__aligned(32) ; + +void chacha20_fpu_init (void ); + +static inline void chacha20_init (struct chacha20_ctx *state, +const u8 key[CHACHA20_KEY_SIZE ], +const u64 nonce) +{ + state->key[0 ] =get_unaligned_le32 (key +0 ); + state->key[1 ] =get_unaligned_le32 (key +4 ); + state->key[2 ] =get_unaligned_le32 (key +8 ); + state->key[3 ] =get_unaligned_le32 (key +12 ); + state->key[4 ] =get_unaligned_le32 (key +16 ); + state->key[5 ] =get_unaligned_le32 (key +20 ); + state->key[6 ] =get_unaligned_le32 (key +24 ); + state->key[7 ] =get_unaligned_le32 (key +28 ); + state->counter[0 ] = state->counter[1 ] =0 ; + state->counter[2 ] = nonce &U32_MAX ; + state->counter[3 ] = nonce >>32 ; +} +void chacha20 (struct chacha20_ctx *state,u8 *dst,const u8 *src,u32 len, +simd_context_t simd_context); + +/* Derived key should be 32-bit aligned */ +void hchacha20 (u8 derived_key[CHACHA20_KEY_SIZE ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t simd_context); + +#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +5 -0
@@ -1,6 +1,11 @@ config ZINC tristate +config ZINC_CHACHA20 + bool + select ZINC + select CRYPTO_ALGAPI + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" depends on ZINC
lib/zinc/Makefile +4 -0
@@ -3,6 +3,10 @@ ccflags-y += -Wframe-larger-than=8192 ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG +ifeq ($(CONFIG_ZINC_CHACHA20),y) +zinc-y += chacha20/chacha20.o +endif + zinc-y += main.o obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/chacha20/chacha20.c new file +168 -0
168 lines not on this page · raw message
lib/zinc/main.c +5 -0
@@ -3,6 +3,8 @@* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ +#include <zinc/chacha20.h> +#include <linux/init.h> #include <linux/module.h>
@@ -17,6 +19,9 @@static int __init mod_init (void ) { +#ifdef CONFIG_ZINC_CHACHA20 +chacha20_fpu_init (); +#endif return 0 ; }
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 4/20 zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:
- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed (prefixed with .L) to fit kernel conventions.
- Constants have been rearranged so that they are closer to the code that is using them. [ARM only]
- The neon code can jump to the scalar code when it makes sense to do so.
- The neon_512 function as a separate function has been removed, leaving the decision up to the main neon entry point. [ARM64 only]
After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.
ARM:
-ChaCha20_ctr32:
-.LChaCha20_ctr32:
+ENTRY(chacha20_arm)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
- sub r14,pc,#16 @ ChaCha20_ctr32
- adr r14,.LChaCha20_ctr32
cmp r2,#0 @ len==0?
itt eq
addeq sp,sp,#4*3
- beq .Lno_data
- cmp r2,#192 @ test len
- bls .Lshort
- ldr r4,[r14,#-32]
- ldr r4,[r14,r4]
- ldr r4,[r4]
- tst r4,#ARMV7_NEON
- bne .LChaCha20_neon
+ beq .Lno_data_arm
.Lshort:
ldmia r12,{r4-r7} @ load counter and nonce
sub sp,sp,#4*(16) @ off-load area
- sub r14,r14,#64 @ .Lsigma
+ sub r14,pc,#100 @ .Lsigma
+ adr r14,.Lsigma @ .Lsigma
stmdb sp!,{r4-r7} @ copy counter and nonce
ldmia r3,{r4-r11} @ load key
ldmia r14,{r0-r3} @ load sigma
@@ -617,14 +615,25 @@
.Ldone:
add sp,sp,#4*(32+3)
-.Lno_data:
+.Lno_data_arm:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_arm)
-ChaCha20_neon:
+ENTRY(chacha20_neon)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
-.LChaCha20_neon:
- adr r14,.Lsigma
+ cmp r2,#0 @ len==0?
+ itt eq
+ addeq sp,sp,#4*3
+ beq .Lno_data_neon
+ cmp r2,#192 @ test len
+ bls .Lshort
+.Lchacha20_neon_begin:
+ adr r14,.Lsigma2
vstmdb sp!,{d8-d15} @ ABI spec says so
stmdb sp!,{r0-r3}
@@ -1265,4 +1274,6 @@
add sp,sp,#4*(32+4)
vldmia sp,{d8-d15}
add sp,sp,#4*(16+3)
+.Lno_data_neon:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_neon)
ARM64:
-ChaCha20_ctr32:
+ENTRY(chacha20_arm)
cbz x2,.Labort
- adr x5,.LOPENSSL_armcap_P
- cmp x2,#192
- b.lo .Lshort
- ldrsw x6,[x5]
- ldr x6,[x5]
- ldr w17,[x6,x5]
- tst w17,#ARMV7_NEON
- b.ne ChaCha20_neon
-
.Lshort:
stp x29,x30,[sp,#-96]!
add x29,sp,#0
@@ -279,8 +274,13 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
+ENDPROC(chacha20_arm)
+
+ENTRY(chacha20_neon)
+ cbz x2,.Labort_neon
+ cmp x2,#192
+ b.lo .Lshort
-ChaCha20_neon:
stp x29,x30,[sp,#-96]!
add x29,sp,#0
@@ -763,16 +763,6 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
-ChaCha20_512_neon:
- stp x29,x30,[sp,#-96]!
- add x29,sp,#0
-
- adr x5,.Lsigma
- stp x19,x20,[sp,#16]
- stp x21,x22,[sp,#32]
- stp x23,x24,[sp,#48]
- stp x25,x26,[sp,#64]
- stp x27,x28,[sp,#80]
.L512_or_more_neon:
sub sp,sp,#128+64
@@ -1920,4 +1910,6 @@
ldp x25,x26,[x29,#64]
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
+.Labort_neon:
ret
+ENDPROC(chacha20_neon)
--- lib/zinc/Makefile | 8 + lib/zinc/chacha20/chacha20-arm-glue.h | 50 + lib/zinc/chacha20/chacha20-arm.S | 1473 +++++++++++++++++++ lib/zinc/chacha20/chacha20-arm64.S | 1942 +++++++++++++++++++++++++ 4 files changed, 3473 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h create mode 100644 lib/zinc/chacha20/chacha20-arm.S create mode 100644 lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/Makefile +8 -0
@@ -5,6 +5,14 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG ifeq ($(CONFIG_ZINC_CHACHA20),y) zinc-y += chacha20/chacha20.o +ifeq ($(CONFIG_ZINC_ARCH_ARM),y) +zinc-y += chacha20/chacha20-arm.o +CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h +endif +ifeq ($(CONFIG_ZINC_ARCH_ARM64),y) +zinc-y += chacha20/chacha20-arm64.o +CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h +endif endif zinc-y += main.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +50 -0
50 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm.S new file +1473 -0
1,473 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm64.S new file +1942 -0
1,463 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v4 5/20 zinc: ChaCha20 x86_64 implementation
In reply to Jason A. DonenfeldThis provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.
- When aligning the stack, the original code was saving %rsp to %r9.
To keep objtool happy, we use instead the DRAP idiom to save %rsp
to %r10:
leaq 8(%rsp),%r10
... code here ...
leaq -8(%r10),%rsp - The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.
- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
--- lib/zinc/Makefile | 4 + lib/zinc/chacha20/chacha20-x86_64-glue.h | 102 + lib/zinc/chacha20/chacha20-x86_64.S | 2632 ++++++++++++++++++++++ 3 files changed, 2738 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +4 -0
@@ -5,6 +5,10 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG ifeq ($(CONFIG_ZINC_CHACHA20),y) zinc-y += chacha20/chacha20.o +ifeq ($(CONFIG_ZINC_ARCH_X86_64),y) +zinc-y += chacha20/chacha20-x86_64.o +CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-x86_64-glue.h +endif ifeq ($(CONFIG_ZINC_ARCH_ARM),y) zinc-y += chacha20/chacha20-arm.o CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +102 -0
102 lines not on this page · raw message
lib/zinc/chacha20/chacha20-x86_64.S new file +2632 -0
2,632 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 6/20 zinc: ChaCha20 MIPS32r2 implementation
In reply to Jason A. DonenfeldThis MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets.
--- lib/zinc/Makefile | 4 + lib/zinc/chacha20/chacha20-mips-glue.h | 28 ++ lib/zinc/chacha20/chacha20-mips.S | 474 +++++++++++++++++++++++++ 3 files changed, 506 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-mips-glue.h create mode 100644 lib/zinc/chacha20/chacha20-mips.S
lib/zinc/Makefile +4 -0
@@ -17,6 +17,10 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM64),y) zinc-y += chacha20/chacha20-arm64.o CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h endif +ifeq ($(CONFIG_ZINC_ARCH_MIPS)$(CONFIG_CPU_MIPS32_R2),yy) +zinc-y += chacha20/chacha20-mips.o +CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-mips-glue.h +endif endif zinc-y += main.o
lib/zinc/chacha20/chacha20-mips-glue.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <zinc/chacha20.h> + +asmlinkage void chacha20_mips(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +void __init chacha20_fpu_init (void ) +{ +} + +static inline bool chacha20_arch (u8 *dst,const u8 *src,const size_t len, +const u32 key[8 ],const u32 counter[4 ], +simd_context_t simd_context) +{ +chacha20_mips (dst, src, len, key, counter); +return true; +} + +static inline bool hchacha20_arch (u8 *derived_key,const u8 *nonce, +const u8 *key,simd_context_t simd_context) +{ +return false; +} + +#define HAVE_CHACHA20_ARCH_IMPLEMENTATION
lib/zinc/chacha20/chacha20-mips.S new file +474 -0
474 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 7/20 zinc: Poly1305 generic C implementations and selftest
In reply to Jason A. DonenfeldThese two C implementations -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel and for usage with accelerated primitives.
Information: https://cr.yp.to/mac.html
--- include/zinc/poly1305.h | 38 ++ lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 4 + lib/zinc/main.c | 5 + lib/zinc/poly1305/poly1305-donna32.h | 205 +++++++ lib/zinc/poly1305/poly1305-donna64.h | 182 ++++++ lib/zinc/poly1305/poly1305.c | 131 ++++ lib/zinc/selftest/poly1305.h | 876 +++++++++++++++++++++++++++ 8 files changed, 1445 insertions(+) create mode 100644 include/zinc/poly1305.h create mode 100644 lib/zinc/poly1305/poly1305-donna32.h create mode 100644 lib/zinc/poly1305/poly1305-donna64.h create mode 100644 lib/zinc/poly1305/poly1305.c create mode 100644 lib/zinc/selftest/poly1305.h
include/zinc/poly1305.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_POLY1305_H +#define _ZINC_POLY1305_H + +#include <linux/simd.h> +#include <linux/types.h> + +enum poly1305_lengths { +POLY1305_BLOCK_SIZE =16 , +POLY1305_KEY_SIZE =32 , +POLY1305_MAC_SIZE =16 +}; + +struct poly1305_ctx { +u8 opaque[24 *sizeof (u64)]; +u32 nonce[4 ]; +u8 data[POLY1305_BLOCK_SIZE ]; +size_t num; +}__aligned(8) ; + +void poly1305_fpu_init (void ); + +void poly1305_init (struct poly1305_ctx *ctx,const u8 key[POLY1305_KEY_SIZE ], +simd_context_t simd_context); +void poly1305_update (struct poly1305_ctx *ctx,const u8 *input,size_t len, +simd_context_t simd_context); +void poly1305_final (struct poly1305_ctx *ctx,u8 mac[POLY1305_MAC_SIZE ], +simd_context_t simd_context); + +#ifdef DEBUG +bool poly1305_selftest (void ); +#endif + +#endif /* _ZINC_POLY1305_H */
lib/zinc/Kconfig +4 -0
@@ -6,6 +6,10 @@ config ZINC_CHACHA20 select ZINC select CRYPTO_ALGAPI +config ZINC_POLY1305 + bool + select ZINC + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" depends on ZINC
lib/zinc/Makefile +4 -0
@@ -23,6 +23,10 @@ CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-mips-glue.h endif endif +ifeq ($(CONFIG_ZINC_POLY1305),y) +zinc-y += poly1305/poly1305.o +endif + zinc-y += main.o obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/main.c +5 -0
@@ -4,6 +4,7 @@*/ #include <zinc/chacha20.h> +#include <zinc/poly1305.h> #include <linux/init.h> #include <linux/module.h>
@@ -21,6 +22,10 @@ static int __init mod_init(void) {#ifdef CONFIG_ZINC_CHACHA20 chacha20_fpu_init (); +#endif +#ifdef CONFIG_ZINC_POLY1305 +poly1305_fpu_init (); +selftest (poly1305);#endif return 0 ; }
lib/zinc/poly1305/poly1305-donna32.h new file +205 -0
205 lines not on this page · raw message
lib/zinc/poly1305/poly1305-donna64.h new file +182 -0
182 lines not on this page · raw message
lib/zinc/poly1305/poly1305.c new file +131 -0
131 lines not on this page · raw message
lib/zinc/selftest/poly1305.h new file +876 -0
876 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 8/20 zinc: Poly1305 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:
- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed to fit kernel conventions.
- The neon code can jump to the scalar code when it makes sense to do so.
After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.
ARM:
-poly1305_init:
-.Lpoly1305_init:
+ENTRY(poly1305_init_arm)
stmdb sp!,{r4-r11}
eor r3,r3,r3
@@ -18,8 +25,6 @@
moveq r0,#0
beq .Lno_key
- adr r11,.Lpoly1305_init
- ldr r12,.LOPENSSL_armcap
ldrb r4,[r1,#0]
mov r10,#0x0fffffff
ldrb r5,[r1,#1]
@@ -34,8 +39,6 @@
ldrb r7,[r1,#6]
and r4,r4,r10
- ldr r12,[r11,r12] @ OPENSSL_armcap_P
- ldr r12,[r12]
ldrb r8,[r1,#7]
orr r5,r5,r6,lsl#8
ldrb r6,[r1,#8]
@@ -45,22 +48,6 @@
ldrb r8,[r1,#10]
and r5,r5,r3
- tst r12,#ARMV7_NEON @ check for NEON
- adr r9,poly1305_blocks_neon
- adr r11,poly1305_blocks
- it ne
- movne r11,r9
- adr r12,poly1305_emit
- adr r10,poly1305_emit_neon
- it ne
- movne r12,r10
- itete eq
- addeq r12,r11,#(poly1305_emit-.Lpoly1305_init)
- addne r12,r11,#(poly1305_emit_neon-.Lpoly1305_init)
- addeq r11,r11,#(poly1305_blocks-.Lpoly1305_init)
- addne r11,r11,#(poly1305_blocks_neon-.Lpoly1305_init)
- orr r12,r12,#1 @ thumb-ify address
- orr r11,r11,#1
ldrb r9,[r1,#11]
orr r6,r6,r7,lsl#8
ldrb r7,[r1,#12]
@@ -79,17 +66,16 @@
str r6,[r0,#8]
and r7,r7,r3
str r7,[r0,#12]
- stmia r2,{r11,r12} @ fill functions table
- mov r0,#1
- mov r0,#0
.Lno_key:
ldmia sp!,{r4-r11}
bx lr @ bx lr
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_blocks:
-.Lpoly1305_blocks:
+ENDPROC(poly1305_init_arm)
+
+ENTRY(poly1305_blocks_arm)
+.Lpoly1305_blocks_arm:
stmdb sp!,{r3-r11,lr}
ands r2,r2,#-16
@@ -231,10 +217,11 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_emit:
+ENDPROC(poly1305_blocks_arm)
+
+ENTRY(poly1305_emit_arm)
stmdb sp!,{r4-r11}
.Lpoly1305_emit_enter:
-
ldmia r0,{r3-r7}
adds r8,r3,#5 @ compare to modulus
adcs r9,r4,#0
@@ -305,8 +292,12 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
+ENDPROC(poly1305_emit_arm)
+
+
-poly1305_init_neon:
+ENTRY(poly1305_init_neon)
+.Lpoly1305_init_neon:
ldr r4,[r0,#20] @ load key base 2^32
ldr r5,[r0,#24]
ldr r6,[r0,#28]
@@ -515,8 +506,9 @@
vst1.32 {d8[1]},[r7]
bx lr @ bx lr
+ENDPROC(poly1305_init_neon)
-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr ip,[r0,#36] @ is_base2_26
ands r2,r2,#-16
beq .Lno_data_neon
@@ -524,7 +516,7 @@
cmp r2,#64
bhs .Lenter_neon
tst ip,ip @ is_base2_26?
- beq .Lpoly1305_blocks
+ beq .Lpoly1305_blocks_arm
.Lenter_neon:
stmdb sp!,{r4-r7}
@@ -534,7 +526,7 @@
bne .Lbase2_26_neon
stmdb sp!,{r1-r3,lr}
- bl poly1305_init_neon
+ bl .Lpoly1305_init_neon
ldr r4,[r0,#0] @ load hash value base 2^32
ldr r5,[r0,#4]
@@ -989,8 +981,9 @@
ldmia sp!,{r4-r7}
.Lno_data_neon:
bx lr @ bx lr
+ENDPROC(poly1305_blocks_neon)
-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr ip,[r0,#36] @ is_base2_26
stmdb sp!,{r4-r11}
@@ -1055,6 +1048,6 @@
ldmia sp!,{r4-r11}
bx lr @ bx lr
+ENDPROC(poly1305_emit_neon)
ARM64:
-poly1305_init:
+ENTRY(poly1305_init_arm)
cmp x1,xzr
stp xzr,xzr,[x0] // zero hash value
stp xzr,xzr,[x0,#16] // [along with is_base2_26]
@@ -11,14 +15,9 @@
csel x0,xzr,x0,eq
b.eq .Lno_key
- ldrsw x11,.LOPENSSL_armcap_P
- ldr x11,.LOPENSSL_armcap_P
- adr x10,.LOPENSSL_armcap_P
-
ldp x7,x8,[x1] // load key
mov x9,#0xfffffffc0fffffff
movk x9,#0x0fff,lsl#48
- ldr w17,[x10,x11]
rev x7,x7 // flip bytes
rev x8,x8
and x7,x7,x9 // &=0ffffffc0fffffff
@@ -26,24 +25,11 @@
and x8,x8,x9 // &=0ffffffc0ffffffc
stp x7,x8,[x0,#32] // save key value
- tst w17,#ARMV7_NEON
-
- adr x12,poly1305_blocks
- adr x7,poly1305_blocks_neon
- adr x13,poly1305_emit
- adr x8,poly1305_emit_neon
-
- csel x12,x12,x7,eq
- csel x13,x13,x8,eq
-
- stp w12,w13,[x2]
- stp x12,x13,[x2]
-
- mov x0,#1
.Lno_key:
ret
+ENDPROC(poly1305_init_arm)
-poly1305_blocks:
+ENTRY(poly1305_blocks_arm)
ands x2,x2,#-16
b.eq .Lno_data
@@ -100,8 +86,9 @@
.Lno_data:
ret
+ENDPROC(poly1305_blocks_arm)
-poly1305_emit:
+ENTRY(poly1305_emit_arm)
ldp x4,x5,[x0] // load hash base 2^64
ldr x6,[x0,#16]
ldp x10,x11,[x2] // load nonce
@@ -124,7 +111,9 @@
stp x4,x5,[x1] // write result
ret
-poly1305_mult:
+ENDPROC(poly1305_emit_arm)
+
+__poly1305_mult:
mul x12,x4,x7 // h0*r0
umulh x13,x4,x7
@@ -158,7 +147,7 @@
ret
-poly1305_splat:
+__poly1305_splat:
and x12,x4,#0x03ffffff // base 2^64 -> base 2^26
ubfx x13,x4,#26,#26
extr x14,x5,x4,#52
@@ -182,11 +171,11 @@
ret
-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr x17,[x0,#24]
cmp x2,#128
b.hs .Lblocks_neon
- cbz x17,poly1305_blocks
+ cbz x17,poly1305_blocks_arm
.Lblocks_neon:
stp x29,x30,[sp,#-80]!
@@ -232,7 +221,7 @@
adcs x5,x5,x13
adc x6,x6,x3
- bl poly1305_mult
+ bl __poly1305_mult
ldr x30,[sp,#8]
cbz x3,.Lstore_base2_64_neon
@@ -274,7 +263,7 @@
adcs x5,x5,x13
adc x6,x6,x3
- bl poly1305_mult
+ bl __poly1305_mult
.Linit_neon:
and x10,x4,#0x03ffffff // base 2^64 -> base 2^26
@@ -301,19 +290,19 @@
mov x5,x8
mov x6,xzr
add x0,x0,#48+12
- bl poly1305_splat
+ bl __poly1305_splat
- bl poly1305_mult // r^2
+ bl __poly1305_mult // r^2
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat
- bl poly1305_mult // r^3
+ bl __poly1305_mult // r^3
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat
- bl poly1305_mult // r^4
+ bl __poly1305_mult // r^4
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat
ldr x30,[sp,#8]
add x16,x1,#32
@@ -743,10 +732,11 @@
.Lno_data_neon:
ldr x29,[sp],#80
ret
+ENDPROC(poly1305_blocks_neon)
-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr x17,[x0,#24]
- cbz x17,poly1305_emit
+ cbz x17,poly1305_emit_arm
ldp w10,w11,[x0] // load hash value base 2^26
ldp w12,w13,[x0,#8]
@@ -788,6 +778,6 @@
stp x4,x5,[x1] // write result
ret
+ENDPROC(poly1305_emit_neon)
--- lib/zinc/Makefile | 8 + lib/zinc/poly1305/poly1305-arm-glue.h | 69 ++ lib/zinc/poly1305/poly1305-arm.S | 1117 +++++++++++++++++++++++++ lib/zinc/poly1305/poly1305-arm64.S | 822 ++++++++++++++++++ 4 files changed, 2016 insertions(+) create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.h create mode 100644 lib/zinc/poly1305/poly1305-arm.S create mode 100644 lib/zinc/poly1305/poly1305-arm64.S
lib/zinc/Makefile +8 -0
@@ -25,6 +25,14 @@ endif ifeq ($(CONFIG_ZINC_POLY1305),y) zinc-y += poly1305/poly1305.o +ifeq ($(CONFIG_ZINC_ARCH_ARM),y) +zinc-y += poly1305/poly1305-arm.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h +endif +ifeq ($(CONFIG_ZINC_ARCH_ARM64),y) +zinc-y += poly1305/poly1305-arm64.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h +endif endif zinc-y += main.o
lib/zinc/poly1305/poly1305-arm-glue.h new file +69 -0
69 lines not on this page · raw message
lib/zinc/poly1305/poly1305-arm.S new file +1117 -0
1,117 lines not on this page · raw message
lib/zinc/poly1305/poly1305-arm64.S new file +822 -0
822 lines not on this page · raw message
Signature · 1 line
2.19.0
Re: [PATCH net-next v4 08/20] zinc: Poly1305 ARM and ARM64 implementations
Quoted 12 lines · Jason A. Donenfeld
These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:
- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed to fit kernel conventions.
- The neon code can jump to the scalar code when it makes sense to do so.
Quoted 3 lines
After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.
As I asked in response to v3, could we please have this as a separate patch on top? The diff below is corrupted.
Also, both Andy and Eric have offered to get involved in upstreaming these changes to OpenSSL, so there is no delta to begin with.
Quoted 153 lines
ARM:
(quoted diff) +20
-34-poly1305_init: -.Lpoly1305_init: +ENTRY(poly1305_init_arm) stmdb sp!,{r4-r11} eor r3,r3,r3@@ -18,8 +25,6 @@ moveq r0,#0 beq .Lno_key - adr r11,.Lpoly1305_init - ldr r12,.LOPENSSL_armcap ldrb r4,[r1,#0] mov r10,#0x0fffffff ldrb r5,[r1,#1]@@ -34,8 +39,6 @@ ldrb r7,[r1,#6] and r4,r4,r10 - ldr r12,[r11,r12] @ OPENSSL_armcap_P - ldr r12,[r12] ldrb r8,[r1,#7] orr r5,r5,r6,lsl#8 ldrb r6,[r1,#8]@@ -45,22 +48,6 @@ ldrb r8,[r1,#10] and r5,r5,r3 - tst r12,#ARMV7_NEON @ check for NEON - adr r9,poly1305_blocks_neon - adr r11,poly1305_blocks - it ne - movne r11,r9 - adr r12,poly1305_emit - adr r10,poly1305_emit_neon - it ne - movne r12,r10 - itete eq - addeq r12,r11,#(poly1305_emit-.Lpoly1305_init) - addne r12,r11,#(poly1305_emit_neon-.Lpoly1305_init) - addeq r11,r11,#(poly1305_blocks-.Lpoly1305_init) - addne r11,r11,#(poly1305_blocks_neon-.Lpoly1305_init) - orr r12,r12,#1 @ thumb-ify address - orr r11,r11,#1 ldrb r9,[r1,#11] orr r6,r6,r7,lsl#8 ldrb r7,[r1,#12]@@ -79,17 +66,16 @@ str r6,[r0,#8] and r7,r7,r3 str r7,[r0,#12] - stmia r2,{r11,r12} @ fill functions table - mov r0,#1 - mov r0,#0 .Lno_key: ldmia sp!,{r4-r11} bx lr @ bx lr tst lr,#1 moveq pc,lr @ be binary compatible with V4, yet .word 0xe12fff1e @ interoperable with Thumb ISA:-) -poly1305_blocks: -.Lpoly1305_blocks: +ENDPROC(poly1305_init_arm) + +ENTRY(poly1305_blocks_arm) +.Lpoly1305_blocks_arm: stmdb sp!,{r3-r11,lr} ands r2,r2,#-16@@ -231,10 +217,11 @@ tst lr,#1 moveq pc,lr @ be binary compatible with V4, yet .word 0xe12fff1e @ interoperable with Thumb ISA:-) -poly1305_emit: +ENDPROC(poly1305_blocks_arm) + +ENTRY(poly1305_emit_arm) stmdb sp!,{r4-r11} .Lpoly1305_emit_enter: - ldmia r0,{r3-r7} adds r8,r3,#5 @ compare to modulus adcs r9,r4,#0@@ -305,8 +292,12 @@ tst lr,#1 moveq pc,lr @ be binary compatible with V4, yet .word 0xe12fff1e @ interoperable with Thumb ISA:-) +ENDPROC(poly1305_emit_arm) + + -poly1305_init_neon: +ENTRY(poly1305_init_neon) +.Lpoly1305_init_neon: ldr r4,[r0,#20] @ load key base 2^32 ldr r5,[r0,#24] ldr r6,[r0,#28]@@ -515,8 +506,9 @@ vst1.32 {d8[1]},[r7] bx lr @ bx lr +ENDPROC(poly1305_init_neon) -poly1305_blocks_neon: +ENTRY(poly1305_blocks_neon) ldr ip,[r0,#36] @ is_base2_26 ands r2,r2,#-16 beq .Lno_data_neon@@ -524,7 +516,7 @@ cmp r2,#64 bhs .Lenter_neon tst ip,ip @ is_base2_26? - beq .Lpoly1305_blocks + beq .Lpoly1305_blocks_arm .Lenter_neon: stmdb sp!,{r4-r7}@@ -534,7 +526,7 @@ bne .Lbase2_26_neon stmdb sp!,{r1-r3,lr} - bl poly1305_init_neon + bl .Lpoly1305_init_neon ldr r4,[r0,#0] @ load hash value base 2^32 ldr r5,[r0,#4]@@ -989,8 +981,9 @@ ldmia sp!,{r4-r7} .Lno_data_neon: bx lr @ bx lr +ENDPROC(poly1305_blocks_neon) -poly1305_emit_neon: +ENTRY(poly1305_emit_neon) ldr ip,[r0,#36] @ is_base2_26 stmdb sp!,{r4-r11}@@ -1055,6 +1048,6 @@ ldmia sp!,{r4-r11} bx lr @ bx lr +ENDPROC(poly1305_emit_neon)ARM64:
(quoted diff) +1
-1-poly1305_init: +ENTRY(poly1305_init_arm) cmp x1,xzr stp xzr,xzr,[x0] // zero hash value stp xzr,xzr,[x0,#16] // [along with is_base2_26]
Review line 15
Quoted 5 lines
(quoted diff) +0
-2@@ -11,14 +15,9 @@ csel x0,xzr,x0,eq b.eq .Lno_key - ldrsw x11,.LOPENSSL_armcap_P - ldr x11,.LOPENSSL_armcap_P
In the original, this looks like
#ifdef __ILP32__
ldrsw $t1,.LOPENSSL_armcap_P
#else
ldr $t1,.LOPENSSL_armcap_P
#endif
so I guess git commit ate those lines.
Quoted 172 lines
(quoted diff) +23
-34- adr x10,.LOPENSSL_armcap_P - ldp x7,x8,[x1] // load key mov x9,#0xfffffffc0fffffff movk x9,#0x0fff,lsl#48 - ldr w17,[x10,x11] rev x7,x7 // flip bytes rev x8,x8 and x7,x7,x9 // &=0ffffffc0fffffff@@ -26,24 +25,11 @@ and x8,x8,x9 // &=0ffffffc0ffffffc stp x7,x8,[x0,#32] // save key value - tst w17,#ARMV7_NEON - - adr x12,poly1305_blocks - adr x7,poly1305_blocks_neon - adr x13,poly1305_emit - adr x8,poly1305_emit_neon - - csel x12,x12,x7,eq - csel x13,x13,x8,eq - - stp w12,w13,[x2] - stp x12,x13,[x2] - - mov x0,#1 .Lno_key: ret +ENDPROC(poly1305_init_arm) -poly1305_blocks: +ENTRY(poly1305_blocks_arm) ands x2,x2,#-16 b.eq .Lno_data@@ -100,8 +86,9 @@ .Lno_data: ret +ENDPROC(poly1305_blocks_arm) -poly1305_emit: +ENTRY(poly1305_emit_arm) ldp x4,x5,[x0] // load hash base 2^64 ldr x6,[x0,#16] ldp x10,x11,[x2] // load nonce@@ -124,7 +111,9 @@ stp x4,x5,[x1] // write result ret -poly1305_mult: +ENDPROC(poly1305_emit_arm) + +__poly1305_mult: mul x12,x4,x7 // h0*r0 umulh x13,x4,x7@@ -158,7 +147,7 @@ ret -poly1305_splat: +__poly1305_splat: and x12,x4,#0x03ffffff // base 2^64 -> base 2^26 ubfx x13,x4,#26,#26 extr x14,x5,x4,#52@@ -182,11 +171,11 @@ ret -poly1305_blocks_neon: +ENTRY(poly1305_blocks_neon) ldr x17,[x0,#24] cmp x2,#128 b.hs .Lblocks_neon - cbz x17,poly1305_blocks + cbz x17,poly1305_blocks_arm .Lblocks_neon: stp x29,x30,[sp,#-80]!@@ -232,7 +221,7 @@ adcs x5,x5,x13 adc x6,x6,x3 - bl poly1305_mult + bl __poly1305_mult ldr x30,[sp,#8] cbz x3,.Lstore_base2_64_neon@@ -274,7 +263,7 @@ adcs x5,x5,x13 adc x6,x6,x3 - bl poly1305_mult + bl __poly1305_mult .Linit_neon: and x10,x4,#0x03ffffff // base 2^64 -> base 2^26@@ -301,19 +290,19 @@ mov x5,x8 mov x6,xzr add x0,x0,#48+12 - bl poly1305_splat + bl __poly1305_splat - bl poly1305_mult // r^2 + bl __poly1305_mult // r^2 sub x0,x0,#4 - bl poly1305_splat + bl __poly1305_splat - bl poly1305_mult // r^3 + bl __poly1305_mult // r^3 sub x0,x0,#4 - bl poly1305_splat + bl __poly1305_splat - bl poly1305_mult // r^4 + bl __poly1305_mult // r^4 sub x0,x0,#4 - bl poly1305_splat + bl __poly1305_splat ldr x30,[sp,#8] add x16,x1,#32@@ -743,10 +732,11 @@ .Lno_data_neon: ldr x29,[sp],#80 ret +ENDPROC(poly1305_blocks_neon) -poly1305_emit_neon: +ENTRY(poly1305_emit_neon) ldr x17,[x0,#24] - cbz x17,poly1305_emit + cbz x17,poly1305_emit_arm ldp w10,w11,[x0] // load hash value base 2^26 ldp w12,w13,[x0,#8]@@ -788,6 +778,6 @@ stp x4,x5,[x1] // write result ret +ENDPROC(poly1305_emit_neon)Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Andy Polyakov <[email protected]>Cc Russell King <[email protected]>Cc linux-arm-kernel at lists.infradead.org--- lib/zinc/Makefile | 8 + lib/zinc/poly1305/poly1305-arm-glue.h | 69 ++ lib/zinc/poly1305/poly1305-arm.S | 1117 +++++++++++++++++++++++++ lib/zinc/poly1305/poly1305-arm64.S | 822 ++++++++++++++++++ 4 files changed, 2016 insertions(+) create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.h create mode 100644 lib/zinc/poly1305/poly1305-arm.S create mode 100644 lib/zinc/poly1305/poly1305-arm64.Slib/zinc/Makefile +0
-0@@ -25,6 +25,14 @@ endif ifeq ($(CONFIG_ZINC_POLY1305),y) zinc-y += poly1305/poly1305.o
lib/zinc/Makefile endif · line 37
Quoted 10 lines
lib/zinc/Makefile +8
-0+ifeq ($(CONFIG_ZINC_ARCH_ARM),y) +zinc-y += poly1305/poly1305-arm.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h +endif +ifeq ($(CONFIG_ZINC_ARCH_ARM64),y) +zinc-y += poly1305/poly1305-arm64.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h +endif endif
I still don't like the GCC -includes, especially because these .h files contain function and variable definitions so they are not actually header files to begin with.
Also, you mentioned in the commit log that you got rid of defines and made the code more modular, but as far as I can tell, libzinc is still a single monolithic binary that is essentially always builtin once we move random.c to it.
Quoted 66 lines
lib/zinc/Makefile +0
-0zinc-y += main.olib/zinc/poly1305/poly1305-arm-glue.h new file +59
-0@@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <zinc/poly1305.h> +#include <asm/hwcap.h> +#include <asm/neon.h> + +asmlinkage void poly1305_init_arm(void *ctx, const u8 key[16]); +asmlinkage void poly1305_blocks_arm(void *ctx, const u8 *inp, const size_t len, +const u32 padbit); +asmlinkage void poly1305_emit_arm(void *ctx, u8 mac[16], const u32 nonce[4]); +#if IS_ENABLED (CONFIG_KERNEL_MODE_NEON ) && \ + (defined(CONFIG_64BIT ) || __LINUX_ARM_ARCH__ >=7 ) +#define ARM_USE_NEON +asmlinkage void poly1305_blocks_neon(void *ctx, const u8 *inp, const size_t len, +const u32 padbit); +asmlinkage void poly1305_emit_neon(void *ctx, u8 mac[16], const u32 nonce[4]); +#endif + +static bool poly1305_use_neon__ro_after_init ; + +void __init poly1305_fpu_init (void ) +{ +#if defined(CONFIG_ARM64 ) + poly1305_use_neon = elf_hwcap &HWCAP_ASIMD ; +#elif defined(CONFIG_ARM ) + poly1305_use_neon = elf_hwcap &HWCAP_NEON ; +#endif +} + +static inline bool poly1305_init_arch (void *ctx, +const u8 key[POLY1305_KEY_SIZE ], +simd_context_t simd_context) +{ +poly1305_init_arm (ctx, key); +return true; +} + +static inline bool poly1305_blocks_arch (void *ctx,const u8 *inp, +const size_t len,const u32 padbit, +simd_context_t simd_context) +{ +#if defined(ARM_USE_NEON ) +if (simd_context ==HAVE_FULL_SIMD && poly1305_use_neon) { +poly1305_blocks_neon (ctx, inp, len, padbit); +return true; + } +#endif +poly1305_blocks_arm (ctx, inp, len, padbit); +return true; +} + +static inline bool poly1305_emit_arch (void *ctx,u8 mac[POLY1305_MAC_SIZE ], +const u32 nonce[4 ], +simd_context_t simd_context) +{ +#if defined(ARM_USE_NEON )
lib/zinc/poly1305/poly1305-arm-glue.h line 69
Quoted 10 lines
lib/zinc/poly1305/poly1305-arm-glue.h new file +10
-0+if (simd_context ==HAVE_FULL_SIMD && poly1305_use_neon) { +poly1305_emit_neon (ctx, mac, nonce); +return true; + } +#endif +poly1305_emit_arm (ctx, mac, nonce); +return true; +} + +#define HAVE_POLY1305_ARCH_IMPLEMENTATION
We shouldn't #define HAVE_xxx constants in code but only in Kconfig.
Quoted 1,954 lines
lib/zinc/poly1305/poly1305-arm.S new file +1117
-0@@ -0,0 +1,1117 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 + * + * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. + * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + * + * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS. + */ + +#include <linux/linkage.h> + +.text +#if defined(__thumb2__) +.syntax unified +.thumb +#else +.code 32 +#endif + +.align 5 +ENTRY(poly1305_init_arm) + stmdb sp!,{r4-r11} + + eor r3,r3,r3 + cmp r1,#0 + str r3,[r0,#0] @ zero hash value + str r3,[r0,#4] + str r3,[r0,#8] + str r3,[r0,#12] + str r3,[r0,#16] + str r3,[r0,#36] @ is_base2_26 + add r0,r0,#20 + +#ifdef __thumb2__ + it eq +#endif + moveq r0,#0 + beq .Lno_key + + ldrb r4,[r1,#0] + mov r10,#0x0fffffff + ldrb r5,[r1,#1] + and r3,r10,#-4 @ 0x0ffffffc + ldrb r6,[r1,#2] + ldrb r7,[r1,#3] + orr r4,r4,r5,lsl#8 + ldrb r5,[r1,#4] + orr r4,r4,r6,lsl#16 + ldrb r6,[r1,#5] + orr r4,r4,r7,lsl#24 + ldrb r7,[r1,#6] + and r4,r4,r10 + + ldrb r8,[r1,#7] + orr r5,r5,r6,lsl#8 + ldrb r6,[r1,#8] + orr r5,r5,r7,lsl#16 + ldrb r7,[r1,#9] + orr r5,r5,r8,lsl#24 + ldrb r8,[r1,#10] + and r5,r5,r3 + + ldrb r9,[r1,#11] + orr r6,r6,r7,lsl#8 + ldrb r7,[r1,#12] + orr r6,r6,r8,lsl#16 + ldrb r8,[r1,#13] + orr r6,r6,r9,lsl#24 + ldrb r9,[r1,#14] + and r6,r6,r3 + + ldrb r10,[r1,#15] + orr r7,r7,r8,lsl#8 + str r4,[r0,#0] + orr r7,r7,r9,lsl#16 + str r5,[r0,#4] + orr r7,r7,r10,lsl#24 + str r6,[r0,#8] + and r7,r7,r3 + str r7,[r0,#12] +.Lno_key: + ldmia sp!,{r4-r11} +#if __LINUX_ARM_ARCH__ >= 5 + bx lr @ bx lr +#else + tst lr,#1 + moveq pc,lr @ be binary compatible with V4, yet + .word 0xe12fff1e @ interoperable with Thumb ISA:-) +#endif +ENDPROC(poly1305_init_arm) + +.align 5 +ENTRY(poly1305_blocks_arm) +.Lpoly1305_blocks_arm: + stmdb sp!,{r3-r11,lr} + + ands r2,r2,#-16 + beq .Lno_data + + cmp r3,#0 + add r2,r2,r1 @ end pointer + sub sp,sp,#32 + + ldmia r0,{r4-r12} @ load context + + str r0,[sp,#12] @ offload stuff + mov lr,r1 + str r2,[sp,#16] + str r10,[sp,#20] + str r11,[sp,#24] + str r12,[sp,#28] + b .Loop + +.Loop: +#if __LINUX_ARM_ARCH__ < 7 + ldrb r0,[lr],#16 @ load input +#ifdef __thumb2__ + it hi +#endif + addhi r8,r8,#1 @ 1<<128 + ldrb r1,[lr,#-15] + ldrb r2,[lr,#-14] + ldrb r3,[lr,#-13] + orr r1,r0,r1,lsl#8 + ldrb r0,[lr,#-12] + orr r2,r1,r2,lsl#16 + ldrb r1,[lr,#-11] + orr r3,r2,r3,lsl#24 + ldrb r2,[lr,#-10] + adds r4,r4,r3 @ accumulate input + + ldrb r3,[lr,#-9] + orr r1,r0,r1,lsl#8 + ldrb r0,[lr,#-8] + orr r2,r1,r2,lsl#16 + ldrb r1,[lr,#-7] + orr r3,r2,r3,lsl#24 + ldrb r2,[lr,#-6] + adcs r5,r5,r3 + + ldrb r3,[lr,#-5] + orr r1,r0,r1,lsl#8 + ldrb r0,[lr,#-4] + orr r2,r1,r2,lsl#16 + ldrb r1,[lr,#-3] + orr r3,r2,r3,lsl#24 + ldrb r2,[lr,#-2] + adcs r6,r6,r3 + + ldrb r3,[lr,#-1] + orr r1,r0,r1,lsl#8 + str lr,[sp,#8] @ offload input pointer + orr r2,r1,r2,lsl#16 + add r10,r10,r10,lsr#2 + orr r3,r2,r3,lsl#24 +#else + ldr r0,[lr],#16 @ load input +#ifdef __thumb2__ + it hi +#endif + addhi r8,r8,#1 @ padbit + ldr r1,[lr,#-12] + ldr r2,[lr,#-8] + ldr r3,[lr,#-4] +#ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +#endif + adds r4,r4,r0 @ accumulate input + str lr,[sp,#8] @ offload input pointer + adcs r5,r5,r1 + add r10,r10,r10,lsr#2 + adcs r6,r6,r2 +#endif + add r11,r11,r11,lsr#2 + adcs r7,r7,r3 + add r12,r12,r12,lsr#2 + + umull r2,r3,r5,r9 + adc r8,r8,#0 + umull r0,r1,r4,r9 + umlal r2,r3,r8,r10 + umlal r0,r1,r7,r10 + ldr r10,[sp,#20] @ reload r10 + umlal r2,r3,r6,r12 + umlal r0,r1,r5,r12 + umlal r2,r3,r7,r11 + umlal r0,r1,r6,r11 + umlal r2,r3,r4,r10 + str r0,[sp,#0] @ future r4 + mul r0,r11,r8 + ldr r11,[sp,#24] @ reload r11 + adds r2,r2,r1 @ d1+=d0>>32 + eor r1,r1,r1 + adc lr,r3,#0 @ future r6 + str r2,[sp,#4] @ future r5 + + mul r2,r12,r8 + eor r3,r3,r3 + umlal r0,r1,r7,r12 + ldr r12,[sp,#28] @ reload r12 + umlal r2,r3,r7,r9 + umlal r0,r1,r6,r9 + umlal r2,r3,r6,r10 + umlal r0,r1,r5,r10 + umlal r2,r3,r5,r11 + umlal r0,r1,r4,r11 + umlal r2,r3,r4,r12 + ldr r4,[sp,#0] + mul r8,r9,r8 + ldr r5,[sp,#4] + + adds r6,lr,r0 @ d2+=d1>>32 + ldr lr,[sp,#8] @ reload input pointer + adc r1,r1,#0 + adds r7,r2,r1 @ d3+=d2>>32 + ldr r0,[sp,#16] @ reload end pointer + adc r3,r3,#0 + add r8,r8,r3 @ h4+=d3>>32 + + and r1,r8,#-4 + and r8,r8,#3 + add r1,r1,r1,lsr#2 @ *=5 + adds r4,r4,r1 + adcs r5,r5,#0 + adcs r6,r6,#0 + adcs r7,r7,#0 + adc r8,r8,#0 + + cmp r0,lr @ done yet? + bhi .Loop + + ldr r0,[sp,#12] + add sp,sp,#32 + stmia r0,{r4-r8} @ store the result + +.Lno_data: +#if __LINUX_ARM_ARCH__ >= 5 + ldmia sp!,{r3-r11,pc} +#else + ldmia sp!,{r3-r11,lr} + tst lr,#1 + moveq pc,lr @ be binary compatible with V4, yet + .word 0xe12fff1e @ interoperable with Thumb ISA:-) +#endif +ENDPROC(poly1305_blocks_arm) + +.align 5 +ENTRY(poly1305_emit_arm) + stmdb sp!,{r4-r11} +.Lpoly1305_emit_enter: + ldmia r0,{r3-r7} + adds r8,r3,#5 @ compare to modulus + adcs r9,r4,#0 + adcs r10,r5,#0 + adcs r11,r6,#0 + adc r7,r7,#0 + tst r7,#4 @ did it carry/borrow? + +#ifdef __thumb2__ + it ne +#endif + movne r3,r8 + ldr r8,[r2,#0] +#ifdef __thumb2__ + it ne +#endif + movne r4,r9 + ldr r9,[r2,#4] +#ifdef __thumb2__ + it ne +#endif + movne r5,r10 + ldr r10,[r2,#8] +#ifdef __thumb2__ + it ne +#endif + movne r6,r11 + ldr r11,[r2,#12] + + adds r3,r3,r8 + adcs r4,r4,r9 + adcs r5,r5,r10 + adc r6,r6,r11 + +#if __LINUX_ARM_ARCH__ >= 7 +#ifdef __ARMEB__ + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 +#endif + str r3,[r1,#0] + str r4,[r1,#4] + str r5,[r1,#8] + str r6,[r1,#12] +#else + strb r3,[r1,#0] + mov r3,r3,lsr#8 + strb r4,[r1,#4] + mov r4,r4,lsr#8 + strb r5,[r1,#8] + mov r5,r5,lsr#8 + strb r6,[r1,#12] + mov r6,r6,lsr#8 + + strb r3,[r1,#1] + mov r3,r3,lsr#8 + strb r4,[r1,#5] + mov r4,r4,lsr#8 + strb r5,[r1,#9] + mov r5,r5,lsr#8 + strb r6,[r1,#13] + mov r6,r6,lsr#8 + + strb r3,[r1,#2] + mov r3,r3,lsr#8 + strb r4,[r1,#6] + mov r4,r4,lsr#8 + strb r5,[r1,#10] + mov r5,r5,lsr#8 + strb r6,[r1,#14] + mov r6,r6,lsr#8 + + strb r3,[r1,#3] + strb r4,[r1,#7] + strb r5,[r1,#11] + strb r6,[r1,#15] +#endif + ldmia sp!,{r4-r11} +#if __LINUX_ARM_ARCH__ >= 5 + bx lr @ bx lr +#else + tst lr,#1 + moveq pc,lr @ be binary compatible with V4, yet + .word 0xe12fff1e @ interoperable with Thumb ISA:-) +#endif +ENDPROC(poly1305_emit_arm) + + +#if __LINUX_ARM_ARCH__ >= 7 +.fpu neon + +.align 5 +ENTRY(poly1305_init_neon) +.Lpoly1305_init_neon: + ldr r4,[r0,#20] @ load key base 2^32 + ldr r5,[r0,#24] + ldr r6,[r0,#28] + ldr r7,[r0,#32] + + and r2,r4,#0x03ffffff @ base 2^32 -> base 2^26 + mov r3,r4,lsr#26 + mov r4,r5,lsr#20 + orr r3,r3,r5,lsl#6 + mov r5,r6,lsr#14 + orr r4,r4,r6,lsl#12 + mov r6,r7,lsr#8 + orr r5,r5,r7,lsl#18 + and r3,r3,#0x03ffffff + and r4,r4,#0x03ffffff + and r5,r5,#0x03ffffff + + vdup.32 d0,r2 @ r^1 in both lanes + add r2,r3,r3,lsl#2 @ *5 + vdup.32 d1,r3 + add r3,r4,r4,lsl#2 + vdup.32 d2,r2 + vdup.32 d3,r4 + add r4,r5,r5,lsl#2 + vdup.32 d4,r3 + vdup.32 d5,r5 + add r5,r6,r6,lsl#2 + vdup.32 d6,r4 + vdup.32 d7,r6 + vdup.32 d8,r5 + + mov r5,#2 @ counter + +.Lsquare_neon: + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ d0 = h0*r0 + h4*5*r1 + h3*5*r2 + h2*5*r3 + h1*5*r4 + @ d1 = h1*r0 + h0*r1 + h4*5*r2 + h3*5*r3 + h2*5*r4 + @ d2 = h2*r0 + h1*r1 + h0*r2 + h4*5*r3 + h3*5*r4 + @ d3 = h3*r0 + h2*r1 + h1*r2 + h0*r3 + h4*5*r4 + @ d4 = h4*r0 + h3*r1 + h2*r2 + h1*r3 + h0*r4 + + vmull.u32 q5,d0,d0[1] + vmull.u32 q6,d1,d0[1] + vmull.u32 q7,d3,d0[1] + vmull.u32 q8,d5,d0[1] + vmull.u32 q9,d7,d0[1] + + vmlal.u32 q5,d7,d2[1] + vmlal.u32 q6,d0,d1[1] + vmlal.u32 q7,d1,d1[1] + vmlal.u32 q8,d3,d1[1] + vmlal.u32 q9,d5,d1[1] + + vmlal.u32 q5,d5,d4[1] + vmlal.u32 q6,d7,d4[1] + vmlal.u32 q8,d1,d3[1] + vmlal.u32 q7,d0,d3[1] + vmlal.u32 q9,d3,d3[1] + + vmlal.u32 q5,d3,d6[1] + vmlal.u32 q8,d0,d5[1] + vmlal.u32 q6,d5,d6[1] + vmlal.u32 q7,d7,d6[1] + vmlal.u32 q9,d1,d5[1] + + vmlal.u32 q8,d7,d8[1] + vmlal.u32 q5,d1,d8[1] + vmlal.u32 q6,d3,d8[1] + vmlal.u32 q7,d5,d8[1] + vmlal.u32 q9,d0,d7[1] + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ lazy reduction as discussed in "NEON crypto" by D.J. Bernstein + @ and P. Schwabe + @ + @ H0>>+H1>>+H2>>+H3>>+H4 + @ H3>>+H4>>*5+H0>>+H1 + @ + @ Trivia. + @ + @ Result of multiplication of n-bit number by m-bit number is + @ n+m bits wide. However! Even though 2^n is a n+1-bit number, + @ m-bit number multiplied by 2^n is still n+m bits wide. + @ + @ Sum of two n-bit numbers is n+1 bits wide, sum of three - n+2, + @ and so is sum of four. Sum of 2^m n-m-bit numbers and n-bit + @ one is n+1 bits wide. + @ + @ >>+ denotes Hnext += Hn>>26, Hn &= 0x3ffffff. This means that + @ H0, H2, H3 are guaranteed to be 26 bits wide, while H1 and H4 + @ can be 27. However! In cases when their width exceeds 26 bits + @ they are limited by 2^26+2^6. This in turn means that *sum* + @ of the products with these values can still be viewed as sum + @ of 52-bit numbers as long as the amount of addends is not a + @ power of 2. For example, + @ + @ H4 = H4*R0 + H3*R1 + H2*R2 + H1*R3 + H0 * R4, + @ + @ which can't be larger than 5 * (2^26 + 2^6) * (2^26 + 2^6), or + @ 5 * (2^52 + 2*2^32 + 2^12), which in turn is smaller than + @ 8 * (2^52) or 2^55. However, the value is then multiplied by + @ by 5, so we should be looking at 5 * 5 * (2^52 + 2^33 + 2^12), + @ which is less than 32 * (2^52) or 2^57. And when processing + @ data we are looking at triple as many addends... + @ + @ In key setup procedure pre-reduced H0 is limited by 5*4+1 and + @ 5*H4 - by 5*5 52-bit addends, or 57 bits. But when hashing the + @ input H0 is limited by (5*4+1)*3 addends, or 58 bits, while + @ 5*H4 by 5*5*3, or 59[!] bits. How is this relevant? vmlal.u32 + @ instruction accepts 2x32-bit input and writes 2x64-bit result. + @ This means that result of reduction have to be compressed upon + @ loop wrap-around. This can be done in the process of reduction + @ to minimize amount of instructions [as well as amount of + @ 128-bit instructions, which benefits low-end processors], but + @ one has to watch for H2 (which is narrower than H0) and 5*H4 + @ not being wider than 58 bits, so that result of right shift + @ by 26 bits fits in 32 bits. This is also useful on x86, + @ because it allows to use paddd in place for paddq, which + @ benefits Atom, where paddq is ridiculously slow. + + vshr.u64 q15,q8,#26 + vmovn.i64 d16,q8 + vshr.u64 q4,q5,#26 + vmovn.i64 d10,q5 + vadd.i64 q9,q9,q15 @ h3 -> h4 + vbic.i32 d16,#0xfc000000 @ &=0x03ffffff + vadd.i64 q6,q6,q4 @ h0 -> h1 + vbic.i32 d10,#0xfc000000 + + vshrn.u64 d30,q9,#26 + vmovn.i64 d18,q9 + vshr.u64 q4,q6,#26 + vmovn.i64 d12,q6 + vadd.i64 q7,q7,q4 @ h1 -> h2 + vbic.i32 d18,#0xfc000000 + vbic.i32 d12,#0xfc000000 + + vadd.i32 d10,d10,d30 + vshl.u32 d30,d30,#2 + vshrn.u64 d8,q7,#26 + vmovn.i64 d14,q7 + vadd.i32 d10,d10,d30 @ h4 -> h0 + vadd.i32 d16,d16,d8 @ h2 -> h3 + vbic.i32 d14,#0xfc000000 + + vshr.u32 d30,d10,#26 + vbic.i32 d10,#0xfc000000 + vshr.u32 d8,d16,#26 + vbic.i32 d16,#0xfc000000 + vadd.i32 d12,d12,d30 @ h0 -> h1 + vadd.i32 d18,d18,d8 @ h3 -> h4 + + subs r5,r5,#1 + beq .Lsquare_break_neon + + add r6,r0,#(48+0*9*4) + add r7,r0,#(48+1*9*4) + + vtrn.32 d0,d10 @ r^2:r^1 + vtrn.32 d3,d14 + vtrn.32 d5,d16 + vtrn.32 d1,d12 + vtrn.32 d7,d18 + + vshl.u32 d4,d3,#2 @ *5 + vshl.u32 d6,d5,#2 + vshl.u32 d2,d1,#2 + vshl.u32 d8,d7,#2 + vadd.i32 d4,d4,d3 + vadd.i32 d2,d2,d1 + vadd.i32 d6,d6,d5 + vadd.i32 d8,d8,d7 + + vst4.32 {d0[0],d1[0],d2[0],d3[0]},[r6]! + vst4.32 {d0[1],d1[1],d2[1],d3[1]},[r7]! + vst4.32 {d4[0],d5[0],d6[0],d7[0]},[r6]! + vst4.32 {d4[1],d5[1],d6[1],d7[1]},[r7]! + vst1.32 {d8[0]},[r6,:32] + vst1.32 {d8[1]},[r7,:32] + + b .Lsquare_neon + +.align 4 +.Lsquare_break_neon: + add r6,r0,#(48+2*4*9) + add r7,r0,#(48+3*4*9) + + vmov d0,d10 @ r^4:r^3 + vshl.u32 d2,d12,#2 @ *5 + vmov d1,d12 + vshl.u32 d4,d14,#2 + vmov d3,d14 + vshl.u32 d6,d16,#2 + vmov d5,d16 + vshl.u32 d8,d18,#2 + vmov d7,d18 + vadd.i32 d2,d2,d12 + vadd.i32 d4,d4,d14 + vadd.i32 d6,d6,d16 + vadd.i32 d8,d8,d18 + + vst4.32 {d0[0],d1[0],d2[0],d3[0]},[r6]! + vst4.32 {d0[1],d1[1],d2[1],d3[1]},[r7]! + vst4.32 {d4[0],d5[0],d6[0],d7[0]},[r6]! + vst4.32 {d4[1],d5[1],d6[1],d7[1]},[r7]! + vst1.32 {d8[0]},[r6] + vst1.32 {d8[1]},[r7] + + bx lr @ bx lr +ENDPROC(poly1305_init_neon) + +.align 5 +ENTRY(poly1305_blocks_neon) + ldr ip,[r0,#36] @ is_base2_26 + ands r2,r2,#-16 + beq .Lno_data_neon + + cmp r2,#64 + bhs .Lenter_neon + tst ip,ip @ is_base2_26? + beq .Lpoly1305_blocks_arm + +.Lenter_neon: + stmdb sp!,{r4-r7} + vstmdb sp!,{d8-d15} @ ABI specification says so + + tst ip,ip @ is_base2_26? + bne .Lbase2_26_neon + + stmdb sp!,{r1-r3,lr} + bl .Lpoly1305_init_neon + + ldr r4,[r0,#0] @ load hash value base 2^32 + ldr r5,[r0,#4] + ldr r6,[r0,#8] + ldr r7,[r0,#12] + ldr ip,[r0,#16] + + and r2,r4,#0x03ffffff @ base 2^32 -> base 2^26 + mov r3,r4,lsr#26 + veor d10,d10,d10 + mov r4,r5,lsr#20 + orr r3,r3,r5,lsl#6 + veor d12,d12,d12 + mov r5,r6,lsr#14 + orr r4,r4,r6,lsl#12 + veor d14,d14,d14 + mov r6,r7,lsr#8 + orr r5,r5,r7,lsl#18 + veor d16,d16,d16 + and r3,r3,#0x03ffffff + orr r6,r6,ip,lsl#24 + veor d18,d18,d18 + and r4,r4,#0x03ffffff + mov r1,#1 + and r5,r5,#0x03ffffff + str r1,[r0,#36] @ is_base2_26 + + vmov.32 d10[0],r2 + vmov.32 d12[0],r3 + vmov.32 d14[0],r4 + vmov.32 d16[0],r5 + vmov.32 d18[0],r6 + adr r5,.Lzeros + + ldmia sp!,{r1-r3,lr} + b .Lbase2_32_neon + +.align 4 +.Lbase2_26_neon: + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ load hash value + + veor d10,d10,d10 + veor d12,d12,d12 + veor d14,d14,d14 + veor d16,d16,d16 + veor d18,d18,d18 + vld4.32 {d10[0],d12[0],d14[0],d16[0]},[r0]! + adr r5,.Lzeros + vld1.32 {d18[0]},[r0] + sub r0,r0,#16 @ rewind + +.Lbase2_32_neon: + add r4,r1,#32 + mov r3,r3,lsl#24 + tst r2,#31 + beq .Leven + + vld4.32 {d20[0],d22[0],d24[0],d26[0]},[r1]! + vmov.32 d28[0],r3 + sub r2,r2,#16 + add r4,r1,#32 + +#ifdef __ARMEB__ + vrev32.8 q10,q10 + vrev32.8 q13,q13 + vrev32.8 q11,q11 + vrev32.8 q12,q12 +#endif + vsri.u32 d28,d26,#8 @ base 2^32 -> base 2^26 + vshl.u32 d26,d26,#18 + + vsri.u32 d26,d24,#14 + vshl.u32 d24,d24,#12 + vadd.i32 d29,d28,d18 @ add hash value and move to #hi + + vbic.i32 d26,#0xfc000000 + vsri.u32 d24,d22,#20 + vshl.u32 d22,d22,#6 + + vbic.i32 d24,#0xfc000000 + vsri.u32 d22,d20,#26 + vadd.i32 d27,d26,d16 + + vbic.i32 d20,#0xfc000000 + vbic.i32 d22,#0xfc000000 + vadd.i32 d25,d24,d14 + + vadd.i32 d21,d20,d10 + vadd.i32 d23,d22,d12 + + mov r7,r5 + add r6,r0,#48 + + cmp r2,r2 + b .Long_tail + +.align 4 +.Leven: + subs r2,r2,#64 + it lo + movlo r4,r5 + + vmov.i32 q14,#1<<24 @ padbit, yes, always + vld4.32 {d20,d22,d24,d26},[r1] @ inp[0:1] + add r1,r1,#64 + vld4.32 {d21,d23,d25,d27},[r4] @ inp[2:3] (or 0) + add r4,r4,#64 + itt hi + addhi r7,r0,#(48+1*9*4) + addhi r6,r0,#(48+3*9*4) + +#ifdef __ARMEB__ + vrev32.8 q10,q10 + vrev32.8 q13,q13 + vrev32.8 q11,q11 + vrev32.8 q12,q12 +#endif + vsri.u32 q14,q13,#8 @ base 2^32 -> base 2^26 + vshl.u32 q13,q13,#18 + + vsri.u32 q13,q12,#14 + vshl.u32 q12,q12,#12 + + vbic.i32 q13,#0xfc000000 + vsri.u32 q12,q11,#20 + vshl.u32 q11,q11,#6 + + vbic.i32 q12,#0xfc000000 + vsri.u32 q11,q10,#26 + + vbic.i32 q10,#0xfc000000 + vbic.i32 q11,#0xfc000000 + + bls .Lskip_loop + + vld4.32 {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^2 + vld4.32 {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^4 + vld4.32 {d4[1],d5[1],d6[1],d7[1]},[r7]! + vld4.32 {d4[0],d5[0],d6[0],d7[0]},[r6]! + b .Loop_neon + +.align 5 +.Loop_neon: + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2 + @ ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^3+inp[7]*r + @ ___________________/ + @ ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2+inp[8])*r^2 + @ ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^4+inp[7]*r^2+inp[9])*r + @ ___________________/ ____________________/ + @ + @ Note that we start with inp[2:3]*r^2. This is because it + @ doesn't depend on reduction in previous iteration. + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ d4 = h4*r0 + h3*r1 + h2*r2 + h1*r3 + h0*r4 + @ d3 = h3*r0 + h2*r1 + h1*r2 + h0*r3 + h4*5*r4 + @ d2 = h2*r0 + h1*r1 + h0*r2 + h4*5*r3 + h3*5*r4 + @ d1 = h1*r0 + h0*r1 + h4*5*r2 + h3*5*r3 + h2*5*r4 + @ d0 = h0*r0 + h4*5*r1 + h3*5*r2 + h2*5*r3 + h1*5*r4 + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ inp[2:3]*r^2 + + vadd.i32 d24,d24,d14 @ accumulate inp[0:1] + vmull.u32 q7,d25,d0[1] + vadd.i32 d20,d20,d10 + vmull.u32 q5,d21,d0[1] + vadd.i32 d26,d26,d16 + vmull.u32 q8,d27,d0[1] + vmlal.u32 q7,d23,d1[1] + vadd.i32 d22,d22,d12 + vmull.u32 q6,d23,d0[1] + + vadd.i32 d28,d28,d18 + vmull.u32 q9,d29,d0[1] + subs r2,r2,#64 + vmlal.u32 q5,d29,d2[1] + it lo + movlo r4,r5 + vmlal.u32 q8,d25,d1[1] + vld1.32 d8[1],[r7,:32] + vmlal.u32 q6,d21,d1[1] + vmlal.u32 q9,d27,d1[1] + + vmlal.u32 q5,d27,d4[1] + vmlal.u32 q8,d23,d3[1] + vmlal.u32 q9,d25,d3[1] + vmlal.u32 q6,d29,d4[1] + vmlal.u32 q7,d21,d3[1] + + vmlal.u32 q8,d21,d5[1] + vmlal.u32 q5,d25,d6[1] + vmlal.u32 q9,d23,d5[1] + vmlal.u32 q6,d27,d6[1] + vmlal.u32 q7,d29,d6[1] + + vmlal.u32 q8,d29,d8[1] + vmlal.u32 q5,d23,d8[1] + vmlal.u32 q9,d21,d7[1] + vmlal.u32 q6,d25,d8[1] + vmlal.u32 q7,d27,d8[1] + + vld4.32 {d21,d23,d25,d27},[r4] @ inp[2:3] (or 0) + add r4,r4,#64 + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ (hash+inp[0:1])*r^4 and accumulate + + vmlal.u32 q8,d26,d0[0] + vmlal.u32 q5,d20,d0[0] + vmlal.u32 q9,d28,d0[0] + vmlal.u32 q6,d22,d0[0] + vmlal.u32 q7,d24,d0[0] + vld1.32 d8[0],[r6,:32] + + vmlal.u32 q8,d24,d1[0] + vmlal.u32 q5,d28,d2[0] + vmlal.u32 q9,d26,d1[0] + vmlal.u32 q6,d20,d1[0] + vmlal.u32 q7,d22,d1[0] + + vmlal.u32 q8,d22,d3[0] + vmlal.u32 q5,d26,d4[0] + vmlal.u32 q9,d24,d3[0] + vmlal.u32 q6,d28,d4[0] + vmlal.u32 q7,d20,d3[0] + + vmlal.u32 q8,d20,d5[0] + vmlal.u32 q5,d24,d6[0] + vmlal.u32 q9,d22,d5[0] + vmlal.u32 q6,d26,d6[0] + vmlal.u32 q8,d28,d8[0] + + vmlal.u32 q7,d28,d6[0] + vmlal.u32 q5,d22,d8[0] + vmlal.u32 q9,d20,d7[0] + vmov.i32 q14,#1<<24 @ padbit, yes, always + vmlal.u32 q6,d24,d8[0] + vmlal.u32 q7,d26,d8[0] + + vld4.32 {d20,d22,d24,d26},[r1] @ inp[0:1] + add r1,r1,#64 +#ifdef __ARMEB__ + vrev32.8 q10,q10 + vrev32.8 q11,q11 + vrev32.8 q12,q12 + vrev32.8 q13,q13 +#endif + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ lazy reduction interleaved with base 2^32 -> base 2^26 of + @ inp[0:3] previously loaded to q10-q13 and smashed to q10-q14. + + vshr.u64 q15,q8,#26 + vmovn.i64 d16,q8 + vshr.u64 q4,q5,#26 + vmovn.i64 d10,q5 + vadd.i64 q9,q9,q15 @ h3 -> h4 + vbic.i32 d16,#0xfc000000 + vsri.u32 q14,q13,#8 @ base 2^32 -> base 2^26 + vadd.i64 q6,q6,q4 @ h0 -> h1 + vshl.u32 q13,q13,#18 + vbic.i32 d10,#0xfc000000 + + vshrn.u64 d30,q9,#26 + vmovn.i64 d18,q9 + vshr.u64 q4,q6,#26 + vmovn.i64 d12,q6 + vadd.i64 q7,q7,q4 @ h1 -> h2 + vsri.u32 q13,q12,#14 + vbic.i32 d18,#0xfc000000 + vshl.u32 q12,q12,#12 + vbic.i32 d12,#0xfc000000 + + vadd.i32 d10,d10,d30 + vshl.u32 d30,d30,#2 + vbic.i32 q13,#0xfc000000 + vshrn.u64 d8,q7,#26 + vmovn.i64 d14,q7 + vaddl.u32 q5,d10,d30 @ h4 -> h0 [widen for a sec] + vsri.u32 q12,q11,#20 + vadd.i32 d16,d16,d8 @ h2 -> h3 + vshl.u32 q11,q11,#6 + vbic.i32 d14,#0xfc000000 + vbic.i32 q12,#0xfc000000 + + vshrn.u64 d30,q5,#26 @ re-narrow + vmovn.i64 d10,q5 + vsri.u32 q11,q10,#26 + vbic.i32 q10,#0xfc000000 + vshr.u32 d8,d16,#26 + vbic.i32 d16,#0xfc000000 + vbic.i32 d10,#0xfc000000 + vadd.i32 d12,d12,d30 @ h0 -> h1 + vadd.i32 d18,d18,d8 @ h3 -> h4 + vbic.i32 q11,#0xfc000000 + + bhi .Loop_neon + +.Lskip_loop: + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ multiply (inp[0:1]+hash) or inp[2:3] by r^2:r^1 + + add r7,r0,#(48+0*9*4) + add r6,r0,#(48+1*9*4) + adds r2,r2,#32 + it ne + movne r2,#0 + bne .Long_tail + + vadd.i32 d25,d24,d14 @ add hash value and move to #hi + vadd.i32 d21,d20,d10 + vadd.i32 d27,d26,d16 + vadd.i32 d23,d22,d12 + vadd.i32 d29,d28,d18 + +.Long_tail: + vld4.32 {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^1 + vld4.32 {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^2 + + vadd.i32 d24,d24,d14 @ can be redundant + vmull.u32 q7,d25,d0 + vadd.i32 d20,d20,d10 + vmull.u32 q5,d21,d0 + vadd.i32 d26,d26,d16 + vmull.u32 q8,d27,d0 + vadd.i32 d22,d22,d12 + vmull.u32 q6,d23,d0 + vadd.i32 d28,d28,d18 + vmull.u32 q9,d29,d0 + + vmlal.u32 q5,d29,d2 + vld4.32 {d4[1],d5[1],d6[1],d7[1]},[r7]! + vmlal.u32 q8,d25,d1 + vld4.32 {d4[0],d5[0],d6[0],d7[0]},[r6]! + vmlal.u32 q6,d21,d1 + vmlal.u32 q9,d27,d1 + vmlal.u32 q7,d23,d1 + + vmlal.u32 q8,d23,d3 + vld1.32 d8[1],[r7,:32] + vmlal.u32 q5,d27,d4 + vld1.32 d8[0],[r6,:32] + vmlal.u32 q9,d25,d3 + vmlal.u32 q6,d29,d4 + vmlal.u32 q7,d21,d3 + + vmlal.u32 q8,d21,d5 + it ne + addne r7,r0,#(48+2*9*4) + vmlal.u32 q5,d25,d6 + it ne + addne r6,r0,#(48+3*9*4) + vmlal.u32 q9,d23,d5 + vmlal.u32 q6,d27,d6 + vmlal.u32 q7,d29,d6 + + vmlal.u32 q8,d29,d8 + vorn q0,q0,q0 @ all-ones, can be redundant + vmlal.u32 q5,d23,d8 + vshr.u64 q0,q0,#38 + vmlal.u32 q9,d21,d7 + vmlal.u32 q6,d25,d8 + vmlal.u32 q7,d27,d8 + + beq .Lshort_tail + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ (hash+inp[0:1])*r^4:r^3 and accumulate + + vld4.32 {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^3 + vld4.32 {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^4 + + vmlal.u32 q7,d24,d0 + vmlal.u32 q5,d20,d0 + vmlal.u32 q8,d26,d0 + vmlal.u32 q6,d22,d0 + vmlal.u32 q9,d28,d0 + + vmlal.u32 q5,d28,d2 + vld4.32 {d4[1],d5[1],d6[1],d7[1]},[r7]! + vmlal.u32 q8,d24,d1 + vld4.32 {d4[0],d5[0],d6[0],d7[0]},[r6]! + vmlal.u32 q6,d20,d1 + vmlal.u32 q9,d26,d1 + vmlal.u32 q7,d22,d1 + + vmlal.u32 q8,d22,d3 + vld1.32 d8[1],[r7,:32] + vmlal.u32 q5,d26,d4 + vld1.32 d8[0],[r6,:32] + vmlal.u32 q9,d24,d3 + vmlal.u32 q6,d28,d4 + vmlal.u32 q7,d20,d3 + + vmlal.u32 q8,d20,d5 + vmlal.u32 q5,d24,d6 + vmlal.u32 q9,d22,d5 + vmlal.u32 q6,d26,d6 + vmlal.u32 q7,d28,d6 + + vmlal.u32 q8,d28,d8 + vorn q0,q0,q0 @ all-ones + vmlal.u32 q5,d22,d8 + vshr.u64 q0,q0,#38 + vmlal.u32 q9,d20,d7 + vmlal.u32 q6,d24,d8 + vmlal.u32 q7,d26,d8 + +.Lshort_tail: + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ horizontal addition + + vadd.i64 d16,d16,d17 + vadd.i64 d10,d10,d11 + vadd.i64 d18,d18,d19 + vadd.i64 d12,d12,d13 + vadd.i64 d14,d14,d15 + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ lazy reduction, but without narrowing + + vshr.u64 q15,q8,#26 + vand.i64 q8,q8,q0 + vshr.u64 q4,q5,#26 + vand.i64 q5,q5,q0 + vadd.i64 q9,q9,q15 @ h3 -> h4 + vadd.i64 q6,q6,q4 @ h0 -> h1 + + vshr.u64 q15,q9,#26 + vand.i64 q9,q9,q0 + vshr.u64 q4,q6,#26 + vand.i64 q6,q6,q0 + vadd.i64 q7,q7,q4 @ h1 -> h2 + + vadd.i64 q5,q5,q15 + vshl.u64 q15,q15,#2 + vshr.u64 q4,q7,#26 + vand.i64 q7,q7,q0 + vadd.i64 q5,q5,q15 @ h4 -> h0 + vadd.i64 q8,q8,q4 @ h2 -> h3 + + vshr.u64 q15,q5,#26 + vand.i64 q5,q5,q0 + vshr.u64 q4,q8,#26 + vand.i64 q8,q8,q0 + vadd.i64 q6,q6,q15 @ h0 -> h1 + vadd.i64 q9,q9,q4 @ h3 -> h4 + + cmp r2,#0 + bne .Leven + + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @ store hash value + + vst4.32 {d10[0],d12[0],d14[0],d16[0]},[r0]! + vst1.32 {d18[0]},[r0] + + vldmia sp!,{d8-d15} @ epilogue + ldmia sp!,{r4-r7} +.Lno_data_neon: + bx lr @ bx lr +ENDPROC(poly1305_blocks_neon) + +.align 5 +ENTRY(poly1305_emit_neon) + ldr ip,[r0,#36] @ is_base2_26 + + stmdb sp!,{r4-r11} + + tst ip,ip + beq .Lpoly1305_emit_enter + + ldmia r0,{r3-r7} + eor r8,r8,r8 + + adds r3,r3,r4,lsl#26 @ base 2^26 -> base 2^32 + mov r4,r4,lsr#6 + adcs r4,r4,r5,lsl#20 + mov r5,r5,lsr#12 + adcs r5,r5,r6,lsl#14 + mov r6,r6,lsr#18 + adcs r6,r6,r7,lsl#8 + adc r7,r8,r7,lsr#24 @ can be partially reduced ... + + and r8,r7,#-4 @ ... so reduce + and r7,r6,#3 + add r8,r8,r8,lsr#2 @ *= 5 + adds r3,r3,r8 + adcs r4,r4,#0 + adcs r5,r5,#0 + adcs r6,r6,#0 + adc r7,r7,#0 + + adds r8,r3,#5 @ compare to modulus + adcs r9,r4,#0 + adcs r10,r5,#0 + adcs r11,r6,#0 + adc r7,r7,#0 + tst r7,#4 @ did it carry/borrow? + + it ne + movne r3,r8 + ldr r8,[r2,#0] + it ne + movne r4,r9 + ldr r9,[r2,#4] + it ne + movne r5,r10 + ldr r10,[r2,#8] + it ne + movne r6,r11 + ldr r11,[r2,#12] + + adds r3,r3,r8 @ accumulate nonce + adcs r4,r4,r9 + adcs r5,r5,r10 + adc r6,r6,r11 + +#ifdef __ARMEB__ + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 +#endif + str r3,[r1,#0] @ store the result + str r4,[r1,#4] + str r5,[r1,#8] + str r6,[r1,#12] + + ldmia sp!,{r4-r11} + bx lr @ bx lr +ENDPROC(poly1305_emit_neon) + +.align 5 +.Lzeros: +.long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +#endiflib/zinc/poly1305/poly1305-arm64.S new file +822
-1@@ -0,0 +1,822 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 + * + * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. + * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + * + * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS. + */ + +#include <linux/linkage.h> +.text + +.align 5 +ENTRY(poly1305_init_arm) + cmp x1,xzr + stp xzr,xzr,[x0] // zero hash value + stp xzr,xzr,[x0,#16] // [along with is_base2_26] + + csel x0,xzr,x0,eq + b.eq .Lno_key + + ldp x7,x8,[x1] // load key + mov x9,#0xfffffffc0fffffff + movk x9,#0x0fff,lsl#48 +#ifdef __ARMEB__ + rev x7,x7 // flip bytes + rev x8,x8 +#endif + and x7,x7,x9 // &=0ffffffc0fffffff + and x9,x9,#-4 + and x8,x8,x9 // &=0ffffffc0ffffffc + stp x7,x8,[x0,#32] // save key value + +.Lno_key: + ret +ENDPROC(poly1305_init_arm) + +.align 5 +ENTRY(poly1305_blocks_arm) + ands x2,x2,#-16 + b.eq .Lno_data + + ldp x4,x5,[x0] // load hash value + ldp x7,x8,[x0,#32] // load key value + ldr x6,[x0,#16] + add x9,x8,x8,lsr#2 // s1 = r1 + (r1 >> 2) + b .Loop + +.align 5 +.Loop: + ldp x10,x11,[x1],#16 // load input + sub x2,x2,#16 +#ifdef __ARMEB__ + rev x10,x10 + rev x11,x11 +#endif + adds x4,x4,x10 // accumulate input + adcs x5,x5,x11 + + mul x12,x4,x7 // h0*r0 + adc x6,x6,x3 + umulh x13,x4,x7 + + mul x10,x5,x9 // h1*5*r1 + umulh x11,x5,x9 + + adds x12,x12,x10 + mul x10,x4,x8 // h0*r1 + adc x13,x13,x11 + umulh x14,x4,x8 + + adds x13,x13,x10 + mul x10,x5,x7 // h1*r0 + adc x14,x14,xzr + umulh x11,x5,x7 + + adds x13,x13,x10 + mul x10,x6,x9 // h2*5*r1 + adc x14,x14,x11 + mul x11,x6,x7 // h2*r0 + + adds x13,x13,x10 + adc x14,x14,x11 + + and x10,x14,#-4 // final reduction + and x6,x14,#3 + add x10,x10,x14,lsr#2 + adds x4,x12,x10 + adcs x5,x13,xzr + adc x6,x6,xzr + + cbnz x2,.Loop + + stp x4,x5,[x0] // store hash value + str x6,[x0,#16] + +.Lno_data: + ret +ENDPROC(poly1305_blocks_arm) + +.align 5 +ENTRY(poly1305_emit_arm) + ldp x4,x5,[x0] // load hash base 2^64 + ldr x6,[x0,#16] + ldp x10,x11,[x2] // load nonce + + adds x12,x4,#5 // compare to modulus + adcs x13,x5,xzr + adc x14,x6,xzr + + tst x14,#-4 // see if it's carried/borrowed + + csel x4,x4,x12,eq + csel x5,x5,x13,eq + +#ifdef __ARMEB__ + ror x10,x10,#32 // flip nonce words + ror x11,x11,#32 +#endif + adds x4,x4,x10 // accumulate nonce + adc x5,x5,x11 +#ifdef __ARMEB__ + rev x4,x4 // flip output bytes + rev x5,x5 +#endif + stp x4,x5,[x1] // write result + + ret +ENDPROC(poly1305_emit_arm) + +.align 5 +__poly1305_mult: + mul x12,x4,x7 // h0*r0 + umulh x13,x4,x7 + + mul x10,x5,x9 // h1*5*r1 + umulh x11,x5,x9 + + adds x12,x12,x10 + mul x10,x4,x8 // h0*r1 + adc x13,x13,x11 + umulh x14,x4,x8 + + adds x13,x13,x10 + mul x10,x5,x7 // h1*r0 + adc x14,x14,xzr + umulh x11,x5,x7 + + adds x13,x13,x10 + mul x10,x6,x9 // h2*5*r1 + adc x14,x14,x11 + mul x11,x6,x7 // h2*r0 + + adds x13,x13,x10 + adc x14,x14,x11 + + and x10,x14,#-4 // final reduction + and x6,x14,#3 + add x10,x10,x14,lsr#2 + adds x4,x12,x10 + adcs x5,x13,xzr + adc x6,x6,xzr + + ret + +__poly1305_splat: + and x12,x4,#0x03ffffff // base 2^64 -> base 2^26 + ubfx x13,x4,#26,#26 + extr x14,x5,x4,#52 + and x14,x14,#0x03ffffff + ubfx x15,x5,#14,#26 + extr x16,x6,x5,#40 + + str w12,[x0,#16*0] // r0 + add w12,w13,w13,lsl#2 // r1*5 + str w13,[x0,#16*1] // r1 + add w13,w14,w14,lsl#2 // r2*5 + str w12,[x0,#16*2] // s1 + str w14,[x0,#16*3] // r2 + add w14,w15,w15,lsl#2 // r3*5 + str w13,[x0,#16*4] // s2 + str w15,[x0,#16*5] // r3 + add w15,w16,w16,lsl#2 // r4*5 + str w14,[x0,#16*6] // s3 + str w16,[x0,#16*7] // r4 + str w15,[x0,#16*8] // s4 + + ret + +.align 5 +ENTRY(poly1305_blocks_neon) + ldr x17,[x0,#24] + cmp x2,#128 + b.hs .Lblocks_neon + cbz x17,poly1305_blocks_arm + +.Lblocks_neon: + stp x29,x30,[sp,#-80]! + add x29,sp,#0 + + ands x2,x2,#-16 + b.eq .Lno_data_neon + + cbz x17,.Lbase2_64_neon + + ldp w10,w11,[x0] // load hash value base 2^26 + ldp w12,w13,[x0,#8] + ldr w14,[x0,#16] + + tst x2,#31 + b.eq .Leven_neon + + ldp x7,x8,[x0,#32] // load key value + + add x4,x10,x11,lsl#26 // base 2^26 -> base 2^64 + lsr x5,x12,#12 + adds x4,x4,x12,lsl#52 + add x5,x5,x13,lsl#14 + adc x5,x5,xzr + lsr x6,x14,#24 + adds x5,x5,x14,lsl#40 + adc x14,x6,xzr // can be partially reduced... + + ldp x12,x13,[x1],#16 // load input + sub x2,x2,#16 + add x9,x8,x8,lsr#2 // s1 = r1 + (r1 >> 2) + + and x10,x14,#-4 // ... so reduce + and x6,x14,#3 + add x10,x10,x14,lsr#2 + adds x4,x4,x10 + adcs x5,x5,xzr + adc x6,x6,xzr + +#ifdef __ARMEB__ + rev x12,x12 + rev x13,x13 +#endif + adds x4,x4,x12 // accumulate input + adcs x5,x5,x13 + adc x6,x6,x3 + + bl __poly1305_mult + ldr x30,[sp,#8] + + cbz x3,.Lstore_base2_64_neon + + and x10,x4,#0x03ffffff // base 2^64 -> base 2^26 + ubfx x11,x4,#26,#26 + extr x12,x5,x4,#52 + and x12,x12,#0x03ffffff + ubfx x13,x5,#14,#26 + extr x14,x6,x5,#40 + + cbnz x2,.Leven_neon + + stp w10,w11,[x0] // store hash value base 2^26 + stp w12,w13,[x0,#8] + str w14,[x0,#16] + b .Lno_data_neon + +.align 4 +.Lstore_base2_64_neon: + stp x4,x5,[x0] // store hash value base 2^64 + stp x6,xzr,[x0,#16] // note that is_base2_26 is zeroed + b .Lno_data_neon + +.align 4 +.Lbase2_64_neon: + ldp x7,x8,[x0,#32] // load key value + + ldp x4,x5,[x0] // load hash value base 2^64 + ldr x6,[x0,#16] + + tst x2,#31 + b.eq .Linit_neon + + ldp x12,x13,[x1],#16 // load input + sub x2,x2,#16 + add x9,x8,x8,lsr#2 // s1 = r1 + (r1 >> 2) +#ifdef __ARMEB__ + rev x12,x12 + rev x13,x13 +#endif + adds x4,x4,x12 // accumulate input + adcs x5,x5,x13 + adc x6,x6,x3 + + bl __poly1305_mult + +.Linit_neon: + and x10,x4,#0x03ffffff // base 2^64 -> base 2^26 + ubfx x11,x4,#26,#26 + extr x12,x5,x4,#52 + and x12,x12,#0x03ffffff + ubfx x13,x5,#14,#26 + extr x14,x6,x5,#40 + + stp d8,d9,[sp,#16] // meet ABI requirements + stp d10,d11,[sp,#32] + stp d12,d13,[sp,#48] + stp d14,d15,[sp,#64] + + fmov d24,x10 + fmov d25,x11 + fmov d26,x12 + fmov d27,x13 + fmov d28,x14 + + ////////////////////////////////// initialize r^n table + mov x4,x7 // r^1 + add x9,x8,x8,lsr#2 // s1 = r1 + (r1 >> 2) + mov x5,x8 + mov x6,xzr + add x0,x0,#48+12 + bl __poly1305_splat + + bl __poly1305_mult // r^2 + sub x0,x0,#4 + bl __poly1305_splat + + bl __poly1305_mult // r^3 + sub x0,x0,#4 + bl __poly1305_splat + + bl __poly1305_mult // r^4 + sub x0,x0,#4 + bl __poly1305_splat + ldr x30,[sp,#8] + + add x16,x1,#32 + adr x17,.Lzeros + subs x2,x2,#64 + csel x16,x17,x16,lo + + mov x4,#1 + str x4,[x0,#-24] // set is_base2_26 + sub x0,x0,#48 // restore original x0 + b .Ldo_neon + +.align 4 +.Leven_neon: + add x16,x1,#32 + adr x17,.Lzeros + subs x2,x2,#64 + csel x16,x17,x16,lo + + stp d8,d9,[sp,#16] // meet ABI requirements + stp d10,d11,[sp,#32] + stp d12,d13,[sp,#48] + stp d14,d15,[sp,#64] + + fmov d24,x10 + fmov d25,x11 + fmov d26,x12 + fmov d27,x13 + fmov d28,x14 + +.Ldo_neon: + ldp x8,x12,[x16],#16 // inp[2:3] (or zero) + ldp x9,x13,[x16],#48 + + lsl x3,x3,#24 + add x15,x0,#48 + +#ifdef __ARMEB__ + rev x8,x8 + rev x12,x12 + rev x9,x9 + rev x13,x13 +#endif + and x4,x8,#0x03ffffff // base 2^64 -> base 2^26 + and x5,x9,#0x03ffffff + ubfx x6,x8,#26,#26 + ubfx x7,x9,#26,#26 + add x4,x4,x5,lsl#32 // bfi x4,x5,#32,#32 + extr x8,x12,x8,#52 + extr x9,x13,x9,#52 + add x6,x6,x7,lsl#32 // bfi x6,x7,#32,#32 + fmov d14,x4 + and x8,x8,#0x03ffffff + and x9,x9,#0x03ffffff + ubfx x10,x12,#14,#26 + ubfx x11,x13,#14,#26 + add x12,x3,x12,lsr#40 + add x13,x3,x13,lsr#40 + add x8,x8,x9,lsl#32 // bfi x8,x9,#32,#32 + fmov d15,x6 + add x10,x10,x11,lsl#32 // bfi x10,x11,#32,#32 + add x12,x12,x13,lsl#32 // bfi x12,x13,#32,#32 + fmov d16,x8 + fmov d17,x10 + fmov d18,x12 + + ldp x8,x12,[x1],#16 // inp[0:1] + ldp x9,x13,[x1],#48 + + ld1 {v0.4s,v1.4s,v2.4s,v3.4s},[x15],#64 + ld1 {v4.4s,v5.4s,v6.4s,v7.4s},[x15],#64 + ld1 {v8.4s},[x15] + +#ifdef __ARMEB__ + rev x8,x8 + rev x12,x12 + rev x9,x9 + rev x13,x13 +#endif + and x4,x8,#0x03ffffff // base 2^64 -> base 2^26 + and x5,x9,#0x03ffffff + ubfx x6,x8,#26,#26 + ubfx x7,x9,#26,#26 + add x4,x4,x5,lsl#32 // bfi x4,x5,#32,#32 + extr x8,x12,x8,#52 + extr x9,x13,x9,#52 + add x6,x6,x7,lsl#32 // bfi x6,x7,#32,#32 + fmov d9,x4 + and x8,x8,#0x03ffffff + and x9,x9,#0x03ffffff + ubfx x10,x12,#14,#26 + ubfx x11,x13,#14,#26 + add x12,x3,x12,lsr#40 + add x13,x3,x13,lsr#40 + add x8,x8,x9,lsl#32 // bfi x8,x9,#32,#32 + fmov d10,x6 + add x10,x10,x11,lsl#32 // bfi x10,x11,#32,#32 + add x12,x12,x13,lsl#32 // bfi x12,x13,#32,#32 + movi v31.2d,#-1 + fmov d11,x8 + fmov d12,x10 + fmov d13,x12 + ushr v31.2d,v31.2d,#38 + + b.ls .Lskip_loop + +.align 4 +.Loop_neon: + //////////////////////////////////////////////////////////////// + // ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2 + // ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^3+inp[7]*r + // ___________________/ + // ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2+inp[8])*r^2 + // ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^4+inp[7]*r^2+inp[9])*r + // ___________________/ ____________________/ + // + // Note that we start with inp[2:3]*r^2. This is because it + // doesn't depend on reduction in previous iteration. + //////////////////////////////////////////////////////////////// + // d4 = h0*r4 + h1*r3 + h2*r2 + h3*r1 + h4*r0 + // d3 = h0*r3 + h1*r2 + h2*r1 + h3*r0 + h4*5*r4 + // d2 = h0*r2 + h1*r1 + h2*r0 + h3*5*r4 + h4*5*r3 + // d1 = h0*r1 + h1*r0 + h2*5*r4 + h3*5*r3 + h4*5*r2 + // d0 = h0*r0 + h1*5*r4 + h2*5*r3 + h3*5*r2 + h4*5*r1 + + subs x2,x2,#64 + umull v23.2d,v14.2s,v7.s[2] + csel x16,x17,x16,lo + umull v22.2d,v14.2s,v5.s[2] + umull v21.2d,v14.2s,v3.s[2] + ldp x8,x12,[x16],#16 // inp[2:3] (or zero) + umull v20.2d,v14.2s,v1.s[2] + ldp x9,x13,[x16],#48 + umull v19.2d,v14.2s,v0.s[2] +#ifdef __ARMEB__ + rev x8,x8 + rev x12,x12 + rev x9,x9 + rev x13,x13 +#endif + + umlal v23.2d,v15.2s,v5.s[2] + and x4,x8,#0x03ffffff // base 2^64 -> base 2^26 + umlal v22.2d,v15.2s,v3.s[2] + and x5,x9,#0x03ffffff + umlal v21.2d,v15.2s,v1.s[2] + ubfx x6,x8,#26,#26 + umlal v20.2d,v15.2s,v0.s[2] + ubfx x7,x9,#26,#26 + umlal v19.2d,v15.2s,v8.s[2] + add x4,x4,x5,lsl#32 // bfi x4,x5,#32,#32 + + umlal v23.2d,v16.2s,v3.s[2] + extr x8,x12,x8,#52 + umlal v22.2d,v16.2s,v1.s[2] + extr x9,x13,x9,#52 + umlal v21.2d,v16.2s,v0.s[2] + add x6,x6,x7,lsl#32 // bfi x6,x7,#32,#32 + umlal v20.2d,v16.2s,v8.s[2] + fmov d14,x4 + umlal v19.2d,v16.2s,v6.s[2] + and x8,x8,#0x03ffffff + + umlal v23.2d,v17.2s,v1.s[2] + and x9,x9,#0x03ffffff + umlal v22.2d,v17.2s,v0.s[2] + ubfx x10,x12,#14,#26 + umlal v21.2d,v17.2s,v8.s[2] + ubfx x11,x13,#14,#26 + umlal v20.2d,v17.2s,v6.s[2] + add x8,x8,x9,lsl#32 // bfi x8,x9,#32,#32 + umlal v19.2d,v17.2s,v4.s[2] + fmov d15,x6 + + add v11.2s,v11.2s,v26.2s + add x12,x3,x12,lsr#40 + umlal v23.2d,v18.2s,v0.s[2] + add x13,x3,x13,lsr#40 + umlal v22.2d,v18.2s,v8.s[2] + add x10,x10,x11,lsl#32 // bfi x10,x11,#32,#32 + umlal v21.2d,v18.2s,v6.s[2] + add x12,x12,x13,lsl#32 // bfi x12,x13,#32,#32 + umlal v20.2d,v18.2s,v4.s[2] + fmov d16,x8 + umlal v19.2d,v18.2s,v2.s[2] + fmov d17,x10 + + //////////////////////////////////////////////////////////////// + // (hash+inp[0:1])*r^4 and accumulate + + add v9.2s,v9.2s,v24.2s + fmov d18,x12 + umlal v22.2d,v11.2s,v1.s[0] + ldp x8,x12,[x1],#16 // inp[0:1] + umlal v19.2d,v11.2s,v6.s[0] + ldp x9,x13,[x1],#48 + umlal v23.2d,v11.2s,v3.s[0] + umlal v20.2d,v11.2s,v8.s[0] + umlal v21.2d,v11.2s,v0.s[0] +#ifdef __ARMEB__ + rev x8,x8 + rev x12,x12 + rev x9,x9 + rev x13,x13 +#endif + + add v10.2s,v10.2s,v25.2s + umlal v22.2d,v9.2s,v5.s[0] + umlal v23.2d,v9.2s,v7.s[0] + and x4,x8,#0x03ffffff // base 2^64 -> base 2^26 + umlal v21.2d,v9.2s,v3.s[0] + and x5,x9,#0x03ffffff + umlal v19.2d,v9.2s,v0.s[0] + ubfx x6,x8,#26,#26 + umlal v20.2d,v9.2s,v1.s[0] + ubfx x7,x9,#26,#26 + + add v12.2s,v12.2s,v27.2s + add x4,x4,x5,lsl#32 // bfi x4,x5,#32,#32 + umlal v22.2d,v10.2s,v3.s[0] + extr x8,x12,x8,#52 + umlal v23.2d,v10.2s,v5.s[0] + extr x9,x13,x9,#52 + umlal v19.2d,v10.2s,v8.s[0] + add x6,x6,x7,lsl#32 // bfi x6,x7,#32,#32 + umlal v21.2d,v10.2s,v1.s[0] + fmov d9,x4 + umlal v20.2d,v10.2s,v0.s[0] + and x8,x8,#0x03ffffff + + add v13.2s,v13.2s,v28.2s + and x9,x9,#0x03ffffff + umlal v22.2d,v12.2s,v0.s[0] + ubfx x10,x12,#14,#26 + umlal v19.2d,v12.2s,v4.s[0] + ubfx x11,x13,#14,#26 + umlal v23.2d,v12.2s,v1.s[0] + add x8,x8,x9,lsl#32 // bfi x8,x9,#32,#32 + umlal v20.2d,v12.2s,v6.s[0] + fmov d10,x6 + umlal v21.2d,v12.2s,v8.s[0] + add x12,x3,x12,lsr#40 + + umlal v22.2d,v13.2s,v8.s[0] + add x13,x3,x13,lsr#40 + umlal v19.2d,v13.2s,v2.s[0] + add x10,x10,x11,lsl#32 // bfi x10,x11,#32,#32 + umlal v23.2d,v13.2s,v0.s[0] + add x12,x12,x13,lsl#32 // bfi x12,x13,#32,#32 + umlal v20.2d,v13.2s,v4.s[0] + fmov d11,x8 + umlal v21.2d,v13.2s,v6.s[0] + fmov d12,x10 + fmov d13,x12 + + ///////////////////////////////////////////////////////////////// + // lazy reduction as discussed in "NEON crypto" by D.J. Bernstein + // and P. Schwabe + // + // [see discussion in poly1305-armv4 module] + + ushr v29.2d,v22.2d,#26 + xtn v27.2s,v22.2d + ushr v30.2d,v19.2d,#26 + and v19.16b,v19.16b,v31.16b + add v23.2d,v23.2d,v29.2d // h3 -> h4 + bic v27.2s,#0xfc,lsl#24 // &=0x03ffffff + add v20.2d,v20.2d,v30.2d // h0 -> h1 + + ushr v29.2d,v23.2d,#26 + xtn v28.2s,v23.2d + ushr v30.2d,v20.2d,#26 + xtn v25.2s,v20.2d + bic v28.2s,#0xfc,lsl#24 + add v21.2d,v21.2d,v30.2d // h1 -> h2 + + add v19.2d,v19.2d,v29.2d + shl v29.2d,v29.2d,#2 + shrn v30.2s,v21.2d,#26 + xtn v26.2s,v21.2d + add v19.2d,v19.2d,v29.2d // h4 -> h0 + bic v25.2s,#0xfc,lsl#24 + add v27.2s,v27.2s,v30.2s // h2 -> h3 + bic v26.2s,#0xfc,lsl#24 + + shrn v29.2s,v19.2d,#26 + xtn v24.2s,v19.2d + ushr v30.2s,v27.2s,#26 + bic v27.2s,#0xfc,lsl#24 + bic v24.2s,#0xfc,lsl#24 + add v25.2s,v25.2s,v29.2s // h0 -> h1 + add v28.2s,v28.2s,v30.2s // h3 -> h4 + + b.hi .Loop_neon + +.Lskip_loop: + dup v16.2d,v16.d[0] + add v11.2s,v11.2s,v26.2s + + //////////////////////////////////////////////////////////////// + // multiply (inp[0:1]+hash) or inp[2:3] by r^2:r^1 + + adds x2,x2,#32 + b.ne .Long_tail + + dup v16.2d,v11.d[0] + add v14.2s,v9.2s,v24.2s + add v17.2s,v12.2s,v27.2s + add v15.2s,v10.2s,v25.2s + add v18.2s,v13.2s,v28.2s + +.Long_tail: + dup v14.2d,v14.d[0] + umull2 v19.2d,v16.4s,v6.4s + umull2 v22.2d,v16.4s,v1.4s + umull2 v23.2d,v16.4s,v3.4s + umull2 v21.2d,v16.4s,v0.4s + umull2 v20.2d,v16.4s,v8.4s + + dup v15.2d,v15.d[0] + umlal2 v19.2d,v14.4s,v0.4s + umlal2 v21.2d,v14.4s,v3.4s + umlal2 v22.2d,v14.4s,v5.4s + umlal2 v23.2d,v14.4s,v7.4s + umlal2 v20.2d,v14.4s,v1.4s + + dup v17.2d,v17.d[0] + umlal2 v19.2d,v15.4s,v8.4s + umlal2 v22.2d,v15.4s,v3.4s + umlal2 v21.2d,v15.4s,v1.4s + umlal2 v23.2d,v15.4s,v5.4s + umlal2 v20.2d,v15.4s,v0.4s + + dup v18.2d,v18.d[0] + umlal2 v22.2d,v17.4s,v0.4s + umlal2 v23.2d,v17.4s,v1.4s + umlal2 v19.2d,v17.4s,v4.4s + umlal2 v20.2d,v17.4s,v6.4s + umlal2 v21.2d,v17.4s,v8.4s + + umlal2 v22.2d,v18.4s,v8.4s + umlal2 v19.2d,v18.4s,v2.4s + umlal2 v23.2d,v18.4s,v0.4s + umlal2 v20.2d,v18.4s,v4.4s + umlal2 v21.2d,v18.4s,v6.4s + + b.eq .Lshort_tail + + //////////////////////////////////////////////////////////////// + // (hash+inp[0:1])*r^4:r^3 and accumulate + + add v9.2s,v9.2s,v24.2s + umlal v22.2d,v11.2s,v1.2s + umlal v19.2d,v11.2s,v6.2s + umlal v23.2d,v11.2s,v3.2s + umlal v20.2d,v11.2s,v8.2s + umlal v21.2d,v11.2s,v0.2s + + add v10.2s,v10.2s,v25.2s + umlal v22.2d,v9.2s,v5.2s + umlal v19.2d,v9.2s,v0.2s + umlal v23.2d,v9.2s,v7.2s + umlal v20.2d,v9.2s,v1.2s + umlal v21.2d,v9.2s,v3.2s + + add v12.2s,v12.2s,v27.2s + umlal v22.2d,v10.2s,v3.2s + umlal v19.2d,v10.2s,v8.2s + umlal v23.2d,v10.2s,v5.2s + umlal v20.2d,v10.2s,v0.2s + umlal v21.2d,v10.2s,v1.2s + + add v13.2s,v13.2s,v28.2s + umlal v22.2d,v12.2s,v0.2s + umlal v19.2d,v12.2s,v4.2s + umlal v23.2d,v12.2s,v1.2s + umlal v20.2d,v12.2s,v6.2s + umlal v21.2d,v12.2s,v8.2s + + umlal v22.2d,v13.2s,v8.2s + umlal v19.2d,v13.2s,v2.2s + umlal v23.2d,v13.2s,v0.2s + umlal v20.2d,v13.2s,v4.2s + umlal v21.2d,v13.2s,v6.2s + +.Lshort_tail: + //////////////////////////////////////////////////////////////// + // horizontal add + + addp v22.2d,v22.2d,v22.2d + ldp d8,d9,[sp,#16] // meet ABI requirements + addp v19.2d,v19.2d,v19.2d + ldp d10,d11,[sp,#32] + addp v23.2d,v23.2d,v23.2d + ldp d12,d13,[sp,#48] + addp v20.2d,v20.2d,v20.2d + ldp d14,d15,[sp,#64] + addp v21.2d,v21.2d,v21.2d + + //////////////////////////////////////////////////////////////// + // lazy reduction, but without narrowing + + ushr v29.2d,v22.2d,#26 + and v22.16b,v22.16b,v31.16b + ushr v30.2d,v19.2d,#26 + and v19.16b,v19.16b,v31.16b + + add v23.2d,v23.2d,v29.2d // h3 -> h4 + add v20.2d,v20.2d,v30.2d // h0 -> h1 + + ushr v29.2d,v23.2d,#26 + and v23.16b,v23.16b,v31.16b + ushr v30.2d,v20.2d,#26 + and v20.16b,v20.16b,v31.16b + add v21.2d,v21.2d,v30.2d // h1 -> h2 + + add v19.2d,v19.2d,v29.2d + shl v29.2d,v29.2d,#2 + ushr v30.2d,v21.2d,#26 + and v21.16b,v21.16b,v31.16b + add v19.2d,v19.2d,v29.2d // h4 -> h0 + add v22.2d,v22.2d,v30.2d // h2 -> h3 + + ushr v29.2d,v19.2d,#26 + and v19.16b,v19.16b,v31.16b + ushr v30.2d,v22.2d,#26 + and v22.16b,v22.16b,v31.16b + add v20.2d,v20.2d,v29.2d // h0 -> h1 + add v23.2d,v23.2d,v30.2d // h3 -> h4 + + //////////////////////////////////////////////////////////////// + // write the result, can be partially reduced + + st4 {v19.s,v20.s,v21.s,v22.s}[0],[x0],#16 + st1 {v23.s}[0],[x0] + +.Lno_data_neon: + ldr x29,[sp],#80 + ret +ENDPROC(poly1305_blocks_neon) + +.align 5 +ENTRY(poly1305_emit_neon) + ldr x17,[x0,#24] + cbz x17,poly1305_emit_arm + + ldp w10,w11,[x0] // load hash value base 2^26 + ldp w12,w13,[x0,#8] + ldr w14,[x0,#16] + + add x4,x10,x11,lsl#26 // base 2^26 -> base 2^64 + lsr x5,x12,#12 + adds x4,x4,x12,lsl#52 + add x5,x5,x13,lsl#14 + adc x5,x5,xzr + lsr x6,x14,#24 + adds x5,x5,x14,lsl#40 + adc x6,x6,xzr // can be partially reduced... + + ldp x10,x11,[x2] // load nonce + + and x12,x6,#-4 // ... so reduce + add x12,x12,x6,lsr#2 + and x6,x6,#3 + adds x4,x4,x12 + adcs x5,x5,xzr + adc x6,x6,xzr + + adds x12,x4,#5 // compare to modulus + adcs x13,x5,xzr + adc x14,x6,xzr + + tst x14,#-4 // see if it's carried/borrowed + + csel x4,x4,x12,eq + csel x5,x5,x13,eq + +#ifdef __ARMEB__ + ror x10,x10,#32 // flip nonce words + ror x11,x11,#32 +#endif + adds x4,x4,x10 // accumulate nonce + adc x5,x5,x11 +#ifdef __ARMEB__ + rev x4,x4 // flip output bytes + rev x5,x5 +#endif + stp x4,x5,[x1] // write result + + ret +ENDPROC(poly1305_emit_neon) + +.align 5 +.Lzeros: +.long 0,0,0,0,0,0,0,0 --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 08/20] zinc: Poly1305 ARM and ARM64 implementations
Hi Ard,
Quoted 2 lines · Ard Biesheuvel
As I asked in response to v3, could we please have this as a separate patch on top? The diff below is corrupted.
I had played with that originally, but thought it made things actually harder to review, whereas here you have the changes presented pretty straight forwardly, and I'd appreciate your review of them. If you and Eric both prefer I split this into two commits, with the first one just plopping down the CRYPTOGAMS code as is and the second one bringing it up to kernel-snuff, I can do that.
Quoted 2 lines
Also, both Andy and Eric have offered to get involved in upstreaming these changes to OpenSSL, so there is no delta to begin with.
Yes, I think this is probably a good long-term plan, which we can act on sometime after Zinc is merged.
Quoted 3 lines
I still don't like the GCC -includes, especially because these .h files contain function and variable definitions so they are not actually header files to begin with.
I very very strongly disagree with you here. I think doing it via -include is significantly cleaner than any of the alternatives, and allows the code to be cleanly expressed as conditionals that the optimizer trivially compiles out in the case of stub functions returning false and branch optimizes when the stub functions return true. It is extremely important that these compile together as one compilation unit. Yes, this is a different design than the crypto API's approach, but I believe the approach presented here poses significant improvements and is a lot cleaner.
Quoted 4 lines
Also, you mentioned in the commit log that you got rid of defines and made the code more modular, but as far as I can tell, libzinc is still a single monolithic binary that is essentially always builtin once we move
random.cto it.
Yes, it's still monolithic, but it's now trivial to split up when the time comes to do that. If you and AndyL think that it should be split into multiple modules _now_, then I can go ahead and do that for v5. But if it's not essential, it seems simpler to keep it as is. I'll wait for word from you two on this.
Jason
Jason A. Donenfeld
maintainerPATCH v4 9/20 zinc: Poly1305 x86_64 implementation
In reply to Jason A. DonenfeldThis provides AVX, AVX-2, and AVX-512F implementations for Poly1305. The AVX-512F implementation is disabled on Skylake, due to throttling. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- poly1305_blocks_avx512 jumped to the middle of the poly1305_blocks_avx2 for the final blocks. To appease objtool, the relevant tail avx2 code was duplicated for the avx512 function.
- The original uses %rbp as a scratch register. However, the kernel expects %rbp to be a valid frame pointer at any given time in order to do proper unwinding. Thus we need to alter the code in order to preserve it. The most straightforward manner in which this was accomplished was by replacing $d3, formerly %r10, by %rdi, and replacing %rbp by %r10. Because %rdi, a pointer to the context structure, does not change and is not used by poly1305_iteration, it is safe to use it here, and the overhead of saving and restoring it should be minimal.
- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
--- lib/zinc/Makefile | 4 + lib/zinc/poly1305/poly1305-x86_64-glue.h | 109 + lib/zinc/poly1305/poly1305-x86_64.S | 2792 ++++++++++++++++++++++ 3 files changed, 2905 insertions(+) create mode 100644 lib/zinc/poly1305/poly1305-x86_64-glue.h create mode 100644 lib/zinc/poly1305/poly1305-x86_64.S
lib/zinc/Makefile +4 -0
@@ -25,6 +25,10 @@ endif ifeq ($(CONFIG_ZINC_POLY1305),y) zinc-y += poly1305/poly1305.o +ifeq ($(CONFIG_ZINC_ARCH_X86_64),y) +zinc-y += poly1305/poly1305-x86_64.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-x86_64-glue.h +endif ifeq ($(CONFIG_ZINC_ARCH_ARM),y) zinc-y += poly1305/poly1305-arm.o CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
lib/zinc/poly1305/poly1305-x86_64-glue.h new file +109 -0
109 lines not on this page · raw message
lib/zinc/poly1305/poly1305-x86_64.S new file +2792 -0
2,792 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 10/20 zinc: Poly1305 MIPS32r2 and MIPS64 implementations
In reply to Jason A. DonenfeldThis MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets. The MIPS64 implementation comes from Andy Polyakov results in a nice speedup on commodity Octeon hardware, and has been modified slightly from the original:
- The function names have been renamed to fit kernel conventions.
- A comment has been added.
No changes have been made to the actual instructions.
--- lib/zinc/Makefile | 8 + lib/zinc/poly1305/poly1305-mips-glue.h | 40 +++ lib/zinc/poly1305/poly1305-mips.S | 417 +++++++++++++++++++++++++ lib/zinc/poly1305/poly1305-mips64.S | 359 +++++++++++++++++++++ 4 files changed, 824 insertions(+) create mode 100644 lib/zinc/poly1305/poly1305-mips-glue.h create mode 100644 lib/zinc/poly1305/poly1305-mips.S create mode 100644 lib/zinc/poly1305/poly1305-mips64.S
lib/zinc/Makefile +8 -0
@@ -37,6 +37,14 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM64),y) zinc-y += poly1305/poly1305-arm64.o CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h endif +ifeq ($(CONFIG_ZINC_ARCH_MIPS)$(CONFIG_CPU_MIPS32_R2),yy) +zinc-y += poly1305/poly1305-mips.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h +endif +ifeq ($(CONFIG_ZINC_ARCH_MIPS64),y) +zinc-y += poly1305/poly1305-mips64.o +CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h +endif endif zinc-y += main.o
lib/zinc/poly1305/poly1305-mips-glue.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <zinc/poly1305.h> + +asmlinkage void poly1305_init_mips(void *ctx, const u8 key[16]); +asmlinkage void poly1305_blocks_mips(void *ctx, const u8 *inp, const size_t len, +const u32 padbit); +asmlinkage void poly1305_emit_mips(void *ctx, u8 mac[16], const u32 nonce[4]); +void __init poly1305_fpu_init (void ) +{ +} + +static inline bool poly1305_init_arch (void *ctx, +const u8 key[POLY1305_KEY_SIZE ], +simd_context_t simd_context) +{ +poly1305_init_mips (ctx, key); +return true; +} + +static inline bool poly1305_blocks_arch (void *ctx,const u8 *inp, +const size_t len,const u32 padbit, +simd_context_t simd_context) +{ +poly1305_blocks_mips (ctx, inp, len, padbit); +return true; +} + +static inline bool poly1305_emit_arch (void *ctx,u8 mac[POLY1305_MAC_SIZE ], +const u32 nonce[4 ], +simd_context_t simd_context) +{ +poly1305_emit_mips (ctx, mac, nonce); +return true; +} + +#define HAVE_POLY1305_ARCH_IMPLEMENTATION
lib/zinc/poly1305/poly1305-mips.S new file +417 -0
417 lines not on this page · raw message
lib/zinc/poly1305/poly1305-mips64.S new file +359 -0
359 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 11/20 zinc: ChaCha20Poly1305 construction and selftest
In reply to Jason A. DonenfeldThis is an implementation of the ChaCha20Poly1305 AEAD, with an easy API for encrypting either contiguous buffers or scatter gather lists (such as those created from skb_to_sgvec).
Information: https://tools.ietf.org/html/rfc8439
--- include/zinc/chacha20poly1305.h | 54 + lib/zinc/Kconfig | 7 + lib/zinc/Makefile | 4 + lib/zinc/chacha20poly1305.c | 333 ++ lib/zinc/main.c | 4 + lib/zinc/selftest/chacha20poly1305.h | 7852 ++++++++++++++++++++++++++ 6 files changed, 8254 insertions(+) create mode 100644 include/zinc/chacha20poly1305.h create mode 100644 lib/zinc/chacha20poly1305.c create mode 100644 lib/zinc/selftest/chacha20poly1305.h
include/zinc/chacha20poly1305.h new file +54 -0
@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20POLY1305_H +#define _ZINC_CHACHA20POLY1305_H + +#include <linux/simd.h> +#include <linux/types.h> + +struct scatterlist ; + +enum chacha20poly1305_lengths { +XCHACHA20POLY1305_NONCELEN =24 , +CHACHA20POLY1305_KEYLEN =32 , +CHACHA20POLY1305_AUTHTAGLEN =16 +}; + +void chacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u64 nonce, +const u8 key[CHACHA20POLY1305_KEYLEN ]); + +bool __must_check chacha20poly1305_encrypt_sg ( +struct scatterlist *dst,struct scatterlist *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEYLEN ],simd_context_t simd_context); + +bool __must_check +chacha20poly1305_decrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEYLEN ]); + +bool __must_check chacha20poly1305_decrypt_sg ( +struct scatterlist *dst,struct scatterlist *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEYLEN ],simd_context_t simd_context); + +void xchacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u8 nonce[XCHACHA20POLY1305_NONCELEN ], +const u8 key[CHACHA20POLY1305_KEYLEN ]); + +bool __must_check xchacha20poly1305_decrypt ( +u8 *dst,const u8 *src,const size_t src_len,const u8 *ad, +const size_t ad_len,const u8 nonce[XCHACHA20POLY1305_NONCELEN ], +const u8 key[CHACHA20POLY1305_KEYLEN ]); + +#ifdef DEBUG +bool chacha20poly1305_selftest (void ); +#endif + +#endif /* _ZINC_CHACHA20POLY1305_H */
lib/zinc/Kconfig +7 -0
@@ -10,6 +10,13 @@ config ZINC_POLY1305 bool select ZINC +config ZINC_CHACHA20POLY1305 + bool + select ZINC + select ZINC_CHACHA20 + select ZINC_POLY1305 + select CRYPTO_BLKCIPHER + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" depends on ZINC
lib/zinc/Makefile +4 -0
@@ -47,6 +47,10 @@ CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h endif endif +ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y) +zinc-y += chacha20poly1305.o +endif + zinc-y += main.o obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/chacha20poly1305.c new file +333 -0
333 lines not on this page · raw message
lib/zinc/main.c +4 -0
@@ -3,6 +3,7 @@* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ +#include <zinc/chacha20poly1305.h> #include <zinc/chacha20.h> #include <zinc/poly1305.h>
@@ -26,6 +27,9 @@ static int __init mod_init(void)#ifdef CONFIG_ZINC_POLY1305 poly1305_fpu_init ();selftest (poly1305); +#endif +#ifdef CONFIG_ZINC_CHACHA20POLY1305 +selftest (chacha20poly1305);#endif return 0 ; }
lib/zinc/selftest/chacha20poly1305.h new file +7852 -0
2,574 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v4 12/20 zinc: BLAKE2s generic C implementation and selftest
In reply to Jason A. DonenfeldThe C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.
Information: https://blake2.net/
--- include/zinc/blake2s.h | 101 ++ lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 4 + lib/zinc/blake2s/blake2s.c | 274 +++++ lib/zinc/main.c | 5 + lib/zinc/selftest/blake2s.h | 2095 +++++++++++++++++++++++++++++++++++ 6 files changed, 2483 insertions(+) create mode 100644 include/zinc/blake2s.h create mode 100644 lib/zinc/blake2s/blake2s.c create mode 100644 lib/zinc/selftest/blake2s.h
include/zinc/blake2s.h new file +101 -0
@@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_BLAKE2S_H +#define _ZINC_BLAKE2S_H + +#include <linux/types.h> +#include <linux/kernel.h> +#include <crypto/algapi.h> + +enum blake2s_lengths { +BLAKE2S_BLOCKBYTES =64 , +BLAKE2S_OUTBYTES =32 , +BLAKE2S_KEYBYTES =32 +}; + +struct blake2s_state { +u32 h[8 ]; +u32 t[2 ]; +u32 f[2 ]; +u8 buf[BLAKE2S_BLOCKBYTES ]; +size_t buflen; +u8 last_node; +}; + +void blake2s_init (struct blake2s_state *state,const size_t outlen); +void blake2s_init_key (struct blake2s_state *state,const size_t outlen, +const void *key,const size_t keylen); +void blake2s_update (struct blake2s_state *state,const u8 *in,size_t inlen); +void __blake2s_final (struct blake2s_state *state); +static inline void blake2s_final (struct blake2s_state *state,u8 *out, +const size_t outlen) +{ +int i; + +#ifdef DEBUG +BUG_ON (!out || !outlen || outlen >BLAKE2S_OUTBYTES ); +#endif +__blake2s_final (state); + +if (__builtin_constant_p (outlen) && !(outlen %sizeof (u32))) { +if (IS_ENABLED (CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS ) || +IS_ALIGNED ((unsigned long )out, __alignof__(u32 ))) { +__le32 *outwords = (__le32 *)out; + +for (i =0 ; i < outlen /sizeof (u32); ++i) + outwords[i] =cpu_to_le32 (state->h[i]); + }else { +__le32 buffer[BLAKE2S_OUTBYTES ]; + +for (i =0 ; i < outlen /sizeof (u32); ++i) + buffer[i] =cpu_to_le32 (state->h[i]); +memcpy (out, buffer, outlen); +memzero_explicit (buffer,sizeof (buffer)); + } + }else { +u8 buffer[BLAKE2S_OUTBYTES ]__aligned(__alignof__(u32)) ; +__le32 *outwords = (__le32 *)buffer; + +for (i =0 ; i <8 ; ++i) + outwords[i] =cpu_to_le32 (state->h[i]); +memcpy (out, buffer, outlen); +memzero_explicit (buffer,sizeof (buffer)); + } + +memzero_explicit (state,sizeof (*state)); +} + +static inline void blake2s (u8 *out,const u8 *in,const u8 *key, +const size_t outlen,const size_t inlen, +const size_t keylen) +{ +struct blake2s_state state; + +#ifdef DEBUG +BUG_ON ((!in && inlen >0 ) || !out || !outlen || + outlen >BLAKE2S_OUTBYTES || keylen >BLAKE2S_KEYBYTES || + (!key && keylen)); +#endif + +if (keylen) +blake2s_init_key (&state, outlen, key, keylen); +else +blake2s_init (&state, outlen); + +blake2s_update (&state, in, inlen); +blake2s_final (&state, out, outlen); +} + +void blake2s_hmac (u8 *out,const u8 *in,const u8 *key,const size_t outlen, +const size_t inlen,const size_t keylen); + +void blake2s_fpu_init (void ); + +#ifdef DEBUG +bool blake2s_selftest (void ); +#endif + +#endif /* _ZINC_BLAKE2S_H */
lib/zinc/Kconfig +4 -0
@@ -17,6 +17,10 @@ config ZINC_CHACHA20POLY1305 select ZINC_POLY1305 select CRYPTO_BLKCIPHER +config ZINC_BLAKE2S + bool + select ZINC + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" depends on ZINC
lib/zinc/Makefile +4 -0
@@ -51,6 +51,10 @@ ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y) zinc-y += chacha20poly1305.o endif +ifeq ($(CONFIG_ZINC_BLAKE2S),y) +zinc-y += blake2s/blake2s.o +endif + zinc-y += main.o obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/blake2s/blake2s.c new file +274 -0
274 lines not on this page · raw message
lib/zinc/main.c +5 -0
@@ -6,6 +6,7 @@#include <zinc/chacha20poly1305.h> #include <zinc/chacha20.h> #include <zinc/poly1305.h> +#include <zinc/blake2s.h> #include <linux/init.h> #include <linux/module.h>
@@ -30,6 +31,10 @@ static int __init mod_init(void)#endif #ifdef CONFIG_ZINC_CHACHA20POLY1305 selftest (chacha20poly1305); +#endif +#ifdef CONFIG_ZINC_BLAKE2S +blake2s_fpu_init (); +selftest (blake2s);#endif return 0 ; }
lib/zinc/selftest/blake2s.h new file +2095 -0
2,095 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 13/20 zinc: BLAKE2s x86_64 implementation
In reply to Jason A. DonenfeldThese implementations from Samuel Neves support AVX and AVX-512VL. Originally this used AVX-512F, but Skylake thermal throttling made AVX-512VL more attractive and possible to do with negligable difference.
--- lib/zinc/Makefile | 4 + lib/zinc/blake2s/blake2s-x86_64-glue.h | 62 +++ lib/zinc/blake2s/blake2s-x86_64.S | 685 +++++++++++++++++++++++++ 3 files changed, 751 insertions(+) create mode 100644 lib/zinc/blake2s/blake2s-x86_64-glue.h create mode 100644 lib/zinc/blake2s/blake2s-x86_64.S
lib/zinc/Makefile +4 -0
@@ -53,6 +53,10 @@ endif ifeq ($(CONFIG_ZINC_BLAKE2S),y) zinc-y += blake2s/blake2s.o +ifeq ($(CONFIG_ZINC_ARCH_X86_64),y) +zinc-y += blake2s/blake2s-x86_64.o +CFLAGS_blake2s.o += -include $(srctree)/$(src)/blake2s/blake2s-x86_64-glue.h +endif endif zinc-y += main.o
lib/zinc/blake2s/blake2s-x86_64-glue.h new file +62 -0
62 lines not on this page · raw message
lib/zinc/blake2s/blake2s-x86_64.S new file +685 -0
685 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 14/20 zinc: Curve25519 generic C implementations and selftest
In reply to Jason A. DonenfeldThis contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic.
The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star
The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto
Information: https://cr.yp.to/ecdh.html
--- include/zinc/curve25519.h | 28 + lib/zinc/Kconfig | 5 + lib/zinc/Makefile | 4 + lib/zinc/curve25519/curve25519-fiat32.h | 862 +++++++++++++++ lib/zinc/curve25519/curve25519-hacl64.h | 785 ++++++++++++++ lib/zinc/curve25519/curve25519.c | 83 ++ lib/zinc/main.c | 5 + lib/zinc/selftest/curve25519.h | 1321 +++++++++++++++++++++++ 8 files changed, 3093 insertions(+) create mode 100644 include/zinc/curve25519.h create mode 100644 lib/zinc/curve25519/curve25519-fiat32.h create mode 100644 lib/zinc/curve25519/curve25519-hacl64.h create mode 100644 lib/zinc/curve25519/curve25519.c create mode 100644 lib/zinc/selftest/curve25519.h
include/zinc/curve25519.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CURVE25519_H +#define _ZINC_CURVE25519_H + +#include <linux/types.h> + +enum curve25519_lengths { +CURVE25519_POINT_SIZE =32 +}; + +bool __must_check curve25519 (u8 mypublic[CURVE25519_POINT_SIZE ], +const u8 secret[CURVE25519_POINT_SIZE ], +const u8 basepoint[CURVE25519_POINT_SIZE ]); +void curve25519_generate_secret (u8 secret[CURVE25519_POINT_SIZE ]); +bool __must_check curve25519_generate_public ( +u8 pub[CURVE25519_POINT_SIZE ],const u8 secret[CURVE25519_POINT_SIZE ]); + +void curve25519_fpu_init (void ); + +#ifdef DEBUG +bool curve25519_selftest (void ); +#endif + +#endif /* _ZINC_CURVE25519_H */
lib/zinc/Kconfig +5 -0
@@ -21,6 +21,11 @@ config ZINC_BLAKE2S bool select ZINC +config ZINC_CURVE25519 + bool + select ZINC + select CONFIG_CRYPTO + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" depends on ZINC
lib/zinc/Makefile +4 -0
@@ -51,6 +51,10 @@ ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y) zinc-y += chacha20poly1305.o endif +ifeq ($(CONFIG_ZINC_CURVE25519),y) +zinc-y += curve25519/curve25519.o +endif + ifeq ($(CONFIG_ZINC_BLAKE2S),y) zinc-y += blake2s/blake2s.o ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
lib/zinc/curve25519/curve25519-fiat32.h new file +862 -0
862 lines not on this page · raw message
lib/zinc/curve25519/curve25519-hacl64.h new file +785 -0
785 lines not on this page · raw message
lib/zinc/curve25519/curve25519.c new file +83 -0
83 lines not on this page · raw message
lib/zinc/main.c +5 -0
@@ -7,6 +7,7 @@#include <zinc/chacha20.h> #include <zinc/poly1305.h> #include <zinc/blake2s.h> +#include <zinc/curve25519.h> #include <linux/init.h> #include <linux/module.h>
@@ -35,6 +36,10 @@ static int __init mod_init(void)#ifdef CONFIG_ZINC_BLAKE2S blake2s_fpu_init ();selftest (blake2s); +#endif +#ifdef CONFIG_ZINC_CURVE25519 +curve25519_fpu_init (); +selftest (curve25519);#endif return 0 ; }
lib/zinc/selftest/curve25519.h new file +1321 -0
1,204 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v4 15/20 zinc: Curve25519 ARM implementation
In reply to Jason A. DonenfeldThis comes from Dan Bernstein and Peter Schwabe's public domain NEON code, and has been modified to be friendly for kernel space, as well as removing some qhasm strangeness to be more idiomatic.
--- lib/zinc/Makefile | 4 + lib/zinc/curve25519/curve25519-arm-glue.h | 46 + lib/zinc/curve25519/curve25519-arm.S | 2095 +++++++++++++++++++++ 3 files changed, 2145 insertions(+) create mode 100644 lib/zinc/curve25519/curve25519-arm-glue.h create mode 100644 lib/zinc/curve25519/curve25519-arm.S
lib/zinc/Makefile +4 -0
@@ -53,6 +53,10 @@ endif ifeq ($(CONFIG_ZINC_CURVE25519),y) zinc-y += curve25519/curve25519.o +ifeq ($(CONFIG_ZINC_ARCH_ARM)$(CONFIG_KERNEL_MODE_NEON),yy) +zinc-y += curve25519/curve25519-arm.o +CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-arm-glue.h +endif endif ifeq ($(CONFIG_ZINC_BLAKE2S),y)
lib/zinc/curve25519/curve25519-arm-glue.h new file +46 -0
46 lines not on this page · raw message
lib/zinc/curve25519/curve25519-arm.S new file +2095 -0
2,095 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 16/20 zinc: Curve25519 x86_64 implementation
In reply to Jason A. DonenfeldThis implementation is the fastest available x86_64 implementation, and unlike Sandy2x, it doesn't requie use of the floating point registers at all. Instead it makes use of BMI2 and ADX, available on recent microarchitectures. The implementation was written by Armando Faz-Hernández with contributions (upstream) from Samuel Neves and me, in addition to further changes in the kernel implementation from us.
--- lib/zinc/Makefile | 3 + lib/zinc/curve25519/curve25519-x86_64-glue.h | 49 + lib/zinc/curve25519/curve25519-x86_64.h | 2333 ++++++++++++++++++ 3 files changed, 2385 insertions(+) create mode 100644 lib/zinc/curve25519/curve25519-x86_64-glue.h create mode 100644 lib/zinc/curve25519/curve25519-x86_64.h
lib/zinc/Makefile +3 -0
@@ -57,6 +57,9 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM)$(CONFIG_KERNEL_MODE_NEON),yy) zinc-y += curve25519/curve25519-arm.o CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-arm-glue.h endif +ifeq ($(CONFIG_ZINC_ARCH_X86_64),y) +CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-x86_64-glue.h +endif endif ifeq ($(CONFIG_ZINC_BLAKE2S),y)
lib/zinc/curve25519/curve25519-x86_64-glue.h new file +49 -0
49 lines not on this page · raw message
lib/zinc/curve25519/curve25519-x86_64.h new file +2333 -0
2,333 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v4 17/20 crypto: port Poly1305 to Zinc
In reply to Jason A. DonenfeldNow that Poly1305 is in Zinc, we can have the crypto API code simply call into it. We have to do a little bit of book keeping here, because the crypto API receives the key in the first few calls to update.
--- arch/x86/crypto/Makefile | 3 - arch/x86/crypto/poly1305-avx2-x86_64.S | 388 ---------------- arch/x86/crypto/poly1305-sse2-x86_64.S | 584 ------------------------- arch/x86/crypto/poly1305_glue.c | 205 --------- crypto/Kconfig | 15 +- crypto/Makefile | 2 +- crypto/chacha20poly1305.c | 12 +- crypto/poly1305_generic.c | 304 ------------- crypto/poly1305_zinc.c | 92 ++++ include/crypto/poly1305.h | 40 -- 10 files changed, 101 insertions(+), 1544 deletions(-) delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S delete mode 100644 arch/x86/crypto/poly1305_glue.c delete mode 100644 crypto/poly1305_generic.c create mode 100644 crypto/poly1305_zinc.c delete mode 100644 include/crypto/poly1305.h
arch/x86/crypto/Makefile +0 -3
@@ -34,7 +34,6 @@ obj-$(CONFIG_CRYPTO_CRC32_PCLMUL) += crc32-pclmul.o obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o -obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o obj-$(CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2) += aegis128l-aesni.o
@@ -110,10 +109,8 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o -poly1305-x86_64-y := poly1305-sse2-x86_64.o poly1305_glue.o ifeq ($(avx2_supported),yes) sha1-ssse3-y += sha1_avx2_x86_64_asm.o -poly1305-x86_64-y += poly1305-avx2-x86_64.o endif ifeq ($(sha1_ni_supported),yes) sha1-ssse3-y += sha1_ni_asm.o
arch/x86/crypto/poly1305-avx2-x86_64.S deleted +0 -388
388 lines not on this page · raw message
arch/x86/crypto/poly1305-sse2-x86_64.S deleted +0 -584
584 lines not on this page · raw message
arch/x86/crypto/poly1305_glue.c deleted +0 -205
205 lines not on this page · raw message
crypto/Kconfig +2 -13
@@ -656,24 +656,13 @@ config CRYPTO_GHASH config CRYPTO_POLY1305 tristate "Poly1305 authenticator algorithm" select CRYPTO_HASH + select ZINC_POLY1305 help Poly1305 authenticator algorithm, RFC7539. Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein. It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use - in IETF protocols. This is the portable C implementation of Poly1305. - -config CRYPTO_POLY1305_X86_64 - tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)" - depends on X86 && 64BIT - select CRYPTO_POLY1305 - help - Poly1305 authenticator algorithm, RFC7539. - - Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein. - It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use - in IETF protocols. This is the x86_64 assembler implementation using SIMD - instructions. + in IETF protocols. config CRYPTO_MD4 tristate "MD4 digest algorithm"
crypto/Makefile +1 -1
@@ -118,7 +118,7 @@ obj-$(CONFIG_CRYPTO_SEED) += seed.o obj-$(CONFIG_CRYPTO_SPECK) += speck.o obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o -obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o +obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o
crypto/chacha20poly1305.c +6 -6
@@ -14,7 +14,7 @@#include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> #include <crypto/chacha20.h> -#include <crypto/poly1305.h> +#include <zinc/poly1305.h> #include <linux/err.h> #include <linux/init.h> #include <linux/kernel.h>
@@ -62,7 +62,7 @@ struct chachapoly_req_ctx {/* the key we generate for Poly1305 using Chacha20 */ u8 key[POLY1305_KEY_SIZE ];/* calculated Poly1305 tag */ -u8 tag[POLY1305_DIGEST_SIZE ]; +u8 tag[POLY1305_MAC_SIZE ];/* length of data to en/decrypt, without ICV */ unsigned int cryptlen;/* Actual AD, excluding IV */
@@ -471,7 +471,7 @@ static int chachapoly_decrypt(struct aead_request *req) {struct chachapoly_req_ctx *rctx =aead_request_ctx (req); - rctx->cryptlen = req->cryptlen -POLY1305_DIGEST_SIZE ; + rctx->cryptlen = req->cryptlen -POLY1305_MAC_SIZE ;/* decrypt call chain: * - poly_genkey/done()
@@ -513,7 +513,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,static int chachapoly_setauthsize (struct crypto_aead *tfm,unsigned int authsize) { -if (authsize !=POLY1305_DIGEST_SIZE ) +if (authsize !=POLY1305_MAC_SIZE )return -EINVAL ;return 0 ;
@@ -613,7 +613,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb, poly_hash =__crypto_hash_alg_common (poly); err = -EINVAL ; -if (poly_hash->digestsize !=POLY1305_DIGEST_SIZE ) +if (poly_hash->digestsize !=POLY1305_MAC_SIZE ) goto out_put_poly; err = -ENOMEM ;
@@ -666,7 +666,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb, ctx->saltlen; inst->alg.ivsize = ivsize; inst->alg.chunksize =crypto_skcipher_alg_chunksize (chacha); - inst->alg.maxauthsize =POLY1305_DIGEST_SIZE ; + inst->alg.maxauthsize =POLY1305_MAC_SIZE ; inst->alg.init = chachapoly_init; inst->alg.exit = chachapoly_exit; inst->alg.encrypt = chachapoly_encrypt;
crypto/poly1305_generic.c deleted +0 -304
304 lines not on this page · raw message
crypto/poly1305_zinc.c new file +92 -0
92 lines not on this page · raw message
include/crypto/poly1305.h deleted +0 -40
@@ -1,40 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* -* Common values for the Poly1305 algorithm -*/ - -#ifndef _CRYPTO_POLY1305_H -#define _CRYPTO_POLY1305_H - -#include <linux/types.h> -#include <linux/crypto.h> - -#define POLY1305_BLOCK_SIZE 16 -#define POLY1305_KEY_SIZE 32 -#define POLY1305_DIGEST_SIZE 16 - -struct poly1305_desc_ctx { -/* key */ -u32 r[5 ]; -/* finalize key */ -u32 s[4 ]; -/* accumulator */ -u32 h[5 ]; -/* partial buffer */ -u8 buf[POLY1305_BLOCK_SIZE ]; -/* bytes used in partial buffer */ -unsigned int buflen; -/* r key has been set */ -bool rset; -/* s key has been set */ -bool sset; -}; - -int crypto_poly1305_init (struct shash_desc *desc); -unsigned int crypto_poly1305_setdesckey (struct poly1305_desc_ctx *dctx, -const u8 *src,unsigned int srclen); -int crypto_poly1305_update (struct shash_desc *desc, -const u8 *src,unsigned int srclen); -int crypto_poly1305_final (struct shash_desc *desc,u8 *dst); - -#endif
Signature · 1 line
2.19.0
kbuild test robot
build botRe: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc
I love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64
All errors (new ones prefixed by >>):
fs/sysfs/Kconfig:1:error: recursive dependency detected! fs/sysfs/Kconfig:1: symbol SYSFS is selected by CONFIGFS_FS fs/configfs/Kconfig:1: symbol CONFIGFS_FS is selected by ACPI_CONFIGFS drivers/acpi/Kconfig:542: symbol ACPI_CONFIGFS depends on ACPI
>> drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI >> drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI >> arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON >> arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM >> lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC
lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_POLY1305 lib/zinc/Kconfig:9: symbol ZINC_POLY1305 is selected by CRYPTO_POLY1305 crypto/Kconfig:656: symbol CRYPTO_POLY1305 depends on CRYPTO crypto/Kconfig:16: symbol CRYPTO is selected by IP_SCTP net/sctp/Kconfig:5: symbol IP_SCTP is selected by DLM fs/dlm/Kconfig:1: symbol DLM depends on SYSFS
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
vim +9 drivers/acpi/Kconfig
^1da177e4 Linus Torvalds 2005-04-16 5 f5d707ede Arnd Bergmann 2018-08-21 @6 config ARCH_SUPPORTS_ACPI f5d707ede Arnd Bergmann 2018-08-21 7 bool f5d707ede Arnd Bergmann 2018-08-21 8 3f2c48c9b Jan Engelhardt 2007-07-03 @9 menuconfig ACPI 355ee5eb6 Frans Pop 2007-10-29 10 bool "ACPI (Advanced Configuration and Power Interface) Support" 2c870e611 Arnd Bergmann 2018-07-24 11 depends on ARCH_SUPPORTS_ACPI 1300124f6 Adrian Bunk 2006-03-28 12 depends on PCI 243b66e76 Len Brown 2007-02-15 13 select PNP 2c870e611 Arnd Bergmann 2018-07-24 14 default y if X86 1c48aa36e Bjorn Helgaas 2009-02-19 15 help ^1da177e4 Linus Torvalds 2005-04-16 16 Advanced Configuration and Power Interface (ACPI) support for 1c48aa36e Bjorn Helgaas 2009-02-19 17 Linux requires an ACPI-compliant platform (hardware/firmware), ^1da177e4 Linus Torvalds 2005-04-16 18 and assumes the presence of OS-directed configuration and power ^1da177e4 Linus Torvalds 2005-04-16 19 management (OSPM) software. This option will enlarge your ^1da177e4 Linus Torvalds 2005-04-16 20 kernel by about 70K. ^1da177e4 Linus Torvalds 2005-04-16 21 ^1da177e4 Linus Torvalds 2005-04-16 22 Linux ACPI provides a robust functional replacement for several ^1da177e4 Linus Torvalds 2005-04-16 23 legacy configuration and power management interfaces, including ^1da177e4 Linus Torvalds 2005-04-16 24 the Plug-and-Play BIOS specification (PnP BIOS), the ^1da177e4 Linus Torvalds 2005-04-16 25 MultiProcessor Specification (MPS), and the Advanced Power ^1da177e4 Linus Torvalds 2005-04-16 26 Management (APM) specification. If both ACPI and APM support 1c48aa36e Bjorn Helgaas 2009-02-19 27 are configured, ACPI is used. ^1da177e4 Linus Torvalds 2005-04-16 28 1c48aa36e Bjorn Helgaas 2009-02-19 29 The project home page for the Linux ACPI subsystem is here: aaf3d29fe Rafael J. Wysocki 2013-10-10 30 <https://01.org/linux-acpi> ^1da177e4 Linus Torvalds 2005-04-16 31 ^1da177e4 Linus Torvalds 2005-04-16 32 Linux support for ACPI is based on Intel Corporation's ACPI 1c48aa36e Bjorn Helgaas 2009-02-19 33 Component Architecture (ACPI CA). For more information on the 1c48aa36e Bjorn Helgaas 2009-02-19 34 ACPI CA, see: 1c48aa36e Bjorn Helgaas 2009-02-19 35 <http://acpica.org/> ^1da177e4 Linus Torvalds 2005-04-16 36 c7f5220d0 Hanjun Guo 2014-04-08 37 ACPI is an open industry specification originally co-developed by c7f5220d0 Hanjun Guo 2014-04-08 38 Hewlett-Packard, Intel, Microsoft, Phoenix, and Toshiba. Currently, c7f5220d0 Hanjun Guo 2014-04-08 39 it is developed by the ACPI Specification Working Group (ASWG) under c7f5220d0 Hanjun Guo 2014-04-08 40 the UEFI Forum and any UEFI member can join the ASWG and contribute c7f5220d0 Hanjun Guo 2014-04-08 41 to the ACPI specification. 1c48aa36e Bjorn Helgaas 2009-02-19 42 The specification is available at: ^1da177e4 Linus Torvalds 2005-04-16 43 <http://www.acpi.info> c7f5220d0 Hanjun Guo 2014-04-08 44 <http://www.uefi.org/acpi/specs> ^1da177e4 Linus Torvalds 2005-04-16 45
:::::: The code at line 9 was first introduced by commit
:::::: 3f2c48c9b48423d1411695da066d525cca2a27db ACPI: Use menuconfig objects
:::::: TO: Jan Engelhardt <[email protected]> :::::: CC: Len Brown <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc
Greetings Mr. Ro Bot,
Another one of your robot friends also caught this, and the offending code has been removed for v5.
Thanks for botting,
Jason
kbuild test robot
build botRe: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc
In reply to Jason A. DonenfeldI love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64
All errors (new ones prefixed by >>):
>> fs/sysfs/Kconfig:1:error: recursive dependency detected! >> fs/sysfs/Kconfig:1: symbol SYSFS is selected by CONFIGFS_FS >> fs/configfs/Kconfig:1: symbol CONFIGFS_FS is selected by ACPI_CONFIGFS >> drivers/acpi/Kconfig:542: symbol ACPI_CONFIGFS depends on ACPI drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC >> lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_POLY1305 >> lib/zinc/Kconfig:9: symbol ZINC_POLY1305 is selected by CRYPTO_POLY1305 >> crypto/Kconfig:656: symbol CRYPTO_POLY1305 depends on CRYPTO >> crypto/Kconfig:16: symbol CRYPTO is selected by IP_SCTP >> net/sctp/Kconfig:5: symbol IP_SCTP is selected by DLM >> fs/dlm/Kconfig:1: symbol DLM depends on SYSFS
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
vim +1 lib/zinc/Kconfig
32bbe22e Jason A. Donenfeld 2018-09-14 @1 config ZINC
32bbe22e Jason A. Donenfeld 2018-09-14 2 tristate 32bbe22e Jason A. Donenfeld 2018-09-14 3 35f45248 Jason A. Donenfeld 2018-09-14 4 config ZINC_CHACHA20 35f45248 Jason A. Donenfeld 2018-09-14 5 bool 35f45248 Jason A. Donenfeld 2018-09-14 6 select ZINC 35f45248 Jason A. Donenfeld 2018-09-14 7 select CRYPTO_ALGAPI 35f45248 Jason A. Donenfeld 2018-09-14 8 0a36c146 Jason A. Donenfeld 2018-09-14 @9 config ZINC_POLY1305 0a36c146 Jason A. Donenfeld 2018-09-14 10 bool 0a36c146 Jason A. Donenfeld 2018-09-14 11 select ZINC 0a36c146 Jason A. Donenfeld 2018-09-14 12 1b5dbb86 Jason A. Donenfeld 2018-09-14 13 config ZINC_CHACHA20POLY1305 1b5dbb86 Jason A. Donenfeld 2018-09-14 14 bool 1b5dbb86 Jason A. Donenfeld 2018-09-14 15 select ZINC 1b5dbb86 Jason A. Donenfeld 2018-09-14 16 select ZINC_CHACHA20 1b5dbb86 Jason A. Donenfeld 2018-09-14 17 select ZINC_POLY1305 1b5dbb86 Jason A. Donenfeld 2018-09-14 18 select CRYPTO_BLKCIPHER 1b5dbb86 Jason A. Donenfeld 2018-09-14 19 a740374c Jason A. Donenfeld 2018-09-14 20 config ZINC_BLAKE2S a740374c Jason A. Donenfeld 2018-09-14 21 bool a740374c Jason A. Donenfeld 2018-09-14 22 select ZINC a740374c Jason A. Donenfeld 2018-09-14 23 cec5aa7c Jason A. Donenfeld 2018-09-14 24 config ZINC_CURVE25519 cec5aa7c Jason A. Donenfeld 2018-09-14 25 bool cec5aa7c Jason A. Donenfeld 2018-09-14 26 select ZINC cec5aa7c Jason A. Donenfeld 2018-09-14 27 select CONFIG_CRYPTO cec5aa7c Jason A. Donenfeld 2018-09-14 28 32bbe22e Jason A. Donenfeld 2018-09-14 29 config ZINC_DEBUG 32bbe22e Jason A. Donenfeld 2018-09-14 30 bool "Zinc cryptography library debugging and self-tests" 32bbe22e Jason A. Donenfeld 2018-09-14 31 depends on ZINC 32bbe22e Jason A. Donenfeld 2018-09-14 32 help 32bbe22e Jason A. Donenfeld 2018-09-14 33 This builds a series of self-tests for the Zinc crypto library, which 32bbe22e Jason A. Donenfeld 2018-09-14 34 help diagnose any cryptographic algorithm implementation issues that 32bbe22e Jason A. Donenfeld 2018-09-14 35 might be at the root cause of potential bugs. It also adds various 32bbe22e Jason A. Donenfeld 2018-09-14 36 debugging traps. 32bbe22e Jason A. Donenfeld 2018-09-14 37 32bbe22e Jason A. Donenfeld 2018-09-14 38 Unless you're developing and testing cryptographic routines, or are 32bbe22e Jason A. Donenfeld 2018-09-14 39 especially paranoid about correctness on your hardware, you may say 32bbe22e Jason A. Donenfeld 2018-09-14 40 N here. 32bbe22e Jason A. Donenfeld 2018-09-14 41 32bbe22e Jason A. Donenfeld 2018-09-14 @42 config ZINC_ARCH_ARM 32bbe22e Jason A. Donenfeld 2018-09-14 43 def_bool y 32bbe22e Jason A. Donenfeld 2018-09-14 44 depends on ARM 32bbe22e Jason A. Donenfeld 2018-09-14 45 depends on ZINC 32bbe22e Jason A. Donenfeld 2018-09-14 46 imply VFP 32bbe22e Jason A. Donenfeld 2018-09-14 47 imply VFPv3 if CPU_V7 32bbe22e Jason A. Donenfeld 2018-09-14 48 imply NEON if CPU_V7 32bbe22e Jason A. Donenfeld 2018-09-14 49 imply KERNEL_MODE_NEON if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14 50
:::::: The code at line 1 was first introduced by commit
:::::: 32bbe22ec6fddc21b70274aab7d2830d69d51578 zinc: introduce minimal cryptography library
:::::: TO: Jason A. Donenfeld <[email protected]> :::::: CC: 0day robot <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerPATCH v4 18/20 crypto: port ChaCha20 to Zinc
In reply to Jason A. DonenfeldNow that ChaCha20 is in Zinc, we can have the crypto API code simply call into it. The crypto API expects to have a stored key per instance and independent nonces, so we follow suite and store the key and initialize the nonce independently.
--- arch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/multi_v7_defconfig | 1 - arch/arm/configs/omap2plus_defconfig | 1 - arch/arm/crypto/Kconfig | 6 - arch/arm/crypto/Makefile | 2 - arch/arm/crypto/chacha20-neon-core.S | 521 -------------------- arch/arm/crypto/chacha20-neon-glue.c | 127 ----- arch/arm64/configs/defconfig | 1 - arch/arm64/crypto/Kconfig | 6 - arch/arm64/crypto/Makefile | 3 - arch/arm64/crypto/chacha20-neon-core.S | 450 ----------------- arch/arm64/crypto/chacha20-neon-glue.c | 133 ----- arch/x86/crypto/Makefile | 3 - arch/x86/crypto/chacha20-avx2-x86_64.S | 448 ----------------- arch/x86/crypto/chacha20-ssse3-x86_64.S | 630 ------------------------ arch/x86/crypto/chacha20_glue.c | 146 ------ crypto/Kconfig | 16 - crypto/Makefile | 2 +- crypto/chacha20_generic.c | 136 ----- crypto/chacha20_zinc.c | 100 ++++ crypto/chacha20poly1305.c | 2 +- include/crypto/chacha20.h | 12 - 22 files changed, 102 insertions(+), 2645 deletions(-) delete mode 100644 arch/arm/crypto/chacha20-neon-core.S delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c delete mode 100644 arch/arm64/crypto/chacha20-neon-core.S delete mode 100644 arch/arm64/crypto/chacha20-neon-glue.c delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S delete mode 100644 arch/x86/crypto/chacha20_glue.c delete mode 100644 crypto/chacha20_generic.c create mode 100644 crypto/chacha20_zinc.c
arch/arm/configs/exynos_defconfig +0 -1
@@ -350,7 +350,6 @@ CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA256_ARM=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRC_CCITT=y CONFIG_FONTS=y CONFIG_FONT_7x14=y
arch/arm/configs/multi_v7_defconfig +0 -1
@@ -1000,4 +1000,3 @@ CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_CRC32_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/omap2plus_defconfig +0 -1
@@ -547,7 +547,6 @@ CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_GHASH_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRC_CCITT=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=y
arch/arm/crypto/Kconfig +0 -6
@@ -115,12 +115,6 @@ config CRYPTO_CRC32_ARM_CE depends on KERNEL_MODE_NEON && CRC32 select CRYPTO_HASH -config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" - depends on KERNEL_MODE_NEON - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - config CRYPTO_SPECK_NEON tristate "NEON accelerated Speck cipher algorithms" depends on KERNEL_MODE_NEON
arch/arm/crypto/Makefile +0 -2
@@ -9,7 +9,6 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -53,7 +52,6 @@ aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o speck-neon-y := speck-neon-core.o speck-neon-glue.o ifdef REGENERATE_ARM_CRYPTO
arch/arm/crypto/chacha20-neon-core.S deleted +0 -521
521 lines not on this page · raw message
arch/arm/crypto/chacha20-neon-glue.c deleted +0 -127
127 lines not on this page · raw message
arch/arm64/configs/defconfig +0 -1
@@ -709,5 +709,4 @@ CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m CONFIG_CRYPTO_CRC32_ARM64_CE=m CONFIG_CRYPTO_AES_ARM64_CE_CCM=y CONFIG_CRYPTO_AES_ARM64_CE_BLK=y -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_AES_ARM64_BS=m
arch/arm64/crypto/Kconfig +0 -6
@@ -105,12 +105,6 @@ config CRYPTO_AES_ARM64_NEON_BLK select CRYPTO_AES select CRYPTO_SIMD -config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" - depends on KERNEL_MODE_NEON - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - config CRYPTO_AES_ARM64_BS tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm" depends on KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +0 -3
@@ -53,9 +53,6 @@ sha256-arm64-y := sha256-glue.o sha256-core.o obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o sha512-arm64-y := sha512-glue.o sha512-core.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o - obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o speck-neon-y := speck-neon-core.o speck-neon-glue.o
arch/arm64/crypto/chacha20-neon-core.S deleted +0 -450
450 lines not on this page · raw message
arch/arm64/crypto/chacha20-neon-glue.c deleted +0 -133
133 lines not on this page · raw message
arch/x86/crypto/Makefile +0 -3
@@ -23,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o -obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
@@ -76,7 +75,6 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o -chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -99,7 +97,6 @@ endif ifeq ($(avx2_supported),yes) camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o - chacha20-x86_64-y += chacha20-avx2-x86_64.o serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
arch/x86/crypto/chacha20-avx2-x86_64.S deleted +0 -448
448 lines not on this page · raw message
arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0 -630
630 lines not on this page · raw message
arch/x86/crypto/chacha20_glue.c deleted +0 -146
146 lines not on this page · raw message
crypto/Kconfig +0 -16
@@ -1433,22 +1433,6 @@ config CRYPTO_CHACHA20 ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the portable C implementation of ChaCha20. - - See also: - <http://cr.yp.to/chacha/chacha-20080128.pdf> - -config CRYPTO_CHACHA20_X86_64 - tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)" - depends on X86 && 64BIT - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - help - ChaCha20 cipher algorithm, RFC7539. - - ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. - Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the x86_64 assembler implementation using SIMD instructions. See also: <http://cr.yp.to/chacha/chacha-20080128.pdf>
crypto/Makefile +1 -1
@@ -117,7 +117,7 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o obj-$(CONFIG_CRYPTO_SPECK) += speck.o obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o -obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o +obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_zinc.o obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
crypto/chacha20_generic.c deleted +0 -136
136 lines not on this page · raw message
crypto/chacha20_zinc.c new file +100 -0
100 lines not on this page · raw message
crypto/chacha20poly1305.c +1 -1
@@ -13,7 +13,7 @@#include <crypto/internal/hash.h> #include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> -#include <crypto/chacha20.h> +#include <zinc/chacha20.h> #include <zinc/poly1305.h> #include <linux/err.h> #include <linux/init.h>
include/crypto/chacha20.h +0 -12
@@ -6,23 +6,11 @@#ifndef _CRYPTO_CHACHA20_H#define _CRYPTO_CHACHA20_H -#include <crypto/skcipher.h> -#include <linux/types.h> -#include <linux/crypto.h> -#define CHACHA20_IV_SIZE 16#define CHACHA20_KEY_SIZE 32#define CHACHA20_BLOCK_SIZE 64#define CHACHA20_BLOCK_WORDS (CHACHA20_BLOCK_SIZE / sizeof(u32)) -struct chacha20_ctx { -u32 key[8 ]; -}; -void chacha20_block (u32 *state,u32 *stream); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv); -int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize); -int crypto_chacha20_crypt (struct skcipher_request *req);#endif
Signature · 1 line
2.19.0
Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
Quoted 17 lines · Jason A. Donenfeld
Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it. The crypto API expects to have a stored key per instance and independent nonces, so we follow suite and store the key and initialize the nonce independently.
From our exchange re v3:
Quoted 8 lines
Then there is the performance claim. We know for instance that the OpenSSL ARM NEON code for ChaCha20 is faster on cores that happen to possess a micro-architectural property that ALU instructions are essentially free when they are interleaved with SIMD instructions. But we also know that a) Cortex-A7, which is a relevant target, is not one of those cores, and b) that chip designers are not likely to optimize for that particular usage pattern so relying on it in generic code is unwise in general.
Quoted 6 lines
That's interesting. I'll bring this up with AndyP. FWIW, if you think you have a real and compelling claim here, I'd be much more likely to accept a different ChaCha20 implementation than I would be to accept a different Poly1305 implementation. (It's a *lot* harder to screw up ChaCha20 than it is to screw up Poly1305.)
so could we please bring that discussion to a close before we drop the ARM code?
I am fine with dropping the arm64 code btw.
Quoted 3,008 lines
Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Eric Biggers <[email protected]>--- arch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/multi_v7_defconfig | 1 - arch/arm/configs/omap2plus_defconfig | 1 - arch/arm/crypto/Kconfig | 6 - arch/arm/crypto/Makefile | 2 - arch/arm/crypto/chacha20-neon-core.S | 521 -------------------- arch/arm/crypto/chacha20-neon-glue.c | 127 ----- arch/arm64/configs/defconfig | 1 - arch/arm64/crypto/Kconfig | 6 - arch/arm64/crypto/Makefile | 3 - arch/arm64/crypto/chacha20-neon-core.S | 450 ----------------- arch/arm64/crypto/chacha20-neon-glue.c | 133 ----- arch/x86/crypto/Makefile | 3 - arch/x86/crypto/chacha20-avx2-x86_64.S | 448 ----------------- arch/x86/crypto/chacha20-ssse3-x86_64.S | 630 ------------------------ arch/x86/crypto/chacha20_glue.c | 146 ------ crypto/Kconfig | 16 - crypto/Makefile | 2 +- crypto/chacha20_generic.c | 136 ----- crypto/chacha20_zinc.c | 100 ++++ crypto/chacha20poly1305.c | 2 +- include/crypto/chacha20.h | 12 - 22 files changed, 102 insertions(+), 2645 deletions(-) delete mode 100644 arch/arm/crypto/chacha20-neon-core.S delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c delete mode 100644 arch/arm64/crypto/chacha20-neon-core.S delete mode 100644 arch/arm64/crypto/chacha20-neon-glue.c delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S delete mode 100644 arch/x86/crypto/chacha20_glue.c delete mode 100644 crypto/chacha20_generic.c create mode 100644 crypto/chacha20_zinc.carch/arm/configs/exynos_defconfig +0
-1@@ -350,7 +350,6 @@ CONFIG_CRYPTO_SHA1_ARM_NEON=m CONFIG_CRYPTO_SHA256_ARM=m CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRC_CCITT=y CONFIG_FONTS=y CONFIG_FONT_7x14=yarch/arm/configs/multi_v7_defconfig +0
-1@@ -1000,4 +1000,3 @@ CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_AES_ARM_CE=m CONFIG_CRYPTO_GHASH_ARM_CE=m CONFIG_CRYPTO_CRC32_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=march/arm/configs/omap2plus_defconfig +0
-1@@ -547,7 +547,6 @@ CONFIG_CRYPTO_SHA512_ARM=m CONFIG_CRYPTO_AES_ARM=m CONFIG_CRYPTO_AES_ARM_BS=m CONFIG_CRYPTO_GHASH_ARM_CE=m -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRC_CCITT=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=yarch/arm/crypto/Kconfig +0
-6@@ -115,12 +115,6 @@ config CRYPTO_CRC32_ARM_CE depends on KERNEL_MODE_NEON && CRC32 select CRYPTO_HASH -config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" - depends on KERNEL_MODE_NEON - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - config CRYPTO_SPECK_NEON tristate "NEON accelerated Speck cipher algorithms" depends on KERNEL_MODE_NEONarch/arm/crypto/Makefile +0
-2@@ -9,7 +9,6 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o@@ -53,7 +52,6 @@ aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o speck-neon-y := speck-neon-core.o speck-neon-glue.o ifdef REGENERATE_ARM_CRYPTOarch/arm/crypto/chacha20-neon-core.S deleted +0
-521@@ -1,521 +0,0 @@ -/* - * ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions - * - * Copyright (C) 2016 Linaro, Ltd. <[email protected]> - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * Based on: - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSE3 functions - * - * Copyright (C) 2015 Martin Willi - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include <linux/linkage.h> - - .text - .fpu neon - .align 5 - -ENTRY(chacha20_block_xor_neon) - // r0: Input state matrix, s - // r1: 1 data block output, o - // r2: 1 data block input, i - - // - // This function encrypts one ChaCha20 block by loading the state matrix - // in four NEON registers. It performs matrix operation on four words in - // parallel, but requireds shuffling to rearrange the words after each - // round. - // - - // x0..3 = s0..3 - add ip, r0, #0x20 - vld1.32 {q0-q1}, [r0] - vld1.32 {q2-q3}, [ip] - - vmov q8, q0 - vmov q9, q1 - vmov q10, q2 - vmov q11, q3 - - mov r3, #10 - -.Ldoubleround: - // x0 += x1, x3 = rotl32(x3 ^ x0, 16) - vadd.i32 q0, q0, q1 - veor q3, q3, q0 - vrev32.16 q3, q3 - - // x2 += x3, x1 = rotl32(x1 ^ x2, 12) - vadd.i32 q2, q2, q3 - veor q4, q1, q2 - vshl.u32 q1, q4, #12 - vsri.u32 q1, q4, #20 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 8) - vadd.i32 q0, q0, q1 - veor q4, q3, q0 - vshl.u32 q3, q4, #8 - vsri.u32 q3, q4, #24 - - // x2 += x3, x1 = rotl32(x1 ^ x2, 7) - vadd.i32 q2, q2, q3 - veor q4, q1, q2 - vshl.u32 q1, q4, #7 - vsri.u32 q1, q4, #25 - - // x1 = shuffle32(x1, MASK(0, 3, 2, 1)) - vext.8 q1, q1, q1, #4 - // x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - vext.8 q2, q2, q2, #8 - // x3 = shuffle32(x3, MASK(2, 1, 0, 3)) - vext.8 q3, q3, q3, #12 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 16) - vadd.i32 q0, q0, q1 - veor q3, q3, q0 - vrev32.16 q3, q3 - - // x2 += x3, x1 = rotl32(x1 ^ x2, 12) - vadd.i32 q2, q2, q3 - veor q4, q1, q2 - vshl.u32 q1, q4, #12 - vsri.u32 q1, q4, #20 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 8) - vadd.i32 q0, q0, q1 - veor q4, q3, q0 - vshl.u32 q3, q4, #8 - vsri.u32 q3, q4, #24 - - // x2 += x3, x1 = rotl32(x1 ^ x2, 7) - vadd.i32 q2, q2, q3 - veor q4, q1, q2 - vshl.u32 q1, q4, #7 - vsri.u32 q1, q4, #25 - - // x1 = shuffle32(x1, MASK(2, 1, 0, 3)) - vext.8 q1, q1, q1, #12 - // x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - vext.8 q2, q2, q2, #8 - // x3 = shuffle32(x3, MASK(0, 3, 2, 1)) - vext.8 q3, q3, q3, #4 - - subs r3, r3, #1 - bne .Ldoubleround - - add ip, r2, #0x20 - vld1.8 {q4-q5}, [r2] - vld1.8 {q6-q7}, [ip] - - // o0 = i0 ^ (x0 + s0) - vadd.i32 q0, q0, q8 - veor q0, q0, q4 - - // o1 = i1 ^ (x1 + s1) - vadd.i32 q1, q1, q9 - veor q1, q1, q5 - - // o2 = i2 ^ (x2 + s2) - vadd.i32 q2, q2, q10 - veor q2, q2, q6 - - // o3 = i3 ^ (x3 + s3) - vadd.i32 q3, q3, q11 - veor q3, q3, q7 - - add ip, r1, #0x20 - vst1.8 {q0-q1}, [r1] - vst1.8 {q2-q3}, [ip] - - bx lr -ENDPROC(chacha20_block_xor_neon) - - .align 5 -ENTRY(chacha20_4block_xor_neon) - push {r4-r6, lr} - mov ip, sp // preserve the stack pointer - sub r3, sp, #0x20 // allocate a 32 byte buffer - bic r3, r3, #0x1f // aligned to 32 bytes - mov sp, r3 - - // r0: Input state matrix, s - // r1: 4 data blocks output, o - // r2: 4 data blocks input, i - - // - // This function encrypts four consecutive ChaCha20 blocks by loading - // the state matrix in NEON registers four times. The algorithm performs - // each operation on the corresponding word of each state matrix, hence - // requires no word shuffling. For final XORing step we transpose the - // matrix by interleaving 32- and then 64-bit words, which allows us to - // do XOR in NEON registers. - // - - // x0..15[0-3] = s0..3[0..3] - add r3, r0, #0x20 - vld1.32 {q0-q1}, [r0] - vld1.32 {q2-q3}, [r3] - - adr r3, CTRINC - vdup.32 q15, d7[1] - vdup.32 q14, d7[0] - vld1.32 {q11}, [r3, :128] - vdup.32 q13, d6[1] - vdup.32 q12, d6[0] - vadd.i32 q12, q12, q11 // x12 += counter values 0-3 - vdup.32 q11, d5[1] - vdup.32 q10, d5[0] - vdup.32 q9, d4[1] - vdup.32 q8, d4[0] - vdup.32 q7, d3[1] - vdup.32 q6, d3[0] - vdup.32 q5, d2[1] - vdup.32 q4, d2[0] - vdup.32 q3, d1[1] - vdup.32 q2, d1[0] - vdup.32 q1, d0[1] - vdup.32 q0, d0[0] - - mov r3, #10 - -.Ldoubleround4: - // x0 += x4, x12 = rotl32(x12 ^ x0, 16) - // x1 += x5, x13 = rotl32(x13 ^ x1, 16) - // x2 += x6, x14 = rotl32(x14 ^ x2, 16) - // x3 += x7, x15 = rotl32(x15 ^ x3, 16) - vadd.i32 q0, q0, q4 - vadd.i32 q1, q1, q5 - vadd.i32 q2, q2, q6 - vadd.i32 q3, q3, q7 - - veor q12, q12, q0 - veor q13, q13, q1 - veor q14, q14, q2 - veor q15, q15, q3 - - vrev32.16 q12, q12 - vrev32.16 q13, q13 - vrev32.16 q14, q14 - vrev32.16 q15, q15 - - // x8 += x12, x4 = rotl32(x4 ^ x8, 12) - // x9 += x13, x5 = rotl32(x5 ^ x9, 12) - // x10 += x14, x6 = rotl32(x6 ^ x10, 12) - // x11 += x15, x7 = rotl32(x7 ^ x11, 12) - vadd.i32 q8, q8, q12 - vadd.i32 q9, q9, q13 - vadd.i32 q10, q10, q14 - vadd.i32 q11, q11, q15 - - vst1.32 {q8-q9}, [sp, :256] - - veor q8, q4, q8 - veor q9, q5, q9 - vshl.u32 q4, q8, #12 - vshl.u32 q5, q9, #12 - vsri.u32 q4, q8, #20 - vsri.u32 q5, q9, #20 - - veor q8, q6, q10 - veor q9, q7, q11 - vshl.u32 q6, q8, #12 - vshl.u32 q7, q9, #12 - vsri.u32 q6, q8, #20 - vsri.u32 q7, q9, #20 - - // x0 += x4, x12 = rotl32(x12 ^ x0, 8) - // x1 += x5, x13 = rotl32(x13 ^ x1, 8) - // x2 += x6, x14 = rotl32(x14 ^ x2, 8) - // x3 += x7, x15 = rotl32(x15 ^ x3, 8) - vadd.i32 q0, q0, q4 - vadd.i32 q1, q1, q5 - vadd.i32 q2, q2, q6 - vadd.i32 q3, q3, q7 - - veor q8, q12, q0 - veor q9, q13, q1 - vshl.u32 q12, q8, #8 - vshl.u32 q13, q9, #8 - vsri.u32 q12, q8, #24 - vsri.u32 q13, q9, #24 - - veor q8, q14, q2 - veor q9, q15, q3 - vshl.u32 q14, q8, #8 - vshl.u32 q15, q9, #8 - vsri.u32 q14, q8, #24 - vsri.u32 q15, q9, #24 - - vld1.32 {q8-q9}, [sp, :256] - - // x8 += x12, x4 = rotl32(x4 ^ x8, 7) - // x9 += x13, x5 = rotl32(x5 ^ x9, 7) - // x10 += x14, x6 = rotl32(x6 ^ x10, 7) - // x11 += x15, x7 = rotl32(x7 ^ x11, 7) - vadd.i32 q8, q8, q12 - vadd.i32 q9, q9, q13 - vadd.i32 q10, q10, q14 - vadd.i32 q11, q11, q15 - - vst1.32 {q8-q9}, [sp, :256] - - veor q8, q4, q8 - veor q9, q5, q9 - vshl.u32 q4, q8, #7 - vshl.u32 q5, q9, #7 - vsri.u32 q4, q8, #25 - vsri.u32 q5, q9, #25 - - veor q8, q6, q10 - veor q9, q7, q11 - vshl.u32 q6, q8, #7 - vshl.u32 q7, q9, #7 - vsri.u32 q6, q8, #25 - vsri.u32 q7, q9, #25 - - vld1.32 {q8-q9}, [sp, :256] - - // x0 += x5, x15 = rotl32(x15 ^ x0, 16) - // x1 += x6, x12 = rotl32(x12 ^ x1, 16) - // x2 += x7, x13 = rotl32(x13 ^ x2, 16) - // x3 += x4, x14 = rotl32(x14 ^ x3, 16) - vadd.i32 q0, q0, q5 - vadd.i32 q1, q1, q6 - vadd.i32 q2, q2, q7 - vadd.i32 q3, q3, q4 - - veor q15, q15, q0 - veor q12, q12, q1 - veor q13, q13, q2 - veor q14, q14, q3 - - vrev32.16 q15, q15 - vrev32.16 q12, q12 - vrev32.16 q13, q13 - vrev32.16 q14, q14 - - // x10 += x15, x5 = rotl32(x5 ^ x10, 12) - // x11 += x12, x6 = rotl32(x6 ^ x11, 12) - // x8 += x13, x7 = rotl32(x7 ^ x8, 12) - // x9 += x14, x4 = rotl32(x4 ^ x9, 12) - vadd.i32 q10, q10, q15 - vadd.i32 q11, q11, q12 - vadd.i32 q8, q8, q13 - vadd.i32 q9, q9, q14 - - vst1.32 {q8-q9}, [sp, :256] - - veor q8, q7, q8 - veor q9, q4, q9 - vshl.u32 q7, q8, #12 - vshl.u32 q4, q9, #12 - vsri.u32 q7, q8, #20 - vsri.u32 q4, q9, #20 - - veor q8, q5, q10 - veor q9, q6, q11 - vshl.u32 q5, q8, #12 - vshl.u32 q6, q9, #12 - vsri.u32 q5, q8, #20 - vsri.u32 q6, q9, #20 - - // x0 += x5, x15 = rotl32(x15 ^ x0, 8) - // x1 += x6, x12 = rotl32(x12 ^ x1, 8) - // x2 += x7, x13 = rotl32(x13 ^ x2, 8) - // x3 += x4, x14 = rotl32(x14 ^ x3, 8) - vadd.i32 q0, q0, q5 - vadd.i32 q1, q1, q6 - vadd.i32 q2, q2, q7 - vadd.i32 q3, q3, q4 - - veor q8, q15, q0 - veor q9, q12, q1 - vshl.u32 q15, q8, #8 - vshl.u32 q12, q9, #8 - vsri.u32 q15, q8, #24 - vsri.u32 q12, q9, #24 - - veor q8, q13, q2 - veor q9, q14, q3 - vshl.u32 q13, q8, #8 - vshl.u32 q14, q9, #8 - vsri.u32 q13, q8, #24 - vsri.u32 q14, q9, #24 - - vld1.32 {q8-q9}, [sp, :256] - - // x10 += x15, x5 = rotl32(x5 ^ x10, 7) - // x11 += x12, x6 = rotl32(x6 ^ x11, 7) - // x8 += x13, x7 = rotl32(x7 ^ x8, 7) - // x9 += x14, x4 = rotl32(x4 ^ x9, 7) - vadd.i32 q10, q10, q15 - vadd.i32 q11, q11, q12 - vadd.i32 q8, q8, q13 - vadd.i32 q9, q9, q14 - - vst1.32 {q8-q9}, [sp, :256] - - veor q8, q7, q8 - veor q9, q4, q9 - vshl.u32 q7, q8, #7 - vshl.u32 q4, q9, #7 - vsri.u32 q7, q8, #25 - vsri.u32 q4, q9, #25 - - veor q8, q5, q10 - veor q9, q6, q11 - vshl.u32 q5, q8, #7 - vshl.u32 q6, q9, #7 - vsri.u32 q5, q8, #25 - vsri.u32 q6, q9, #25 - - subs r3, r3, #1 - beq 0f - - vld1.32 {q8-q9}, [sp, :256] - b .Ldoubleround4 - - // x0[0-3] += s0[0] - // x1[0-3] += s0[1] - // x2[0-3] += s0[2] - // x3[0-3] += s0[3] -0: ldmia r0!, {r3-r6} - vdup.32 q8, r3 - vdup.32 q9, r4 - vadd.i32 q0, q0, q8 - vadd.i32 q1, q1, q9 - vdup.32 q8, r5 - vdup.32 q9, r6 - vadd.i32 q2, q2, q8 - vadd.i32 q3, q3, q9 - - // x4[0-3] += s1[0] - // x5[0-3] += s1[1] - // x6[0-3] += s1[2] - // x7[0-3] += s1[3] - ldmia r0!, {r3-r6} - vdup.32 q8, r3 - vdup.32 q9, r4 - vadd.i32 q4, q4, q8 - vadd.i32 q5, q5, q9 - vdup.32 q8, r5 - vdup.32 q9, r6 - vadd.i32 q6, q6, q8 - vadd.i32 q7, q7, q9 - - // interleave 32-bit words in state n, n+1 - vzip.32 q0, q1 - vzip.32 q2, q3 - vzip.32 q4, q5 - vzip.32 q6, q7 - - // interleave 64-bit words in state n, n+2 - vswp d1, d4 - vswp d3, d6 - vswp d9, d12 - vswp d11, d14 - - // xor with corresponding input, write to output - vld1.8 {q8-q9}, [r2]! - veor q8, q8, q0 - veor q9, q9, q4 - vst1.8 {q8-q9}, [r1]! - - vld1.32 {q8-q9}, [sp, :256] - - // x8[0-3] += s2[0] - // x9[0-3] += s2[1] - // x10[0-3] += s2[2] - // x11[0-3] += s2[3] - ldmia r0!, {r3-r6} - vdup.32 q0, r3 - vdup.32 q4, r4 - vadd.i32 q8, q8, q0 - vadd.i32 q9, q9, q4 - vdup.32 q0, r5 - vdup.32 q4, r6 - vadd.i32 q10, q10, q0 - vadd.i32 q11, q11, q4 - - // x12[0-3] += s3[0] - // x13[0-3] += s3[1] - // x14[0-3] += s3[2] - // x15[0-3] += s3[3] - ldmia r0!, {r3-r6} - vdup.32 q0, r3 - vdup.32 q4, r4 - adr r3, CTRINC - vadd.i32 q12, q12, q0 - vld1.32 {q0}, [r3, :128] - vadd.i32 q13, q13, q4 - vadd.i32 q12, q12, q0 // x12 += counter values 0-3 - - vdup.32 q0, r5 - vdup.32 q4, r6 - vadd.i32 q14, q14, q0 - vadd.i32 q15, q15, q4 - - // interleave 32-bit words in state n, n+1 - vzip.32 q8, q9 - vzip.32 q10, q11 - vzip.32 q12, q13 - vzip.32 q14, q15 - - // interleave 64-bit words in state n, n+2 - vswp d17, d20 - vswp d19, d22 - vswp d25, d28 - vswp d27, d30 - - vmov q4, q1 - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q8 - veor q1, q1, q12 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q2 - veor q1, q1, q6 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q10 - veor q1, q1, q14 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q4 - veor q1, q1, q5 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q9 - veor q1, q1, q13 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2]! - veor q0, q0, q3 - veor q1, q1, q7 - vst1.8 {q0-q1}, [r1]! - - vld1.8 {q0-q1}, [r2] - veor q0, q0, q11 - veor q1, q1, q15 - vst1.8 {q0-q1}, [r1] - - mov sp, ip - pop {r4-r6, pc} -ENDPROC(chacha20_4block_xor_neon) - - .align 4 -CTRINC: .word 0, 1, 2, 3arch/arm/crypto/chacha20-neon-glue.c deleted +0
-127@@ -1,127 +0,0 @@ -/* -* ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions -* -* Copyright (C) 2016 Linaro, Ltd. <[email protected]> -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -* -* Based on: -* ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code -* -* Copyright (C) 2015 Martin Willi -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License as published by -* the Free Software Foundation; either version 2 of the License, or -* (at your option) any later version. -*/ - -#include <crypto/algapi.h> -#include <crypto/chacha20.h> -#include <crypto/internal/skcipher.h> -#include <linux/kernel.h> -#include <linux/module.h> - -#include <asm/hwcap.h> -#include <asm/neon.h> -#include <asm/simd.h> - -asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); -asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); - -static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src, -unsigned int bytes) -{ -u8 buf[CHACHA20_BLOCK_SIZE ]; - -while (bytes >=CHACHA20_BLOCK_SIZE *4 ) { -chacha20_4block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE *4 ; - src +=CHACHA20_BLOCK_SIZE *4 ; - dst +=CHACHA20_BLOCK_SIZE *4 ; - state[12 ] +=4 ; - } -while (bytes >=CHACHA20_BLOCK_SIZE ) { -chacha20_block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE ; - src +=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; - state[12 ]++; - } -if (bytes) { -memcpy (buf, src, bytes); -chacha20_block_xor_neon (state, buf, buf); -memcpy (dst, buf, bytes); - } -} - -static int chacha20_neon (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -u32 state[16 ]; -int err; - -if (req->cryptlen <=CHACHA20_BLOCK_SIZE || !may_use_simd ()) -return crypto_chacha20_crypt (req); - - err =skcipher_walk_virt (&walk, req, true); - -crypto_chacha20_init (state, ctx, walk.iv); - -kernel_neon_begin (); -while (walk.nbytes >0 ) { -unsigned int nbytes = walk.nbytes; - -if (nbytes < walk.total) - nbytes =round_down (nbytes, walk.stride); - -chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); - err =skcipher_walk_done (&walk, walk.nbytes - nbytes); - } -kernel_neon_end (); - -return err; -} - -static struct skcipher_alg alg = { - .base.cra_name ="chacha20" , - .base.cra_driver_name ="chacha20-neon" , - .base.cra_priority =300 , - .base.cra_blocksize =1 , - .base.cra_ctxsize =sizeof (struct chacha20_ctx ), - .base.cra_module =THIS_MODULE , - - .min_keysize =CHACHA20_KEY_SIZE , - .max_keysize =CHACHA20_KEY_SIZE , - .ivsize =CHACHA20_IV_SIZE , - .chunksize =CHACHA20_BLOCK_SIZE , - .walksize =4 *CHACHA20_BLOCK_SIZE , - .setkey = crypto_chacha20_setkey, - .encrypt = chacha20_neon, - .decrypt = chacha20_neon, -}; - -static int __init chacha20_simd_mod_init (void ) -{ -if (!(elf_hwcap &HWCAP_NEON )) -return -ENODEV ; - -return crypto_register_skcipher (&alg); -} - -static void __exit chacha20_simd_mod_fini(void) -{ -crypto_unregister_skcipher (&alg); -} - -module_init (chacha20_simd_mod_init); -module_exit (chacha20_simd_mod_fini); - -MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" ); -MODULE_LICENSE ("GPL v2" ); -MODULE_ALIAS_CRYPTO ("chacha20" );arch/arm64/configs/defconfig +0
-1@@ -709,5 +709,4 @@ CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m CONFIG_CRYPTO_CRC32_ARM64_CE=m CONFIG_CRYPTO_AES_ARM64_CE_CCM=y CONFIG_CRYPTO_AES_ARM64_CE_BLK=y -CONFIG_CRYPTO_CHACHA20_NEON=m CONFIG_CRYPTO_AES_ARM64_BS=march/arm64/crypto/Kconfig +0
-6@@ -105,12 +105,6 @@ config CRYPTO_AES_ARM64_NEON_BLK select CRYPTO_AES select CRYPTO_SIMD -config CRYPTO_CHACHA20_NEON - tristate "NEON accelerated ChaCha20 symmetric cipher" - depends on KERNEL_MODE_NEON - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - config CRYPTO_AES_ARM64_BS tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm" depends on KERNEL_MODE_NEONarch/arm64/crypto/Makefile +0
-3@@ -53,9 +53,6 @@ sha256-arm64-y := sha256-glue.o sha256-core.o obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o sha512-arm64-y := sha512-glue.o sha512-core.o -obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o -chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o - obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o speck-neon-y := speck-neon-core.o speck-neon-glue.oarch/arm64/crypto/chacha20-neon-core.S deleted +0
-450@@ -1,450 +0,0 @@ -/* - * ChaCha20 256-bit cipher algorithm, RFC7539, arm64 NEON functions - * - * Copyright (C) 2016 Linaro, Ltd. <[email protected]> - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * Based on: - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSSE3 functions - * - * Copyright (C) 2015 Martin Willi - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include <linux/linkage.h> - - .text - .align 6 - -ENTRY(chacha20_block_xor_neon) - // x0: Input state matrix, s - // x1: 1 data block output, o - // x2: 1 data block input, i - - // - // This function encrypts one ChaCha20 block by loading the state matrix - // in four NEON registers. It performs matrix operation on four words in - // parallel, but requires shuffling to rearrange the words after each - // round. - // - - // x0..3 = s0..3 - adr x3, ROT8 - ld1 {v0.4s-v3.4s}, [x0] - ld1 {v8.4s-v11.4s}, [x0] - ld1 {v12.4s}, [x3] - - mov x3, #10 - -.Ldoubleround: - // x0 += x1, x3 = rotl32(x3 ^ x0, 16) - add v0.4s, v0.4s, v1.4s - eor v3.16b, v3.16b, v0.16b - rev32 v3.8h, v3.8h - - // x2 += x3, x1 = rotl32(x1 ^ x2, 12) - add v2.4s, v2.4s, v3.4s - eor v4.16b, v1.16b, v2.16b - shl v1.4s, v4.4s, #12 - sri v1.4s, v4.4s, #20 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 8) - add v0.4s, v0.4s, v1.4s - eor v3.16b, v3.16b, v0.16b - tbl v3.16b, {v3.16b}, v12.16b - - // x2 += x3, x1 = rotl32(x1 ^ x2, 7) - add v2.4s, v2.4s, v3.4s - eor v4.16b, v1.16b, v2.16b - shl v1.4s, v4.4s, #7 - sri v1.4s, v4.4s, #25 - - // x1 = shuffle32(x1, MASK(0, 3, 2, 1)) - ext v1.16b, v1.16b, v1.16b, #4 - // x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - ext v2.16b, v2.16b, v2.16b, #8 - // x3 = shuffle32(x3, MASK(2, 1, 0, 3)) - ext v3.16b, v3.16b, v3.16b, #12 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 16) - add v0.4s, v0.4s, v1.4s - eor v3.16b, v3.16b, v0.16b - rev32 v3.8h, v3.8h - - // x2 += x3, x1 = rotl32(x1 ^ x2, 12) - add v2.4s, v2.4s, v3.4s - eor v4.16b, v1.16b, v2.16b - shl v1.4s, v4.4s, #12 - sri v1.4s, v4.4s, #20 - - // x0 += x1, x3 = rotl32(x3 ^ x0, 8) - add v0.4s, v0.4s, v1.4s - eor v3.16b, v3.16b, v0.16b - tbl v3.16b, {v3.16b}, v12.16b - - // x2 += x3, x1 = rotl32(x1 ^ x2, 7) - add v2.4s, v2.4s, v3.4s - eor v4.16b, v1.16b, v2.16b - shl v1.4s, v4.4s, #7 - sri v1.4s, v4.4s, #25 - - // x1 = shuffle32(x1, MASK(2, 1, 0, 3)) - ext v1.16b, v1.16b, v1.16b, #12 - // x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - ext v2.16b, v2.16b, v2.16b, #8 - // x3 = shuffle32(x3, MASK(0, 3, 2, 1)) - ext v3.16b, v3.16b, v3.16b, #4 - - subs x3, x3, #1 - b.ne .Ldoubleround - - ld1 {v4.16b-v7.16b}, [x2] - - // o0 = i0 ^ (x0 + s0) - add v0.4s, v0.4s, v8.4s - eor v0.16b, v0.16b, v4.16b - - // o1 = i1 ^ (x1 + s1) - add v1.4s, v1.4s, v9.4s - eor v1.16b, v1.16b, v5.16b - - // o2 = i2 ^ (x2 + s2) - add v2.4s, v2.4s, v10.4s - eor v2.16b, v2.16b, v6.16b - - // o3 = i3 ^ (x3 + s3) - add v3.4s, v3.4s, v11.4s - eor v3.16b, v3.16b, v7.16b - - st1 {v0.16b-v3.16b}, [x1] - - ret -ENDPROC(chacha20_block_xor_neon) - - .align 6 -ENTRY(chacha20_4block_xor_neon) - // x0: Input state matrix, s - // x1: 4 data blocks output, o - // x2: 4 data blocks input, i - - // - // This function encrypts four consecutive ChaCha20 blocks by loading - // the state matrix in NEON registers four times. The algorithm performs - // each operation on the corresponding word of each state matrix, hence - // requires no word shuffling. For final XORing step we transpose the - // matrix by interleaving 32- and then 64-bit words, which allows us to - // do XOR in NEON registers. - // - adr x3, CTRINC // ... and ROT8 - ld1 {v30.4s-v31.4s}, [x3] - - // x0..15[0-3] = s0..3[0..3] - mov x4, x0 - ld4r { v0.4s- v3.4s}, [x4], #16 - ld4r { v4.4s- v7.4s}, [x4], #16 - ld4r { v8.4s-v11.4s}, [x4], #16 - ld4r {v12.4s-v15.4s}, [x4] - - // x12 += counter values 0-3 - add v12.4s, v12.4s, v30.4s - - mov x3, #10 - -.Ldoubleround4: - // x0 += x4, x12 = rotl32(x12 ^ x0, 16) - // x1 += x5, x13 = rotl32(x13 ^ x1, 16) - // x2 += x6, x14 = rotl32(x14 ^ x2, 16) - // x3 += x7, x15 = rotl32(x15 ^ x3, 16) - add v0.4s, v0.4s, v4.4s - add v1.4s, v1.4s, v5.4s - add v2.4s, v2.4s, v6.4s - add v3.4s, v3.4s, v7.4s - - eor v12.16b, v12.16b, v0.16b - eor v13.16b, v13.16b, v1.16b - eor v14.16b, v14.16b, v2.16b - eor v15.16b, v15.16b, v3.16b - - rev32 v12.8h, v12.8h - rev32 v13.8h, v13.8h - rev32 v14.8h, v14.8h - rev32 v15.8h, v15.8h - - // x8 += x12, x4 = rotl32(x4 ^ x8, 12) - // x9 += x13, x5 = rotl32(x5 ^ x9, 12) - // x10 += x14, x6 = rotl32(x6 ^ x10, 12) - // x11 += x15, x7 = rotl32(x7 ^ x11, 12) - add v8.4s, v8.4s, v12.4s - add v9.4s, v9.4s, v13.4s - add v10.4s, v10.4s, v14.4s - add v11.4s, v11.4s, v15.4s - - eor v16.16b, v4.16b, v8.16b - eor v17.16b, v5.16b, v9.16b - eor v18.16b, v6.16b, v10.16b - eor v19.16b, v7.16b, v11.16b - - shl v4.4s, v16.4s, #12 - shl v5.4s, v17.4s, #12 - shl v6.4s, v18.4s, #12 - shl v7.4s, v19.4s, #12 - - sri v4.4s, v16.4s, #20 - sri v5.4s, v17.4s, #20 - sri v6.4s, v18.4s, #20 - sri v7.4s, v19.4s, #20 - - // x0 += x4, x12 = rotl32(x12 ^ x0, 8) - // x1 += x5, x13 = rotl32(x13 ^ x1, 8) - // x2 += x6, x14 = rotl32(x14 ^ x2, 8) - // x3 += x7, x15 = rotl32(x15 ^ x3, 8) - add v0.4s, v0.4s, v4.4s - add v1.4s, v1.4s, v5.4s - add v2.4s, v2.4s, v6.4s - add v3.4s, v3.4s, v7.4s - - eor v12.16b, v12.16b, v0.16b - eor v13.16b, v13.16b, v1.16b - eor v14.16b, v14.16b, v2.16b - eor v15.16b, v15.16b, v3.16b - - tbl v12.16b, {v12.16b}, v31.16b - tbl v13.16b, {v13.16b}, v31.16b - tbl v14.16b, {v14.16b}, v31.16b - tbl v15.16b, {v15.16b}, v31.16b - - // x8 += x12, x4 = rotl32(x4 ^ x8, 7) - // x9 += x13, x5 = rotl32(x5 ^ x9, 7) - // x10 += x14, x6 = rotl32(x6 ^ x10, 7) - // x11 += x15, x7 = rotl32(x7 ^ x11, 7) - add v8.4s, v8.4s, v12.4s - add v9.4s, v9.4s, v13.4s - add v10.4s, v10.4s, v14.4s - add v11.4s, v11.4s, v15.4s - - eor v16.16b, v4.16b, v8.16b - eor v17.16b, v5.16b, v9.16b - eor v18.16b, v6.16b, v10.16b - eor v19.16b, v7.16b, v11.16b - - shl v4.4s, v16.4s, #7 - shl v5.4s, v17.4s, #7 - shl v6.4s, v18.4s, #7 - shl v7.4s, v19.4s, #7 - - sri v4.4s, v16.4s, #25 - sri v5.4s, v17.4s, #25 - sri v6.4s, v18.4s, #25 - sri v7.4s, v19.4s, #25 - - // x0 += x5, x15 = rotl32(x15 ^ x0, 16) - // x1 += x6, x12 = rotl32(x12 ^ x1, 16) - // x2 += x7, x13 = rotl32(x13 ^ x2, 16) - // x3 += x4, x14 = rotl32(x14 ^ x3, 16) - add v0.4s, v0.4s, v5.4s - add v1.4s, v1.4s, v6.4s - add v2.4s, v2.4s, v7.4s - add v3.4s, v3.4s, v4.4s - - eor v15.16b, v15.16b, v0.16b - eor v12.16b, v12.16b, v1.16b - eor v13.16b, v13.16b, v2.16b - eor v14.16b, v14.16b, v3.16b - - rev32 v15.8h, v15.8h - rev32 v12.8h, v12.8h - rev32 v13.8h, v13.8h - rev32 v14.8h, v14.8h - - // x10 += x15, x5 = rotl32(x5 ^ x10, 12) - // x11 += x12, x6 = rotl32(x6 ^ x11, 12) - // x8 += x13, x7 = rotl32(x7 ^ x8, 12) - // x9 += x14, x4 = rotl32(x4 ^ x9, 12) - add v10.4s, v10.4s, v15.4s - add v11.4s, v11.4s, v12.4s - add v8.4s, v8.4s, v13.4s - add v9.4s, v9.4s, v14.4s - - eor v16.16b, v5.16b, v10.16b - eor v17.16b, v6.16b, v11.16b - eor v18.16b, v7.16b, v8.16b - eor v19.16b, v4.16b, v9.16b - - shl v5.4s, v16.4s, #12 - shl v6.4s, v17.4s, #12 - shl v7.4s, v18.4s, #12 - shl v4.4s, v19.4s, #12 - - sri v5.4s, v16.4s, #20 - sri v6.4s, v17.4s, #20 - sri v7.4s, v18.4s, #20 - sri v4.4s, v19.4s, #20 - - // x0 += x5, x15 = rotl32(x15 ^ x0, 8) - // x1 += x6, x12 = rotl32(x12 ^ x1, 8) - // x2 += x7, x13 = rotl32(x13 ^ x2, 8) - // x3 += x4, x14 = rotl32(x14 ^ x3, 8) - add v0.4s, v0.4s, v5.4s - add v1.4s, v1.4s, v6.4s - add v2.4s, v2.4s, v7.4s - add v3.4s, v3.4s, v4.4s - - eor v15.16b, v15.16b, v0.16b - eor v12.16b, v12.16b, v1.16b - eor v13.16b, v13.16b, v2.16b - eor v14.16b, v14.16b, v3.16b - - tbl v15.16b, {v15.16b}, v31.16b - tbl v12.16b, {v12.16b}, v31.16b - tbl v13.16b, {v13.16b}, v31.16b - tbl v14.16b, {v14.16b}, v31.16b - - // x10 += x15, x5 = rotl32(x5 ^ x10, 7) - // x11 += x12, x6 = rotl32(x6 ^ x11, 7) - // x8 += x13, x7 = rotl32(x7 ^ x8, 7) - // x9 += x14, x4 = rotl32(x4 ^ x9, 7) - add v10.4s, v10.4s, v15.4s - add v11.4s, v11.4s, v12.4s - add v8.4s, v8.4s, v13.4s - add v9.4s, v9.4s, v14.4s - - eor v16.16b, v5.16b, v10.16b - eor v17.16b, v6.16b, v11.16b - eor v18.16b, v7.16b, v8.16b - eor v19.16b, v4.16b, v9.16b - - shl v5.4s, v16.4s, #7 - shl v6.4s, v17.4s, #7 - shl v7.4s, v18.4s, #7 - shl v4.4s, v19.4s, #7 - - sri v5.4s, v16.4s, #25 - sri v6.4s, v17.4s, #25 - sri v7.4s, v18.4s, #25 - sri v4.4s, v19.4s, #25 - - subs x3, x3, #1 - b.ne .Ldoubleround4 - - ld4r {v16.4s-v19.4s}, [x0], #16 - ld4r {v20.4s-v23.4s}, [x0], #16 - - // x12 += counter values 0-3 - add v12.4s, v12.4s, v30.4s - - // x0[0-3] += s0[0] - // x1[0-3] += s0[1] - // x2[0-3] += s0[2] - // x3[0-3] += s0[3] - add v0.4s, v0.4s, v16.4s - add v1.4s, v1.4s, v17.4s - add v2.4s, v2.4s, v18.4s - add v3.4s, v3.4s, v19.4s - - ld4r {v24.4s-v27.4s}, [x0], #16 - ld4r {v28.4s-v31.4s}, [x0] - - // x4[0-3] += s1[0] - // x5[0-3] += s1[1] - // x6[0-3] += s1[2] - // x7[0-3] += s1[3] - add v4.4s, v4.4s, v20.4s - add v5.4s, v5.4s, v21.4s - add v6.4s, v6.4s, v22.4s - add v7.4s, v7.4s, v23.4s - - // x8[0-3] += s2[0] - // x9[0-3] += s2[1] - // x10[0-3] += s2[2] - // x11[0-3] += s2[3] - add v8.4s, v8.4s, v24.4s - add v9.4s, v9.4s, v25.4s - add v10.4s, v10.4s, v26.4s - add v11.4s, v11.4s, v27.4s - - // x12[0-3] += s3[0] - // x13[0-3] += s3[1] - // x14[0-3] += s3[2] - // x15[0-3] += s3[3] - add v12.4s, v12.4s, v28.4s - add v13.4s, v13.4s, v29.4s - add v14.4s, v14.4s, v30.4s - add v15.4s, v15.4s, v31.4s - - // interleave 32-bit words in state n, n+1 - zip1 v16.4s, v0.4s, v1.4s - zip2 v17.4s, v0.4s, v1.4s - zip1 v18.4s, v2.4s, v3.4s - zip2 v19.4s, v2.4s, v3.4s - zip1 v20.4s, v4.4s, v5.4s - zip2 v21.4s, v4.4s, v5.4s - zip1 v22.4s, v6.4s, v7.4s - zip2 v23.4s, v6.4s, v7.4s - zip1 v24.4s, v8.4s, v9.4s - zip2 v25.4s, v8.4s, v9.4s - zip1 v26.4s, v10.4s, v11.4s - zip2 v27.4s, v10.4s, v11.4s - zip1 v28.4s, v12.4s, v13.4s - zip2 v29.4s, v12.4s, v13.4s - zip1 v30.4s, v14.4s, v15.4s - zip2 v31.4s, v14.4s, v15.4s - - // interleave 64-bit words in state n, n+2 - zip1 v0.2d, v16.2d, v18.2d - zip2 v4.2d, v16.2d, v18.2d - zip1 v8.2d, v17.2d, v19.2d - zip2 v12.2d, v17.2d, v19.2d - ld1 {v16.16b-v19.16b}, [x2], #64 - - zip1 v1.2d, v20.2d, v22.2d - zip2 v5.2d, v20.2d, v22.2d - zip1 v9.2d, v21.2d, v23.2d - zip2 v13.2d, v21.2d, v23.2d - ld1 {v20.16b-v23.16b}, [x2], #64 - - zip1 v2.2d, v24.2d, v26.2d - zip2 v6.2d, v24.2d, v26.2d - zip1 v10.2d, v25.2d, v27.2d - zip2 v14.2d, v25.2d, v27.2d - ld1 {v24.16b-v27.16b}, [x2], #64 - - zip1 v3.2d, v28.2d, v30.2d - zip2 v7.2d, v28.2d, v30.2d - zip1 v11.2d, v29.2d, v31.2d - zip2 v15.2d, v29.2d, v31.2d - ld1 {v28.16b-v31.16b}, [x2] - - // xor with corresponding input, write to output - eor v16.16b, v16.16b, v0.16b - eor v17.16b, v17.16b, v1.16b - eor v18.16b, v18.16b, v2.16b - eor v19.16b, v19.16b, v3.16b - eor v20.16b, v20.16b, v4.16b - eor v21.16b, v21.16b, v5.16b - st1 {v16.16b-v19.16b}, [x1], #64 - eor v22.16b, v22.16b, v6.16b - eor v23.16b, v23.16b, v7.16b - eor v24.16b, v24.16b, v8.16b - eor v25.16b, v25.16b, v9.16b - st1 {v20.16b-v23.16b}, [x1], #64 - eor v26.16b, v26.16b, v10.16b - eor v27.16b, v27.16b, v11.16b - eor v28.16b, v28.16b, v12.16b - st1 {v24.16b-v27.16b}, [x1], #64 - eor v29.16b, v29.16b, v13.16b - eor v30.16b, v30.16b, v14.16b - eor v31.16b, v31.16b, v15.16b - st1 {v28.16b-v31.16b}, [x1] - - ret -ENDPROC(chacha20_4block_xor_neon) - -CTRINC: .word 0, 1, 2, 3 -ROT8: .word 0x02010003, 0x06050407, 0x0a09080b, 0x0e0d0c0farch/arm64/crypto/chacha20-neon-glue.c deleted +0
-133@@ -1,133 +0,0 @@ -/* -* ChaCha20 256-bit cipher algorithm, RFC7539, arm64 NEON functions -* -* Copyright (C) 2016 - 2017 Linaro, Ltd. <[email protected]> -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -* -* Based on: -* ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code -* -* Copyright (C) 2015 Martin Willi -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License as published by -* the Free Software Foundation; either version 2 of the License, or -* (at your option) any later version. -*/ - -#include <crypto/algapi.h> -#include <crypto/chacha20.h> -#include <crypto/internal/skcipher.h> -#include <linux/kernel.h> -#include <linux/module.h> - -#include <asm/hwcap.h> -#include <asm/neon.h> -#include <asm/simd.h> - -asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src); -asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src); - -static void chacha20_doneon (u32 *state,u8 *dst,const u8 *src, -unsigned int bytes) -{ -u8 buf[CHACHA20_BLOCK_SIZE ]; - -while (bytes >=CHACHA20_BLOCK_SIZE *4 ) { -kernel_neon_begin (); -chacha20_4block_xor_neon (state, dst, src); -kernel_neon_end (); - bytes -=CHACHA20_BLOCK_SIZE *4 ; - src +=CHACHA20_BLOCK_SIZE *4 ; - dst +=CHACHA20_BLOCK_SIZE *4 ; - state[12 ] +=4 ; - } - -if (!bytes) -return ; - -kernel_neon_begin (); -while (bytes >=CHACHA20_BLOCK_SIZE ) { -chacha20_block_xor_neon (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE ; - src +=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; - state[12 ]++; - } -if (bytes) { -memcpy (buf, src, bytes); -chacha20_block_xor_neon (state, buf, buf); -memcpy (dst, buf, bytes); - } -kernel_neon_end (); -} - -static int chacha20_neon (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -u32 state[16 ]; -int err; - -if (!may_use_simd () || req->cryptlen <=CHACHA20_BLOCK_SIZE ) -return crypto_chacha20_crypt (req); - - err =skcipher_walk_virt (&walk, req, false); - -crypto_chacha20_init (state, ctx, walk.iv); - -while (walk.nbytes >0 ) { -unsigned int nbytes = walk.nbytes; - -if (nbytes < walk.total) - nbytes =round_down (nbytes, walk.stride); - -chacha20_doneon (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); - err =skcipher_walk_done (&walk, walk.nbytes - nbytes); - } - -return err; -} - -static struct skcipher_alg alg = { - .base.cra_name ="chacha20" , - .base.cra_driver_name ="chacha20-neon" , - .base.cra_priority =300 , - .base.cra_blocksize =1 , - .base.cra_ctxsize =sizeof (struct chacha20_ctx ), - .base.cra_module =THIS_MODULE , - - .min_keysize =CHACHA20_KEY_SIZE , - .max_keysize =CHACHA20_KEY_SIZE , - .ivsize =CHACHA20_IV_SIZE , - .chunksize =CHACHA20_BLOCK_SIZE , - .walksize =4 *CHACHA20_BLOCK_SIZE , - .setkey = crypto_chacha20_setkey, - .encrypt = chacha20_neon, - .decrypt = chacha20_neon, -}; - -static int __init chacha20_simd_mod_init (void ) -{ -if (!(elf_hwcap &HWCAP_ASIMD )) -return -ENODEV ; - -return crypto_register_skcipher (&alg); -} - -static void __exit chacha20_simd_mod_fini(void) -{ -crypto_unregister_skcipher (&alg); -} - -module_init (chacha20_simd_mod_init); -module_exit (chacha20_simd_mod_fini); - -MODULE_AUTHOR ("Ard Biesheuvel <[email protected]>" ); -MODULE_LICENSE ("GPL v2" ); -MODULE_ALIAS_CRYPTO ("chacha20" );arch/x86/crypto/Makefile +0
-3@@ -23,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o -obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o@@ -76,7 +75,6 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o -chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o@@ -99,7 +97,6 @@ endif ifeq ($(avx2_supported),yes) camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o - chacha20-x86_64-y += chacha20-avx2-x86_64.o serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.oarch/x86/crypto/chacha20-avx2-x86_64.S deleted +0
-448@@ -1,448 +0,0 @@ -/* - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 AVX2 functions - * - * Copyright (C) 2015 Martin Willi - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include <linux/linkage.h> - -.section .rodata.cst32.ROT8, "aM", @progbits, 32 -.align 32 -ROT8: .octa 0x0e0d0c0f0a09080b0605040702010003 - .octa 0x0e0d0c0f0a09080b0605040702010003 - -.section .rodata.cst32.ROT16, "aM", @progbits, 32 -.align 32 -ROT16: .octa 0x0d0c0f0e09080b0a0504070601000302 - .octa 0x0d0c0f0e09080b0a0504070601000302 - -.section .rodata.cst32.CTRINC, "aM", @progbits, 32 -.align 32 -CTRINC: .octa 0x00000003000000020000000100000000 - .octa 0x00000007000000060000000500000004 - -.text - -ENTRY(chacha20_8block_xor_avx2) - # %rdi: Input state matrix, s - # %rsi: 8 data blocks output, o - # %rdx: 8 data blocks input, i - - # This function encrypts eight consecutive ChaCha20 blocks by loading - # the state matrix in AVX registers eight times. As we need some - # scratch registers, we save the first four registers on the stack. The - # algorithm performs each operation on the corresponding word of each - # state matrix, hence requires no word shuffling. For final XORing step - # we transpose the matrix by interleaving 32-, 64- and then 128-bit - # words, which allows us to do XOR in AVX registers. 8/16-bit word - # rotation is done with the slightly better performing byte shuffling, - # 7/12-bit word rotation uses traditional shift+OR. - - vzeroupper - # 4 * 32 byte stack, 32-byte aligned - lea 8(%rsp),%r10 - and $~31, %rsp - sub $0x80, %rsp - - # x0..15[0-7] = s[0..15] - vpbroadcastd 0x00(%rdi),%ymm0 - vpbroadcastd 0x04(%rdi),%ymm1 - vpbroadcastd 0x08(%rdi),%ymm2 - vpbroadcastd 0x0c(%rdi),%ymm3 - vpbroadcastd 0x10(%rdi),%ymm4 - vpbroadcastd 0x14(%rdi),%ymm5 - vpbroadcastd 0x18(%rdi),%ymm6 - vpbroadcastd 0x1c(%rdi),%ymm7 - vpbroadcastd 0x20(%rdi),%ymm8 - vpbroadcastd 0x24(%rdi),%ymm9 - vpbroadcastd 0x28(%rdi),%ymm10 - vpbroadcastd 0x2c(%rdi),%ymm11 - vpbroadcastd 0x30(%rdi),%ymm12 - vpbroadcastd 0x34(%rdi),%ymm13 - vpbroadcastd 0x38(%rdi),%ymm14 - vpbroadcastd 0x3c(%rdi),%ymm15 - # x0..3 on stack - vmovdqa %ymm0,0x00(%rsp) - vmovdqa %ymm1,0x20(%rsp) - vmovdqa %ymm2,0x40(%rsp) - vmovdqa %ymm3,0x60(%rsp) - - vmovdqa CTRINC(%rip),%ymm1 - vmovdqa ROT8(%rip),%ymm2 - vmovdqa ROT16(%rip),%ymm3 - - # x12 += counter values 0-3 - vpaddd %ymm1,%ymm12,%ymm12 - - mov $10,%ecx - -.Ldoubleround8: - # x0 += x4, x12 = rotl32(x12 ^ x0, 16) - vpaddd 0x00(%rsp),%ymm4,%ymm0 - vmovdqa %ymm0,0x00(%rsp) - vpxor %ymm0,%ymm12,%ymm12 - vpshufb %ymm3,%ymm12,%ymm12 - # x1 += x5, x13 = rotl32(x13 ^ x1, 16) - vpaddd 0x20(%rsp),%ymm5,%ymm0 - vmovdqa %ymm0,0x20(%rsp) - vpxor %ymm0,%ymm13,%ymm13 - vpshufb %ymm3,%ymm13,%ymm13 - # x2 += x6, x14 = rotl32(x14 ^ x2, 16) - vpaddd 0x40(%rsp),%ymm6,%ymm0 - vmovdqa %ymm0,0x40(%rsp) - vpxor %ymm0,%ymm14,%ymm14 - vpshufb %ymm3,%ymm14,%ymm14 - # x3 += x7, x15 = rotl32(x15 ^ x3, 16) - vpaddd 0x60(%rsp),%ymm7,%ymm0 - vmovdqa %ymm0,0x60(%rsp) - vpxor %ymm0,%ymm15,%ymm15 - vpshufb %ymm3,%ymm15,%ymm15 - - # x8 += x12, x4 = rotl32(x4 ^ x8, 12) - vpaddd %ymm12,%ymm8,%ymm8 - vpxor %ymm8,%ymm4,%ymm4 - vpslld $12,%ymm4,%ymm0 - vpsrld $20,%ymm4,%ymm4 - vpor %ymm0,%ymm4,%ymm4 - # x9 += x13, x5 = rotl32(x5 ^ x9, 12) - vpaddd %ymm13,%ymm9,%ymm9 - vpxor %ymm9,%ymm5,%ymm5 - vpslld $12,%ymm5,%ymm0 - vpsrld $20,%ymm5,%ymm5 - vpor %ymm0,%ymm5,%ymm5 - # x10 += x14, x6 = rotl32(x6 ^ x10, 12) - vpaddd %ymm14,%ymm10,%ymm10 - vpxor %ymm10,%ymm6,%ymm6 - vpslld $12,%ymm6,%ymm0 - vpsrld $20,%ymm6,%ymm6 - vpor %ymm0,%ymm6,%ymm6 - # x11 += x15, x7 = rotl32(x7 ^ x11, 12) - vpaddd %ymm15,%ymm11,%ymm11 - vpxor %ymm11,%ymm7,%ymm7 - vpslld $12,%ymm7,%ymm0 - vpsrld $20,%ymm7,%ymm7 - vpor %ymm0,%ymm7,%ymm7 - - # x0 += x4, x12 = rotl32(x12 ^ x0, 8) - vpaddd 0x00(%rsp),%ymm4,%ymm0 - vmovdqa %ymm0,0x00(%rsp) - vpxor %ymm0,%ymm12,%ymm12 - vpshufb %ymm2,%ymm12,%ymm12 - # x1 += x5, x13 = rotl32(x13 ^ x1, 8) - vpaddd 0x20(%rsp),%ymm5,%ymm0 - vmovdqa %ymm0,0x20(%rsp) - vpxor %ymm0,%ymm13,%ymm13 - vpshufb %ymm2,%ymm13,%ymm13 - # x2 += x6, x14 = rotl32(x14 ^ x2, 8) - vpaddd 0x40(%rsp),%ymm6,%ymm0 - vmovdqa %ymm0,0x40(%rsp) - vpxor %ymm0,%ymm14,%ymm14 - vpshufb %ymm2,%ymm14,%ymm14 - # x3 += x7, x15 = rotl32(x15 ^ x3, 8) - vpaddd 0x60(%rsp),%ymm7,%ymm0 - vmovdqa %ymm0,0x60(%rsp) - vpxor %ymm0,%ymm15,%ymm15 - vpshufb %ymm2,%ymm15,%ymm15 - - # x8 += x12, x4 = rotl32(x4 ^ x8, 7) - vpaddd %ymm12,%ymm8,%ymm8 - vpxor %ymm8,%ymm4,%ymm4 - vpslld $7,%ymm4,%ymm0 - vpsrld $25,%ymm4,%ymm4 - vpor %ymm0,%ymm4,%ymm4 - # x9 += x13, x5 = rotl32(x5 ^ x9, 7) - vpaddd %ymm13,%ymm9,%ymm9 - vpxor %ymm9,%ymm5,%ymm5 - vpslld $7,%ymm5,%ymm0 - vpsrld $25,%ymm5,%ymm5 - vpor %ymm0,%ymm5,%ymm5 - # x10 += x14, x6 = rotl32(x6 ^ x10, 7) - vpaddd %ymm14,%ymm10,%ymm10 - vpxor %ymm10,%ymm6,%ymm6 - vpslld $7,%ymm6,%ymm0 - vpsrld $25,%ymm6,%ymm6 - vpor %ymm0,%ymm6,%ymm6 - # x11 += x15, x7 = rotl32(x7 ^ x11, 7) - vpaddd %ymm15,%ymm11,%ymm11 - vpxor %ymm11,%ymm7,%ymm7 - vpslld $7,%ymm7,%ymm0 - vpsrld $25,%ymm7,%ymm7 - vpor %ymm0,%ymm7,%ymm7 - - # x0 += x5, x15 = rotl32(x15 ^ x0, 16) - vpaddd 0x00(%rsp),%ymm5,%ymm0 - vmovdqa %ymm0,0x00(%rsp) - vpxor %ymm0,%ymm15,%ymm15 - vpshufb %ymm3,%ymm15,%ymm15 - # x1 += x6, x12 = rotl32(x12 ^ x1, 16)%ymm0 - vpaddd 0x20(%rsp),%ymm6,%ymm0 - vmovdqa %ymm0,0x20(%rsp) - vpxor %ymm0,%ymm12,%ymm12 - vpshufb %ymm3,%ymm12,%ymm12 - # x2 += x7, x13 = rotl32(x13 ^ x2, 16) - vpaddd 0x40(%rsp),%ymm7,%ymm0 - vmovdqa %ymm0,0x40(%rsp) - vpxor %ymm0,%ymm13,%ymm13 - vpshufb %ymm3,%ymm13,%ymm13 - # x3 += x4, x14 = rotl32(x14 ^ x3, 16) - vpaddd 0x60(%rsp),%ymm4,%ymm0 - vmovdqa %ymm0,0x60(%rsp) - vpxor %ymm0,%ymm14,%ymm14 - vpshufb %ymm3,%ymm14,%ymm14 - - # x10 += x15, x5 = rotl32(x5 ^ x10, 12) - vpaddd %ymm15,%ymm10,%ymm10 - vpxor %ymm10,%ymm5,%ymm5 - vpslld $12,%ymm5,%ymm0 - vpsrld $20,%ymm5,%ymm5 - vpor %ymm0,%ymm5,%ymm5 - # x11 += x12, x6 = rotl32(x6 ^ x11, 12) - vpaddd %ymm12,%ymm11,%ymm11 - vpxor %ymm11,%ymm6,%ymm6 - vpslld $12,%ymm6,%ymm0 - vpsrld $20,%ymm6,%ymm6 - vpor %ymm0,%ymm6,%ymm6 - # x8 += x13, x7 = rotl32(x7 ^ x8, 12) - vpaddd %ymm13,%ymm8,%ymm8 - vpxor %ymm8,%ymm7,%ymm7 - vpslld $12,%ymm7,%ymm0 - vpsrld $20,%ymm7,%ymm7 - vpor %ymm0,%ymm7,%ymm7 - # x9 += x14, x4 = rotl32(x4 ^ x9, 12) - vpaddd %ymm14,%ymm9,%ymm9 - vpxor %ymm9,%ymm4,%ymm4 - vpslld $12,%ymm4,%ymm0 - vpsrld $20,%ymm4,%ymm4 - vpor %ymm0,%ymm4,%ymm4 - - # x0 += x5, x15 = rotl32(x15 ^ x0, 8) - vpaddd 0x00(%rsp),%ymm5,%ymm0 - vmovdqa %ymm0,0x00(%rsp) - vpxor %ymm0,%ymm15,%ymm15 - vpshufb %ymm2,%ymm15,%ymm15 - # x1 += x6, x12 = rotl32(x12 ^ x1, 8) - vpaddd 0x20(%rsp),%ymm6,%ymm0 - vmovdqa %ymm0,0x20(%rsp) - vpxor %ymm0,%ymm12,%ymm12 - vpshufb %ymm2,%ymm12,%ymm12 - # x2 += x7, x13 = rotl32(x13 ^ x2, 8) - vpaddd 0x40(%rsp),%ymm7,%ymm0 - vmovdqa %ymm0,0x40(%rsp) - vpxor %ymm0,%ymm13,%ymm13 - vpshufb %ymm2,%ymm13,%ymm13 - # x3 += x4, x14 = rotl32(x14 ^ x3, 8) - vpaddd 0x60(%rsp),%ymm4,%ymm0 - vmovdqa %ymm0,0x60(%rsp) - vpxor %ymm0,%ymm14,%ymm14 - vpshufb %ymm2,%ymm14,%ymm14 - - # x10 += x15, x5 = rotl32(x5 ^ x10, 7) - vpaddd %ymm15,%ymm10,%ymm10 - vpxor %ymm10,%ymm5,%ymm5 - vpslld $7,%ymm5,%ymm0 - vpsrld $25,%ymm5,%ymm5 - vpor %ymm0,%ymm5,%ymm5 - # x11 += x12, x6 = rotl32(x6 ^ x11, 7) - vpaddd %ymm12,%ymm11,%ymm11 - vpxor %ymm11,%ymm6,%ymm6 - vpslld $7,%ymm6,%ymm0 - vpsrld $25,%ymm6,%ymm6 - vpor %ymm0,%ymm6,%ymm6 - # x8 += x13, x7 = rotl32(x7 ^ x8, 7) - vpaddd %ymm13,%ymm8,%ymm8 - vpxor %ymm8,%ymm7,%ymm7 - vpslld $7,%ymm7,%ymm0 - vpsrld $25,%ymm7,%ymm7 - vpor %ymm0,%ymm7,%ymm7 - # x9 += x14, x4 = rotl32(x4 ^ x9, 7) - vpaddd %ymm14,%ymm9,%ymm9 - vpxor %ymm9,%ymm4,%ymm4 - vpslld $7,%ymm4,%ymm0 - vpsrld $25,%ymm4,%ymm4 - vpor %ymm0,%ymm4,%ymm4 - - dec %ecx - jnz .Ldoubleround8 - - # x0..15[0-3] += s[0..15] - vpbroadcastd 0x00(%rdi),%ymm0 - vpaddd 0x00(%rsp),%ymm0,%ymm0 - vmovdqa %ymm0,0x00(%rsp) - vpbroadcastd 0x04(%rdi),%ymm0 - vpaddd 0x20(%rsp),%ymm0,%ymm0 - vmovdqa %ymm0,0x20(%rsp) - vpbroadcastd 0x08(%rdi),%ymm0 - vpaddd 0x40(%rsp),%ymm0,%ymm0 - vmovdqa %ymm0,0x40(%rsp) - vpbroadcastd 0x0c(%rdi),%ymm0 - vpaddd 0x60(%rsp),%ymm0,%ymm0 - vmovdqa %ymm0,0x60(%rsp) - vpbroadcastd 0x10(%rdi),%ymm0 - vpaddd %ymm0,%ymm4,%ymm4 - vpbroadcastd 0x14(%rdi),%ymm0 - vpaddd %ymm0,%ymm5,%ymm5 - vpbroadcastd 0x18(%rdi),%ymm0 - vpaddd %ymm0,%ymm6,%ymm6 - vpbroadcastd 0x1c(%rdi),%ymm0 - vpaddd %ymm0,%ymm7,%ymm7 - vpbroadcastd 0x20(%rdi),%ymm0 - vpaddd %ymm0,%ymm8,%ymm8 - vpbroadcastd 0x24(%rdi),%ymm0 - vpaddd %ymm0,%ymm9,%ymm9 - vpbroadcastd 0x28(%rdi),%ymm0 - vpaddd %ymm0,%ymm10,%ymm10 - vpbroadcastd 0x2c(%rdi),%ymm0 - vpaddd %ymm0,%ymm11,%ymm11 - vpbroadcastd 0x30(%rdi),%ymm0 - vpaddd %ymm0,%ymm12,%ymm12 - vpbroadcastd 0x34(%rdi),%ymm0 - vpaddd %ymm0,%ymm13,%ymm13 - vpbroadcastd 0x38(%rdi),%ymm0 - vpaddd %ymm0,%ymm14,%ymm14 - vpbroadcastd 0x3c(%rdi),%ymm0 - vpaddd %ymm0,%ymm15,%ymm15 - - # x12 += counter values 0-3 - vpaddd %ymm1,%ymm12,%ymm12 - - # interleave 32-bit words in state n, n+1 - vmovdqa 0x00(%rsp),%ymm0 - vmovdqa 0x20(%rsp),%ymm1 - vpunpckldq %ymm1,%ymm0,%ymm2 - vpunpckhdq %ymm1,%ymm0,%ymm1 - vmovdqa %ymm2,0x00(%rsp) - vmovdqa %ymm1,0x20(%rsp) - vmovdqa 0x40(%rsp),%ymm0 - vmovdqa 0x60(%rsp),%ymm1 - vpunpckldq %ymm1,%ymm0,%ymm2 - vpunpckhdq %ymm1,%ymm0,%ymm1 - vmovdqa %ymm2,0x40(%rsp) - vmovdqa %ymm1,0x60(%rsp) - vmovdqa %ymm4,%ymm0 - vpunpckldq %ymm5,%ymm0,%ymm4 - vpunpckhdq %ymm5,%ymm0,%ymm5 - vmovdqa %ymm6,%ymm0 - vpunpckldq %ymm7,%ymm0,%ymm6 - vpunpckhdq %ymm7,%ymm0,%ymm7 - vmovdqa %ymm8,%ymm0 - vpunpckldq %ymm9,%ymm0,%ymm8 - vpunpckhdq %ymm9,%ymm0,%ymm9 - vmovdqa %ymm10,%ymm0 - vpunpckldq %ymm11,%ymm0,%ymm10 - vpunpckhdq %ymm11,%ymm0,%ymm11 - vmovdqa %ymm12,%ymm0 - vpunpckldq %ymm13,%ymm0,%ymm12 - vpunpckhdq %ymm13,%ymm0,%ymm13 - vmovdqa %ymm14,%ymm0 - vpunpckldq %ymm15,%ymm0,%ymm14 - vpunpckhdq %ymm15,%ymm0,%ymm15 - - # interleave 64-bit words in state n, n+2 - vmovdqa 0x00(%rsp),%ymm0 - vmovdqa 0x40(%rsp),%ymm2 - vpunpcklqdq %ymm2,%ymm0,%ymm1 - vpunpckhqdq %ymm2,%ymm0,%ymm2 - vmovdqa %ymm1,0x00(%rsp) - vmovdqa %ymm2,0x40(%rsp) - vmovdqa 0x20(%rsp),%ymm0 - vmovdqa 0x60(%rsp),%ymm2 - vpunpcklqdq %ymm2,%ymm0,%ymm1 - vpunpckhqdq %ymm2,%ymm0,%ymm2 - vmovdqa %ymm1,0x20(%rsp) - vmovdqa %ymm2,0x60(%rsp) - vmovdqa %ymm4,%ymm0 - vpunpcklqdq %ymm6,%ymm0,%ymm4 - vpunpckhqdq %ymm6,%ymm0,%ymm6 - vmovdqa %ymm5,%ymm0 - vpunpcklqdq %ymm7,%ymm0,%ymm5 - vpunpckhqdq %ymm7,%ymm0,%ymm7 - vmovdqa %ymm8,%ymm0 - vpunpcklqdq %ymm10,%ymm0,%ymm8 - vpunpckhqdq %ymm10,%ymm0,%ymm10 - vmovdqa %ymm9,%ymm0 - vpunpcklqdq %ymm11,%ymm0,%ymm9 - vpunpckhqdq %ymm11,%ymm0,%ymm11 - vmovdqa %ymm12,%ymm0 - vpunpcklqdq %ymm14,%ymm0,%ymm12 - vpunpckhqdq %ymm14,%ymm0,%ymm14 - vmovdqa %ymm13,%ymm0 - vpunpcklqdq %ymm15,%ymm0,%ymm13 - vpunpckhqdq %ymm15,%ymm0,%ymm15 - - # interleave 128-bit words in state n, n+4 - vmovdqa 0x00(%rsp),%ymm0 - vperm2i128 $0x20,%ymm4,%ymm0,%ymm1 - vperm2i128 $0x31,%ymm4,%ymm0,%ymm4 - vmovdqa %ymm1,0x00(%rsp) - vmovdqa 0x20(%rsp),%ymm0 - vperm2i128 $0x20,%ymm5,%ymm0,%ymm1 - vperm2i128 $0x31,%ymm5,%ymm0,%ymm5 - vmovdqa %ymm1,0x20(%rsp) - vmovdqa 0x40(%rsp),%ymm0 - vperm2i128 $0x20,%ymm6,%ymm0,%ymm1 - vperm2i128 $0x31,%ymm6,%ymm0,%ymm6 - vmovdqa %ymm1,0x40(%rsp) - vmovdqa 0x60(%rsp),%ymm0 - vperm2i128 $0x20,%ymm7,%ymm0,%ymm1 - vperm2i128 $0x31,%ymm7,%ymm0,%ymm7 - vmovdqa %ymm1,0x60(%rsp) - vperm2i128 $0x20,%ymm12,%ymm8,%ymm0 - vperm2i128 $0x31,%ymm12,%ymm8,%ymm12 - vmovdqa %ymm0,%ymm8 - vperm2i128 $0x20,%ymm13,%ymm9,%ymm0 - vperm2i128 $0x31,%ymm13,%ymm9,%ymm13 - vmovdqa %ymm0,%ymm9 - vperm2i128 $0x20,%ymm14,%ymm10,%ymm0 - vperm2i128 $0x31,%ymm14,%ymm10,%ymm14 - vmovdqa %ymm0,%ymm10 - vperm2i128 $0x20,%ymm15,%ymm11,%ymm0 - vperm2i128 $0x31,%ymm15,%ymm11,%ymm15 - vmovdqa %ymm0,%ymm11 - - # xor with corresponding input, write to output - vmovdqa 0x00(%rsp),%ymm0 - vpxor 0x0000(%rdx),%ymm0,%ymm0 - vmovdqu %ymm0,0x0000(%rsi) - vmovdqa 0x20(%rsp),%ymm0 - vpxor 0x0080(%rdx),%ymm0,%ymm0 - vmovdqu %ymm0,0x0080(%rsi) - vmovdqa 0x40(%rsp),%ymm0 - vpxor 0x0040(%rdx),%ymm0,%ymm0 - vmovdqu %ymm0,0x0040(%rsi) - vmovdqa 0x60(%rsp),%ymm0 - vpxor 0x00c0(%rdx),%ymm0,%ymm0 - vmovdqu %ymm0,0x00c0(%rsi) - vpxor 0x0100(%rdx),%ymm4,%ymm4 - vmovdqu %ymm4,0x0100(%rsi) - vpxor 0x0180(%rdx),%ymm5,%ymm5 - vmovdqu %ymm5,0x00180(%rsi) - vpxor 0x0140(%rdx),%ymm6,%ymm6 - vmovdqu %ymm6,0x0140(%rsi) - vpxor 0x01c0(%rdx),%ymm7,%ymm7 - vmovdqu %ymm7,0x01c0(%rsi) - vpxor 0x0020(%rdx),%ymm8,%ymm8 - vmovdqu %ymm8,0x0020(%rsi) - vpxor 0x00a0(%rdx),%ymm9,%ymm9 - vmovdqu %ymm9,0x00a0(%rsi) - vpxor 0x0060(%rdx),%ymm10,%ymm10 - vmovdqu %ymm10,0x0060(%rsi) - vpxor 0x00e0(%rdx),%ymm11,%ymm11 - vmovdqu %ymm11,0x00e0(%rsi) - vpxor 0x0120(%rdx),%ymm12,%ymm12 - vmovdqu %ymm12,0x0120(%rsi) - vpxor 0x01a0(%rdx),%ymm13,%ymm13 - vmovdqu %ymm13,0x01a0(%rsi) - vpxor 0x0160(%rdx),%ymm14,%ymm14 - vmovdqu %ymm14,0x0160(%rsi) - vpxor 0x01e0(%rdx),%ymm15,%ymm15 - vmovdqu %ymm15,0x01e0(%rsi) - - vzeroupper - lea -8(%r10),%rsp - ret -ENDPROC(chacha20_8block_xor_avx2)arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0
-630@@ -1,630 +0,0 @@ -/* - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSSE3 functions - * - * Copyright (C) 2015 Martin Willi - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include <linux/linkage.h> - -.section .rodata.cst16.ROT8, "aM", @progbits, 16 -.align 16 -ROT8: .octa 0x0e0d0c0f0a09080b0605040702010003 -.section .rodata.cst16.ROT16, "aM", @progbits, 16 -.align 16 -ROT16: .octa 0x0d0c0f0e09080b0a0504070601000302 -.section .rodata.cst16.CTRINC, "aM", @progbits, 16 -.align 16 -CTRINC: .octa 0x00000003000000020000000100000000 - -.text - -ENTRY(chacha20_block_xor_ssse3) - # %rdi: Input state matrix, s - # %rsi: 1 data block output, o - # %rdx: 1 data block input, i - - # This function encrypts one ChaCha20 block by loading the state matrix - # in four SSE registers. It performs matrix operation on four words in - # parallel, but requireds shuffling to rearrange the words after each - # round. 8/16-bit word rotation is done with the slightly better - # performing SSSE3 byte shuffling, 7/12-bit word rotation uses - # traditional shift+OR. - - # x0..3 = s0..3 - movdqa 0x00(%rdi),%xmm0 - movdqa 0x10(%rdi),%xmm1 - movdqa 0x20(%rdi),%xmm2 - movdqa 0x30(%rdi),%xmm3 - movdqa %xmm0,%xmm8 - movdqa %xmm1,%xmm9 - movdqa %xmm2,%xmm10 - movdqa %xmm3,%xmm11 - - movdqa ROT8(%rip),%xmm4 - movdqa ROT16(%rip),%xmm5 - - mov $10,%ecx - -.Ldoubleround: - - # x0 += x1, x3 = rotl32(x3 ^ x0, 16) - paddd %xmm1,%xmm0 - pxor %xmm0,%xmm3 - pshufb %xmm5,%xmm3 - - # x2 += x3, x1 = rotl32(x1 ^ x2, 12) - paddd %xmm3,%xmm2 - pxor %xmm2,%xmm1 - movdqa %xmm1,%xmm6 - pslld $12,%xmm6 - psrld $20,%xmm1 - por %xmm6,%xmm1 - - # x0 += x1, x3 = rotl32(x3 ^ x0, 8) - paddd %xmm1,%xmm0 - pxor %xmm0,%xmm3 - pshufb %xmm4,%xmm3 - - # x2 += x3, x1 = rotl32(x1 ^ x2, 7) - paddd %xmm3,%xmm2 - pxor %xmm2,%xmm1 - movdqa %xmm1,%xmm7 - pslld $7,%xmm7 - psrld $25,%xmm1 - por %xmm7,%xmm1 - - # x1 = shuffle32(x1, MASK(0, 3, 2, 1)) - pshufd $0x39,%xmm1,%xmm1 - # x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - pshufd $0x4e,%xmm2,%xmm2 - # x3 = shuffle32(x3, MASK(2, 1, 0, 3)) - pshufd $0x93,%xmm3,%xmm3 - - # x0 += x1, x3 = rotl32(x3 ^ x0, 16) - paddd %xmm1,%xmm0 - pxor %xmm0,%xmm3 - pshufb %xmm5,%xmm3 - - # x2 += x3, x1 = rotl32(x1 ^ x2, 12) - paddd %xmm3,%xmm2 - pxor %xmm2,%xmm1 - movdqa %xmm1,%xmm6 - pslld $12,%xmm6 - psrld $20,%xmm1 - por %xmm6,%xmm1 - - # x0 += x1, x3 = rotl32(x3 ^ x0, 8) - paddd %xmm1,%xmm0 - pxor %xmm0,%xmm3 - pshufb %xmm4,%xmm3 - - # x2 += x3, x1 = rotl32(x1 ^ x2, 7) - paddd %xmm3,%xmm2 - pxor %xmm2,%xmm1 - movdqa %xmm1,%xmm7 - pslld $7,%xmm7 - psrld $25,%xmm1 - por %xmm7,%xmm1 - - # x1 = shuffle32(x1, MASK(2, 1, 0, 3)) - pshufd $0x93,%xmm1,%xmm1 - # x2 = shuffle32(x2, MASK(1, 0, 3, 2)) - pshufd $0x4e,%xmm2,%xmm2 - # x3 = shuffle32(x3, MASK(0, 3, 2, 1)) - pshufd $0x39,%xmm3,%xmm3 - - dec %ecx - jnz .Ldoubleround - - # o0 = i0 ^ (x0 + s0) - movdqu 0x00(%rdx),%xmm4 - paddd %xmm8,%xmm0 - pxor %xmm4,%xmm0 - movdqu %xmm0,0x00(%rsi) - # o1 = i1 ^ (x1 + s1) - movdqu 0x10(%rdx),%xmm5 - paddd %xmm9,%xmm1 - pxor %xmm5,%xmm1 - movdqu %xmm1,0x10(%rsi) - # o2 = i2 ^ (x2 + s2) - movdqu 0x20(%rdx),%xmm6 - paddd %xmm10,%xmm2 - pxor %xmm6,%xmm2 - movdqu %xmm2,0x20(%rsi) - # o3 = i3 ^ (x3 + s3) - movdqu 0x30(%rdx),%xmm7 - paddd %xmm11,%xmm3 - pxor %xmm7,%xmm3 - movdqu %xmm3,0x30(%rsi) - - ret -ENDPROC(chacha20_block_xor_ssse3) - -ENTRY(chacha20_4block_xor_ssse3) - # %rdi: Input state matrix, s - # %rsi: 4 data blocks output, o - # %rdx: 4 data blocks input, i - - # This function encrypts four consecutive ChaCha20 blocks by loading the - # the state matrix in SSE registers four times. As we need some scratch - # registers, we save the first four registers on the stack. The - # algorithm performs each operation on the corresponding word of each - # state matrix, hence requires no word shuffling. For final XORing step - # we transpose the matrix by interleaving 32- and then 64-bit words, - # which allows us to do XOR in SSE registers. 8/16-bit word rotation is - # done with the slightly better performing SSSE3 byte shuffling, - # 7/12-bit word rotation uses traditional shift+OR. - - lea 8(%rsp),%r10 - sub $0x80,%rsp - and $~63,%rsp - - # x0..15[0-3] = s0..3[0..3] - movq 0x00(%rdi),%xmm1 - pshufd $0x00,%xmm1,%xmm0 - pshufd $0x55,%xmm1,%xmm1 - movq 0x08(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - movq 0x10(%rdi),%xmm5 - pshufd $0x00,%xmm5,%xmm4 - pshufd $0x55,%xmm5,%xmm5 - movq 0x18(%rdi),%xmm7 - pshufd $0x00,%xmm7,%xmm6 - pshufd $0x55,%xmm7,%xmm7 - movq 0x20(%rdi),%xmm9 - pshufd $0x00,%xmm9,%xmm8 - pshufd $0x55,%xmm9,%xmm9 - movq 0x28(%rdi),%xmm11 - pshufd $0x00,%xmm11,%xmm10 - pshufd $0x55,%xmm11,%xmm11 - movq 0x30(%rdi),%xmm13 - pshufd $0x00,%xmm13,%xmm12 - pshufd $0x55,%xmm13,%xmm13 - movq 0x38(%rdi),%xmm15 - pshufd $0x00,%xmm15,%xmm14 - pshufd $0x55,%xmm15,%xmm15 - # x0..3 on stack - movdqa %xmm0,0x00(%rsp) - movdqa %xmm1,0x10(%rsp) - movdqa %xmm2,0x20(%rsp) - movdqa %xmm3,0x30(%rsp) - - movdqa CTRINC(%rip),%xmm1 - movdqa ROT8(%rip),%xmm2 - movdqa ROT16(%rip),%xmm3 - - # x12 += counter values 0-3 - paddd %xmm1,%xmm12 - - mov $10,%ecx - -.Ldoubleround4: - # x0 += x4, x12 = rotl32(x12 ^ x0, 16) - movdqa 0x00(%rsp),%xmm0 - paddd %xmm4,%xmm0 - movdqa %xmm0,0x00(%rsp) - pxor %xmm0,%xmm12 - pshufb %xmm3,%xmm12 - # x1 += x5, x13 = rotl32(x13 ^ x1, 16) - movdqa 0x10(%rsp),%xmm0 - paddd %xmm5,%xmm0 - movdqa %xmm0,0x10(%rsp) - pxor %xmm0,%xmm13 - pshufb %xmm3,%xmm13 - # x2 += x6, x14 = rotl32(x14 ^ x2, 16) - movdqa 0x20(%rsp),%xmm0 - paddd %xmm6,%xmm0 - movdqa %xmm0,0x20(%rsp) - pxor %xmm0,%xmm14 - pshufb %xmm3,%xmm14 - # x3 += x7, x15 = rotl32(x15 ^ x3, 16) - movdqa 0x30(%rsp),%xmm0 - paddd %xmm7,%xmm0 - movdqa %xmm0,0x30(%rsp) - pxor %xmm0,%xmm15 - pshufb %xmm3,%xmm15 - - # x8 += x12, x4 = rotl32(x4 ^ x8, 12) - paddd %xmm12,%xmm8 - pxor %xmm8,%xmm4 - movdqa %xmm4,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm4 - por %xmm0,%xmm4 - # x9 += x13, x5 = rotl32(x5 ^ x9, 12) - paddd %xmm13,%xmm9 - pxor %xmm9,%xmm5 - movdqa %xmm5,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm5 - por %xmm0,%xmm5 - # x10 += x14, x6 = rotl32(x6 ^ x10, 12) - paddd %xmm14,%xmm10 - pxor %xmm10,%xmm6 - movdqa %xmm6,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm6 - por %xmm0,%xmm6 - # x11 += x15, x7 = rotl32(x7 ^ x11, 12) - paddd %xmm15,%xmm11 - pxor %xmm11,%xmm7 - movdqa %xmm7,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm7 - por %xmm0,%xmm7 - - # x0 += x4, x12 = rotl32(x12 ^ x0, 8) - movdqa 0x00(%rsp),%xmm0 - paddd %xmm4,%xmm0 - movdqa %xmm0,0x00(%rsp) - pxor %xmm0,%xmm12 - pshufb %xmm2,%xmm12 - # x1 += x5, x13 = rotl32(x13 ^ x1, 8) - movdqa 0x10(%rsp),%xmm0 - paddd %xmm5,%xmm0 - movdqa %xmm0,0x10(%rsp) - pxor %xmm0,%xmm13 - pshufb %xmm2,%xmm13 - # x2 += x6, x14 = rotl32(x14 ^ x2, 8) - movdqa 0x20(%rsp),%xmm0 - paddd %xmm6,%xmm0 - movdqa %xmm0,0x20(%rsp) - pxor %xmm0,%xmm14 - pshufb %xmm2,%xmm14 - # x3 += x7, x15 = rotl32(x15 ^ x3, 8) - movdqa 0x30(%rsp),%xmm0 - paddd %xmm7,%xmm0 - movdqa %xmm0,0x30(%rsp) - pxor %xmm0,%xmm15 - pshufb %xmm2,%xmm15 - - # x8 += x12, x4 = rotl32(x4 ^ x8, 7) - paddd %xmm12,%xmm8 - pxor %xmm8,%xmm4 - movdqa %xmm4,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm4 - por %xmm0,%xmm4 - # x9 += x13, x5 = rotl32(x5 ^ x9, 7) - paddd %xmm13,%xmm9 - pxor %xmm9,%xmm5 - movdqa %xmm5,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm5 - por %xmm0,%xmm5 - # x10 += x14, x6 = rotl32(x6 ^ x10, 7) - paddd %xmm14,%xmm10 - pxor %xmm10,%xmm6 - movdqa %xmm6,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm6 - por %xmm0,%xmm6 - # x11 += x15, x7 = rotl32(x7 ^ x11, 7) - paddd %xmm15,%xmm11 - pxor %xmm11,%xmm7 - movdqa %xmm7,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm7 - por %xmm0,%xmm7 - - # x0 += x5, x15 = rotl32(x15 ^ x0, 16) - movdqa 0x00(%rsp),%xmm0 - paddd %xmm5,%xmm0 - movdqa %xmm0,0x00(%rsp) - pxor %xmm0,%xmm15 - pshufb %xmm3,%xmm15 - # x1 += x6, x12 = rotl32(x12 ^ x1, 16) - movdqa 0x10(%rsp),%xmm0 - paddd %xmm6,%xmm0 - movdqa %xmm0,0x10(%rsp) - pxor %xmm0,%xmm12 - pshufb %xmm3,%xmm12 - # x2 += x7, x13 = rotl32(x13 ^ x2, 16) - movdqa 0x20(%rsp),%xmm0 - paddd %xmm7,%xmm0 - movdqa %xmm0,0x20(%rsp) - pxor %xmm0,%xmm13 - pshufb %xmm3,%xmm13 - # x3 += x4, x14 = rotl32(x14 ^ x3, 16) - movdqa 0x30(%rsp),%xmm0 - paddd %xmm4,%xmm0 - movdqa %xmm0,0x30(%rsp) - pxor %xmm0,%xmm14 - pshufb %xmm3,%xmm14 - - # x10 += x15, x5 = rotl32(x5 ^ x10, 12) - paddd %xmm15,%xmm10 - pxor %xmm10,%xmm5 - movdqa %xmm5,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm5 - por %xmm0,%xmm5 - # x11 += x12, x6 = rotl32(x6 ^ x11, 12) - paddd %xmm12,%xmm11 - pxor %xmm11,%xmm6 - movdqa %xmm6,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm6 - por %xmm0,%xmm6 - # x8 += x13, x7 = rotl32(x7 ^ x8, 12) - paddd %xmm13,%xmm8 - pxor %xmm8,%xmm7 - movdqa %xmm7,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm7 - por %xmm0,%xmm7 - # x9 += x14, x4 = rotl32(x4 ^ x9, 12) - paddd %xmm14,%xmm9 - pxor %xmm9,%xmm4 - movdqa %xmm4,%xmm0 - pslld $12,%xmm0 - psrld $20,%xmm4 - por %xmm0,%xmm4 - - # x0 += x5, x15 = rotl32(x15 ^ x0, 8) - movdqa 0x00(%rsp),%xmm0 - paddd %xmm5,%xmm0 - movdqa %xmm0,0x00(%rsp) - pxor %xmm0,%xmm15 - pshufb %xmm2,%xmm15 - # x1 += x6, x12 = rotl32(x12 ^ x1, 8) - movdqa 0x10(%rsp),%xmm0 - paddd %xmm6,%xmm0 - movdqa %xmm0,0x10(%rsp) - pxor %xmm0,%xmm12 - pshufb %xmm2,%xmm12 - # x2 += x7, x13 = rotl32(x13 ^ x2, 8) - movdqa 0x20(%rsp),%xmm0 - paddd %xmm7,%xmm0 - movdqa %xmm0,0x20(%rsp) - pxor %xmm0,%xmm13 - pshufb %xmm2,%xmm13 - # x3 += x4, x14 = rotl32(x14 ^ x3, 8) - movdqa 0x30(%rsp),%xmm0 - paddd %xmm4,%xmm0 - movdqa %xmm0,0x30(%rsp) - pxor %xmm0,%xmm14 - pshufb %xmm2,%xmm14 - - # x10 += x15, x5 = rotl32(x5 ^ x10, 7) - paddd %xmm15,%xmm10 - pxor %xmm10,%xmm5 - movdqa %xmm5,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm5 - por %xmm0,%xmm5 - # x11 += x12, x6 = rotl32(x6 ^ x11, 7) - paddd %xmm12,%xmm11 - pxor %xmm11,%xmm6 - movdqa %xmm6,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm6 - por %xmm0,%xmm6 - # x8 += x13, x7 = rotl32(x7 ^ x8, 7) - paddd %xmm13,%xmm8 - pxor %xmm8,%xmm7 - movdqa %xmm7,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm7 - por %xmm0,%xmm7 - # x9 += x14, x4 = rotl32(x4 ^ x9, 7) - paddd %xmm14,%xmm9 - pxor %xmm9,%xmm4 - movdqa %xmm4,%xmm0 - pslld $7,%xmm0 - psrld $25,%xmm4 - por %xmm0,%xmm4 - - dec %ecx - jnz .Ldoubleround4 - - # x0[0-3] += s0[0] - # x1[0-3] += s0[1] - movq 0x00(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd 0x00(%rsp),%xmm2 - movdqa %xmm2,0x00(%rsp) - paddd 0x10(%rsp),%xmm3 - movdqa %xmm3,0x10(%rsp) - # x2[0-3] += s0[2] - # x3[0-3] += s0[3] - movq 0x08(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd 0x20(%rsp),%xmm2 - movdqa %xmm2,0x20(%rsp) - paddd 0x30(%rsp),%xmm3 - movdqa %xmm3,0x30(%rsp) - - # x4[0-3] += s1[0] - # x5[0-3] += s1[1] - movq 0x10(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm4 - paddd %xmm3,%xmm5 - # x6[0-3] += s1[2] - # x7[0-3] += s1[3] - movq 0x18(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm6 - paddd %xmm3,%xmm7 - - # x8[0-3] += s2[0] - # x9[0-3] += s2[1] - movq 0x20(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm8 - paddd %xmm3,%xmm9 - # x10[0-3] += s2[2] - # x11[0-3] += s2[3] - movq 0x28(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm10 - paddd %xmm3,%xmm11 - - # x12[0-3] += s3[0] - # x13[0-3] += s3[1] - movq 0x30(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm12 - paddd %xmm3,%xmm13 - # x14[0-3] += s3[2] - # x15[0-3] += s3[3] - movq 0x38(%rdi),%xmm3 - pshufd $0x00,%xmm3,%xmm2 - pshufd $0x55,%xmm3,%xmm3 - paddd %xmm2,%xmm14 - paddd %xmm3,%xmm15 - - # x12 += counter values 0-3 - paddd %xmm1,%xmm12 - - # interleave 32-bit words in state n, n+1 - movdqa 0x00(%rsp),%xmm0 - movdqa 0x10(%rsp),%xmm1 - movdqa %xmm0,%xmm2 - punpckldq %xmm1,%xmm2 - punpckhdq %xmm1,%xmm0 - movdqa %xmm2,0x00(%rsp) - movdqa %xmm0,0x10(%rsp) - movdqa 0x20(%rsp),%xmm0 - movdqa 0x30(%rsp),%xmm1 - movdqa %xmm0,%xmm2 - punpckldq %xmm1,%xmm2 - punpckhdq %xmm1,%xmm0 - movdqa %xmm2,0x20(%rsp) - movdqa %xmm0,0x30(%rsp) - movdqa %xmm4,%xmm0 - punpckldq %xmm5,%xmm4 - punpckhdq %xmm5,%xmm0 - movdqa %xmm0,%xmm5 - movdqa %xmm6,%xmm0 - punpckldq %xmm7,%xmm6 - punpckhdq %xmm7,%xmm0 - movdqa %xmm0,%xmm7 - movdqa %xmm8,%xmm0 - punpckldq %xmm9,%xmm8 - punpckhdq %xmm9,%xmm0 - movdqa %xmm0,%xmm9 - movdqa %xmm10,%xmm0 - punpckldq %xmm11,%xmm10 - punpckhdq %xmm11,%xmm0 - movdqa %xmm0,%xmm11 - movdqa %xmm12,%xmm0 - punpckldq %xmm13,%xmm12 - punpckhdq %xmm13,%xmm0 - movdqa %xmm0,%xmm13 - movdqa %xmm14,%xmm0 - punpckldq %xmm15,%xmm14 - punpckhdq %xmm15,%xmm0 - movdqa %xmm0,%xmm15 - - # interleave 64-bit words in state n, n+2 - movdqa 0x00(%rsp),%xmm0 - movdqa 0x20(%rsp),%xmm1 - movdqa %xmm0,%xmm2 - punpcklqdq %xmm1,%xmm2 - punpckhqdq %xmm1,%xmm0 - movdqa %xmm2,0x00(%rsp) - movdqa %xmm0,0x20(%rsp) - movdqa 0x10(%rsp),%xmm0 - movdqa 0x30(%rsp),%xmm1 - movdqa %xmm0,%xmm2 - punpcklqdq %xmm1,%xmm2 - punpckhqdq %xmm1,%xmm0 - movdqa %xmm2,0x10(%rsp) - movdqa %xmm0,0x30(%rsp) - movdqa %xmm4,%xmm0 - punpcklqdq %xmm6,%xmm4 - punpckhqdq %xmm6,%xmm0 - movdqa %xmm0,%xmm6 - movdqa %xmm5,%xmm0 - punpcklqdq %xmm7,%xmm5 - punpckhqdq %xmm7,%xmm0 - movdqa %xmm0,%xmm7 - movdqa %xmm8,%xmm0 - punpcklqdq %xmm10,%xmm8 - punpckhqdq %xmm10,%xmm0 - movdqa %xmm0,%xmm10 - movdqa %xmm9,%xmm0 - punpcklqdq %xmm11,%xmm9 - punpckhqdq %xmm11,%xmm0 - movdqa %xmm0,%xmm11 - movdqa %xmm12,%xmm0 - punpcklqdq %xmm14,%xmm12 - punpckhqdq %xmm14,%xmm0 - movdqa %xmm0,%xmm14 - movdqa %xmm13,%xmm0 - punpcklqdq %xmm15,%xmm13 - punpckhqdq %xmm15,%xmm0 - movdqa %xmm0,%xmm15 - - # xor with corresponding input, write to output - movdqa 0x00(%rsp),%xmm0 - movdqu 0x00(%rdx),%xmm1 - pxor %xmm1,%xmm0 - movdqu %xmm0,0x00(%rsi) - movdqa 0x10(%rsp),%xmm0 - movdqu 0x80(%rdx),%xmm1 - pxor %xmm1,%xmm0 - movdqu %xmm0,0x80(%rsi) - movdqa 0x20(%rsp),%xmm0 - movdqu 0x40(%rdx),%xmm1 - pxor %xmm1,%xmm0 - movdqu %xmm0,0x40(%rsi) - movdqa 0x30(%rsp),%xmm0 - movdqu 0xc0(%rdx),%xmm1 - pxor %xmm1,%xmm0 - movdqu %xmm0,0xc0(%rsi) - movdqu 0x10(%rdx),%xmm1 - pxor %xmm1,%xmm4 - movdqu %xmm4,0x10(%rsi) - movdqu 0x90(%rdx),%xmm1 - pxor %xmm1,%xmm5 - movdqu %xmm5,0x90(%rsi) - movdqu 0x50(%rdx),%xmm1 - pxor %xmm1,%xmm6 - movdqu %xmm6,0x50(%rsi) - movdqu 0xd0(%rdx),%xmm1 - pxor %xmm1,%xmm7 - movdqu %xmm7,0xd0(%rsi) - movdqu 0x20(%rdx),%xmm1 - pxor %xmm1,%xmm8 - movdqu %xmm8,0x20(%rsi) - movdqu 0xa0(%rdx),%xmm1 - pxor %xmm1,%xmm9 - movdqu %xmm9,0xa0(%rsi) - movdqu 0x60(%rdx),%xmm1 - pxor %xmm1,%xmm10 - movdqu %xmm10,0x60(%rsi) - movdqu 0xe0(%rdx),%xmm1 - pxor %xmm1,%xmm11 - movdqu %xmm11,0xe0(%rsi) - movdqu 0x30(%rdx),%xmm1 - pxor %xmm1,%xmm12 - movdqu %xmm12,0x30(%rsi) - movdqu 0xb0(%rdx),%xmm1 - pxor %xmm1,%xmm13 - movdqu %xmm13,0xb0(%rsi) - movdqu 0x70(%rdx),%xmm1 - pxor %xmm1,%xmm14 - movdqu %xmm14,0x70(%rsi) - movdqu 0xf0(%rdx),%xmm1 - pxor %xmm1,%xmm15 - movdqu %xmm15,0xf0(%rsi) - - lea -8(%r10),%rsp - ret -ENDPROC(chacha20_4block_xor_ssse3)arch/x86/crypto/chacha20_glue.c deleted +0
-146@@ -1,146 +0,0 @@ -/* -* ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code -* -* Copyright (C) 2015 Martin Willi -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License as published by -* the Free Software Foundation; either version 2 of the License, or -* (at your option) any later version. -*/ - -#include <crypto/algapi.h> -#include <crypto/chacha20.h> -#include <crypto/internal/skcipher.h> -#include <linux/kernel.h> -#include <linux/module.h> -#include <asm/fpu/api.h> -#include <asm/simd.h> - -#define CHACHA20_STATE_ALIGN 16 - -asmlinkage void chacha20_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src); -asmlinkage void chacha20_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src); -#ifdef CONFIG_AS_AVX2 -asmlinkage void chacha20_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src); -static bool chacha20_use_avx2; -#endif - -static void chacha20_dosimd (u32 *state,u8 *dst,const u8 *src, -unsigned int bytes) -{ -u8 buf[CHACHA20_BLOCK_SIZE ]; - -#ifdef CONFIG_AS_AVX2 -if (chacha20_use_avx2) { -while (bytes >=CHACHA20_BLOCK_SIZE *8 ) { -chacha20_8block_xor_avx2 (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE *8 ; - src +=CHACHA20_BLOCK_SIZE *8 ; - dst +=CHACHA20_BLOCK_SIZE *8 ; - state[12 ] +=8 ; - } - } -#endif -while (bytes >=CHACHA20_BLOCK_SIZE *4 ) { -chacha20_4block_xor_ssse3 (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE *4 ; - src +=CHACHA20_BLOCK_SIZE *4 ; - dst +=CHACHA20_BLOCK_SIZE *4 ; - state[12 ] +=4 ; - } -while (bytes >=CHACHA20_BLOCK_SIZE ) { -chacha20_block_xor_ssse3 (state, dst, src); - bytes -=CHACHA20_BLOCK_SIZE ; - src +=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; - state[12 ]++; - } -if (bytes) { -memcpy (buf, src, bytes); -chacha20_block_xor_ssse3 (state, buf, buf); -memcpy (dst, buf, bytes); - } -} - -static int chacha20_simd (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); -u32 *state, state_buf[16 +2 ]__aligned(8) ; -struct skcipher_walk walk; -int err; - -BUILD_BUG_ON (CHACHA20_STATE_ALIGN !=16 ); - state =PTR_ALIGN (state_buf +0 ,CHACHA20_STATE_ALIGN ); - -if (req->cryptlen <=CHACHA20_BLOCK_SIZE || !may_use_simd ()) -return crypto_chacha20_crypt (req); - - err =skcipher_walk_virt (&walk, req, true); - -crypto_chacha20_init (state, ctx, walk.iv); - -kernel_fpu_begin (); - -while (walk.nbytes >=CHACHA20_BLOCK_SIZE ) { -chacha20_dosimd (state, walk.dst.virt.addr, walk.src.virt.addr, -rounddown (walk.nbytes,CHACHA20_BLOCK_SIZE )); - err =skcipher_walk_done (&walk, - walk.nbytes %CHACHA20_BLOCK_SIZE ); - } - -if (walk.nbytes) { -chacha20_dosimd (state, walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes); - err =skcipher_walk_done (&walk,0 ); - } - -kernel_fpu_end (); - -return err; -} - -static struct skcipher_alg alg = { - .base.cra_name ="chacha20" , - .base.cra_driver_name ="chacha20-simd" , - .base.cra_priority =300 , - .base.cra_blocksize =1 , - .base.cra_ctxsize =sizeof (struct chacha20_ctx ), - .base.cra_module =THIS_MODULE , - - .min_keysize =CHACHA20_KEY_SIZE , - .max_keysize =CHACHA20_KEY_SIZE , - .ivsize =CHACHA20_IV_SIZE , - .chunksize =CHACHA20_BLOCK_SIZE , - .setkey = crypto_chacha20_setkey, - .encrypt = chacha20_simd, - .decrypt = chacha20_simd, -}; - -static int __init chacha20_simd_mod_init (void ) -{ -if (!boot_cpu_has (X86_FEATURE_SSSE3 )) -return -ENODEV ; - -#ifdef CONFIG_AS_AVX2 - chacha20_use_avx2 =boot_cpu_has (X86_FEATURE_AVX ) && -boot_cpu_has (X86_FEATURE_AVX2 ) && -cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL ); -#endif -return crypto_register_skcipher (&alg); -} - -static void __exit chacha20_simd_mod_fini(void) -{ -crypto_unregister_skcipher (&alg); -} - -module_init (chacha20_simd_mod_init); -module_exit (chacha20_simd_mod_fini); - -MODULE_LICENSE ("GPL" ); -MODULE_AUTHOR ("Martin Willi <[email protected]>" ); -MODULE_DESCRIPTION ("chacha20 cipher algorithm, SIMD accelerated" ); -MODULE_ALIAS_CRYPTO ("chacha20" ); -MODULE_ALIAS_CRYPTO ("chacha20-simd" );crypto/Kconfig +0
-16@@ -1433,22 +1433,6 @@ config CRYPTO_CHACHA20 ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the portable C implementation of ChaCha20. - - See also: - <http://cr.yp.to/chacha/chacha-20080128.pdf> - -config CRYPTO_CHACHA20_X86_64 - tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)" - depends on X86 && 64BIT - select CRYPTO_BLKCIPHER - select CRYPTO_CHACHA20 - help - ChaCha20 cipher algorithm, RFC7539. - - ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. - Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the x86_64 assembler implementation using SIMD instructions. See also: <http://cr.yp.to/chacha/chacha-20080128.pdf>crypto/Makefile +1
-1@@ -117,7 +117,7 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_SEED) += seed.o obj-$(CONFIG_CRYPTO_SPECK) += speck.o obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o -obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o +obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_zinc.o obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.ocrypto/chacha20_generic.c deleted +0
-136@@ -1,136 +0,0 @@ -/* -* ChaCha20 256-bit cipher algorithm, RFC7539 -* -* Copyright (C) 2015 Martin Willi -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License as published by -* the Free Software Foundation; either version 2 of the License, or -* (at your option) any later version. -*/ - -#include <asm/unaligned.h> -#include <crypto/algapi.h> -#include <crypto/chacha20.h> -#include <crypto/internal/skcipher.h> -#include <linux/module.h> - -static void chacha20_docrypt (u32 *state,u8 *dst,const u8 *src, -unsigned int bytes) -{ -u32 stream[CHACHA20_BLOCK_WORDS ]; - -if (dst != src) -memcpy (dst, src, bytes); - -while (bytes >=CHACHA20_BLOCK_SIZE ) { -chacha20_block (state, stream); -crypto_xor (dst, (const u8 *)stream,CHACHA20_BLOCK_SIZE ); - bytes -=CHACHA20_BLOCK_SIZE ; - dst +=CHACHA20_BLOCK_SIZE ; - } -if (bytes) { -chacha20_block (state, stream); -crypto_xor (dst, (const u8 *)stream, bytes); - } -} - -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv) -{ - state[0 ] =0x61707865 ;/* "expa" */ - state[1 ] =0x3320646e ;/* "nd 3" */ - state[2 ] =0x79622d32 ;/* "2-by" */ - state[3 ] =0x6b206574 ;/* "te k" */ - state[4 ] = ctx->key[0 ]; - state[5 ] = ctx->key[1 ]; - state[6 ] = ctx->key[2 ]; - state[7 ] = ctx->key[3 ]; - state[8 ] = ctx->key[4 ]; - state[9 ] = ctx->key[5 ]; - state[10 ] = ctx->key[6 ]; - state[11 ] = ctx->key[7 ]; - state[12 ] =get_unaligned_le32 (iv +0 ); - state[13 ] =get_unaligned_le32 (iv +4 ); - state[14 ] =get_unaligned_le32 (iv +8 ); - state[15 ] =get_unaligned_le32 (iv +12 ); -} -EXPORT_SYMBOL_GPL (crypto_chacha20_init); - -int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize) -{ -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); -int i; - -if (keysize !=CHACHA20_KEY_SIZE ) -return -EINVAL ; - -for (i =0 ; i <ARRAY_SIZE (ctx->key); i++) - ctx->key[i] =get_unaligned_le32 (key + i *sizeof (u32)); - -return 0 ; -} -EXPORT_SYMBOL_GPL (crypto_chacha20_setkey); - -int crypto_chacha20_crypt (struct skcipher_request *req) -{ -struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); -struct chacha20_ctx *ctx =crypto_skcipher_ctx (tfm); -struct skcipher_walk walk; -u32 state[16 ]; -int err; - - err =skcipher_walk_virt (&walk, req, true); - -crypto_chacha20_init (state, ctx, walk.iv); - -while (walk.nbytes >0 ) { -unsigned int nbytes = walk.nbytes; - -if (nbytes < walk.total) - nbytes =round_down (nbytes, walk.stride); - -chacha20_docrypt (state, walk.dst.virt.addr, walk.src.virt.addr, - nbytes); - err =skcipher_walk_done (&walk, walk.nbytes - nbytes); - } - -return err; -} -EXPORT_SYMBOL_GPL (crypto_chacha20_crypt); - -static struct skcipher_alg alg = { - .base.cra_name ="chacha20" , - .base.cra_driver_name ="chacha20-generic" , - .base.cra_priority =100 , - .base.cra_blocksize =1 , - .base.cra_ctxsize =sizeof (struct chacha20_ctx ), - .base.cra_module =THIS_MODULE , - - .min_keysize =CHACHA20_KEY_SIZE , - .max_keysize =CHACHA20_KEY_SIZE , - .ivsize =CHACHA20_IV_SIZE , - .chunksize =CHACHA20_BLOCK_SIZE , - .setkey = crypto_chacha20_setkey, - .encrypt = crypto_chacha20_crypt, - .decrypt = crypto_chacha20_crypt, -}; - -static int __init chacha20_generic_mod_init (void ) -{ -return crypto_register_skcipher (&alg); -} - -static void __exit chacha20_generic_mod_fini(void) -{ -crypto_unregister_skcipher (&alg); -} - -module_init (chacha20_generic_mod_init); -module_exit (chacha20_generic_mod_fini); - -MODULE_LICENSE ("GPL" ); -MODULE_AUTHOR ("Martin Willi <[email protected]>" ); -MODULE_DESCRIPTION ("chacha20 cipher algorithm" ); -MODULE_ALIAS_CRYPTO ("chacha20" ); -MODULE_ALIAS_CRYPTO ("chacha20-generic" );crypto/chacha20_zinc.c new file +100
-0@@ -0,0 +1,100 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <asm/unaligned.h> +#include <crypto/algapi.h> +#include <crypto/internal/skcipher.h> +#include <zinc/chacha20.h> +#include <linux/module.h> + +struct chacha20_key_ctx { +u32 key[8 ]; +}; + +static int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, +unsigned int keysize) +{ +struct chacha20_key_ctx *key_ctx =crypto_skcipher_ctx (tfm); +int i; + +if (keysize !=CHACHA20_KEY_SIZE ) +return -EINVAL ; + +for (i =0 ; i <ARRAY_SIZE (key_ctx->key); ++i) + key_ctx->key[i] =get_unaligned_le32 (key + i *sizeof (u32)); + +return 0 ; +} + +static int crypto_chacha20_crypt (struct skcipher_request *req) +{ +struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); +struct chacha20_key_ctx *key_ctx =crypto_skcipher_ctx (tfm); +struct chacha20_ctx ctx; +struct skcipher_walk walk; +simd_context_t simd_context; +int err, i; + + err =skcipher_walk_virt (&walk, req, true); +if (unlikely (err)) +return err; + +memcpy (ctx.key, key_ctx->key,sizeof (ctx.key)); +for (i =0 ; i <ARRAY_SIZE (ctx.counter); ++i) + ctx.counter[i] =get_unaligned_le32 (walk.iv + i *sizeof (u32)); + + simd_context =simd_get (); +while (walk.nbytes >0 ) { +unsigned int nbytes = walk.nbytes; + +if (nbytes < walk.total) + nbytes =round_down (nbytes, walk.stride); + +chacha20 (&ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes, + simd_context); + + err =skcipher_walk_done (&walk, walk.nbytes - nbytes); + simd_context =simd_relax (simd_context); + } +simd_put (simd_context); + +return err; +} + +static struct skcipher_alg alg = { + .base.cra_name ="chacha20" , + .base.cra_driver_name ="chacha20-software" , + .base.cra_priority =100 , + .base.cra_blocksize =1 , + .base.cra_ctxsize =sizeof (struct chacha20_key_ctx ), + .base.cra_module =THIS_MODULE , + + .min_keysize =CHACHA20_KEY_SIZE , + .max_keysize =CHACHA20_KEY_SIZE , + .ivsize =CHACHA20_IV_SIZE , + .chunksize =CHACHA20_BLOCK_SIZE , + .setkey = crypto_chacha20_setkey, + .encrypt = crypto_chacha20_crypt, + .decrypt = crypto_chacha20_crypt, +}; + +static int __init chacha20_mod_init (void ) +{ +return crypto_register_skcipher (&alg); +} + +static void __exit chacha20_mod_exit(void) +{ +crypto_unregister_skcipher (&alg); +} + +module_init (chacha20_mod_init); +module_exit (chacha20_mod_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" ); +MODULE_DESCRIPTION ("ChaCha20 stream cipher" ); +MODULE_ALIAS_CRYPTO ("chacha20" ); +MODULE_ALIAS_CRYPTO ("chacha20-software" );crypto/chacha20poly1305.c +1
-1@@ -13,7 +13,7 @@#include <crypto/internal/hash.h> #include <crypto/internal/skcipher.h> #include <crypto/scatterwalk.h> -#include <crypto/chacha20.h> +#include <zinc/chacha20.h> #include <zinc/poly1305.h> #include <linux/err.h> #include <linux/init.h> include/crypto/chacha20.h +0
-13@@ -6,23 +6,11 @@#ifndef _CRYPTO_CHACHA20_H#define _CRYPTO_CHACHA20_H -#include <crypto/skcipher.h> -#include <linux/types.h> -#include <linux/crypto.h> -#define CHACHA20_IV_SIZE 16#define CHACHA20_KEY_SIZE 32#define CHACHA20_BLOCK_SIZE 64#define CHACHA20_BLOCK_WORDS (CHACHA20_BLOCK_SIZE / sizeof(u32)) -struct chacha20_ctx { -u32 key[8 ]; -}; -void chacha20_block (u32 *state,u32 *stream); -void crypto_chacha20_init (u32 *state,struct chacha20_ctx *ctx,u8 *iv); -int crypto_chacha20_setkey (struct crypto_skcipher *tfm,const u8 *key, -unsigned int keysize); -int crypto_chacha20_crypt (struct skcipher_request *req);#endif --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
Quoted 1 line · Ard Biesheuvel
so could we please bring that discussion to a close before we drop the ARM code?
My understanding is that either these will find their way up to AndyP and then back down here, or Eric or you will augment the .S in this patch at a later date with an improvement commit that includes some benchmarks.
Jason
A message that is not in the archive (sent off-list or not yet received): 201809160711.HzjdJedZ%[email protected]
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
Hi Mr. Ro Bot,
Quoted 2 lines · kbuild test robot
crypto/chacha20_zinc.o: In function `crypto_chacha20_crypt':Quoted 1 line
Quoted 1 line
crypto/chacha20_zinc.c:55: undefined reference to `chacha20'
Looks like my Kconfig change didn't get squashed in as intended. Fixed for v5.
Thanks,
Jason
Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
In reply to Jason A. DonenfeldHi Jason,
Quoted 5 lines
Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it.
delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S
I did some trivial benchmarking with tcrypt for the ChaCha20Poly1305 AEAD as used by IPsec. This is on a box with AVX2, which is probably the configuration mostly used these days. With Zinc I get:
Quoted 9 lines
testing speed of
rfc7539esp(chacha20,poly1305)(rfc7539esp(chacha20-software,poly1305-software)) decryption test 0 (288 bit key, 16 byte blocks): 743510 operations in 1 seconds (11896160 bytes)
test 1 (288 bit key, 64 byte blocks): 743190 operations in 1 seconds (47564160 bytes)
test 2 (288 bit key, 256 byte blocks): 701461 operations in 1 seconds (179574016 bytes) test 3 (288 bit key, 512 byte blocks): 681567 operations in 1 seconds (348962304 bytes) test 4 (288 bit key, 1024 byte blocks): 572854 operations in 1 seconds (586602496 bytes) test 5 (288 bit key, 2048 byte blocks): 434477 operations in 1 seconds (889808896 bytes) test 6 (288 bit key, 4096 byte blocks): 293553 operations in 1 seconds (1202393088 bytes) test 7 (288 bit key, 8192 byte blocks): 173351 operations in 1 seconds (1420091392 bytes)
Using the existing implementation, this was:
Quoted 9 lines
testing speed of
rfc7539esp(chacha20,poly1305)(rfc7539esp(chacha20-simd,poly1305-simd)) decryption test 0 (288 bit key, 16 byte blocks): 1064524 operations in 1 seconds (17032384 bytes) test 1 (288 bit key, 64 byte blocks): 1016046 operations in 1 seconds (65026944 bytes) test 2 (288 bit key, 256 byte blocks): 829566 operations in 1 seconds (212368896 bytes) test 3 (288 bit key, 512 byte blocks): 778912 operations in 1 seconds (398802944 bytes) test 4 (288 bit key, 1024 byte blocks): 622331 operations in 1 seconds (637266944 bytes) test 5 (288 bit key, 2048 byte blocks): 441790 operations in 1 seconds (904785920 bytes) test 6 (288 bit key, 4096 byte blocks): 280616 operations in 1 seconds (1149403136 bytes) test 7 (288 bit key, 8192 byte blocks): 158800 operations in 1 seconds (1300889600 bytes)
I've also experimented with the SIMD context save/restore amortization from patch one on the existing implementation:
Quoted 9 lines
testing speed of
rfc7539esp(chacha20,poly1305)(rfc7539esp(chacha20-simd,poly1305-simd)) decryption test 0 (288 bit key, 16 byte blocks): 1088215 operations in 1 seconds (17411440 bytes) test 1 (288 bit key, 64 byte blocks): 1001788 operations in 1 seconds (64114432 bytes) test 2 (288 bit key, 256 byte blocks): 870193 operations in 1 seconds (222769408 bytes) test 3 (288 bit key, 512 byte blocks): 822149 operations in 1 seconds (420940288 bytes) test 4 (288 bit key, 1024 byte blocks): 647447 operations in 1 seconds (662985728 bytes) test 5 (288 bit key, 2048 byte blocks): 454734 operations in 1 seconds (931295232 bytes) test 6 (288 bit key, 4096 byte blocks): 286995 operations in 1 seconds (1175531520 bytes) test 7 (288 bit key, 8192 byte blocks): 162028 operations in 1 seconds (1327333376 bytes)
For large blocks your implementation is faster; for typical IPsec MTUs this degrades performance by ~10% and more.
Martin
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
Hey Martin,
Thanks for running these and pointing this out. I've replicated the results with tcrypt and fixed some issues, and the next patch series should be a lot closer to what you'd expect, instead of the regression you noticed. Most of the slowdown happened as a result of over-eager XSAVEs, which I've now rectified. I'm still working on a few other facets of it, but I believe v5 will be more satisfactory when posted.
Regards,
Jason
kbuild test robot
build botRe: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc
In reply to Jason A. DonenfeldI love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: i386-randconfig-s3-09171149 (attached as .config)
compiler: gcc-6 (Debian 6.4.0-9) 6.4.0 20171026 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All errors (new ones prefixed by >>):
>> ERROR: "chacha20" [crypto/chacha20_zinc.ko] undefined!
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerPATCH v4 19/20 security/keys: rewrite big_key crypto to use Zinc
In reply to Jason A. DonenfeldA while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function. This makes the file considerably more simple; the diffstat alone should justify this commit. It also should be faster, since it no longer requires a mutex around the "aead api object" (nor allocations), allowing us to encrypt multiple items in parallel. We also benefit from being able to pass any type of pointer to Zinc, so we can get rid of the ridiculously complex custom page allocator that big_key really doesn't need.
--- security/keys/Kconfig | 4 +- security/keys/big_key.c | 230 +++++----------------------------------- 2 files changed, 28 insertions(+), 206 deletions(-)
security/keys/Kconfig +1 -3
@@ -45,9 +45,7 @@ config BIG_KEYS bool "Large payload keys" depends on KEYS depends on TMPFS - select CRYPTO - select CRYPTO_AES - select CRYPTO_GCM + select ZINC_CHACHA20POLY1305 help This option provides support for holding large keys within the kernel (for example Kerberos ticket caches). The data may be stored out to
security/keys/big_key.c +27 -203
347 lines not on this page · raw message
Signature · 1 line
2.19.0
kbuild test robot
build botRe: [PATCH net-next v4 19/20] security/keys: rewrite big_key crypto to use Zinc
I love your patch! Yet something to improve:
Hi Jason,
I love your patch! Yet something to improve:
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64
All errors (new ones prefixed by >>):
>> drivers/acpi/Kconfig:9:error: recursive dependency detected! drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC >> lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_CHACHA20 >> lib/zinc/Kconfig:4: symbol ZINC_CHACHA20 is selected by ZINC_CHACHA20POLY1305 >> lib/zinc/Kconfig:13: symbol ZINC_CHACHA20POLY1305 is selected by BIG_KEYS >> security/keys/Kconfig:44: symbol BIG_KEYS depends on KEYS >> security/keys/Kconfig:5: symbol KEYS is selected by FS_ENCRYPTION >> fs/crypto/Kconfig:1: symbol FS_ENCRYPTION is selected by UBIFS_FS_ENCRYPTION >> fs/ubifs/Kconfig:65: symbol UBIFS_FS_ENCRYPTION depends on MISC_FILESYSTEMS >> fs/Kconfig:218: symbol MISC_FILESYSTEMS is selected by ACPI_APEI >> drivers/acpi/apei/Kconfig:8: symbol ACPI_APEI depends on ACPI
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
vim +4 lib/zinc/Kconfig
32bbe22e Jason A. Donenfeld 2018-09-14 @1 config ZINC32bbe22e Jason A. Donenfeld 2018-09-14 2 tristate32bbe22e Jason A. Donenfeld 2018-09-14 3
35f45248 Jason A. Donenfeld 2018-09-14 @4 config ZINC_CHACHA20 35f45248 Jason A. Donenfeld 2018-09-14 5 bool 35f45248 Jason A. Donenfeld 2018-09-14 6 select ZINC 35f45248 Jason A. Donenfeld 2018-09-14 7 select CRYPTO_ALGAPI 35f45248 Jason A. Donenfeld 2018-09-14 8 0a36c146 Jason A. Donenfeld 2018-09-14 9 config ZINC_POLY1305 0a36c146 Jason A. Donenfeld 2018-09-14 10 bool 0a36c146 Jason A. Donenfeld 2018-09-14 11 select ZINC 0a36c146 Jason A. Donenfeld 2018-09-14 12 1b5dbb86 Jason A. Donenfeld 2018-09-14 @13 config ZINC_CHACHA20POLY1305 1b5dbb86 Jason A. Donenfeld 2018-09-14 14 bool 1b5dbb86 Jason A. Donenfeld 2018-09-14 15 select ZINC 1b5dbb86 Jason A. Donenfeld 2018-09-14 16 select ZINC_CHACHA20 1b5dbb86 Jason A. Donenfeld 2018-09-14 17 select ZINC_POLY1305 1b5dbb86 Jason A. Donenfeld 2018-09-14 18 select CRYPTO_BLKCIPHER 1b5dbb86 Jason A. Donenfeld 2018-09-14 19 a740374c Jason A. Donenfeld 2018-09-14 20 config ZINC_BLAKE2S a740374c Jason A. Donenfeld 2018-09-14 21 bool a740374c Jason A. Donenfeld 2018-09-14 22 select ZINC a740374c Jason A. Donenfeld 2018-09-14 23 cec5aa7c Jason A. Donenfeld 2018-09-14 24 config ZINC_CURVE25519 cec5aa7c Jason A. Donenfeld 2018-09-14 25 bool cec5aa7c Jason A. Donenfeld 2018-09-14 26 select ZINC cec5aa7c Jason A. Donenfeld 2018-09-14 27 select CONFIG_CRYPTO cec5aa7c Jason A. Donenfeld 2018-09-14 28 32bbe22e Jason A. Donenfeld 2018-09-14 29 config ZINC_DEBUG 32bbe22e Jason A. Donenfeld 2018-09-14 30 bool "Zinc cryptography library debugging and self-tests" 32bbe22e Jason A. Donenfeld 2018-09-14 31 depends on ZINC 32bbe22e Jason A. Donenfeld 2018-09-14 32 help 32bbe22e Jason A. Donenfeld 2018-09-14 33 This builds a series of self-tests for the Zinc crypto library, which 32bbe22e Jason A. Donenfeld 2018-09-14 34 help diagnose any cryptographic algorithm implementation issues that 32bbe22e Jason A. Donenfeld 2018-09-14 35 might be at the root cause of potential bugs. It also adds various 32bbe22e Jason A. Donenfeld 2018-09-14 36 debugging traps. 32bbe22e Jason A. Donenfeld 2018-09-14 37 32bbe22e Jason A. Donenfeld 2018-09-14 38 Unless you're developing and testing cryptographic routines, or are 32bbe22e Jason A. Donenfeld 2018-09-14 39 especially paranoid about correctness on your hardware, you may say 32bbe22e Jason A. Donenfeld 2018-09-14 40 N here. 32bbe22e Jason A. Donenfeld 2018-09-14 41 32bbe22e Jason A. Donenfeld 2018-09-14 @42 config ZINC_ARCH_ARM 32bbe22e Jason A. Donenfeld 2018-09-14 43 def_bool y 32bbe22e Jason A. Donenfeld 2018-09-14 44 depends on ARM 32bbe22e Jason A. Donenfeld 2018-09-14 45 depends on ZINC 32bbe22e Jason A. Donenfeld 2018-09-14 46 imply VFP 32bbe22e Jason A. Donenfeld 2018-09-14 47 imply VFPv3 if CPU_V7 32bbe22e Jason A. Donenfeld 2018-09-14 48 imply NEON if CPU_V7 32bbe22e Jason A. Donenfeld 2018-09-14 49 imply KERNEL_MODE_NEON if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14 50
:::::: The code at line 4 was first introduced by commit
:::::: 35f45248597b5a2c80f0f4a680344c22c86efe7d zinc: ChaCha20 generic C implementation
:::::: TO: Jason A. Donenfeld <[email protected]> :::::: CC: 0day robot <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 19/20] security/keys: rewrite big_key crypto to use Zinc
Hey again Ro,
Quoted 1 line
32bbe22eJason A. Donenfeld 2018-09-14 49 implyKERNEL_MODE_NEONifCPU_V7
This shouldn't have ever been there anyway. Fixed now for v5.
Thanks,
Jason
Jason A. Donenfeld
maintainerPATCH v4 20/20 net: WireGuard secure network tunnel
In reply to Jason A. DonenfeldWireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:
* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf
This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace have already implemented the API.
This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.
The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:
* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.
* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.
* allowedips.[ch], hashtables.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the specific hashtables we use.
* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.
* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.
* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.
* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.
* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.
* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.
* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.
* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.
* main.c, version.h: Initialization and deinitialization of the module.
* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.
* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.
This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.
We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.
--- MAINTAINERS | 8 + drivers/net/Kconfig | 30 + drivers/net/Makefile | 1 + drivers/net/wireguard/Makefile | 18 + drivers/net/wireguard/allowedips.c | 404 ++++++++++ drivers/net/wireguard/allowedips.h | 55 ++ drivers/net/wireguard/cookie.c | 234 ++++++ drivers/net/wireguard/cookie.h | 59 ++ drivers/net/wireguard/device.c | 438 +++++++++++ drivers/net/wireguard/device.h | 65 ++ drivers/net/wireguard/hashtables.c | 209 +++++ drivers/net/wireguard/hashtables.h | 63 ++ drivers/net/wireguard/main.c | 65 ++ drivers/net/wireguard/messages.h | 128 +++ drivers/net/wireguard/netlink.c | 605 ++++++++++++++ drivers/net/wireguard/netlink.h | 12 + drivers/net/wireguard/noise.c | 784 +++++++++++++++++++ drivers/net/wireguard/noise.h | 129 +++ drivers/net/wireguard/peer.c | 191 +++++ drivers/net/wireguard/peer.h | 87 ++ drivers/net/wireguard/queueing.c | 52 ++ drivers/net/wireguard/queueing.h | 193 +++++ drivers/net/wireguard/ratelimiter.c | 220 ++++++ drivers/net/wireguard/ratelimiter.h | 19 + drivers/net/wireguard/receive.c | 597 ++++++++++++++ drivers/net/wireguard/selftest/allowedips.h | 656 ++++++++++++++++ drivers/net/wireguard/selftest/counter.h | 103 +++ drivers/net/wireguard/selftest/ratelimiter.h | 174 ++++ drivers/net/wireguard/send.c | 420 ++++++++++ drivers/net/wireguard/socket.c | 435 ++++++++++ drivers/net/wireguard/socket.h | 44 ++ drivers/net/wireguard/timers.c | 256 ++++++ drivers/net/wireguard/timers.h | 30 + drivers/net/wireguard/version.h | 1 + include/uapi/linux/wireguard.h | 190 +++++ tools/testing/selftests/wireguard/netns.sh | 499 ++++++++++++ 36 files changed, 7474 insertions(+) create mode 100644 drivers/net/wireguard/Makefile create mode 100644 drivers/net/wireguard/allowedips.c create mode 100644 drivers/net/wireguard/allowedips.h create mode 100644 drivers/net/wireguard/cookie.c create mode 100644 drivers/net/wireguard/cookie.h create mode 100644 drivers/net/wireguard/device.c create mode 100644 drivers/net/wireguard/device.h create mode 100644 drivers/net/wireguard/hashtables.c create mode 100644 drivers/net/wireguard/hashtables.h create mode 100644 drivers/net/wireguard/main.c create mode 100644 drivers/net/wireguard/messages.h create mode 100644 drivers/net/wireguard/netlink.c create mode 100644 drivers/net/wireguard/netlink.h create mode 100644 drivers/net/wireguard/noise.c create mode 100644 drivers/net/wireguard/noise.h create mode 100644 drivers/net/wireguard/peer.c create mode 100644 drivers/net/wireguard/peer.h create mode 100644 drivers/net/wireguard/queueing.c create mode 100644 drivers/net/wireguard/queueing.h create mode 100644 drivers/net/wireguard/ratelimiter.c create mode 100644 drivers/net/wireguard/ratelimiter.h create mode 100644 drivers/net/wireguard/receive.c create mode 100644 drivers/net/wireguard/selftest/allowedips.h create mode 100644 drivers/net/wireguard/selftest/counter.h create mode 100644 drivers/net/wireguard/selftest/ratelimiter.h create mode 100644 drivers/net/wireguard/send.c create mode 100644 drivers/net/wireguard/socket.c create mode 100644 drivers/net/wireguard/socket.h create mode 100644 drivers/net/wireguard/timers.c create mode 100644 drivers/net/wireguard/timers.h create mode 100644 drivers/net/wireguard/version.h create mode 100644 include/uapi/linux/wireguard.h create mode 100755 tools/testing/selftests/wireguard/netns.sh
MAINTAINERS +8 -0
@@ -15813,6 +15813,14 @@ L: [email protected] S: Maintained F: drivers/gpio/gpio-ws16c48.c +WIREGUARD SECURE NETWORK TUNNEL +M: Jason A. Donenfeld <[email protected]> +S: Maintained +F: drivers/net/wireguard/ +F: tools/testing/selftests/wireguard/ +L: [email protected] +L: [email protected] + WISTRON LAPTOP BUTTON DRIVER M: Miloslav Trmac <[email protected]> S: Maintained
drivers/net/Kconfig +30 -0
@@ -70,6 +70,36 @@ config DUMMY To compile this driver as a module, choose M here: the module will be called dummy. +config WIREGUARD + tristate "WireGuard secure network tunnel" + depends on NET && INET + select NET_UDP_TUNNEL + select DST_CACHE + select ZINC_CHACHA20POLY1305 + select ZINC_BLAKE2S + select ZINC_CURVE25519 + default m + help + WireGuard is a secure, fast, and easy to use replacement for IPSec + that uses modern cryptography and clever networking tricks. It's + designed to be fairly general purpose and abstract enough to fit most + use cases, while at the same time remaining extremely simple to + configure. See www.wireguard.com for more info. + + It's safe to say Y or M here, as the driver is very lightweight and + is only in use when an administrator chooses to add an interface. + +config WIREGUARD_DEBUG + bool "Debugging checks and verbose messages" + depends on WIREGUARD + help + This will write log messages for handshake and other events + that occur for a WireGuard interface. It will also perform some + extra validation checks and unit tests at various points. This is + only useful for debugging. + + Say N here unless you know what you're doing. + config EQUALIZER tristate "EQL (serial line load balancing) support" ---help---
drivers/net/Makefile +1 -0
@@ -24,6 +24,7 @@ obj-$(CONFIG_RIONET) += rionet.o obj-$(CONFIG_NET_TEAM) += team/ obj-$(CONFIG_TUN) += tun.o obj-$(CONFIG_TAP) += tap.o +obj-$(CONFIG_WIREGUARD) += wireguard/ obj-$(CONFIG_VETH) += veth.o obj-$(CONFIG_VIRTIO_NET) += virtio_net.o obj-$(CONFIG_VXLAN) += vxlan.o
drivers/net/wireguard/Makefile new file +18 -0
@@ -0,0 +1,18 @@ +ccflags-y := -O3 +ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt' +ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG +wireguard-y := main.o +wireguard-y += noise.o +wireguard-y += device.o +wireguard-y += peer.o +wireguard-y += timers.o +wireguard-y += queueing.o +wireguard-y += send.o +wireguard-y += receive.o +wireguard-y += socket.o +wireguard-y += hashtables.o +wireguard-y += allowedips.o +wireguard-y += ratelimiter.o +wireguard-y += cookie.o +wireguard-y += netlink.o +obj-$(CONFIG_WIREGUARD) := wireguard.o
drivers/net/wireguard/allowedips.c new file +404 -0
404 lines not on this page · raw message
drivers/net/wireguard/allowedips.h new file +55 -0
55 lines not on this page · raw message
drivers/net/wireguard/cookie.c new file +234 -0
234 lines not on this page · raw message
drivers/net/wireguard/cookie.h new file +59 -0
59 lines not on this page · raw message
drivers/net/wireguard/device.c new file +438 -0
438 lines not on this page · raw message
drivers/net/wireguard/device.h new file +65 -0
65 lines not on this page · raw message
drivers/net/wireguard/hashtables.c new file +209 -0
209 lines not on this page · raw message
drivers/net/wireguard/hashtables.h new file +63 -0
63 lines not on this page · raw message
drivers/net/wireguard/main.c new file +65 -0
65 lines not on this page · raw message
drivers/net/wireguard/messages.h new file +128 -0
128 lines not on this page · raw message
drivers/net/wireguard/netlink.c new file +605 -0
605 lines not on this page · raw message
drivers/net/wireguard/netlink.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _WG_NETLINK_H +#define _WG_NETLINK_H + +int genetlink_init (void ); +void genetlink_uninit (void ); + +#endif /* _WG_NETLINK_H */
drivers/net/wireguard/noise.c new file +784 -0
588 lines not on this page · raw message
drivers/net/wireguard/noise.h new file +129 -0
129 lines not on this page · raw message
drivers/net/wireguard/peer.c new file +191 -0
191 lines not on this page · raw message
drivers/net/wireguard/peer.h new file +87 -0
87 lines not on this page · raw message
drivers/net/wireguard/queueing.c new file +52 -0
52 lines not on this page · raw message
drivers/net/wireguard/queueing.h new file +193 -0
193 lines not on this page · raw message
drivers/net/wireguard/ratelimiter.c new file +220 -0
220 lines not on this page · raw message
drivers/net/wireguard/ratelimiter.h new file +19 -0
Lines of this file are not shown. View the raw message.
drivers/net/wireguard/receive.c new file +597 -0
597 lines not on this page · raw message
drivers/net/wireguard/selftest/allowedips.h new file +656 -0
656 lines not on this page · raw message
drivers/net/wireguard/selftest/counter.h new file +103 -0
103 lines not on this page · raw message
drivers/net/wireguard/selftest/ratelimiter.h new file +174 -0
174 lines not on this page · raw message
drivers/net/wireguard/send.c new file +420 -0
420 lines not on this page · raw message
drivers/net/wireguard/socket.c new file +435 -0
435 lines not on this page · raw message
drivers/net/wireguard/socket.h new file +44 -0
44 lines not on this page · raw message
drivers/net/wireguard/timers.c new file +256 -0
256 lines not on this page · raw message
drivers/net/wireguard/timers.h new file +30 -0
Lines of this file are not shown. View the raw message.
drivers/net/wireguard/version.h new file +1 -0
Lines of this file are not shown. View the raw message.
include/uapi/linux/wireguard.h new file +190 -0
190 lines not on this page · raw message
tools/testing/selftests/wireguard/netns.sh new file +499 -0
499 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
kbuild test robot
build botRe: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel
I love your patch! Perhaps something to improve:
Hi Jason,
I love your patch! Perhaps something to improve:
[auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: powerpc-canyonlands_defconfig (attached as .config)
compiler: powerpc-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=powerpc
All warnings (new ones prefixed by >>):
drivers/net/wireguard/send.c: In function 'packet_encrypt_worker':
>> drivers/net/wireguard/send.c:320:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]
}
^
--
drivers/net/wireguard/receive.c: In function 'packet_decrypt_worker':
>> drivers/net/wireguard/receive.c:520:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]
}
^vim +320 drivers/net/wireguard/send.c
294
295 void packet_encrypt_worker(struct work_struct *work)
296 {
297 struct crypt_queue *queue =
298 container_of(work, struct multicore_worker, work)->ptr;
299 struct sk_buff *first, *skb, *next;
300 simd_context_t simd_context = simd_get();
301
302 while ((first = ptr_ring_consume_bh(&queue->ring)) != NULL) {
303 enum packet_state state = PACKET_STATE_CRYPTED;
304
305 skb_walk_null_queue_safe (first, skb, next) {
306 if (likely(skb_encrypt(skb, PACKET_CB(first)->keypair,
307 simd_context)))
308 skb_reset(skb);
309 else {
310 state = PACKET_STATE_DEAD;
311 break;
312 }
313 }
314 queue_enqueue_per_peer(&PACKET_PEER(first)->tx_queue, first,
315 state);
316
317 simd_context = simd_relax(simd_context);
318 }
319 simd_put(simd_context);
> 320 }
321
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel
Hi Mr. Ro Bot,
Quoted 3 lines · kbuild test robot
drivers/net/wireguard/send.c: In function 'packet_encrypt_worker':Quoted 1 line
Quoted 1 line
drivers/net/wireguard/send.c:320:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]}
Thanks, fixed for v5.
Jason
kbuild test robot
build botRe: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel
In reply to Jason A. DonenfeldI love your patch! Perhaps something to improve:
Hi Jason,
I love your patch! Perhaps something to improve:
[auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: x86_64-randconfig-b0-09160521 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64 All warnings (new ones prefixed by >>):
In file included from lib/zinc/curve25519/curve25519.c:45:0:
lib/zinc/curve25519/curve25519-hacl64.h: In function 'curve25519_generic':
>> lib/zinc/curve25519/curve25519-hacl64.h:785:1: warning: the frame size of 13216 bytes is larger than 8192 bytes [-Wframe-larger-than=]
}
^vim +785 lib/zinc/curve25519/curve25519-hacl64.h
cec5aa7c Jason A. Donenfeld 2018-09-14 755
cec5aa7c Jason A. Donenfeld 2018-09-14 756 static void curve25519_generic(u8 mypublic[CURVE25519_POINT_SIZE],
cec5aa7c Jason A. Donenfeld 2018-09-14 757 const u8 secret[CURVE25519_POINT_SIZE],
cec5aa7c Jason A. Donenfeld 2018-09-14 758 const u8 basepoint[CURVE25519_POINT_SIZE])
cec5aa7c Jason A. Donenfeld 2018-09-14 759 {
cec5aa7c Jason A. Donenfeld 2018-09-14 760 u64 buf0[10] __aligned(32) = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14 761 u64 *x0 = buf0;
cec5aa7c Jason A. Donenfeld 2018-09-14 762 u64 *z = buf0 + 5;
cec5aa7c Jason A. Donenfeld 2018-09-14 763 u64 *q;
cec5aa7c Jason A. Donenfeld 2018-09-14 764 format_fexpand(x0, basepoint);
cec5aa7c Jason A. Donenfeld 2018-09-14 765 z[0] = 1;
cec5aa7c Jason A. Donenfeld 2018-09-14 766 q = buf0;
cec5aa7c Jason A. Donenfeld 2018-09-14 767 {
cec5aa7c Jason A. Donenfeld 2018-09-14 768 u8 e[32] __aligned(32) = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14 769 u8 *scalar;
cec5aa7c Jason A. Donenfeld 2018-09-14 770 memcpy(e, secret, 32);
cec5aa7c Jason A. Donenfeld 2018-09-14 771 normalize_secret(e);
cec5aa7c Jason A. Donenfeld 2018-09-14 772 scalar = e;
cec5aa7c Jason A. Donenfeld 2018-09-14 773 {
cec5aa7c Jason A. Donenfeld 2018-09-14 774 u64 buf[15] = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14 775 u64 *nq = buf;
cec5aa7c Jason A. Donenfeld 2018-09-14 776 u64 *x = nq;
cec5aa7c Jason A. Donenfeld 2018-09-14 777 x[0] = 1;
cec5aa7c Jason A. Donenfeld 2018-09-14 778 ladder_cmult(nq, scalar, q);
cec5aa7c Jason A. Donenfeld 2018-09-14 779 format_scalar_of_point(mypublic, nq);
cec5aa7c Jason A. Donenfeld 2018-09-14 780 memzero_explicit(buf, sizeof(buf));
cec5aa7c Jason A. Donenfeld 2018-09-14 781 }
cec5aa7c Jason A. Donenfeld 2018-09-14 782 memzero_explicit(e, sizeof(e));
cec5aa7c Jason A. Donenfeld 2018-09-14 783 }
cec5aa7c Jason A. Donenfeld 2018-09-14 784 memzero_explicit(buf0, sizeof(buf0));cec5aa7c Jason A. Donenfeld 2018-09-14 @785 }
:::::: The code at line 785 was first introduced by commit
:::::: cec5aa7cd396d909f8555d2df80d3e1a0f94688b zinc: Curve25519 generic C implementations and selftest
:::::: TO: Jason A. Donenfeld <[email protected]>
:::::: CC: 0day robot <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Jason A. Donenfeld
maintainerRe: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel
Hi Mr. Ro Bot,
Looks like this is related to stack frames with KASAN. I've fixed this for v5.
Thanks,
Jason