WireGuard: Secure Network Tunnel
Subsystems (28)
Jason A. Donenfeld
maintainerv9 cover WireGuard: Secure Network Tunnel
Changes v8->v9, along with who suggested it.
--------------------------------------------
- [EVERYBODY] Zinc no longer ships generated assembly code. Rather, we now bundle in the original perlasm generator for it. This is ongoing joint work with Andy Polyakov upstream, so that the same .pl files can live in our tree as well as in the CRYPTOGAMS tree. I personally find that the code required to share this in both repositories to be a tiny bit ugly. I think there would be some degree of an advantage to removing that and making the .pl kernel-only, and then carefully tracking Andy's changes (as we already do). Previous opinions on the list, though, were that there's also significant advantage to being able to share the exact same code in both. And I think there's a decent amount of wisdom in that too. Since that appeared to be the prevailing view, and since it also has good reasons arguments, we'll go with that for now.
Meanwhile, Andy is recently back from some time away, and so we'll be improving these even further in the months to come. Already there are a few things ready that haven't been integrated here just yet.
- [Eric Biggers] In Zinc introductory commit, add more details on what Zinc is for and what the inclusion criteria are, as well as some notes on API.
- Clarify the peer removal logic and make lifetimes more precise.
- [Jann Horn] Use READ_ONCE for is_valid and is_dead.
- [Jann Horn] No need to use atomic when the recounter is mutex protected.
- [Andrew Lunn] Fix up macros and annotations in allowedips.
- [Andrew Lunn] Increment drop counter when staged packets are dropped.
- Use static constants instead of enums for 64-bit values in selftest.
- Mark large constants as ULL in poly1305-donna64.
- Fix sparse warnings in allowedips debugging code.
- Do not use wg_peer_get_maybe_zero in timer callbacks, since we now can carefully control the lifetime of these functions and ensure they never execute after dropping the last reference.
- Cleanup hashing in ratelimiter.
- Do not guard timer removals, since del_timer is always okay.
- [Theodore Ts'o, Andrew Lunn, Sultan Alsawaf] We now check for PM_AUTOSLEEP, which makes the clear-on-suspend decision a bit more general.
- Set csum_level to ~0, since the poly1305 authenticator certainly means that no data was modified in transit.
- [Andrejs Hanins] Use CHECKSUM_PARTIAL check for skb_checksum_help instead of skb_checksum_setup check.
- [Ard Biesheuvel] Enable the selftests by default, and leave it to small-system builders to disable this.
- [Sultan Alsawaf] We no longer allocate large hashtables as part of the net_device structure.
- Iteration of allowed IPs per peer now uses a normal boring fast linked list instead of needing to traverse a giant tree. This makes retrieving peer properties over netlink several orders of magnitude faster.
- The removal of all peers now includes two iterations through the peer list with a single synchronize_rcu_bh() in between iterations, as opposed to having one iteration, but a synchronize_rcu_bh() for every single peer.
- [Arnd Bergmann] 64-bit timestamps.
- The test suite now checks the validity of timestamps and checks that private keys are being properly clamped.
- Numerous API changes made in-tree since v8, such as shedding _bh() on a few functions and networking API changes and such.
- Following the Adiantum merge, the two commits that port the old crypto API over to use Zinc have been removed from this series. We can certainly add them back in at some point, but I thought it'd be favorable to at least begin to receive some sign-offs on the Zinc-specific commits, now that (hopefully all of) the previous feedback has been taken care of. The two commits porting it over are fairly standalone as well, so that shouldn't impact the ability to review this. For now those are living in the jd/with-cryptoapi-port branch of kernel.org's zx2c4/linux.git tree. This also allows us to move this all forward a little bit.
-----------------------------------------------------------
This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:
* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard
-----------------------------------------------------------
WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.
Ample information, including documentation, installation instructions, and project details, is available at:
* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf
As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:
* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/
The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. Zinc was presented at Kernel Recipes in September, and a video is available online. Both Zinc and WireGuard were presented at Plumbers in Vancouver in November.
* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf
* Zinc talk: https://www.youtube.com/watch?v=bFhdln8aJ_U
* Netdev talk: https://www.youtube.com/watch?v=54orFwtQ1XY
The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:
* https://www.wireguard.com/formal-verification/
This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows a non-WireGuard use case for Zinc. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.
It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree in cases when an entire development cycle has no relationships with existing code in crypto/; however, if there are any relationships with code in crypto/, then pull requests will be sent to Herbert instead in case there are merge conflicts.
Enjoy,
Jason
Jason A. Donenfeld
maintainerPATCH v9 1/19 asm: simd context helper API
Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:
simd_context_t simd_context;
simd_get(&simd_context);
while ((item = get_item_from_queue()) != NULL) {
encrypt_item(item, &simd_context);
simd_relax(&simd_context);
}
simd_put(&simd_context);The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.
On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:
void encrypt_item(struct item *item, simd_context_t *simd_context)
{
if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
wild_simd_code(item);
else
boring_scalar_code(item);
}The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.
--- arch/alpha/include/asm/Kbuild | 1 + arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/Kbuild | 1 - arch/arm/include/asm/simd.h | 63 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/simd.h | 51 +++++++++++++++++++++--- arch/c6x/include/asm/Kbuild | 1 + arch/h8300/include/asm/Kbuild | 1 + arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nds32/include/asm/Kbuild | 1 + arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 1 + arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 1 + arch/riscv/include/asm/Kbuild | 1 + arch/s390/include/asm/Kbuild | 1 + arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 1 + arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/simd.h | 44 ++++++++++++++++++++- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/simd.h | 20 ++++++++++ include/linux/simd.h | 32 +++++++++++++++ 27 files changed, 224 insertions(+), 8 deletions(-) create mode 100644 arch/arm/include/asm/simd.h create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +1 -0
@@ -13,3 +13,4 @@ generic-y += sections.h generic-y += trace_clock.h generic-y += current.h generic-y += kprobes.h +generic-y += simd.h
arch/arc/include/asm/Kbuild +1 -0
@@ -19,6 +19,7 @@ generic-y += msi.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/arm/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h -generic-y += simd.h generic-y += sizes.h generic-y += timex.h generic-y += trace_clock.h
arch/arm/include/asm/simd.h new file +63 -0
63 lines not on this page · raw message
arch/arm64/include/asm/simd.h +45 -6
71 lines not on this page · raw message
arch/c6x/include/asm/Kbuild +1 -0
@@ -31,6 +31,7 @@ generic-y += preempt.h generic-y += segment.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += tlbflush.h generic-y += topology.h generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +1 -0
@@ -41,6 +41,7 @@ generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += sizes.h generic-y += spinlock.h generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
@@ -31,6 +31,7 @@ generic-y += sections.h generic-y += segment.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += sizes.h generic-y += topology.h generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
@@ -5,6 +5,7 @@ generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h +generic-y += simd.h generic-y += trace_clock.h generic-y += vtime.h generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
@@ -21,6 +21,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += sections.h generic-y += shmparam.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
@@ -27,6 +27,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += syscalls.h generic-y += topology.h generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
@@ -20,6 +20,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += user.h
arch/nds32/include/asm/Kbuild +1 -0
@@ -38,6 +38,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += switch_to.h generic-y += timex.h
arch/nios2/include/asm/Kbuild +1 -0
@@ -33,6 +33,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +1 -0
@@ -35,6 +35,7 @@ generic-y += qrwlock.h generic-y += sections.h generic-y += segment.h generic-y += shmparam.h +generic-y += simd.h generic-y += string.h generic-y += switch_to.h generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
@@ -20,6 +20,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += seccomp.h generic-y += segment.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/powerpc/include/asm/Kbuild +1 -0
@@ -12,3 +12,4 @@ generic-y += preempt.h generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h +generic-y += simd.h
arch/riscv/include/asm/Kbuild +1 -0
@@ -28,6 +28,7 @@ generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += unaligned.h
arch/s390/include/asm/Kbuild +1 -0
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
@@ -18,6 +18,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += trace_clock.h generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
@@ -19,5 +19,6 @@ generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +1 -0
@@ -20,6 +20,7 @@ generic-y += param.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += switch_to.h generic-y += topology.h generic-y += trace_clock.h
arch/unicore32/include/asm/Kbuild +1 -0
@@ -29,6 +29,7 @@ generic-y += sections.h generic-y += segment.h generic-y += serial.h generic-y += shmparam.h +generic-y += simd.h generic-y += sizes.h generic-y += syscalls.h generic-y += topology.h
arch/x86/include/asm/simd.h +43 -1
50 lines not on this page · raw message
arch/xtensa/include/asm/Kbuild +1 -0
@@ -27,6 +27,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += rwsem.h generic-y += sections.h +generic-y += simd.h generic-y += socket.h generic-y += topology.h generic-y += trace_clock.h
include/asm-generic/simd.h +20 -0
@@ -1,5 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H +#include <linux/hardirq.h> /*
@@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void) {return !in_interrupt (); } + +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +return false; +} + +#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _SIMD_H +#define _SIMD_H + +typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; + +#define DONT_USE_SIMD ((simd_context_t []){ HAVE_NO_SIMD }) + +#include <linux/sched.h> +#include <asm/simd.h> + +static inline bool simd_relax (simd_context_t *ctx) +{ +#ifdef CONFIG_PREEMPT +if ((*ctx &HAVE_SIMD_IN_USE ) &&need_resched ()) { +simd_put (ctx); +simd_get (ctx); +return true; + } +#endif +return false; +} + +#endif /* _SIMD_H */
Signature · 1 line
2.21.0
Re: [PATCH net-next v9 01/19] asm: simd context helper API
Jason,
arch/arm/include/asm/simd.h line 1
Quoted 5 lines · Jason A. Donenfeld
index 000000000000..264ed84b41d8
arch/arm/include/asm/simd.h new file +1
-0@@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0
The SPDX identifier has to be in a separate comment line.
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2015-2018 Jason....
*/
Yes, it looks odd, but it's defined that way.
Quoted 26 lines
arch/arm/include/asm/simd.h new file +24
-0+static __must_check inline bool simd_use (simd_context_t *ctx) +{ +if (!(*ctx &HAVE_FULL_SIMD )) +return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_neon_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +} +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +if (!(*ctx &HAVE_FULL_SIMD )) +return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_neon_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +} +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +#if !defined(CONFIG_UML ) +if (!(*ctx &HAVE_FULL_SIMD ))
arch/arm/include/asm/simd.h
Quoted 10 lines
arch/arm/include/asm/simd.h new file +10
-0+return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_fpu_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +#else +return false; +#endif +}
So now we have 3 almost identical copies of the same thing and x86/UM handling it differently than the ARM(64) ones.
That can be completely avoided. Untested patch below.
Thanks,
tglx
8<-------------
arch/arm/include/asm/simd.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H + +#ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> + +static __must_check inline bool may_use_simd (void ) +{ +return !in_nmi () && !in_irq () && !in_serving_softirq (); +} + +static inline void arch_simd_end (void ) +{ +kernel_neon_end (); +} + +static inline void arch_simd_begin (void ) +{ +kernel_neon_begin (); +} + +#else +#include <asm-generic/simd.h> +#endif + +#endif /* _ASM_SIMD_H */
arch/arm64/include/asm/simd.h +15 -8
@@ -1,11 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2017 Linaro Ltd. <[email protected]> -* -* This program is free software; you can redistribute it and/or modify it -* under the terms of the GNU General Public License version 2 as published -* by the Free Software Foundation. +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ +#include <linux/simd.h> #ifndef __ASM_SIMD_H#define __ASM_SIMD_H
@@ -16,6 +15,8 @@#include <linux/types.h> #ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> +#include <asm/simd.h> DECLARE_PER_CPU (bool, kernel_neon_busy);
@@ -40,12 +41,18 @@ static __must_check inline bool may_use_ !this_cpu_read (kernel_neon_busy); } -#else /* ! CONFIG_KERNEL_MODE_NEON */ +static inline void arch_simd_end (void ) +{ +kernel_neon_end (); +} -static __must_check inline bool may_use_simd (void ) { -return false; +static inline void arch_simd_begin (void ) +{ +kernel_neon_begin (); } -#endif /* ! CONFIG_KERNEL_MODE_NEON */ +#else +#include <asm-generic/simd.h> +#endif #endif
arch/x86/include/asm/simd.h +26 -0
@@ -1,4 +1,12 @@/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H + +#ifndef CONFIG_UML #include <asm/fpu/api.h>
@@ -10,3 +18,21 @@ static __must_check inline bool may_use_ {return irq_fpu_usable (); } + +static inline void arch_simd_begin (void ) +{ +kernel_fpu_begin (); +} + +static inline void arch_simd_end (void ) +{ +kernel_fpu_end (); +} + +#else +#include <asm-generic/simd.h> +#endif + +#include <linux/simd.h> + +#endif /* _ASM_SIMD_H */
include/asm-generic/simd.h +12 -0
@@ -1,7 +1,12 @@/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H +#include <linux/hardirq.h> +#define SIMD_GENERIC 1 +/* * may_use_simd - whether it is allowable at this time to issue SIMD * instructions or access the SIMD register file
@@ -13,3 +18,10 @@ static __must_check inline bool may_use_ {return !in_interrupt (); } + +static inline void arch_simd_begin (void ) { } +static inline void arch_simd_end (void ) { } + +#include <linux/simd.h> + +#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +82 -0
82 lines not on this page · raw message
Jason A. Donenfeld
maintainerPATCH v9 2/19 zinc: introduce minimal cryptography library
In reply to Jason A. DonenfeldZinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto". It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.
Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:
* Highly composable "cipher modes" and related abstractions from the 90s did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.
* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.
* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.
This last observation has seen itself play out several times over and over again within the kernel:
* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.
* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.
* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.
Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.
It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.
All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].
Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.
SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.
More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.
Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.
To summarize, Zinc will contain implementations of cryptographic primitives that are:
* Software-based and synchronous.
* Expose a simple API that operate over plain chunks of data and do not need significant cumbersome scaffolding (more below).
* Extremely fast, but also, in order of priority: 1) formally verified, 2) well-known code that's received a lot of eyeballs and has seen significant real-world usage, 3) simple reviewable code that is either obviously correct or hard to screw up given test-vectors that has been subjected to significant amounts of fuzzing and projects like Wycheproof.
The APIs of each implementation are generally expected to take the following forms, with accepted variations for primitives that have non-standard input or output parameters:
* For hash functions the classic init/update/final dance is well-known and clear to implement. Different init functions can instantiate different operating parameters of flexible hash functions (such as blake2s_init and blake2s_init_key).
* Authenticated encryption functions return a simple boolean indicating whether or not decryption succeeded. They take as inputs and outputs either a pointer to a buffer and a size, or they take as inputs and outputs scatter-gather lists (for use with the network subsystem's skb_to_sgvec, for example).
* Functions that are commonly called in loops inside a long-running worker thread may grow to take a simd_context_t parameter, so that the FPU can be twiddled from outside of the function (see prior commit introducing simd_get/put/relax). It is expected that most functions that fit this scenario will be ones that take scatter-gather lists.
In addition to the above implementation and API considerations, inclusion criteria for Zinc will be mostly the same as for other aspects of the kernel: is there a direct user of the primitive or construction's Zinc implementation that would immediately benefit from that kind of API? Certain primitives, like MD5 for example, might be separated off into a legacy/ header subdirectory, to make it clear to new users that if they're using it, it's for a very particular purpose. Zinc is also wary of adding overly newfangled and unvetted primitives that have no immediate uptake or scrutiny: for example, implementations of a new block cipher posted on eprint just yesterday. But beyond that, we recognize that cryptographic functions have many different uses and are required in large variety of standards and circumstances, whose decisions are often made outside the scope of the kernel, and so Zinc will strive to accommodate the writing of clean and effective code and will not discriminate on the basis of fashionability.
Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.
Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.
As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_SELFTEST and CONFIG_ZINC_DEBUG, which enables various selftests and debugging conditions.
[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof
--- MAINTAINERS | 8 ++++++++ lib/Kconfig | 2 ++ lib/Makefile | 2 ++ lib/zinc/Kconfig | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/zinc/Makefile | 3 +++ 5 files changed, 57 insertions(+) create mode 100644 lib/zinc/Kconfig create mode 100644 lib/zinc/Makefile
MAINTAINERS +8 -0
@@ -17213,6 +17213,14 @@ Q: https://patchwork.linuxtv.org/project/linux-media/list/ S: Maintained F: drivers/media/dvb-frontends/zd1301_demod* +ZINC CRYPTOGRAPHY LIBRARY +M: Jason A. Donenfeld <[email protected]> +M: Samuel Neves <[email protected]> +S: Maintained +F: lib/zinc/ +F: include/zinc/ +L: [email protected] + ZPOOL COMPRESSED PAGE STORAGE API M: Dan Streetman <[email protected]> L: [email protected]
lib/Kconfig +2 -0
@@ -496,6 +496,8 @@ config GLOB_SELFTEST module load) by a small amount, so you're welcome to play with it, but you probably don't need it. +source "lib/zinc/Kconfig" + # # Netlink attribute parsing support is select'ed if needed #
lib/Makefile +2 -0
@@ -223,6 +223,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o obj-$(CONFIG_ASN1) += asn1_decoder.o +obj-y += zinc/ + obj-$(CONFIG_FONT_SUPPORT) += fonts/ obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +42 -0
42 lines not on this page · raw message
lib/zinc/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +ccflags-y := -O2 +ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' +ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 3/19 zinc: ChaCha20 generic C implementation and selftest
In reply to Jason A. DonenfeldThis implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
--- include/zinc/chacha20.h | 70 + lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 3 + lib/zinc/chacha20/chacha20.c | 179 +++ lib/zinc/selftest/chacha20.c | 2698 ++++++++++++++++++++++++++++++++++ lib/zinc/selftest/run.h | 48 + 6 files changed, 3002 insertions(+) create mode 100644 include/zinc/chacha20.h create mode 100644 lib/zinc/chacha20/chacha20.c create mode 100644 lib/zinc/selftest/chacha20.c create mode 100644 lib/zinc/selftest/run.h
include/zinc/chacha20.h new file +70 -0
@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20_H +#define _ZINC_CHACHA20_H + +#include <asm/unaligned.h> +#include <linux/simd.h> +#include <linux/kernel.h> +#include <linux/types.h> + +enum chacha20_lengths { +CHACHA20_NONCE_SIZE =16 , +CHACHA20_KEY_SIZE =32 , +CHACHA20_KEY_WORDS =CHACHA20_KEY_SIZE /sizeof (u32), +CHACHA20_BLOCK_SIZE =64 , +CHACHA20_BLOCK_WORDS =CHACHA20_BLOCK_SIZE /sizeof (u32), +HCHACHA20_NONCE_SIZE =CHACHA20_NONCE_SIZE , +HCHACHA20_KEY_SIZE =CHACHA20_KEY_SIZE +}; + +enum chacha20_constants {/* expand 32-byte k */ +CHACHA20_CONSTANT_EXPA =0x61707865U , +CHACHA20_CONSTANT_ND_3 =0x3320646eU , +CHACHA20_CONSTANT_2_BY =0x79622d32U , +CHACHA20_CONSTANT_TE_K =0x6b206574U +}; + +struct chacha20_ctx { +union { +u32 state[16 ]; +struct { +u32 constant[4 ]; +u32 key[8 ]; +u32 counter[4 ]; + }; + }; +}; + +static inline void chacha20_init (struct chacha20_ctx *ctx, +const u8 key[CHACHA20_KEY_SIZE ], +const u64 nonce) +{ + ctx->constant[0 ] =CHACHA20_CONSTANT_EXPA ; + ctx->constant[1 ] =CHACHA20_CONSTANT_ND_3 ; + ctx->constant[2 ] =CHACHA20_CONSTANT_2_BY ; + ctx->constant[3 ] =CHACHA20_CONSTANT_TE_K ; + ctx->key[0 ] =get_unaligned_le32 (key +0 ); + ctx->key[1 ] =get_unaligned_le32 (key +4 ); + ctx->key[2 ] =get_unaligned_le32 (key +8 ); + ctx->key[3 ] =get_unaligned_le32 (key +12 ); + ctx->key[4 ] =get_unaligned_le32 (key +16 ); + ctx->key[5 ] =get_unaligned_le32 (key +20 ); + ctx->key[6 ] =get_unaligned_le32 (key +24 ); + ctx->key[7 ] =get_unaligned_le32 (key +28 ); + ctx->counter[0 ] =0 ; + ctx->counter[1 ] =0 ; + ctx->counter[2 ] = nonce &U32_MAX ; + ctx->counter[3 ] = nonce >>32 ; +} +void chacha20 (struct chacha20_ctx *ctx,u8 *dst,const u8 *src,u32 len, +simd_context_t *simd_context); + +void hchacha20 (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t *simd_context); + +#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
@@ -1,3 +1,7 @@ +config ZINC_CHACHA20 + tristate + select CRYPTO_ALGAPI + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" default y
lib/zinc/Makefile +3 -0
@@ -1,3 +1,6 @@ ccflags-y := -O2 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc_chacha20-y := chacha20/chacha20.o +obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +179 -0
179 lines not on this page · raw message
lib/zinc/selftest/chacha20.c new file +2698 -0
2,698 lines not on this page · raw message
lib/zinc/selftest/run.h new file +48 -0
40 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v9 4/19 zinc: ChaCha20 x86_64 implementation
In reply to Jason A. DonenfeldThese x86_64 vectorized implementations are based on Andy Polyakov's implementations, and support SSSE3, AVX-2, AVX-512F, and AVX-512VL. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead on that platform; other AVX-512 microarchitectures use AVX-512F.
On the left is cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"). On the right are benchmarks on a Xeon Gold 5120 using the AVX-512 codepath. The difference is so stark, because the current crypto api's implementation does not support AVX-512 at all.
AVX-2 AVX-512
--------- -----------size old new size old new
---- ---- ---- ---- ---- ----
0 62 52 0 64 54
16 414 376 16 386 372 32 410 400 32 388 396 48 414 422 48 388 420 64 362 356 64 366 350 80 714 666 80 708 666 96 714 700 96 708 692 112 712 718 112 706 736 128 692 646 128 692 648 144 1042 674 144 1036 682 160 1042 694 160 1036 708 176 1042 726 176 1036 730 192 1018 650 192 1016 658 208 1366 686 208 1360 684 224 1366 696 224 1362 708 240 1366 722 240 1360 732 256 640 656 256 644 500 272 988 1246 272 990 526 288 988 1276 288 988 556 304 992 1296 304 988 576 320 972 1222 320 972 500 336 1318 1256 336 1314 532 352 1318 1276 352 1316 558 368 1316 1294 368 1318 578 384 1294 1218 384 1308 506 400 1642 1258 400 1644 532 416 1642 1282 416 1644 556 432 1642 1302 432 1644 594 448 1628 1224 448 1624 508 464 1970 1258 464 1970 534 480 1970 1280 480 1970 556 496 1970 1300 496 1968 582 512 656 676 512 660 624 528 1010 1290 528 1016 682 544 1010 1306 544 1016 702 560 1010 1332 560 1018 728 576 986 1254 576 998 654 592 1340 1284 592 1344 680 608 1334 1310 608 1344 708 624 1340 1334 624 1344 730 640 1314 1254 640 1326 654 656 1664 1282 656 1670 686 672 1674 1306 672 1670 708 688 1662 1336 688 1670 732 704 1638 1250 704 1652 658 720 1992 1292 720 1998 682 736 1994 1308 736 1998 710 752 1988 1334 752 1996 734 768 1252 1254 768 1256 662 784 1596 1290 784 1606 688 800 1596 1314 800 1606 714 816 1596 1330 816 1606 736 832 1576 1256 832 1584 660 848 1922 1286 848 1948 688 864 1922 1314 864 1950 714 880 1926 1338 880 1948 736 896 1898 1258 896 1912 688 912 2248 1288 912 2258 718 928 2248 1320 928 2258 744 944 2248 1338 944 2256 768 960 2226 1268 960 2238 692 976 2574 1288 976 2584 718 992 2576 1312 992 2584 744
1008 2574 1340 1008 2584 770
While this is CRYPTOGAMS code, the originating code for this happens to be derived from OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da
--- lib/zinc/Makefile | 8 + lib/zinc/chacha20/chacha20-x86_64-glue.c | 103 + lib/zinc/chacha20/chacha20-x86_64.pl | 4106 ++++++++++++++++++++++ lib/zinc/chacha20/chacha20.c | 4 + 4 files changed, 4221 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.c create mode 100644 lib/zinc/chacha20/chacha20-x86_64.pl
lib/zinc/Makefile +8 -0
@@ -3,4 +3,12 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o +perlasm-targets += chacha20/chacha20-x86_64.S obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o + +quiet_cmd_perlasm = PERLASM $@ + cmd_perlasm = $(PERL) $< > $@ +$(addprefix $(obj)/,$(perlasm-targets)): $(obj)/%.S: $(src)/%.pl FORCE + $(call if_changed,perlasm) +targets += $(perlasm-targets)
lib/zinc/chacha20/chacha20-x86_64-glue.c new file +103 -0
103 lines not on this page · raw message
lib/zinc/chacha20/chacha20-x86_64.pl new file +4106 -0
2,885 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v9 5/19 zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese ARM implementations come from Andy Polyakov's implementation, as well as an ultra-fast unrolled scalar version optimized for ARMv7 from Eric Biggers, which performs well on platforms with slow NEON like Cortex A7 and A5.
This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:
- ARM1176JZF-S [ARMv6]
- Cortex-A7 [ARMv7] - Cortex-A8 [ARMv7] - Cortex-A9 [ARMv7] - Cortex-A17 [ARMv7] - Cortex-A53 [ARMv8] - Cortex-A55 [ARMv8] - Cortex-A73 [ARMv8] - Cortex-A75 [ARMv8]
Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 9925a82146f2503b4dd11423d0eba649b43450c0
--- lib/zinc/Makefile | 4 +- lib/zinc/chacha20/chacha20-arm-glue.c | 98 ++ lib/zinc/chacha20/chacha20-arm.pl | 1227 +++++++++++++++++++++ lib/zinc/chacha20/chacha20-arm64.pl | 1163 +++++++++++++++++++ lib/zinc/chacha20/chacha20-unrolled-arm.S | 461 ++++++++ lib/zinc/chacha20/chacha20.c | 2 + 6 files changed, 2954 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.c create mode 100644 lib/zinc/chacha20/chacha20-arm.pl create mode 100644 lib/zinc/chacha20/chacha20-arm64.pl create mode 100644 lib/zinc/chacha20/chacha20-unrolled-arm.S
lib/zinc/Makefile +3 -1
@@ -4,7 +4,9 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o -perlasm-targets += chacha20/chacha20-x86_64.S +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o chacha20/chacha20-unrolled-arm.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o +perlasm-targets += chacha20/chacha20-x86_64.S chacha20/chacha20-arm.S chacha20/chacha20-arm64.S obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/chacha20/chacha20-arm-glue.c new file +98 -0
98 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm.pl new file +1227 -0
1,227 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm64.pl new file +1163 -0
1,163 lines not on this page · raw message
lib/zinc/chacha20/chacha20-unrolled-arm.S new file +461 -0
461 lines not on this page · raw message
lib/zinc/chacha20/chacha20.c +2 -0
@@ -18,6 +18,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "chacha20-x86_64-glue.c" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 ) +#include "chacha20-arm-glue.c" #else static bool *const chacha20_nobs[]__initconst = { };static void __init chacha20_fpu_init (void )
Signature · 1 line
2.21.0
Re: [PATCH net-next v9 05/19] zinc: ChaCha20 ARM and ARM64 implementations
Hi Jason,
I gave this a try with some prelineary patches to enable the integrated assembler for ARM32, and caught a small issue:
Quoted 54 lines · Jason A. Donenfeld
These ARM implementations come from Andy Polyakov's implementation, as well as an ultra-fast unrolled scalar version optimized for ARMv7 from Eric Biggers, which performs well on platforms with slow NEON like Cortex A7 and A5.
This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:
- ARM1176JZF-S [ARMv6]
- Cortex-A7 [ARMv7] - Cortex-A8 [ARMv7] - Cortex-A9 [ARMv7] - Cortex-A17 [ARMv7] - Cortex-A53 [ARMv8] - Cortex-A55 [ARMv8] - Cortex-A73 [ARMv8] - Cortex-A75 [ARMv8]Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
9925a82146f2503b4dd11423d0eba649b43450c0Signed-off-by Jason A. Donenfeld <[email protected]>Co-authored-by Andy Polyakov <[email protected]>Co-authored-by Eric Biggers <[email protected]>Cc Andy Polyakov <[email protected]>Cc Russell King <[email protected]>Cc Samuel Neves <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- lib/zinc/Makefile | 4 +- lib/zinc/chacha20/chacha20-arm-glue.c | 98 ++ lib/zinc/chacha20/chacha20-arm.pl | 1227 +++++++++++++++++++++ lib/zinc/chacha20/chacha20-arm64.pl | 1163 +++++++++++++++++++ lib/zinc/chacha20/chacha20-unrolled-arm.S | 461 ++++++++ lib/zinc/chacha20/chacha20.c | 2 + 6 files changed, 2954 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.c create mode 100644 lib/zinc/chacha20/chacha20-arm.pl create mode 100644 lib/zinc/chacha20/chacha20-arm64.pl create mode 100644 lib/zinc/chacha20/chacha20-unrolled-arm.S
<snip>
Quoted 48 lines
lib/zinc/chacha20/chacha20-unrolled-arm.S new file +22
-0@@ -0,0 +1,461 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2018 Google, Inc. + */ + +#include <linux/linkage.h> +#include <asm/assembler.h> + +/* + * Design notes: + * + * 16 registers would be needed to hold the state matrix, but only 14 are + * available because 'sp' and 'pc' cannot be used. So we spill the elements + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one + * 'ldrd' and one 'strd' instruction per round. + * + * All rotates are performed using the implicit rotate operand accepted by the + * 'add' and 'eor' instructions. This is faster than using explicit rotate + * instructions. To make this work, we allow the values in the second and last + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the + * wrong rotation amount. The rotation amount is then fixed up just in time + * when the values are used. 'brot' is the number of bits the valuesin row 'b'
+ * need to be rotated right to arrive at the correct values, and 'drot'
+ * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such
+ * that they end up as (25, 24) after every round.
+ */lib/zinc/chacha20/chacha20-unrolled-arm.S new file +14
-0+ + // ChaCha state registers + X0 .req r0 + X1 .req r1 + X2 .req r2 + X3 .req r3 + X4 .req r4 + X5 .req r5 + X6 .req r6 + X7 .req r7 + X8_X10 .req r8 // shared by x8 and x10 + X9_X11 .req r9 // shared by x9 and x11 + X12 .req r10 + X13 .req r11
lib/zinc/chacha20/chacha20-unrolled-arm.S line 50
Quoted 10 lines
lib/zinc/chacha20/chacha20-unrolled-arm.S new file +10
-0+ X14 .req r12 + X15 .req r14 + +.Lexpand_32byte_k: + // "expand 32-byte k" + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574 + +#ifdef __thumb2__ +# define adrl adr +#endif
It seems that Clang does not like adrl.
lib/zinc/chacha20/chacha20-unrolled-arm.S:415:2: error: invalid instruction, did you mean: adr?
adrl X3, .Lexpand_32byte_k
^
make[3]: *** [scripts/Makefile.build:369:lib/zinc/chacha20/chacha20-unrolled-arm.o] Error 1 make[2]: *** [scripts/Makefile.build:489: lib/zinc] Error 2 make[1]: *** [/home/sag/projects/linux/Makefile:1045: lib] Error 2 make[1]: *** Waiting for unfinished jobs....
We probably should use
#if defined(__thumb2__) || defined(__clang__)
here?
All other ARM32 assembly assembled fine with the integrated assembler! Also nothing really concerning as there are still plenty of other issues with using the integrated assembler to build ARM32 Linux.
--
Stefan
Quoted 424 lines
lib/zinc/chacha20/chacha20-unrolled-arm.S new file +411
-0+ +.macro __rev out, in, t0, t1, t2 +.if __LINUX_ARM_ARCH__ >= 6 + rev \out, \in +.else + lsl \t0, \in, #24 + and \t1, \in, #0xff00 + and \t2, \in, #0xff0000 + orr \out, \t0, \in, lsr #24 + orr \out, \out, \t1, lsl #8 + orr \out, \out, \t2, lsr #8 +.endif +.endm + +.macro _le32_bswap x, t0, t1, t2 +#ifdef __ARMEB__ + __rev \x, \x, \t0, \t1, \t2 +#endif +.endm + +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2 + _le32_bswap \a, \t0, \t1, \t2 + _le32_bswap \b, \t0, \t1, \t2 + _le32_bswap \c, \t0, \t1, \t2 + _le32_bswap \d, \t0, \t1, \t2 +.endm + +.macro __ldrd a, b, src, offset +#if __LINUX_ARM_ARCH__ >= 6 + ldrd \a, \b, [\src, #\offset] +#else + ldr \a, [\src, #\offset] + ldr \b, [\src, #\offset + 4] +#endif +.endm + +.macro __strd a, b, dst, offset +#if __LINUX_ARM_ARCH__ >= 6 + strd \a, \b, [\dst, #\offset] +#else + str \a, [\dst, #\offset] + str \b, [\dst, #\offset + 4] +#endif +.endm + +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2 + + // a += b; d ^= a; d = rol(d, 16); + add \a1, \a1, \b1, ror #brot + add \a2, \a2, \b2, ror #brot + eor \d1, \a1, \d1, ror #drot + eor \d2, \a2, \d2, ror #drot + // drot == 32 - 16 == 16 + + // c += d; b ^= c; b = rol(b, 12); + add \c1, \c1, \d1, ror #16 + add \c2, \c2, \d2, ror #16 + eor \b1, \c1, \b1, ror #brot + eor \b2, \c2, \b2, ror #brot + // brot == 32 - 12 == 20 + + // a += b; d ^= a; d = rol(d, 8); + add \a1, \a1, \b1, ror #20 + add \a2, \a2, \b2, ror #20 + eor \d1, \a1, \d1, ror #16 + eor \d2, \a2, \d2, ror #16 + // drot == 32 - 8 == 24 + + // c += d; b ^= c; b = rol(b, 7); + add \c1, \c1, \d1, ror #24 + add \c2, \c2, \d2, ror #24 + eor \b1, \c1, \b1, ror #20 + eor \b2, \c2, \b2, ror #20 + // brot == 32 - 7 == 25 +.endm + +.macro _doubleround + + // column round + + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13) + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13 + + // save (x8, x9); restore (x10, x11) + __strd X8_X10, X9_X11, sp, 0 + __ldrd X8_X10, X9_X11, sp, 8 + + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15) + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15 + + .set brot, 25 + .set drot, 24 + + // diagonal round + + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12) + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12 + + // save (x10, x11); restore (x8, x9) + __strd X8_X10, X9_X11, sp, 8 + __ldrd X8_X10, X9_X11, sp, 0 + + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14) + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14 +.endm + +.macro _chacha_permute nrounds + .set brot, 0 + .set drot, 0 + .rept \nrounds / 2 + _doubleround + .endr +.endm + +.macro _chacha nrounds + +.Lnext_block\@: + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + + // Do the core ChaCha permutation to update x0-x15. + _chacha_permute \nrounds + + add sp, #8 + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15). + push {X8_X10, X9_X11, X12, X13, X14, X15} + + // Load (OUT, IN, LEN). + ldr r14, [sp, #96] + ldr r12, [sp, #100] + ldr r11, [sp, #104] + + orr r10, r14, r12 + + // Use slow path if fewer than 64 bytes remain. + cmp r11, #64 + blt .Lxor_slowpath\@ + + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on + // ARMv6+, since ldmia and stmia (used below) still require alignment. + tst r10, #3 + bne .Lxor_slowpath\@ + + // Fast path: XOR 64 bytes of aligned data. + + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // x0-x3 + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + ldmia r12!, {r8-r11} + eor X0, X0, r8 + eor X1, X1, r9 + eor X2, X2, r10 + eor X3, X3, r11 + stmia r14!, {X0-X3} + + // x4-x7 + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + ldmia r12!, {X0-X3} + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + eor X4, X4, X0 + eor X5, X5, X1 + eor X6, X6, X2 + eor X7, X7, X3 + stmia r14!, {X4-X7} + + // x8-x15 + pop {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + ldmia r12!, {r8-r11} + eor r0, r0, r8 // x8 + eor r1, r1, r9 // x9 + eor r6, r6, r10 // x10 + eor r7, r7, r11 // x11 + stmia r14!, {r0,r1,r6,r7} + ldmia r12!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + ldr r9, [sp, #72] // load LEN + eor r2, r2, r0 // x12 + eor r3, r3, r1 // x13 + eor r4, r4, r6 // x14 + eor r5, r5, r7 // x15 + subs r9, #64 // decrement and check LEN + stmia r14!, {r2-r5} + + beq .Ldone\@ + +.Lprepare_for_next_block\@: + + // Stack: x0-x15 OUT IN LEN + + // Increment block counter (x12) + add r8, #1 + + // Store updated (OUT, IN, LEN) + str r14, [sp, #64] + str r12, [sp, #68] + str r9, [sp, #72] + + mov r14, sp + + // Store updated block counter (x12) + str r8, [sp, #48] + + sub sp, #16 + + // Reload state and do next block + ldmia r14!, {r0-r11} // load x0-x11 + __strd r10, r11, sp, 8 // store x10-x11 before state + ldmia r14, {r10-r12,r14} // load x12-x15 + b .Lnext_block\@ + +.Lxor_slowpath\@: + // Slow path: < 64 bytes remaining, or unaligned input or output buffer. + // We handle it by storing the 64 bytes of keystream to the stack, then + // XOR-ing the needed portion with the data. + + // Allocate keystream buffer + sub sp, #64 + mov r14, sp + + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Save keystream for x0-x3 + __ldrd r8, r9, sp, 96 + __ldrd r10, r11, sp, 104 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + stmia r14!, {X0-X3} + + // Save keystream for x4-x7 + __ldrd r8, r9, sp, 112 + __ldrd r10, r11, sp, 120 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + add r8, sp, #64 + stmia r14!, {X4-X7} + + // Save keystream for x8-x15 + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 128 + __ldrd r10, r11, sp, 136 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + stmia r14!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 144 + __ldrd r10, r11, sp, 152 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + stmia r14, {r2-r5} + + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN + // Registers: r8 is block counter, r12 is IN. + + ldr r9, [sp, #168] // LEN + ldr r14, [sp, #160] // OUT + cmp r9, #64 + mov r0, sp + movle r1, r9 + movgt r1, #64 + // r1 is number of bytes to XOR, in range [1, 64] + +.if __LINUX_ARM_ARCH__ < 6 + orr r2, r12, r14 + tst r2, #3 // IN or OUT misaligned? + bne .Lxor_next_byte\@ +.endif + + // XOR a word at a time +.rept 16 + subs r1, #4 + blt .Lxor_words_done\@ + ldr r2, [r12], #4 + ldr r3, [r0], #4 + eor r2, r2, r3 + str r2, [r14], #4 +.endr + b .Lxor_slowpath_done\@ +.Lxor_words_done\@: + ands r1, r1, #3 + beq .Lxor_slowpath_done\@ + + // XOR a byte at a time +.Lxor_next_byte\@: + ldrb r2, [r12], #1 + ldrb r3, [r0], #1 + eor r2, r2, r3 + strb r2, [r14], #1 + subs r1, #1 + bne .Lxor_next_byte\@ + +.Lxor_slowpath_done\@: + subs r9, #64 + add sp, #96 + bgt .Lprepare_for_next_block\@ + +.Ldone\@: +.endm // _chacha + +/* + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8], + * const u32 iv[4]); + */ +ENTRY(chacha20_arm) + cmp r2, #0 // len == 0? + reteq lr + + push {r0-r2,r4-r11,lr} + + // Push state x0-x15 onto stack. + // Also store an extra copy of x10-x11 just before the state. + + ldr r4, [sp, #48] // iv + mov r0, sp + sub sp, #80 + + // iv: x12-x15 + ldm r4, {X12,X13,X14,X15} + stmdb r0!, {X12,X13,X14,X15} + + // key: x4-x11 + __ldrd X8_X10, X9_X11, r3, 24 + __strd X8_X10, X9_X11, sp, 8 + stmdb r0!, {X8_X10, X9_X11} + ldm r3, {X4-X9_X11} + stmdb r0!, {X4-X9_X11} + + // constants: x0-x3 + adrl X3, .Lexpand_32byte_k + ldm X3, {X0-X3} + __strd X0, X1, sp, 16 + __strd X2, X3, sp, 24 + + _chacha 20 + + add sp, #76 + pop {r4-r11, pc} +ENDPROC(chacha20_arm) + +/* + * void hchacha20_arm(const u32 state[16], u32 out[8]); + */ +ENTRY(hchacha20_arm) + push {r1,r4-r11,lr} + + mov r14, r0 + ldmia r14!, {r0-r11} // load x0-x11 + push {r10-r11} // store x10-x11 to stack + ldm r14, {r10-r12,r14} // load x12-x15 + sub sp, #8 + + _chacha_permute 20 + + // Skip over (unused0-unused1, x10-x11) + add sp, #16 + + // Fix up rotations of x12-x15 + ror X12, X12, #drot + ror X13, X13, #drot + pop {r4} // load 'out' + ror X14, X14, #drot + ror X15, X15, #drot + + // Store (x0-x3,x12-x15) to 'out' + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15} + + pop {r4-r11,pc} +ENDPROC(hchacha20_arm)lib/zinc/chacha20/chacha20.c +2
-0@@ -18,6 +18,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "chacha20-x86_64-glue.c" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 ) +#include "chacha20-arm-glue.c" #else static bool *const chacha20_nobs[]__initconst = { };static void __init chacha20_fpu_init (void )
Jason A. Donenfeld
maintainerPATCH v9 6/19 zinc: ChaCha20 MIPS32r2 implementation
In reply to Jason A. DonenfeldThis MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets.
--- lib/zinc/Makefile | 2 + lib/zinc/chacha20/chacha20-mips-glue.c | 27 ++ lib/zinc/chacha20/chacha20-mips.S | 424 +++++++++++++++++++++++++ lib/zinc/chacha20/chacha20.c | 2 + 4 files changed, 455 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-mips-glue.c create mode 100644 lib/zinc/chacha20/chacha20-mips.S
lib/zinc/Makefile +2 -0
@@ -6,6 +6,8 @@ zinc_chacha20-y := chacha20/chacha20.o zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o chacha20/chacha20-unrolled-arm.o zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_MIPS) += chacha20/chacha20-mips.o +AFLAGS_chacha20-mips.o += -O2 # This is required to fill the branch delay slots perlasm-targets += chacha20/chacha20-x86_64.S chacha20/chacha20-arm.S chacha20/chacha20-arm64.S obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-mips-glue.c new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +asmlinkage void chacha20_mips(u32 state[16], u8 *out, const u8 *in, +const size_t len); +static bool *const chacha20_nobs[]__initconst = { }; +static void __init chacha20_fpu_init (void ) +{ +} + +static inline bool chacha20_arch (struct chacha20_ctx *ctx,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{ +chacha20_mips (ctx->state, dst, src, len); +return true; +} + +static inline bool hchacha20_arch (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ], +simd_context_t *simd_context) +{ +return false; +}
lib/zinc/chacha20/chacha20-mips.S new file +424 -0
424 lines not on this page · raw message
lib/zinc/chacha20/chacha20.c +2 -0
@@ -20,6 +20,8 @@#include "chacha20-x86_64-glue.c" #elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 )#include "chacha20-arm-glue.c" +#elif defined(CONFIG_ZINC_ARCH_MIPS ) +#include "chacha20-mips-glue.c" #else static bool *const chacha20_nobs[]__initconst = { };static void __init chacha20_fpu_init (void )
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 7/19 zinc: Poly1305 generic C implementations and selftest
In reply to Jason A. DonenfeldThese two C implementations -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel and for usage with accelerated primitives.
Information: https://cr.yp.to/mac.html
--- include/zinc/poly1305.h | 31 + lib/zinc/Kconfig | 3 + lib/zinc/Makefile | 3 + lib/zinc/poly1305/poly1305-donna32.c | 205 +++++ lib/zinc/poly1305/poly1305-donna64.c | 182 +++++ lib/zinc/poly1305/poly1305.c | 154 ++++ lib/zinc/selftest/poly1305.c | 1107 ++++++++++++++++++++++++++ 7 files changed, 1685 insertions(+) create mode 100644 include/zinc/poly1305.h create mode 100644 lib/zinc/poly1305/poly1305-donna32.c create mode 100644 lib/zinc/poly1305/poly1305-donna64.c create mode 100644 lib/zinc/poly1305/poly1305.c create mode 100644 lib/zinc/selftest/poly1305.c
include/zinc/poly1305.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_POLY1305_H +#define _ZINC_POLY1305_H + +#include <linux/simd.h> +#include <linux/types.h> + +enum poly1305_lengths { +POLY1305_BLOCK_SIZE =16 , +POLY1305_KEY_SIZE =32 , +POLY1305_MAC_SIZE =16 +}; + +struct poly1305_ctx { +u8 opaque[24 *sizeof (u64)]; +u32 nonce[4 ]; +u8 data[POLY1305_BLOCK_SIZE ]; +size_t num; +}__aligned(8) ; + +void poly1305_init (struct poly1305_ctx *ctx,const u8 key[POLY1305_KEY_SIZE ]); +void poly1305_update (struct poly1305_ctx *ctx,const u8 *input,size_t len, +simd_context_t *simd_context); +void poly1305_final (struct poly1305_ctx *ctx,u8 mac[POLY1305_MAC_SIZE ], +simd_context_t *simd_context); + +#endif /* _ZINC_POLY1305_H */
lib/zinc/Kconfig +3 -0
@@ -2,6 +2,9 @@ config ZINC_CHACHA20 tristate select CRYPTO_ALGAPI +config ZINC_POLY1305 + tristate + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" default y
lib/zinc/Makefile +3 -0
@@ -11,6 +11,9 @@ AFLAGS_chacha20-mips.o += -O2 # This is required to fill the branch delay slots perlasm-targets += chacha20/chacha20-x86_64.S chacha20/chacha20-arm.S chacha20/chacha20-arm64.S obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o +zinc_poly1305-y := poly1305/poly1305.o +obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(addprefix $(obj)/,$(perlasm-targets)): $(obj)/%.S: $(src)/%.pl FORCE
lib/zinc/poly1305/poly1305-donna32.c new file +205 -0
205 lines not on this page · raw message
lib/zinc/poly1305/poly1305-donna64.c new file +182 -0
182 lines not on this page · raw message
lib/zinc/poly1305/poly1305.c new file +154 -0
154 lines not on this page · raw message
lib/zinc/selftest/poly1305.c new file +1107 -0
1,107 lines not on this page · raw message
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 8/19 zinc: Poly1305 x86_64 implementation
In reply to Jason A. DonenfeldThese x86_64 vectorized implementations are based on Andy Polyakov's implementation, and support AVX, AVX-2, and AVX512F. The AVX-512F implementation is disabled on Skylake, due to throttling.
The AVX code uses base 2^26, while the scalar code uses base 2^64. If we hit the unfortunate situation of using AVX and then having to go back to scalar -- because the user is silly and has called the update function from two separate contexts -- then we need to convert back to the original base before proceeding. It is possible to reason that the initial reduction below is sufficient given the implementation invariants. However, for an avoidance of doubt and because this is not performance critical, we do the full reduction anyway. This conversion is found in the glue code, and a proof of correctness may be easily obtained from Z3: <https://xn--4db.cc/ltPtHCKN/py>.
On the left is cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"). On the right are benchmarks on a Xeon Gold 5120 using the AVX-512 codepath. The difference is so stark, because the current crypto api's implementation does not support AVX-512 at all.
AVX-2 AVX-512
--------- -----------size old new size old new
---- ---- ---- ---- ---- ----
0 70 68 0 74 70
16 92 90 16 96 92
32 134 104 32 136 106 48 172 120 48 184 124 64 218 136 64 218 138 80 254 158 80 260 160 96 298 174 96 300 176 112 342 192 112 342 194 128 388 212 128 384 212 144 428 228 144 420 226 160 466 246 160 464 248 176 510 264 176 504 264 192 550 282 192 544 282 208 594 302 208 582 300 224 628 316 224 624 318 240 676 334 240 662 338 256 716 354 256 708 358 272 764 374 272 748 372 288 802 352 288 788 358 304 420 366 304 422 370 320 428 360 320 432 364 336 484 378 336 486 380 352 426 384 352 434 390 368 478 400 368 480 408 384 488 394 384 490 398 400 542 408 400 542 412 416 486 416 416 492 426 432 534 430 432 538 436 448 544 422 448 546 432 464 600 438 464 600 448 480 540 448 480 548 456 496 594 464 496 594 476 512 602 456 512 606 470 528 656 476 528 656 480 544 600 480 544 606 498 560 650 494 560 652 512 576 664 490 576 662 508 592 714 508 592 716 522 608 656 514 608 664 538 624 708 532 624 710 552 640 716 524 640 720 516 656 770 536 656 772 526 672 716 548 672 722 544 688 770 562 688 768 556 704 774 552 704 778 556 720 826 568 720 832 568 736 768 574 736 780 584 752 822 592 752 826 600 768 830 584 768 836 560 784 884 602 784 888 572 800 828 610 800 838 588 816 884 628 816 884 604 832 888 618 832 894 598 848 942 632 848 946 612 864 884 644 864 896 628 880 936 660 880 942 644 896 948 652 896 952 608 912 1000 664 912 1004 616 928 942 676 928 954 634 944 994 690 944 1000 646 960 1002 680 960 1008 646 976 1054 694 976 1062 658 992 1002 706 992 1012 674
1008 1052 720 1008 1058 690
While this is CRYPTOGAMS code, the originating code for this happens to be derived from OpenSSL's commit 4dfe4310c31c4483705991d9a798ce9be1ed1c68
--- lib/zinc/Makefile | 2 + lib/zinc/poly1305/poly1305-x86_64-glue.c | 154 + lib/zinc/poly1305/poly1305-x86_64.pl | 4266 ++++++++++++++++++++++ lib/zinc/poly1305/poly1305.c | 4 + 4 files changed, 4426 insertions(+) create mode 100644 lib/zinc/poly1305/poly1305-x86_64-glue.c create mode 100644 lib/zinc/poly1305/poly1305-x86_64.pl
lib/zinc/Makefile +2 -0
@@ -12,6 +12,8 @@ perlasm-targets += chacha20/chacha20-x86_64.S chacha20/chacha20-arm.S chacha20/c obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o zinc_poly1305-y := poly1305/poly1305.o +zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o +perlasm-targets += poly1305/poly1305-x86_64.S obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/poly1305/poly1305-x86_64-glue.c new file +154 -0
154 lines not on this page · raw message
lib/zinc/poly1305/poly1305-x86_64.pl new file +4266 -0
2,838 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v9 9/19 zinc: Poly1305 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese ARM implementations come from Andy Polyakov's implementations, and perform extremely well on ARMv4+, with optimized paths for NEON and non-NEON.
The NEON code uses base 2^26, while the scalar code uses base 2^64 on 64-bit and base 2^32 on 32-bit. If we hit the unfortunate situation of using NEON and then having to go back to scalar -- because the user is silly and has called the update function from two separate contexts -- then we need to convert back to the original base before proceeding. It is possible to reason that the initial reduction below is sufficient given the implementation invariants. However, for an avoidance of doubt and because this is not performance critical, we do the full reduction anyway. This conversion is found in the glue code, and a proof of correctness may be easily obtained from Z3: <https://xn--4db.cc/ltPtHCKN/py>.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 9925a82146f2503b4dd11423d0eba649b43450c0
--- lib/zinc/Makefile | 4 +- lib/zinc/poly1305/poly1305-arm-glue.c | 140 +++ lib/zinc/poly1305/poly1305-arm.pl | 1276 +++++++++++++++++++++++++ lib/zinc/poly1305/poly1305-arm64.pl | 974 +++++++++++++++++++ lib/zinc/poly1305/poly1305.c | 2 + 5 files changed, 2395 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.c create mode 100644 lib/zinc/poly1305/poly1305-arm.pl create mode 100644 lib/zinc/poly1305/poly1305-arm64.pl
lib/zinc/Makefile +3 -1
@@ -13,7 +13,9 @@ obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o zinc_poly1305-y := poly1305/poly1305.o zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o -perlasm-targets += poly1305/poly1305-x86_64.S +zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM) += poly1305/poly1305-arm.o +zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM64) += poly1305/poly1305-arm64.o +perlasm-targets += poly1305/poly1305-x86_64.S poly1305/poly1305-arm.S poly1305/poly1305-arm64.S obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/poly1305/poly1305-arm-glue.c new file +140 -0
140 lines not on this page · raw message
lib/zinc/poly1305/poly1305-arm.pl new file +1276 -0
1,276 lines not on this page · raw message
lib/zinc/poly1305/poly1305-arm64.pl new file +974 -0
974 lines not on this page · raw message
lib/zinc/poly1305/poly1305.c +2 -0
@@ -18,6 +18,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "poly1305-x86_64-glue.c" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 ) +#include "poly1305-arm-glue.c" #else static inline bool poly1305_init_arch (void *ctx,const u8 key[POLY1305_KEY_SIZE ])
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 10/19 zinc: Poly1305 MIPS64 and MIPS32r2 implementations
In reply to Jason A. DonenfeldThe MIPS64 accelerated implementation comes from Andy Polyakov's implementation and provides nice speedups on commodity Octeon hardware. While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 9925a82146f2503b4dd11423d0eba649b43450c0
This MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets.
--- lib/zinc/Makefile | 5 +- lib/zinc/poly1305/poly1305-mips-glue.c | 37 ++ lib/zinc/poly1305/poly1305-mips.S | 407 +++++++++++++++++++++ lib/zinc/poly1305/poly1305-mips64.pl | 467 +++++++++++++++++++++++++ lib/zinc/poly1305/poly1305.c | 2 + 5 files changed, 917 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/poly1305/poly1305-mips-glue.c create mode 100644 lib/zinc/poly1305/poly1305-mips.S create mode 100644 lib/zinc/poly1305/poly1305-mips64.pl
lib/zinc/Makefile +4 -1
@@ -15,7 +15,10 @@ zinc_poly1305-y := poly1305/poly1305.o zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM) += poly1305/poly1305-arm.o zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM64) += poly1305/poly1305-arm64.o -perlasm-targets += poly1305/poly1305-x86_64.S poly1305/poly1305-arm.S poly1305/poly1305-arm64.S +zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS) += poly1305/poly1305-mips.o +AFLAGS_poly1305-mips.o += -O2 # This is required to fill the branch delay slots +zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS64) += poly1305/poly1305-mips64.o +perlasm-targets += poly1305/poly1305-x86_64.S poly1305/poly1305-arm.S poly1305/poly1305-arm64.S poly1305/poly1305-mips64.S obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/poly1305/poly1305-mips-glue.c new file +37 -0
@@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +asmlinkage void poly1305_init_mips(void *ctx, const u8 key[16]); +asmlinkage void poly1305_blocks_mips(void *ctx, const u8 *inp, const size_t len, +const u32 padbit); +asmlinkage void poly1305_emit_mips(void *ctx, u8 mac[16], const u32 nonce[4]); + +static bool *const poly1305_nobs[]__initconst = { }; +static void __init poly1305_fpu_init (void ) +{ +} + +static inline bool poly1305_init_arch (void *ctx, +const u8 key[POLY1305_KEY_SIZE ]) +{ +poly1305_init_mips (ctx, key); +return true; +} + +static inline bool poly1305_blocks_arch (void *ctx,const u8 *inp, +size_t len,const u32 padbit, +simd_context_t *simd_context) +{ +poly1305_blocks_mips (ctx, inp, len, padbit); +return true; +} + +static inline bool poly1305_emit_arch (void *ctx,u8 mac[POLY1305_MAC_SIZE ], +const u32 nonce[4 ], +simd_context_t *simd_context) +{ +poly1305_emit_mips (ctx, mac, nonce); +return true; +}
lib/zinc/poly1305/poly1305-mips.S new file +407 -0
407 lines not on this page · raw message
lib/zinc/poly1305/poly1305-mips64.pl new file +467 -0
467 lines not on this page · raw message
lib/zinc/poly1305/poly1305.c +2 -0
@@ -20,6 +20,8 @@#include "poly1305-x86_64-glue.c" #elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 )#include "poly1305-arm-glue.c" +#elif defined(CONFIG_ZINC_ARCH_MIPS ) || defined(CONFIG_ZINC_ARCH_MIPS64 ) +#include "poly1305-mips-glue.c" #else static inline bool poly1305_init_arch (void *ctx,const u8 key[POLY1305_KEY_SIZE ])
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 11/19 zinc: ChaCha20Poly1305 construction and selftest
In reply to Jason A. DonenfeldThis is an implementation of the ChaCha20Poly1305 AEAD, with an easy API for encrypting either contiguous buffers or scatter gather lists (such as those created from skb_to_sgvec).
Information: https://tools.ietf.org/html/rfc8439
--- include/zinc/chacha20poly1305.h | 50 + lib/zinc/Kconfig | 6 + lib/zinc/Makefile | 3 + lib/zinc/chacha20poly1305.c | 358 + lib/zinc/selftest/chacha20poly1305.c | 9034 ++++++++++++++++++++++++++ 5 files changed, 9451 insertions(+) create mode 100644 include/zinc/chacha20poly1305.h create mode 100644 lib/zinc/chacha20poly1305.c create mode 100644 lib/zinc/selftest/chacha20poly1305.c
include/zinc/chacha20poly1305.h new file +50 -0
@@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20POLY1305_H +#define _ZINC_CHACHA20POLY1305_H + +#include <linux/simd.h> +#include <linux/types.h> + +struct scatterlist ; + +enum chacha20poly1305_lengths { +XCHACHA20POLY1305_NONCE_SIZE =24 , +CHACHA20POLY1305_KEY_SIZE =32 , +CHACHA20POLY1305_AUTHTAG_SIZE =16 +}; + +void chacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool __must_check chacha20poly1305_encrypt_sg ( +struct scatterlist *dst,struct scatterlist *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ],simd_context_t *simd_context); + +bool __must_check +chacha20poly1305_decrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool __must_check chacha20poly1305_decrypt_sg ( +struct scatterlist *dst,struct scatterlist *src,const size_t src_len, +const u8 *ad,const size_t ad_len,const u64 nonce, +const u8 key[CHACHA20POLY1305_KEY_SIZE ],simd_context_t *simd_context); + +void xchacha20poly1305_encrypt (u8 *dst,const u8 *src,const size_t src_len, +const u8 *ad,const size_t ad_len, +const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE ], +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +bool __must_check xchacha20poly1305_decrypt ( +u8 *dst,const u8 *src,const size_t src_len,const u8 *ad, +const size_t ad_len,const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE ], +const u8 key[CHACHA20POLY1305_KEY_SIZE ]); + +#endif /* _ZINC_CHACHA20POLY1305_H */
lib/zinc/Kconfig +6 -0
@@ -5,6 +5,12 @@ config ZINC_CHACHA20 config ZINC_POLY1305 tristate +config ZINC_CHACHA20POLY1305 + tristate + select ZINC_CHACHA20 + select ZINC_POLY1305 + select CRYPTO_BLKCIPHER + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" default y
lib/zinc/Makefile +3 -0
@@ -21,6 +21,9 @@ zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS64) += poly1305/poly1305-mips64.o perlasm-targets += poly1305/poly1305-x86_64.S poly1305/poly1305-arm.S poly1305/poly1305-arm64.S poly1305/poly1305-mips64.S obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o +zinc_chacha20poly1305-y := chacha20poly1305.o +obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(addprefix $(obj)/,$(perlasm-targets)): $(obj)/%.S: $(src)/%.pl FORCE
lib/zinc/chacha20poly1305.c new file +358 -0
358 lines not on this page · raw message
lib/zinc/selftest/chacha20poly1305.c new file +9034 -0
2,571 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v9 12/19 zinc: BLAKE2s generic C implementation and selftest
In reply to Jason A. DonenfeldThe C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.
Information: https://blake2.net/
--- include/zinc/blake2s.h | 56 + lib/zinc/Kconfig | 3 + lib/zinc/Makefile | 3 + lib/zinc/blake2s/blake2s.c | 295 +++++ lib/zinc/selftest/blake2s.c | 2090 +++++++++++++++++++++++++++++++++++ 5 files changed, 2447 insertions(+) create mode 100644 include/zinc/blake2s.h create mode 100644 lib/zinc/blake2s/blake2s.c create mode 100644 lib/zinc/selftest/blake2s.c
include/zinc/blake2s.h new file +56 -0
@@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_BLAKE2S_H +#define _ZINC_BLAKE2S_H + +#include <linux/types.h> +#include <linux/kernel.h> +#include <asm/bug.h> + +enum blake2s_lengths { +BLAKE2S_BLOCK_SIZE =64 , +BLAKE2S_HASH_SIZE =32 , +BLAKE2S_KEY_SIZE =32 +}; + +struct blake2s_state { +u32 h[8 ]; +u32 t[2 ]; +u32 f[2 ]; +u8 buf[BLAKE2S_BLOCK_SIZE ]; +size_t buflen; +u8 last_node; +}; + +void blake2s_init (struct blake2s_state *state,const size_t outlen); +void blake2s_init_key (struct blake2s_state *state,const size_t outlen, +const void *key,const size_t keylen); +void blake2s_update (struct blake2s_state *state,const u8 *in,size_t inlen); +void blake2s_final (struct blake2s_state *state,u8 *out,const size_t outlen); + +static inline void blake2s (u8 *out,const u8 *in,const u8 *key, +const size_t outlen,const size_t inlen, +const size_t keylen) +{ +struct blake2s_state state; + +WARN_ON (IS_ENABLED (DEBUG ) && ((!in && inlen >0 ) || !out || !outlen || + outlen >BLAKE2S_HASH_SIZE || keylen >BLAKE2S_KEY_SIZE || + (!key && keylen))); + +if (keylen) +blake2s_init_key (&state, outlen, key, keylen); +else +blake2s_init (&state, outlen); + +blake2s_update (&state, in, inlen); +blake2s_final (&state, out, outlen); +} + +void blake2s_hmac (u8 *out,const u8 *in,const u8 *key,const size_t outlen, +const size_t inlen,const size_t keylen); + +#endif /* _ZINC_BLAKE2S_H */
lib/zinc/Kconfig +3 -0
@@ -11,6 +11,9 @@ config ZINC_CHACHA20POLY1305 select ZINC_POLY1305 select CRYPTO_BLKCIPHER +config ZINC_BLAKE2S + tristate + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" default y
lib/zinc/Makefile +3 -0
@@ -24,6 +24,9 @@ obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o zinc_chacha20poly1305-y := chacha20poly1305.o obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o +zinc_blake2s-y := blake2s/blake2s.o +obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(addprefix $(obj)/,$(perlasm-targets)): $(obj)/%.S: $(src)/%.pl FORCE
lib/zinc/blake2s/blake2s.c new file +295 -0
295 lines not on this page · raw message
lib/zinc/selftest/blake2s.c new file +2090 -0
2,090 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH net-next v9 12/19] zinc: BLAKE2s generic C implementation and selftest
Hi Jason, a few comments on this patch:
Quoted 40 lines · Jason A. Donenfeld
The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.
Information: https://blake2.net/
Signed-off-by Jason A. Donenfeld <[email protected]>Signed-off-by Samuel Neves <[email protected]>Co-developed-by Samuel Neves <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- include/zinc/blake2s.h | 56 + lib/zinc/Kconfig | 3 + lib/zinc/Makefile | 3 + lib/zinc/blake2s/blake2s.c | 295 +++++ lib/zinc/selftest/blake2s.c | 2090 +++++++++++++++++++++++++++++++++++ 5 files changed, 2447 insertions(+) create mode 100644 include/zinc/blake2s.h create mode 100644 lib/zinc/blake2s/blake2s.c create mode 100644 lib/zinc/selftest/blake2s.cinclude/zinc/blake2s.h new file +6
-0@@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_BLAKE2S_H
include/zinc/blake2s.h line 16
Quoted 10 lines
include/zinc/blake2s.h new file +10
-0+#define _ZINC_BLAKE2S_H + +#include <linux/types.h> +#include <linux/kernel.h> +#include <asm/bug.h> + +enum blake2s_lengths { +BLAKE2S_BLOCK_SIZE =64 , +BLAKE2S_HASH_SIZE =32 , +BLAKE2S_KEY_SIZE =32
Perhaps call the latter two BLAKE2S_MAX_HASH_SIZE and BLAKE2S_MAX_KEY_SIZE, since lower values can be selected too?
include/zinc/blake2s.h line 26
Quoted 10 lines
include/zinc/blake2s.h new file +10
-0+}; + +struct blake2s_state { +u32 h[8 ]; +u32 t[2 ]; +u32 f[2 ]; +u8 buf[BLAKE2S_BLOCK_SIZE ]; +size_t buflen; +u8 last_node; +};
Since this API doesn't actually support any of BLAKE2's tree hashing modes, I think 'last_node' should be removed from this struct for now. It's never set.
include/zinc/blake2s.h line 32
Quoted 6 lines
include/zinc/blake2s.h new file +6
-0+ +void blake2s_init (struct blake2s_state *state,const size_t outlen); +void blake2s_init_key (struct blake2s_state *state,const size_t outlen, +const void *key,const size_t keylen); +void blake2s_update (struct blake2s_state *state,const u8 *in,size_t inlen); +void blake2s_final (struct blake2s_state *state,u8 *out,const size_t outlen);
Since the caller must pass the same 'outlen' into blake2s_init() and blake2s_final(), perhaps that should be verified by the DEBUG checks? It would be easy for someone to get this wrong. Or remove the outlen argument from blake2s_final(), instead using the saved length from blake2s_init()?
lib/zinc/blake2s/blake2s.c line 3
Quoted 9 lines
lib/zinc/blake2s/blake2s.c new file +3
-0@@ -0,0 +1,295 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2012 Samuel Neves <[email protected]>. All Rights Reserved.
If Samuel's reference implementation is "public domain" as you stated in the commit message, why is this copyright statement from 2012 here? Public domain == not copyrighted.
Quoted 18 lines
lib/zinc/blake2s/blake2s.c new file +18
-0+* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +* +* This is an implementation of the BLAKE2s hash and PRF functions. +* +* Information: https://blake2.net/ +* +*/ + +#include <zinc/blake2s.h> +#include "../selftest/run.h" + +#include <linux/types.h> +#include <linux/string.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/bug.h> +#include <asm/unaligned.h>
lib/zinc/blake2s/blake2s.c line 31
Quoted 10 lines
lib/zinc/blake2s/blake2s.c new file +10
-0+ +typedef union { + struct { + u8 digest_length; + u8 key_length; + u8 fanout; + u8 depth; + u32 leaf_length; + u32 node_offset; + u16 xof_length;
Technically the leaf_length, node_offset, and xof_length fields should be __le32, __le32, and __le16 respectively.
Quoted 6 lines
lib/zinc/blake2s/blake2s.c new file +6
-0+void blake2s_update (struct blake2s_state *state,const u8 *in,size_t inlen) +{ +const size_t fill =BLAKE2S_BLOCK_SIZE - state->buflen; + +if (unlikely (!inlen)) +return ;
lib/zinc/blake2s/blake2s.c line 210
Quoted 10 lines
lib/zinc/blake2s/blake2s.c new file +10
-0+if (inlen > fill) { +memcpy (state->buf + state->buflen, in, fill); +blake2s_compress (state, state->buf,1 ,BLAKE2S_BLOCK_SIZE ); + state->buflen =0 ; + in += fill; + inlen -= fill; + } +if (inlen >BLAKE2S_BLOCK_SIZE ) { +const size_t nblocks = + (inlen +BLAKE2S_BLOCK_SIZE -1 ) /BLAKE2S_BLOCK_SIZE ;
This can be written as:
const size_t nblocks = DIV_ROUND_UP(inlen, BLAKE2S_BLOCK_SIZE);
Quoted 9 lines
lib/zinc/blake2s/blake2s.c new file +9
-0+ +static int __init mod_init (void ) +{ +if (!nosimd) +blake2s_fpu_init (); +if (!selftest_run ("blake2s" , blake2s_selftest, blake2s_nobs, +ARRAY_SIZE (blake2s_nobs))) +return -ENOTRECOVERABLE ; +return 0 ;
lib/zinc/blake2s/blake2s.c line 293
Quoted 10 lines
lib/zinc/blake2s/blake2s.c new file +10
-0+} + +static void __exit mod_exit(void) +{ +} + +module_param (nosimd, bool,0 ); +module_init (mod_init); +module_exit (mod_exit); +MODULE_LICENSE ("GPL v2" );
The MODULE_LICENSE() doesn't match the SPDX license tag. Intentional?
Quoted 14 lines
lib/zinc/blake2s/blake2s.c new file +2
-0+MODULE_DESCRIPTION ("BLAKE2s hash function" ); +MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );lib/zinc/selftest/blake2s.c new file +6
-0@@ -0,0 +1,2090 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +static const u8 blake2s_testvecs[][BLAKE2S_HASH_SIZE ]__initconst = {
[snip]
Quoted 3 lines
lib/zinc/selftest/blake2s.c new file +3
-0+}; + +static const u8 blake2s_keyed_testvecs[][BLAKE2S_HASH_SIZE ]__initconst = {
[snip]
Quoted 1 line
+};
There should be a comment mentioning where these test vectors came from. Are they from a published source? Or did you use a different implementation of BLAKE2s to generate them?
Quoted 22 lines
lib/zinc/selftest/blake2s.c new file +22
-0+ +static bool __init blake2s_selftest (void ) +{ +u8 key[BLAKE2S_KEY_SIZE ]; +u8 buf[ARRAY_SIZE (blake2s_testvecs)]; +u8 hash[BLAKE2S_HASH_SIZE ]; +size_t i; +bool success = true; + +for (i =0 ; i <BLAKE2S_KEY_SIZE ; ++i) + key[i] = (u8 )i; + +for (i =0 ; i <ARRAY_SIZE (blake2s_testvecs); ++i) + buf[i] = (u8 )i; + +for (i =0 ; i <ARRAY_SIZE (blake2s_keyed_testvecs); ++i) { +blake2s (hash, buf, key,BLAKE2S_HASH_SIZE , i,BLAKE2S_KEY_SIZE ); +if (memcmp (hash, blake2s_keyed_testvecs[i],BLAKE2S_HASH_SIZE )) { +pr_err ("blake2s keyed self-test %zu: FAIL\n" , i +1 ); + success = false; + } + }
lib/zinc/selftest/blake2s.c line 2090
Quoted 10 lines
lib/zinc/selftest/blake2s.c new file +10
-0+ +for (i =0 ; i <ARRAY_SIZE (blake2s_testvecs); ++i) { +blake2s (hash, buf,NULL ,BLAKE2S_HASH_SIZE , i,0 ); +if (memcmp (hash, blake2s_testvecs[i],BLAKE2S_HASH_SIZE )) { +pr_err ("blake2s unkeyed self-test %zu: FAIL\n" , i + i); + success = false; + } + } +return success; +}
This tests here miss a lot of cases. I can break the implementation in some pretty major ways and it still passes the tests.
We'll get better test coverage of this via testmgr once this is exposed via the "regular" crypto API. E.g. the latest testmgr will automatically test multiple update() calls, and buffers with various alignments. If we also make your new 'simd_get()' function call crypto_simd_usable() rather than may_use_simd() directly, then testmgr will also cover the case where some update() are done in a SIMD-usable context and some aren't.
However, any functionality that won't be exposed by the regular crypto API still needs to be tested here, e.g. the following which aren't currently tested:
- blake2s_hmac()
- Key and digest lengths other than the default
- Eric
Jason A. Donenfeld
maintainerPATCH v9 13/19 zinc: BLAKE2s x86_64 implementation
In reply to Jason A. DonenfeldThese implementations from Samuel Neves support AVX and AVX-512VL. Originally this used AVX-512F, but Skylake thermal throttling made AVX-512VL more attractive and possible to do with negligable difference.
--- lib/zinc/Makefile | 1 + lib/zinc/blake2s/blake2s-x86_64-glue.c | 71 +++ lib/zinc/blake2s/blake2s-x86_64.S | 685 +++++++++++++++++++++++++ lib/zinc/blake2s/blake2s.c | 4 + 4 files changed, 761 insertions(+) create mode 100644 lib/zinc/blake2s/blake2s-x86_64-glue.c create mode 100644 lib/zinc/blake2s/blake2s-x86_64.S
lib/zinc/Makefile +1 -0
@@ -25,6 +25,7 @@ zinc_chacha20poly1305-y := chacha20poly1305.o obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o zinc_blake2s-y := blake2s/blake2s.o +zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/blake2s/blake2s-x86_64-glue.c new file +71 -0
71 lines not on this page · raw message
lib/zinc/blake2s/blake2s-x86_64.S new file +685 -0
685 lines not on this page · raw message
lib/zinc/blake2s/blake2s.c +4 -0
@@ -110,6 +110,9 @@ void blake2s_init_key(struct blake2s_state *state, const size_t outlen, }EXPORT_SYMBOL (blake2s_init_key); +#if defined(CONFIG_ZINC_ARCH_X86_64 ) +#include "blake2s-x86_64-glue.c" +#else static bool *const blake2s_nobs[]__initconst = { };static void __init blake2s_fpu_init (void ) {
@@ -120,6 +123,7 @@ static inline bool blake2s_compress_arch(struct blake2s_state *state, {return false; } +#endif static inline void blake2s_compress(struct blake2s_state *state, const u8 *block, size_t nblocks,
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 14/19 zinc: Curve25519 generic C implementations and selftest
In reply to Jason A. DonenfeldThis contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic.
The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star
The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto
Information: https://cr.yp.to/ecdh.html
--- include/zinc/curve25519.h | 28 + lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 3 + lib/zinc/curve25519/curve25519-fiat32.c | 860 +++++++++++++++ lib/zinc/curve25519/curve25519-hacl64.c | 784 ++++++++++++++ lib/zinc/curve25519/curve25519.c | 101 ++ lib/zinc/selftest/curve25519.c | 1315 +++++++++++++++++++++++ 7 files changed, 3095 insertions(+) create mode 100644 include/zinc/curve25519.h create mode 100644 lib/zinc/curve25519/curve25519-fiat32.c create mode 100644 lib/zinc/curve25519/curve25519-hacl64.c create mode 100644 lib/zinc/curve25519/curve25519.c create mode 100644 lib/zinc/selftest/curve25519.c
include/zinc/curve25519.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CURVE25519_H +#define _ZINC_CURVE25519_H + +#include <linux/types.h> + +enum curve25519_lengths { +CURVE25519_KEY_SIZE =32 +}; + +bool __must_check curve25519 (u8 mypublic[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ], +const u8 basepoint[CURVE25519_KEY_SIZE ]); +void curve25519_generate_secret (u8 secret[CURVE25519_KEY_SIZE ]); +bool __must_check curve25519_generate_public ( +u8 pub[CURVE25519_KEY_SIZE ],const u8 secret[CURVE25519_KEY_SIZE ]); + +static inline void curve25519_clamp_secret (u8 secret[CURVE25519_KEY_SIZE ]) +{ + secret[0 ] &=248 ; + secret[31 ] = (secret[31 ] &127 ) |64 ; +} + +#endif /* _ZINC_CURVE25519_H */
lib/zinc/Kconfig +4 -0
@@ -14,6 +14,10 @@ config ZINC_CHACHA20POLY1305 config ZINC_BLAKE2S tristate +config ZINC_CURVE25519 + tristate + select CONFIG_CRYPTO + config ZINC_SELFTEST bool "Zinc cryptography library self-tests" default y
lib/zinc/Makefile +3 -0
@@ -28,6 +28,9 @@ zinc_blake2s-y := blake2s/blake2s.o zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o +zinc_curve25519-y := curve25519/curve25519.o +obj-$(CONFIG_ZINC_CURVE25519) += zinc_curve25519.o + quiet_cmd_perlasm = PERLASM $@ cmd_perlasm = $(PERL) $< > $@ $(addprefix $(obj)/,$(perlasm-targets)): $(obj)/%.S: $(src)/%.pl FORCE
lib/zinc/curve25519/curve25519-fiat32.c new file +860 -0
860 lines not on this page · raw message
lib/zinc/curve25519/curve25519-hacl64.c new file +784 -0
784 lines not on this page · raw message
lib/zinc/curve25519/curve25519.c new file +101 -0
101 lines not on this page · raw message
lib/zinc/selftest/curve25519.c new file +1315 -0
1,208 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Jason A. Donenfeld
maintainerPATCH v9 15/19 zinc: Curve25519 x86_64 implementation
In reply to Jason A. DonenfeldThis implementation is the fastest available x86_64 implementation, and unlike Sandy2x, it doesn't requie use of the floating point registers at all. Instead it makes use of BMI2 and ADX, available on recent microarchitectures. The implementation was written by Armando Faz-Hernández with contributions (upstream) from Samuel Neves and me, in addition to further changes in the kernel implementation from us.
--- lib/zinc/curve25519/curve25519-x86_64-glue.c | 48 + lib/zinc/curve25519/curve25519-x86_64.c | 2326 ++++++++++++++++++ lib/zinc/curve25519/curve25519.c | 4 + 3 files changed, 2378 insertions(+) create mode 100644 lib/zinc/curve25519/curve25519-x86_64-glue.c create mode 100644 lib/zinc/curve25519/curve25519-x86_64.c
lib/zinc/curve25519/curve25519-x86_64-glue.c new file +48 -0
@@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <asm/cpufeature.h> +#include <asm/processor.h> + +#include "curve25519-x86_64.c" + +static bool curve25519_use_bmi2__ro_after_init ; +static bool curve25519_use_adx__ro_after_init ; +static bool *const curve25519_nobs[]__initconst = { + &curve25519_use_bmi2, &curve25519_use_adx }; + +static void __init curve25519_fpu_init (void ) +{ + curve25519_use_bmi2 =boot_cpu_has (X86_FEATURE_BMI2 ); + curve25519_use_adx =boot_cpu_has (X86_FEATURE_BMI2 ) && +boot_cpu_has (X86_FEATURE_ADX ); +} + +static inline bool curve25519_arch (u8 mypublic[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ], +const u8 basepoint[CURVE25519_KEY_SIZE ]) +{ +if (curve25519_use_adx) { +curve25519_adx (mypublic, secret, basepoint); +return true; + }else if (curve25519_use_bmi2) { +curve25519_bmi2 (mypublic, secret, basepoint); +return true; + } +return false; +} + +static inline bool curve25519_base_arch (u8 pub[CURVE25519_KEY_SIZE ], +const u8 secret[CURVE25519_KEY_SIZE ]) +{ +if (curve25519_use_adx) { +curve25519_adx_base (pub, secret); +return true; + }else if (curve25519_use_bmi2) { +curve25519_bmi2_base (pub, secret); +return true; + } +return false; +}
lib/zinc/curve25519/curve25519-x86_64.c new file +2326 -0
2,326 lines not on this page · raw message
lib/zinc/curve25519/curve25519.c +4 -0
@@ -20,6 +20,9 @@#include <linux/init.h> #include <crypto/algapi.h> // For crypto_memneq. +#if defined(CONFIG_ZINC_ARCH_X86_64 ) +#include "curve25519-x86_64-glue.c" +#else static bool *const curve25519_nobs[]__initconst = { };static void __init curve25519_fpu_init (void ) {
@@ -35,6 +38,7 @@ static inline bool curve25519_base_arch(u8 pub[CURVE25519_KEY_SIZE], {return false; } +#endif #if defined(CONFIG_ARCH_SUPPORTS_INT128 ) && defined(__SIZEOF_INT128__)#include "curve25519-hacl64.c"
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 16/19 zinc: import Bernstein and Schwabe's Curve25519 ARM implementation
In reply to Jason A. DonenfeldThis comes from Dan Bernstein and Peter Schwabe's public domain NEON code, and is included here in raw form so that subsequent commits that fix these up for the kernel can see how it has changed. This code does have some entirely cosmetic formatting differences, adding indentation and so forth, so that when we actually port it for use in the kernel in the subsequent commit, it's obvious what's changed in the process.
This code originates from SUPERCOP 20180818, available at
<https://bench.cr.yp.to/supercop.html>.
--- lib/zinc/curve25519/curve25519-arm-supercop.S | 2105 +++++++++++++++++ 1 file changed, 2105 insertions(+) create mode 100644 lib/zinc/curve25519/curve25519-arm-supercop.S
lib/zinc/curve25519/curve25519-arm-supercop.S new file +2105 -0
@@ -0,0 +1,2105 @@ +/* + * Public domain code from Daniel J. Bernstein and Peter Schwabe, from + * SUPERCOP's curve25519/neon2/scalarmult.s. + */ + +.fpu neon +.text +.align 4 +.global _crypto_scalarmult_curve25519_neon2 +.global crypto_scalarmult_curve25519_neon2 +.type _crypto_scalarmult_curve25519_neon2 STT_FUNC +.type crypto_scalarmult_curve25519_neon2 STT_FUNC + _crypto_scalarmult_curve25519_neon2: + crypto_scalarmult_curve25519_neon2: + vpush {q4, q5, q6, q7} + mov r12, sp + sub sp, sp, #736 + and sp, sp, #0xffffffe0 + strd r4, [sp, #0] + strd r6, [sp, #8] + strd r8, [sp, #16] + strd r10, [sp, #24] + str r12, [sp, #480] + str r14, [sp, #484] + mov r0, r0 + mov r1, r1 + mov r2, r2 + add r3, sp, #32 + ldr r4, =0 + ldr r5, =254 + vmov.i32 q0, #1 + vshr.u64 q1, q0, #7 + vshr.u64 q0, q0, #8 + vmov.i32 d4, #19 + vmov.i32 d5, #38 + add r6, sp, #512 + vst1.8 {d2-d3}, [r6, : 128] + add r6, sp, #528 + vst1.8 {d0-d1}, [r6, : 128] + add r6, sp, #544 + vst1.8 {d4-d5}, [r6, : 128] + add r6, r3, #0 + vmov.i32 q2, #0 + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 d4, [r6, : 64] + add r6, r3, #0 + ldr r7, =960 + sub r7, r7, #2 + neg r7, r7 + sub r7, r7, r7, LSL #7 + str r7, [r6] + add r6, sp, #704 + vld1.8 {d4-d5}, [r1]! + vld1.8 {d6-d7}, [r1] + vst1.8 {d4-d5}, [r6, : 128]! + vst1.8 {d6-d7}, [r6, : 128] + sub r1, r6, #16 + ldrb r6, [r1] + and r6, r6, #248 + strb r6, [r1] + ldrb r6, [r1, #31] + and r6, r6, #127 + orr r6, r6, #64 + strb r6, [r1, #31] + vmov.i64 q2, #0xffffffff + vshr.u64 q3, q2, #7 + vshr.u64 q2, q2, #6 + vld1.8 {d8}, [r2] + vld1.8 {d10}, [r2] + add r2, r2, #6 + vld1.8 {d12}, [r2] + vld1.8 {d14}, [r2] + add r2, r2, #6 + vld1.8 {d16}, [r2] + add r2, r2, #4 + vld1.8 {d18}, [r2] + vld1.8 {d20}, [r2] + add r2, r2, #6 + vld1.8 {d22}, [r2] + add r2, r2, #2 + vld1.8 {d24}, [r2] + vld1.8 {d26}, [r2] + vshr.u64 q5, q5, #26 + vshr.u64 q6, q6, #3 + vshr.u64 q7, q7, #29 + vshr.u64 q8, q8, #6 + vshr.u64 q10, q10, #25 + vshr.u64 q11, q11, #3 + vshr.u64 q12, q12, #12 + vshr.u64 q13, q13, #38 + vand q4, q4, q2 + vand q6, q6, q2 + vand q8, q8, q2 + vand q10, q10, q2 + vand q2, q12, q2 + vand q5, q5, q3 + vand q7, q7, q3 + vand q9, q9, q3 + vand q11, q11, q3 + vand q3, q13, q3 + add r2, r3, #48 + vadd.i64 q12, q4, q1 + vadd.i64 q13, q10, q1 + vshr.s64 q12, q12, #26 + vshr.s64 q13, q13, #26 + vadd.i64 q5, q5, q12 + vshl.i64 q12, q12, #26 + vadd.i64 q14, q5, q0 + vadd.i64 q11, q11, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q11, q0 + vsub.i64 q4, q4, q12 + vshr.s64 q12, q14, #25 + vsub.i64 q10, q10, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q12 + vshl.i64 q12, q12, #25 + vadd.i64 q14, q6, q1 + vadd.i64 q2, q2, q13 + vsub.i64 q5, q5, q12 + vshr.s64 q12, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q1 + vadd.i64 q7, q7, q12 + vshl.i64 q12, q12, #26 + vadd.i64 q15, q7, q0 + vsub.i64 q11, q11, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q12 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q3, q0 + vadd.i64 q8, q8, q12 + vshl.i64 q12, q12, #25 + vadd.i64 q15, q8, q1 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q7, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q9, q9, q12 + vtrn.32 d12, d14 + vshl.i64 q12, q12, #26 + vtrn.32 d13, d15 + vadd.i64 q0, q9, q0 + vadd.i64 q4, q4, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q6, q13, #4 + vsub.i64 q7, q8, q12 + vshr.s64 q0, q0, #25 + vadd.i64 q4, q4, q6 + vadd.i64 q6, q10, q0 + vshl.i64 q0, q0, #25 + vadd.i64 q8, q6, q1 + vadd.i64 q4, q4, q13 + vshl.i64 q10, q13, #25 + vadd.i64 q1, q4, q1 + vsub.i64 q0, q9, q0 + vshr.s64 q8, q8, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d14, d0 + vshr.s64 q1, q1, #26 + vtrn.32 d15, d1 + vadd.i64 q0, q11, q8 + vst1.8 d14, [r2, : 64] + vshl.i64 q7, q8, #26 + vadd.i64 q5, q5, q1 + vtrn.32 d4, d6 + vshl.i64 q1, q1, #26 + vtrn.32 d5, d7 + vsub.i64 q3, q6, q7 + add r2, r2, #16 + vsub.i64 q1, q4, q1 + vst1.8 d4, [r2, : 64] + vtrn.32 d6, d0 + vtrn.32 d7, d1 + sub r2, r2, #8 + vtrn.32 d2, d10 + vtrn.32 d3, d11 + vst1.8 d6, [r2, : 64] + sub r2, r2, #24 + vst1.8 d2, [r2, : 64] + add r2, r3, #96 + vmov.i32 q0, #0 + vmov.i64 d2, #0xff + vmov.i64 d3, #0 + vshr.u32 q1, q1, #7 + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #144 + vmov.i32 q0, #0 + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #240 + vmov.i32 q0, #0 + vmov.i64 d2, #0xff + vmov.i64 d3, #0 + vshr.u32 q1, q1, #7 + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 d0, [r2, : 64] + add r2, r3, #48 + add r6, r3, #192 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r6, : 128]! + vst1.8 {d2-d3}, [r6, : 128]! + vst1.8 d4, [r6, : 64] +._mainloop: + mov r2, r5, LSR #3 + and r6, r5, #7 + ldrb r2, [r1, r2] + mov r2, r2, LSR r6 + and r2, r2, #1 + str r5, [sp, #488] + eor r4, r4, r2 + str r2, [sp, #492] + neg r2, r4 + add r4, r3, #96 + add r5, r3, #192 + add r6, r3, #144 + vld1.8 {d8-d9}, [r4, : 128]! + add r7, r3, #240 + vld1.8 {d10-d11}, [r5, : 128]! + veor q6, q4, q5 + vld1.8 {d14-d15}, [r6, : 128]! + vdup.i32 q8, r2 + vld1.8 {d18-d19}, [r7, : 128]! + veor q10, q7, q9 + vld1.8 {d22-d23}, [r4, : 128]! + vand q6, q6, q8 + vld1.8 {d24-d25}, [r5, : 128]! + vand q10, q10, q8 + vld1.8 {d26-d27}, [r6, : 128]! + veor q4, q4, q6 + vld1.8 {d28-d29}, [r7, : 128]! + veor q5, q5, q6 + vld1.8 {d0}, [r4, : 64] + veor q6, q7, q10 + vld1.8 {d2}, [r5, : 64] + veor q7, q9, q10 + vld1.8 {d4}, [r6, : 64] + veor q9, q11, q12 + vld1.8 {d6}, [r7, : 64] + veor q10, q0, q1 + sub r2, r4, #32 + vand q9, q9, q8 + sub r4, r5, #32 + vand q10, q10, q8 + sub r5, r6, #32 + veor q11, q11, q9 + sub r6, r7, #32 + veor q0, q0, q10 + veor q9, q12, q9 + veor q1, q1, q10 + veor q10, q13, q14 + veor q12, q2, q3 + vand q10, q10, q8 + vand q8, q12, q8 + veor q12, q13, q10 + veor q2, q2, q8 + veor q10, q14, q10 + veor q3, q3, q8 + vadd.i32 q8, q4, q6 + vsub.i32 q4, q4, q6 + vst1.8 {d16-d17}, [r2, : 128]! + vadd.i32 q6, q11, q12 + vst1.8 {d8-d9}, [r5, : 128]! + vsub.i32 q4, q11, q12 + vst1.8 {d12-d13}, [r2, : 128]! + vadd.i32 q6, q0, q2 + vst1.8 {d8-d9}, [r5, : 128]! + vsub.i32 q0, q0, q2 + vst1.8 d12, [r2, : 64] + vadd.i32 q2, q5, q7 + vst1.8 d0, [r5, : 64] + vsub.i32 q0, q5, q7 + vst1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q2, q9, q10 + vst1.8 {d0-d1}, [r6, : 128]! + vsub.i32 q0, q9, q10 + vst1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q2, q1, q3 + vst1.8 {d0-d1}, [r6, : 128]! + vsub.i32 q0, q1, q3 + vst1.8 d4, [r4, : 64] + vst1.8 d0, [r6, : 64] + add r2, sp, #544 + add r4, r3, #96 + add r5, r3, #144 + vld1.8 {d0-d1}, [r2, : 128] + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4-d5}, [r5, : 128]! + vzip.i32 q1, q2 + vld1.8 {d6-d7}, [r4, : 128]! + vld1.8 {d8-d9}, [r5, : 128]! + vshl.i32 q5, q1, #1 + vzip.i32 q3, q4 + vshl.i32 q6, q2, #1 + vld1.8 {d14}, [r4, : 64] + vshl.i32 q8, q3, #1 + vld1.8 {d15}, [r5, : 64] + vshl.i32 q9, q4, #1 + vmul.i32 d21, d7, d1 + vtrn.32 d14, d15 + vmul.i32 q11, q4, q0 + vmul.i32 q0, q7, q0 + vmull.s32 q12, d2, d2 + vmlal.s32 q12, d11, d1 + vmlal.s32 q12, d12, d0 + vmlal.s32 q12, d13, d23 + vmlal.s32 q12, d16, d22 + vmlal.s32 q12, d7, d21 + vmull.s32 q10, d2, d11 + vmlal.s32 q10, d4, d1 + vmlal.s32 q10, d13, d0 + vmlal.s32 q10, d6, d23 + vmlal.s32 q10, d17, d22 + vmull.s32 q13, d10, d4 + vmlal.s32 q13, d11, d3 + vmlal.s32 q13, d13, d1 + vmlal.s32 q13, d16, d0 + vmlal.s32 q13, d17, d23 + vmlal.s32 q13, d8, d22 + vmull.s32 q1, d10, d5 + vmlal.s32 q1, d11, d4 + vmlal.s32 q1, d6, d1 + vmlal.s32 q1, d17, d0 + vmlal.s32 q1, d8, d23 + vmull.s32 q14, d10, d6 + vmlal.s32 q14, d11, d13 + vmlal.s32 q14, d4, d4 + vmlal.s32 q14, d17, d1 + vmlal.s32 q14, d18, d0 + vmlal.s32 q14, d9, d23 + vmull.s32 q11, d10, d7 + vmlal.s32 q11, d11, d6 + vmlal.s32 q11, d12, d5 + vmlal.s32 q11, d8, d1 + vmlal.s32 q11, d19, d0 + vmull.s32 q15, d10, d8 + vmlal.s32 q15, d11, d17 + vmlal.s32 q15, d12, d6 + vmlal.s32 q15, d13, d5 + vmlal.s32 q15, d19, d1 + vmlal.s32 q15, d14, d0 + vmull.s32 q2, d10, d9 + vmlal.s32 q2, d11, d8 + vmlal.s32 q2, d12, d7 + vmlal.s32 q2, d13, d6 + vmlal.s32 q2, d14, d1 + vmull.s32 q0, d15, d1 + vmlal.s32 q0, d10, d14 + vmlal.s32 q0, d11, d19 + vmlal.s32 q0, d12, d8 + vmlal.s32 q0, d13, d17 + vmlal.s32 q0, d6, d6 + add r2, sp, #512 + vld1.8 {d18-d19}, [r2, : 128] + vmull.s32 q3, d16, d7 + vmlal.s32 q3, d10, d15 + vmlal.s32 q3, d11, d14 + vmlal.s32 q3, d12, d9 + vmlal.s32 q3, d13, d8 + add r2, sp, #528 + vld1.8 {d8-d9}, [r2, : 128] + vadd.i64 q5, q12, q9 + vadd.i64 q6, q15, q9 + vshr.s64 q5, q5, #26 + vshr.s64 q6, q6, #26 + vadd.i64 q7, q10, q5 + vshl.i64 q5, q5, #26 + vadd.i64 q8, q7, q4 + vadd.i64 q2, q2, q6 + vshl.i64 q6, q6, #26 + vadd.i64 q10, q2, q4 + vsub.i64 q5, q12, q5 + vshr.s64 q8, q8, #25 + vsub.i64 q6, q15, q6 + vshr.s64 q10, q10, #25 + vadd.i64 q12, q13, q8 + vshl.i64 q8, q8, #25 + vadd.i64 q13, q12, q9 + vadd.i64 q0, q0, q10 + vsub.i64 q7, q7, q8 + vshr.s64 q8, q13, #26 + vshl.i64 q10, q10, #25 + vadd.i64 q13, q0, q9 + vadd.i64 q1, q1, q8 + vshl.i64 q8, q8, #26 + vadd.i64 q15, q1, q4 + vsub.i64 q2, q2, q10 + vshr.s64 q10, q13, #26 + vsub.i64 q8, q12, q8 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q10 + vshl.i64 q10, q10, #26 + vadd.i64 q13, q3, q4 + vadd.i64 q14, q14, q12 + add r2, r3, #288 + vshl.i64 q12, q12, #25 + add r4, r3, #336 + vadd.i64 q15, q14, q9 + add r2, r2, #8 + vsub.i64 q0, q0, q10 + add r4, r4, #8 + vshr.s64 q10, q13, #25 + vsub.i64 q1, q1, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q13, q10, q10 + vadd.i64 q11, q11, q12 + vtrn.32 d16, d2 + vshl.i64 q12, q12, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q11, q4 + vadd.i64 q4, q5, q13 + vst1.8 d16, [r2, : 64]! + vshl.i64 q5, q10, #4 + vst1.8 d17, [r4, : 64]! + vsub.i64 q8, q14, q12 + vshr.s64 q1, q1, #25 + vadd.i64 q4, q4, q5 + vadd.i64 q5, q6, q1 + vshl.i64 q1, q1, #25 + vadd.i64 q6, q5, q9 + vadd.i64 q4, q4, q10 + vshl.i64 q10, q10, #25 + vadd.i64 q9, q4, q9 + vsub.i64 q1, q11, q1 + vshr.s64 q6, q6, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d16, d2 + vshr.s64 q9, q9, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q2, q6 + vst1.8 d16, [r2, : 64] + vshl.i64 q2, q6, #26 + vst1.8 d17, [r4, : 64] + vadd.i64 q6, q7, q9 + vtrn.32 d0, d6 + vshl.i64 q7, q9, #26 + vtrn.32 d1, d7 + vsub.i64 q2, q5, q2 + add r2, r2, #16 + vsub.i64 q3, q4, q7 + vst1.8 d0, [r2, : 64] + add r4, r4, #16 + vst1.8 d1, [r4, : 64] + vtrn.32 d4, d2 + vtrn.32 d5, d3 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d6, d12 + vtrn.32 d7, d13 + vst1.8 d4, [r2, : 64] + vst1.8 d5, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d6, [r2, : 64] + vst1.8 d7, [r4, : 64] + add r2, r3, #240 + add r4, r3, #96 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #144 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #192 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #144 + vshl.i64 q7, q7, #25 + add r4, r3, #96 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + add r2, r3, #288 + add r4, r3, #336 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vsub.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4-d5}, [r4, : 128]! + vsub.i32 q1, q1, q2 + add r5, r3, #240 + vld1.8 {d4}, [r2, : 64] + vld1.8 {d6}, [r4, : 64] + vsub.i32 q2, q2, q3 + vst1.8 {d0-d1}, [r5, : 128]! + vst1.8 {d2-d3}, [r5, : 128]! + vst1.8 d4, [r5, : 64] + add r2, r3, #144 + add r4, r3, #96 + add r5, r3, #144 + add r6, r3, #192 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vsub.i32 q2, q0, q1 + vadd.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d6-d7}, [r4, : 128]! + vsub.i32 q4, q1, q3 + vadd.i32 q1, q1, q3 + vld1.8 {d6}, [r2, : 64] + vld1.8 {d10}, [r4, : 64] + vsub.i32 q6, q3, q5 + vadd.i32 q3, q3, q5 + vst1.8 {d4-d5}, [r5, : 128]! + vst1.8 {d0-d1}, [r6, : 128]! + vst1.8 {d8-d9}, [r5, : 128]! + vst1.8 {d2-d3}, [r6, : 128]! + vst1.8 d12, [r5, : 64] + vst1.8 d6, [r6, : 64] + add r2, r3, #0 + add r4, r3, #240 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #336 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #288 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #288 + vshl.i64 q7, q7, #25 + add r4, r3, #96 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + add r2, sp, #544 + add r4, r3, #144 + add r5, r3, #192 + vld1.8 {d0-d1}, [r2, : 128] + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4-d5}, [r5, : 128]! + vzip.i32 q1, q2 + vld1.8 {d6-d7}, [r4, : 128]! + vld1.8 {d8-d9}, [r5, : 128]! + vshl.i32 q5, q1, #1 + vzip.i32 q3, q4 + vshl.i32 q6, q2, #1 + vld1.8 {d14}, [r4, : 64] + vshl.i32 q8, q3, #1 + vld1.8 {d15}, [r5, : 64] + vshl.i32 q9, q4, #1 + vmul.i32 d21, d7, d1 + vtrn.32 d14, d15 + vmul.i32 q11, q4, q0 + vmul.i32 q0, q7, q0 + vmull.s32 q12, d2, d2 + vmlal.s32 q12, d11, d1 + vmlal.s32 q12, d12, d0 + vmlal.s32 q12, d13, d23 + vmlal.s32 q12, d16, d22 + vmlal.s32 q12, d7, d21 + vmull.s32 q10, d2, d11 + vmlal.s32 q10, d4, d1 + vmlal.s32 q10, d13, d0 + vmlal.s32 q10, d6, d23 + vmlal.s32 q10, d17, d22 + vmull.s32 q13, d10, d4 + vmlal.s32 q13, d11, d3 + vmlal.s32 q13, d13, d1 + vmlal.s32 q13, d16, d0 + vmlal.s32 q13, d17, d23 + vmlal.s32 q13, d8, d22 + vmull.s32 q1, d10, d5 + vmlal.s32 q1, d11, d4 + vmlal.s32 q1, d6, d1 + vmlal.s32 q1, d17, d0 + vmlal.s32 q1, d8, d23 + vmull.s32 q14, d10, d6 + vmlal.s32 q14, d11, d13 + vmlal.s32 q14, d4, d4 + vmlal.s32 q14, d17, d1 + vmlal.s32 q14, d18, d0 + vmlal.s32 q14, d9, d23 + vmull.s32 q11, d10, d7 + vmlal.s32 q11, d11, d6 + vmlal.s32 q11, d12, d5 + vmlal.s32 q11, d8, d1 + vmlal.s32 q11, d19, d0 + vmull.s32 q15, d10, d8 + vmlal.s32 q15, d11, d17 + vmlal.s32 q15, d12, d6 + vmlal.s32 q15, d13, d5 + vmlal.s32 q15, d19, d1 + vmlal.s32 q15, d14, d0 + vmull.s32 q2, d10, d9 + vmlal.s32 q2, d11, d8 + vmlal.s32 q2, d12, d7 + vmlal.s32 q2, d13, d6 + vmlal.s32 q2, d14, d1 + vmull.s32 q0, d15, d1 + vmlal.s32 q0, d10, d14 + vmlal.s32 q0, d11, d19 + vmlal.s32 q0, d12, d8 + vmlal.s32 q0, d13, d17 + vmlal.s32 q0, d6, d6 + add r2, sp, #512 + vld1.8 {d18-d19}, [r2, : 128] + vmull.s32 q3, d16, d7 + vmlal.s32 q3, d10, d15 + vmlal.s32 q3, d11, d14 + vmlal.s32 q3, d12, d9 + vmlal.s32 q3, d13, d8 + add r2, sp, #528 + vld1.8 {d8-d9}, [r2, : 128] + vadd.i64 q5, q12, q9 + vadd.i64 q6, q15, q9 + vshr.s64 q5, q5, #26 + vshr.s64 q6, q6, #26 + vadd.i64 q7, q10, q5 + vshl.i64 q5, q5, #26 + vadd.i64 q8, q7, q4 + vadd.i64 q2, q2, q6 + vshl.i64 q6, q6, #26 + vadd.i64 q10, q2, q4 + vsub.i64 q5, q12, q5 + vshr.s64 q8, q8, #25 + vsub.i64 q6, q15, q6 + vshr.s64 q10, q10, #25 + vadd.i64 q12, q13, q8 + vshl.i64 q8, q8, #25 + vadd.i64 q13, q12, q9 + vadd.i64 q0, q0, q10 + vsub.i64 q7, q7, q8 + vshr.s64 q8, q13, #26 + vshl.i64 q10, q10, #25 + vadd.i64 q13, q0, q9 + vadd.i64 q1, q1, q8 + vshl.i64 q8, q8, #26 + vadd.i64 q15, q1, q4 + vsub.i64 q2, q2, q10 + vshr.s64 q10, q13, #26 + vsub.i64 q8, q12, q8 + vshr.s64 q12, q15, #25 + vadd.i64 q3, q3, q10 + vshl.i64 q10, q10, #26 + vadd.i64 q13, q3, q4 + vadd.i64 q14, q14, q12 + add r2, r3, #144 + vshl.i64 q12, q12, #25 + add r4, r3, #192 + vadd.i64 q15, q14, q9 + add r2, r2, #8 + vsub.i64 q0, q0, q10 + add r4, r4, #8 + vshr.s64 q10, q13, #25 + vsub.i64 q1, q1, q12 + vshr.s64 q12, q15, #26 + vadd.i64 q13, q10, q10 + vadd.i64 q11, q11, q12 + vtrn.32 d16, d2 + vshl.i64 q12, q12, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q11, q4 + vadd.i64 q4, q5, q13 + vst1.8 d16, [r2, : 64]! + vshl.i64 q5, q10, #4 + vst1.8 d17, [r4, : 64]! + vsub.i64 q8, q14, q12 + vshr.s64 q1, q1, #25 + vadd.i64 q4, q4, q5 + vadd.i64 q5, q6, q1 + vshl.i64 q1, q1, #25 + vadd.i64 q6, q5, q9 + vadd.i64 q4, q4, q10 + vshl.i64 q10, q10, #25 + vadd.i64 q9, q4, q9 + vsub.i64 q1, q11, q1 + vshr.s64 q6, q6, #26 + vsub.i64 q3, q3, q10 + vtrn.32 d16, d2 + vshr.s64 q9, q9, #26 + vtrn.32 d17, d3 + vadd.i64 q1, q2, q6 + vst1.8 d16, [r2, : 64] + vshl.i64 q2, q6, #26 + vst1.8 d17, [r4, : 64] + vadd.i64 q6, q7, q9 + vtrn.32 d0, d6 + vshl.i64 q7, q9, #26 + vtrn.32 d1, d7 + vsub.i64 q2, q5, q2 + add r2, r2, #16 + vsub.i64 q3, q4, q7 + vst1.8 d0, [r2, : 64] + add r4, r4, #16 + vst1.8 d1, [r4, : 64] + vtrn.32 d4, d2 + vtrn.32 d5, d3 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d6, d12 + vtrn.32 d7, d13 + vst1.8 d4, [r2, : 64] + vst1.8 d5, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d6, [r2, : 64] + vst1.8 d7, [r4, : 64] + add r2, r3, #336 + add r4, r3, #288 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vadd.i32 q0, q0, q1 + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4-d5}, [r4, : 128]! + vadd.i32 q1, q1, q2 + add r5, r3, #288 + vld1.8 {d4}, [r2, : 64] + vld1.8 {d6}, [r4, : 64] + vadd.i32 q2, q2, q3 + vst1.8 {d0-d1}, [r5, : 128]! + vst1.8 {d2-d3}, [r5, : 128]! + vst1.8 d4, [r5, : 64] + add r2, r3, #48 + add r4, r3, #144 + vld1.8 {d0-d1}, [r4, : 128]! + vld1.8 {d2-d3}, [r4, : 128]! + vld1.8 {d4}, [r4, : 64] + add r4, r3, #288 + vld1.8 {d6-d7}, [r4, : 128]! + vtrn.32 q0, q3 + vld1.8 {d8-d9}, [r4, : 128]! + vshl.i32 q5, q0, #4 + vtrn.32 q1, q4 + vshl.i32 q6, q3, #4 + vadd.i32 q5, q5, q0 + vadd.i32 q6, q6, q3 + vshl.i32 q7, q1, #4 + vld1.8 {d5}, [r4, : 64] + vshl.i32 q8, q4, #4 + vtrn.32 d4, d5 + vadd.i32 q7, q7, q1 + vadd.i32 q8, q8, q4 + vld1.8 {d18-d19}, [r2, : 128]! + vshl.i32 q10, q2, #4 + vld1.8 {d22-d23}, [r2, : 128]! + vadd.i32 q10, q10, q2 + vld1.8 {d24}, [r2, : 64] + vadd.i32 q5, q5, q0 + add r2, r3, #240 + vld1.8 {d26-d27}, [r2, : 128]! + vadd.i32 q6, q6, q3 + vld1.8 {d28-d29}, [r2, : 128]! + vadd.i32 q8, q8, q4 + vld1.8 {d25}, [r2, : 64] + vadd.i32 q10, q10, q2 + vtrn.32 q9, q13 + vadd.i32 q7, q7, q1 + vadd.i32 q5, q5, q0 + vtrn.32 q11, q14 + vadd.i32 q6, q6, q3 + add r2, sp, #560 + vadd.i32 q10, q10, q2 + vtrn.32 d24, d25 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q6, q13, #1 + add r2, sp, #576 + vst1.8 {d20-d21}, [r2, : 128] + vshl.i32 q10, q14, #1 + add r2, sp, #592 + vst1.8 {d12-d13}, [r2, : 128] + vshl.i32 q15, q12, #1 + vadd.i32 q8, q8, q4 + vext.32 d10, d31, d30, #0 + vadd.i32 q7, q7, q1 + add r2, sp, #608 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q8, d18, d5 + vmlal.s32 q8, d26, d4 + vmlal.s32 q8, d19, d9 + vmlal.s32 q8, d27, d3 + vmlal.s32 q8, d22, d8 + vmlal.s32 q8, d28, d2 + vmlal.s32 q8, d23, d7 + vmlal.s32 q8, d29, d1 + vmlal.s32 q8, d24, d6 + vmlal.s32 q8, d25, d0 + add r2, sp, #624 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q2, d18, d4 + vmlal.s32 q2, d12, d9 + vmlal.s32 q2, d13, d8 + vmlal.s32 q2, d19, d3 + vmlal.s32 q2, d22, d2 + vmlal.s32 q2, d23, d1 + vmlal.s32 q2, d24, d0 + add r2, sp, #640 + vst1.8 {d20-d21}, [r2, : 128] + vmull.s32 q7, d18, d9 + vmlal.s32 q7, d26, d3 + vmlal.s32 q7, d19, d8 + vmlal.s32 q7, d27, d2 + vmlal.s32 q7, d22, d7 + vmlal.s32 q7, d28, d1 + vmlal.s32 q7, d23, d6 + vmlal.s32 q7, d29, d0 + add r2, sp, #656 + vst1.8 {d10-d11}, [r2, : 128] + vmull.s32 q5, d18, d3 + vmlal.s32 q5, d19, d2 + vmlal.s32 q5, d22, d1 + vmlal.s32 q5, d23, d0 + vmlal.s32 q5, d12, d8 + add r2, sp, #672 + vst1.8 {d16-d17}, [r2, : 128] + vmull.s32 q4, d18, d8 + vmlal.s32 q4, d26, d2 + vmlal.s32 q4, d19, d7 + vmlal.s32 q4, d27, d1 + vmlal.s32 q4, d22, d6 + vmlal.s32 q4, d28, d0 + vmull.s32 q8, d18, d7 + vmlal.s32 q8, d26, d1 + vmlal.s32 q8, d19, d6 + vmlal.s32 q8, d27, d0 + add r2, sp, #576 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q7, d24, d21 + vmlal.s32 q7, d25, d20 + vmlal.s32 q4, d23, d21 + vmlal.s32 q4, d29, d20 + vmlal.s32 q8, d22, d21 + vmlal.s32 q8, d28, d20 + vmlal.s32 q5, d24, d20 + add r2, sp, #576 + vst1.8 {d14-d15}, [r2, : 128] + vmull.s32 q7, d18, d6 + vmlal.s32 q7, d26, d0 + add r2, sp, #656 + vld1.8 {d30-d31}, [r2, : 128] + vmlal.s32 q2, d30, d21 + vmlal.s32 q7, d19, d21 + vmlal.s32 q7, d27, d20 + add r2, sp, #624 + vld1.8 {d26-d27}, [r2, : 128] + vmlal.s32 q4, d25, d27 + vmlal.s32 q8, d29, d27 + vmlal.s32 q8, d25, d26 + vmlal.s32 q7, d28, d27 + vmlal.s32 q7, d29, d26 + add r2, sp, #608 + vld1.8 {d28-d29}, [r2, : 128] + vmlal.s32 q4, d24, d29 + vmlal.s32 q8, d23, d29 + vmlal.s32 q8, d24, d28 + vmlal.s32 q7, d22, d29 + vmlal.s32 q7, d23, d28 + add r2, sp, #608 + vst1.8 {d8-d9}, [r2, : 128] + add r2, sp, #560 + vld1.8 {d8-d9}, [r2, : 128] + vmlal.s32 q7, d24, d9 + vmlal.s32 q7, d25, d31 + vmull.s32 q1, d18, d2 + vmlal.s32 q1, d19, d1 + vmlal.s32 q1, d22, d0 + vmlal.s32 q1, d24, d27 + vmlal.s32 q1, d23, d20 + vmlal.s32 q1, d12, d7 + vmlal.s32 q1, d13, d6 + vmull.s32 q6, d18, d1 + vmlal.s32 q6, d19, d0 + vmlal.s32 q6, d23, d27 + vmlal.s32 q6, d22, d20 + vmlal.s32 q6, d24, d26 + vmull.s32 q0, d18, d0 + vmlal.s32 q0, d22, d27 + vmlal.s32 q0, d23, d26 + vmlal.s32 q0, d24, d31 + vmlal.s32 q0, d19, d20 + add r2, sp, #640 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q2, d18, d7 + vmlal.s32 q2, d19, d6 + vmlal.s32 q5, d18, d6 + vmlal.s32 q5, d19, d21 + vmlal.s32 q1, d18, d21 + vmlal.s32 q1, d19, d29 + vmlal.s32 q0, d18, d28 + vmlal.s32 q0, d19, d9 + vmlal.s32 q6, d18, d29 + vmlal.s32 q6, d19, d28 + add r2, sp, #592 + vld1.8 {d18-d19}, [r2, : 128] + add r2, sp, #512 + vld1.8 {d22-d23}, [r2, : 128] + vmlal.s32 q5, d19, d7 + vmlal.s32 q0, d18, d21 + vmlal.s32 q0, d19, d29 + vmlal.s32 q6, d18, d6 + add r2, sp, #528 + vld1.8 {d6-d7}, [r2, : 128] + vmlal.s32 q6, d19, d21 + add r2, sp, #576 + vld1.8 {d18-d19}, [r2, : 128] + vmlal.s32 q0, d30, d8 + add r2, sp, #672 + vld1.8 {d20-d21}, [r2, : 128] + vmlal.s32 q5, d30, d29 + add r2, sp, #608 + vld1.8 {d24-d25}, [r2, : 128] + vmlal.s32 q1, d30, d28 + vadd.i64 q13, q0, q11 + vadd.i64 q14, q5, q11 + vmlal.s32 q6, d30, d9 + vshr.s64 q4, q13, #26 + vshr.s64 q13, q14, #26 + vadd.i64 q7, q7, q4 + vshl.i64 q4, q4, #26 + vadd.i64 q14, q7, q3 + vadd.i64 q9, q9, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q15, q9, q3 + vsub.i64 q0, q0, q4 + vshr.s64 q4, q14, #25 + vsub.i64 q5, q5, q13 + vshr.s64 q13, q15, #25 + vadd.i64 q6, q6, q4 + vshl.i64 q4, q4, #25 + vadd.i64 q14, q6, q11 + vadd.i64 q2, q2, q13 + vsub.i64 q4, q7, q4 + vshr.s64 q7, q14, #26 + vshl.i64 q13, q13, #25 + vadd.i64 q14, q2, q11 + vadd.i64 q8, q8, q7 + vshl.i64 q7, q7, #26 + vadd.i64 q15, q8, q3 + vsub.i64 q9, q9, q13 + vshr.s64 q13, q14, #26 + vsub.i64 q6, q6, q7 + vshr.s64 q7, q15, #25 + vadd.i64 q10, q10, q13 + vshl.i64 q13, q13, #26 + vadd.i64 q14, q10, q3 + vadd.i64 q1, q1, q7 + add r2, r3, #240 + vshl.i64 q7, q7, #25 + add r4, r3, #144 + vadd.i64 q15, q1, q11 + add r2, r2, #8 + vsub.i64 q2, q2, q13 + add r4, r4, #8 + vshr.s64 q13, q14, #25 + vsub.i64 q7, q8, q7 + vshr.s64 q8, q15, #26 + vadd.i64 q14, q13, q13 + vadd.i64 q12, q12, q8 + vtrn.32 d12, d14 + vshl.i64 q8, q8, #26 + vtrn.32 d13, d15 + vadd.i64 q3, q12, q3 + vadd.i64 q0, q0, q14 + vst1.8 d12, [r2, : 64]! + vshl.i64 q7, q13, #4 + vst1.8 d13, [r4, : 64]! + vsub.i64 q1, q1, q8 + vshr.s64 q3, q3, #25 + vadd.i64 q0, q0, q7 + vadd.i64 q5, q5, q3 + vshl.i64 q3, q3, #25 + vadd.i64 q6, q5, q11 + vadd.i64 q0, q0, q13 + vshl.i64 q7, q13, #25 + vadd.i64 q8, q0, q11 + vsub.i64 q3, q12, q3 + vshr.s64 q6, q6, #26 + vsub.i64 q7, q10, q7 + vtrn.32 d2, d6 + vshr.s64 q8, q8, #26 + vtrn.32 d3, d7 + vadd.i64 q3, q9, q6 + vst1.8 d2, [r2, : 64] + vshl.i64 q6, q6, #26 + vst1.8 d3, [r4, : 64] + vadd.i64 q1, q4, q8 + vtrn.32 d4, d14 + vshl.i64 q4, q8, #26 + vtrn.32 d5, d15 + vsub.i64 q5, q5, q6 + add r2, r2, #16 + vsub.i64 q0, q0, q4 + vst1.8 d4, [r2, : 64] + add r4, r4, #16 + vst1.8 d5, [r4, : 64] + vtrn.32 d10, d6 + vtrn.32 d11, d7 + sub r2, r2, #8 + sub r4, r4, #8 + vtrn.32 d0, d2 + vtrn.32 d1, d3 + vst1.8 d10, [r2, : 64] + vst1.8 d11, [r4, : 64] + sub r2, r2, #24 + sub r4, r4, #24 + vst1.8 d0, [r2, : 64] + vst1.8 d1, [r4, : 64] + ldr r2, [sp, #488] + ldr r4, [sp, #492] + subs r5, r2, #1 + bge ._mainloop + add r1, r3, #144 + add r2, r3, #336 + vld1.8 {d0-d1}, [r1, : 128]! + vld1.8 {d2-d3}, [r1, : 128]! + vld1.8 {d4}, [r1, : 64] + vst1.8 {d0-d1}, [r2, : 128]! + vst1.8 {d2-d3}, [r2, : 128]! + vst1.8 d4, [r2, : 64] + ldr r1, =0 +._invertloop: + add r2, r3, #144 + ldr r4, =0 + ldr r5, =2 + cmp r1, #1 + ldreq r5, =1 + addeq r2, r3, #336 + addeq r4, r3, #48 + cmp r1, #2 + ldreq r5, =1 + addeq r2, r3, #48 + cmp r1, #3 + ldreq r5, =5 + addeq r4, r3, #336 + cmp r1, #4 + ldreq r5, =10 + cmp r1, #5 + ldreq r5, =20 + cmp r1, #6 + ldreq r5, =10 + addeq r2, r3, #336 + addeq r4, r3, #336 + cmp r1, #7 + ldreq r5, =50 + cmp r1, #8 + ldreq r5, =100 + cmp r1, #9 + ldreq r5, =50 + addeq r2, r3, #336 + cmp r1, #10 + ldreq r5, =5 + addeq r2, r3, #48 + cmp r1, #11 + ldreq r5, =0 + addeq r2, r3, #96 + add r6, r3, #144 + add r7, r3, #288 + vld1.8 {d0-d1}, [r6, : 128]! + vld1.8 {d2-d3}, [r6, : 128]! + vld1.8 {d4}, [r6, : 64] + vst1.8 {d0-d1}, [r7, : 128]! + vst1.8 {d2-d3}, [r7, : 128]! + vst1.8 d4, [r7, : 64] + cmp r5, #0 + beq ._skipsquaringloop +._squaringloop: + add r6, r3, #288 + add r7, r3, #288 + add r8, r3, #288 + vmov.i32 q0, #19 + vmov.i32 q1, #0 + vmov.i32 q2, #1 + vzip.i32 q1, q2 + vld1.8 {d4-d5}, [r7, : 128]! + vld1.8 {d6-d7}, [r7, : 128]! + vld1.8 {d9}, [r7, : 64] + vld1.8 {d10-d11}, [r6, : 128]! + add r7, sp, #416 + vld1.8 {d12-d13}, [r6, : 128]! + vmul.i32 q7, q2, q0 + vld1.8 {d8}, [r6, : 64] + vext.32 d17, d11, d10, #1 + vmul.i32 q9, q3, q0 + vext.32 d16, d10, d8, #1 + vshl.u32 q10, q5, q1 + vext.32 d22, d14, d4, #1 + vext.32 d24, d18, d6, #1 + vshl.u32 q13, q6, q1 + vshl.u32 d28, d8, d2 + vrev64.i32 d22, d22 + vmul.i32 d1, d9, d1 + vrev64.i32 d24, d24 + vext.32 d29, d8, d13, #1 + vext.32 d0, d1, d9, #1 + vrev64.i32 d0, d0 + vext.32 d2, d9, d1, #1 + vext.32 d23, d15, d5, #1 + vmull.s32 q4, d20, d4 + vrev64.i32 d23, d23 + vmlal.s32 q4, d21, d1 + vrev64.i32 d2, d2 + vmlal.s32 q4, d26, d19 + vext.32 d3, d5, d15, #1 + vmlal.s32 q4, d27, d18 + vrev64.i32 d3, d3 + vmlal.s32 q4, d28, d15 + vext.32 d14, d12, d11, #1 + vmull.s32 q5, d16, d23 + vext.32 d15, d13, d12, #1 + vmlal.s32 q5, d17, d4 + vst1.8 d8, [r7, : 64]! + vmlal.s32 q5, d14, d1 + vext.32 d12, d9, d8, #0 + vmlal.s32 q5, d15, d19 + vmov.i64 d13, #0 + vmlal.s32 q5, d29, d18 + vext.32 d25, d19, d7, #1 + vmlal.s32 q6, d20, d5 + vrev64.i32 d25, d25 + vmlal.s32 q6, d21, d4 + vst1.8 d11, [r7, : 64]! + vmlal.s32 q6, d26, d1 + vext.32 d9, d10, d10, #0 + vmlal.s32 q6, d27, d19 + vmov.i64 d8, #0 + vmlal.s32 q6, d28, d18 + vmlal.s32 q4, d16, d24 + vmlal.s32 q4, d17, d5 + vmlal.s32 q4, d14, d4 + vst1.8 d12, [r7, : 64]! + vmlal.s32 q4, d15, d1 + vext.32 d10, d13, d12, #0 + vmlal.s32 q4, d29, d19 + vmov.i64 d11, #0 + vmlal.s32 q5, d20, d6 + vmlal.s32 q5, d21, d5 + vmlal.s32 q5, d26, d4 + vext.32 d13, d8, d8, #0 + vmlal.s32 q5, d27, d1 + vmov.i64 d12, #0 + vmlal.s32 q5, d28, d19 + vst1.8 d9, [r7, : 64]! + vmlal.s32 q6, d16, d25 + vmlal.s32 q6, d17, d6 + vst1.8 d10, [r7, : 64] + vmlal.s32 q6, d14, d5 + vext.32 d8, d11, d10, #0 + vmlal.s32 q6, d15, d4 + vmov.i64 d9, #0 + vmlal.s32 q6, d29, d1 + vmlal.s32 q4, d20, d7 + vmlal.s32 q4, d21, d6 + vmlal.s32 q4, d26, d5 + vext.32 d11, d12, d12, #0 + vmlal.s32 q4, d27, d4 + vmov.i64 d10, #0 + vmlal.s32 q4, d28, d1 + vmlal.s32 q5, d16, d0 + sub r6, r7, #32 + vmlal.s32 q5, d17, d7 + vmlal.s32 q5, d14, d6 + vext.32 d30, d9, d8, #0 + vmlal.s32 q5, d15, d5 + vld1.8 {d31}, [r6, : 64]! + vmlal.s32 q5, d29, d4 + vmlal.s32 q15, d20, d0 + vext.32 d0, d6, d18, #1 + vmlal.s32 q15, d21, d25 + vrev64.i32 d0, d0 + vmlal.s32 q15, d26, d24 + vext.32 d1, d7, d19, #1 + vext.32 d7, d10, d10, #0 + vmlal.s32 q15, d27, d23 + vrev64.i32 d1, d1 + vld1.8 {d6}, [r6, : 64] + vmlal.s32 q15, d28, d22 + vmlal.s32 q3, d16, d4 + add r6, r6, #24 + vmlal.s32 q3, d17, d2 + vext.32 d4, d31, d30, #0 + vmov d17, d11 + vmlal.s32 q3, d14, d1 + vext.32 d11, d13, d13, #0 + vext.32 d13, d30, d30, #0 + vmlal.s32 q3, d15, d0 + vext.32 d1, d8, d8, #0 + vmlal.s32 q3, d29, d3 + vld1.8 {d5}, [r6, : 64] + sub r6, r6, #16 + vext.32 d10, d6, d6, #0 + vmov.i32 q1, #0xffffffff + vshl.i64 q4, q1, #25 + add r7, sp, #512 + vld1.8 {d14-d15}, [r7, : 128] + vadd.i64 q9, q2, q7 + vshl.i64 q1, q1, #26 + vshr.s64 q10, q9, #26 + vld1.8 {d0}, [r6, : 64]! + vadd.i64 q5, q5, q10 + vand q9, q9, q1 + vld1.8 {d16}, [r6, : 64]! + add r6, sp, #528 + vld1.8 {d20-d21}, [r6, : 128] + vadd.i64 q11, q5, q10 + vsub.i64 q2, q2, q9 + vshr.s64 q9, q11, #25 + vext.32 d12, d5, d4, #0 + vand q11, q11, q4 + vadd.i64 q0, q0, q9 + vmov d19, d7 + vadd.i64 q3, q0, q7 + vsub.i64 q5, q5, q11 + vshr.s64 q11, q3, #26 + vext.32 d18, d11, d10, #0 + vand q3, q3, q1 + vadd.i64 q8, q8, q11 + vadd.i64 q11, q8, q10 + vsub.i64 q0, q0, q3 + vshr.s64 q3, q11, #25 + vand q11, q11, q4 + vadd.i64 q3, q6, q3 + vadd.i64 q6, q3, q7 + vsub.i64 q8, q8, q11 + vshr.s64 q11, q6, #26 + vand q6, q6, q1 + vadd.i64 q9, q9, q11 + vadd.i64 d25, d19, d21 + vsub.i64 q3, q3, q6 + vshr.s64 d23, d25, #25 + vand q4, q12, q4 + vadd.i64 d21, d23, d23 + vshl.i64 d25, d23, #4 + vadd.i64 d21, d21, d23 + vadd.i64 d25, d25, d21 + vadd.i64 d4, d4, d25 + vzip.i32 q0, q8 + vadd.i64 d12, d4, d14 + add r6, r8, #8 + vst1.8 d0, [r6, : 64] + vsub.i64 d19, d19, d9 + add r6, r6, #16 + vst1.8 d16, [r6, : 64] + vshr.s64 d22, d12, #26 + vand q0, q6, q1 + vadd.i64 d10, d10, d22 + vzip.i32 q3, q9 + vsub.i64 d4, d4, d0 + sub r6, r6, #8 + vst1.8 d6, [r6, : 64] + add r6, r6, #16 + vst1.8 d18, [r6, : 64] + vzip.i32 q2, q5 + sub r6, r6, #32 + vst1.8 d4, [r6, : 64] + subs r5, r5, #1 + bhi ._squaringloop +._skipsquaringloop: + mov r2, r2 + add r5, r3, #288 + add r6, r3, #144 + vmov.i32 q0, #19 + vmov.i32 q1, #0 + vmov.i32 q2, #1 + vzip.i32 q1, q2 + vld1.8 {d4-d5}, [r5, : 128]! + vld1.8 {d6-d7}, [r5, : 128]! + vld1.8 {d9}, [r5, : 64] + vld1.8 {d10-d11}, [r2, : 128]! + add r5, sp, #416 + vld1.8 {d12-d13}, [r2, : 128]! + vmul.i32 q7, q2, q0 + vld1.8 {d8}, [r2, : 64] + vext.32 d17, d11, d10, #1 + vmul.i32 q9, q3, q0 + vext.32 d16, d10, d8, #1 + vshl.u32 q10, q5, q1 + vext.32 d22, d14, d4, #1 + vext.32 d24, d18, d6, #1 + vshl.u32 q13, q6, q1 + vshl.u32 d28, d8, d2 + vrev64.i32 d22, d22 + vmul.i32 d1, d9, d1 + vrev64.i32 d24, d24 + vext.32 d29, d8, d13, #1 + vext.32 d0, d1, d9, #1 + vrev64.i32 d0, d0 + vext.32 d2, d9, d1, #1 + vext.32 d23, d15, d5, #1 + vmull.s32 q4, d20, d4 + vrev64.i32 d23, d23 + vmlal.s32 q4, d21, d1 + vrev64.i32 d2, d2 + vmlal.s32 q4, d26, d19 + vext.32 d3, d5, d15, #1 + vmlal.s32 q4, d27, d18 + vrev64.i32 d3, d3 + vmlal.s32 q4, d28, d15 + vext.32 d14, d12, d11, #1 + vmull.s32 q5, d16, d23 + vext.32 d15, d13, d12, #1 + vmlal.s32 q5, d17, d4 + vst1.8 d8, [r5, : 64]! + vmlal.s32 q5, d14, d1 + vext.32 d12, d9, d8, #0 + vmlal.s32 q5, d15, d19 + vmov.i64 d13, #0 + vmlal.s32 q5, d29, d18 + vext.32 d25, d19, d7, #1 + vmlal.s32 q6, d20, d5 + vrev64.i32 d25, d25 + vmlal.s32 q6, d21, d4 + vst1.8 d11, [r5, : 64]! + vmlal.s32 q6, d26, d1 + vext.32 d9, d10, d10, #0 + vmlal.s32 q6, d27, d19 + vmov.i64 d8, #0 + vmlal.s32 q6, d28, d18 + vmlal.s32 q4, d16, d24 + vmlal.s32 q4, d17, d5 + vmlal.s32 q4, d14, d4 + vst1.8 d12, [r5, : 64]! + vmlal.s32 q4, d15, d1 + vext.32 d10, d13, d12, #0 + vmlal.s32 q4, d29, d19 + vmov.i64 d11, #0 + vmlal.s32 q5, d20, d6 + vmlal.s32 q5, d21, d5 + vmlal.s32 q5, d26, d4 + vext.32 d13, d8, d8, #0 + vmlal.s32 q5, d27, d1 + vmov.i64 d12, #0 + vmlal.s32 q5, d28, d19 + vst1.8 d9, [r5, : 64]! + vmlal.s32 q6, d16, d25 + vmlal.s32 q6, d17, d6 + vst1.8 d10, [r5, : 64] + vmlal.s32 q6, d14, d5 + vext.32 d8, d11, d10, #0 + vmlal.s32 q6, d15, d4 + vmov.i64 d9, #0 + vmlal.s32 q6, d29, d1 + vmlal.s32 q4, d20, d7 + vmlal.s32 q4, d21, d6 + vmlal.s32 q4, d26, d5 + vext.32 d11, d12, d12, #0 + vmlal.s32 q4, d27, d4 + vmov.i64 d10, #0 + vmlal.s32 q4, d28, d1 + vmlal.s32 q5, d16, d0 + sub r2, r5, #32 + vmlal.s32 q5, d17, d7 + vmlal.s32 q5, d14, d6 + vext.32 d30, d9, d8, #0 + vmlal.s32 q5, d15, d5 + vld1.8 {d31}, [r2, : 64]! + vmlal.s32 q5, d29, d4 + vmlal.s32 q15, d20, d0 + vext.32 d0, d6, d18, #1 + vmlal.s32 q15, d21, d25 + vrev64.i32 d0, d0 + vmlal.s32 q15, d26, d24 + vext.32 d1, d7, d19, #1 + vext.32 d7, d10, d10, #0 + vmlal.s32 q15, d27, d23 + vrev64.i32 d1, d1 + vld1.8 {d6}, [r2, : 64] + vmlal.s32 q15, d28, d22 + vmlal.s32 q3, d16, d4 + add r2, r2, #24 + vmlal.s32 q3, d17, d2 + vext.32 d4, d31, d30, #0 + vmov d17, d11 + vmlal.s32 q3, d14, d1 + vext.32 d11, d13, d13, #0 + vext.32 d13, d30, d30, #0 + vmlal.s32 q3, d15, d0 + vext.32 d1, d8, d8, #0 + vmlal.s32 q3, d29, d3 + vld1.8 {d5}, [r2, : 64] + sub r2, r2, #16 + vext.32 d10, d6, d6, #0 + vmov.i32 q1, #0xffffffff + vshl.i64 q4, q1, #25 + add r5, sp, #512 + vld1.8 {d14-d15}, [r5, : 128] + vadd.i64 q9, q2, q7 + vshl.i64 q1, q1, #26 + vshr.s64 q10, q9, #26 + vld1.8 {d0}, [r2, : 64]! + vadd.i64 q5, q5, q10 + vand q9, q9, q1 + vld1.8 {d16}, [r2, : 64]! + add r2, sp, #528 + vld1.8 {d20-d21}, [r2, : 128] + vadd.i64 q11, q5, q10 + vsub.i64 q2, q2, q9 + vshr.s64 q9, q11, #25 + vext.32 d12, d5, d4, #0 + vand q11, q11, q4 + vadd.i64 q0, q0, q9 + vmov d19, d7 + vadd.i64 q3, q0, q7 + vsub.i64 q5, q5, q11 + vshr.s64 q11, q3, #26 + vext.32 d18, d11, d10, #0 + vand q3, q3, q1 + vadd.i64 q8, q8, q11 + vadd.i64 q11, q8, q10 + vsub.i64 q0, q0, q3 + vshr.s64 q3, q11, #25 + vand q11, q11, q4 + vadd.i64 q3, q6, q3 + vadd.i64 q6, q3, q7 + vsub.i64 q8, q8, q11 + vshr.s64 q11, q6, #26 + vand q6, q6, q1 + vadd.i64 q9, q9, q11 + vadd.i64 d25, d19, d21 + vsub.i64 q3, q3, q6 + vshr.s64 d23, d25, #25 + vand q4, q12, q4 + vadd.i64 d21, d23, d23 + vshl.i64 d25, d23, #4 + vadd.i64 d21, d21, d23 + vadd.i64 d25, d25, d21 + vadd.i64 d4, d4, d25 + vzip.i32 q0, q8 + vadd.i64 d12, d4, d14 + add r2, r6, #8 + vst1.8 d0, [r2, : 64] + vsub.i64 d19, d19, d9 + add r2, r2, #16 + vst1.8 d16, [r2, : 64] + vshr.s64 d22, d12, #26 + vand q0, q6, q1 + vadd.i64 d10, d10, d22 + vzip.i32 q3, q9 + vsub.i64 d4, d4, d0 + sub r2, r2, #8 + vst1.8 d6, [r2, : 64] + add r2, r2, #16 + vst1.8 d18, [r2, : 64] + vzip.i32 q2, q5 + sub r2, r2, #32 + vst1.8 d4, [r2, : 64] + cmp r4, #0 + beq ._skippostcopy + add r2, r3, #144 + mov r4, r4 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r4, : 128]! + vst1.8 {d2-d3}, [r4, : 128]! + vst1.8 d4, [r4, : 64] +._skippostcopy: + cmp r1, #1 + bne ._skipfinalcopy + add r2, r3, #288 + add r4, r3, #144 + vld1.8 {d0-d1}, [r2, : 128]! + vld1.8 {d2-d3}, [r2, : 128]! + vld1.8 {d4}, [r2, : 64] + vst1.8 {d0-d1}, [r4, : 128]! + vst1.8 {d2-d3}, [r4, : 128]! + vst1.8 d4, [r4, : 64] +._skipfinalcopy: + add r1, r1, #1 + cmp r1, #12 + blo ._invertloop + add r1, r3, #144 + ldr r2, [r1], #4 + ldr r3, [r1], #4 + ldr r4, [r1], #4 + ldr r5, [r1], #4 + ldr r6, [r1], #4 + ldr r7, [r1], #4 + ldr r8, [r1], #4 + ldr r9, [r1], #4 + ldr r10, [r1], #4 + ldr r1, [r1] + add r11, r1, r1, LSL #4 + add r11, r11, r1, LSL #1 + add r11, r11, #16777216 + mov r11, r11, ASR #25 + add r11, r11, r2 + mov r11, r11, ASR #26 + add r11, r11, r3 + mov r11, r11, ASR #25 + add r11, r11, r4 + mov r11, r11, ASR #26 + add r11, r11, r5 + mov r11, r11, ASR #25 + add r11, r11, r6 + mov r11, r11, ASR #26 + add r11, r11, r7 + mov r11, r11, ASR #25 + add r11, r11, r8 + mov r11, r11, ASR #26 + add r11, r11, r9 + mov r11, r11, ASR #25 + add r11, r11, r10 + mov r11, r11, ASR #26 + add r11, r11, r1 + mov r11, r11, ASR #25 + add r2, r2, r11 + add r2, r2, r11, LSL #1 + add r2, r2, r11, LSL #4 + mov r11, r2, ASR #26 + add r3, r3, r11 + sub r2, r2, r11, LSL #26 + mov r11, r3, ASR #25 + add r4, r4, r11 + sub r3, r3, r11, LSL #25 + mov r11, r4, ASR #26 + add r5, r5, r11 + sub r4, r4, r11, LSL #26 + mov r11, r5, ASR #25 + add r6, r6, r11 + sub r5, r5, r11, LSL #25 + mov r11, r6, ASR #26 + add r7, r7, r11 + sub r6, r6, r11, LSL #26 + mov r11, r7, ASR #25 + add r8, r8, r11 + sub r7, r7, r11, LSL #25 + mov r11, r8, ASR #26 + add r9, r9, r11 + sub r8, r8, r11, LSL #26 + mov r11, r9, ASR #25 + add r10, r10, r11 + sub r9, r9, r11, LSL #25 + mov r11, r10, ASR #26 + add r1, r1, r11 + sub r10, r10, r11, LSL #26 + mov r11, r1, ASR #25 + sub r1, r1, r11, LSL #25 + add r2, r2, r3, LSL #26 + mov r3, r3, LSR #6 + add r3, r3, r4, LSL #19 + mov r4, r4, LSR #13 + add r4, r4, r5, LSL #13 + mov r5, r5, LSR #19 + add r5, r5, r6, LSL #6 + add r6, r7, r8, LSL #25 + mov r7, r8, LSR #7 + add r7, r7, r9, LSL #19 + mov r8, r9, LSR #13 + add r8, r8, r10, LSL #12 + mov r9, r10, LSR #20 + add r1, r9, r1, LSL #6 + str r2, [r0], #4 + str r3, [r0], #4 + str r4, [r0], #4 + str r5, [r0], #4 + str r6, [r0], #4 + str r7, [r0], #4 + str r8, [r0], #4 + str r1, [r0] + ldrd r4, [sp, #0] + ldrd r6, [sp, #8] + ldrd r8, [sp, #16] + ldrd r10, [sp, #24] + ldr r12, [sp, #480] + ldr r14, [sp, #484] + ldr r0, =0 + mov sp, r12 + vpop {q4, q5, q6, q7} + bx lr
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 17/19 zinc: Curve25519 ARM implementation
In reply to Jason A. DonenfeldThis ports the SUPERCOP implementation for usage in kernel space. In addition to the usual header, macro, and style changes required for kernel space, it makes a few small changes to the code:
- The stack alignment is relaxed to 16 bytes.
- Superfluous mov statements have been removed.
- ldr for constants has been replaced with movw.
- ldreq has been replaced with moveq.
- The str epilogue has been made more idiomatic.
- SIMD registers are not pushed and popped at the beginning and end.
- The prologue and epilogue have been made idiomatic.
- A hole has been removed from the stack, saving 32 bytes.
- We write-back the base register whenever possible for vld1.8.
- Some multiplications have been reordered for better A7 performance.
There are more opportunities for cleanup, since this code is from qhasm, which doesn't always do the most opportune thing. But even prior to extensive hand optimizations, this code delivers significant performance improvements (given in get_cycles() per call):
----------- ------------- | generic C | this commit | ------------ ----------- ------------- | Cortex-A7 | 49136 | 22395 | ------------ ----------- ------------- | Cortex-A17 | 17326 | 4983 | ------------ ----------- -------------
---
lib/zinc/Makefile | 1 +
lib/zinc/curve25519/curve25519-arm-glue.c | 43 +++
...e25519-arm-supercop.S => curve25519-arm.S} | 349 ++++++++----------
lib/zinc/curve25519/curve25519.c | 2 +
4 files changed, 200 insertions(+), 195 deletions(-)
create mode 100644 lib/zinc/curve25519/curve25519-arm-glue.c
rename lib/zinc/curve25519/{curve25519-arm-supercop.S => curve25519-arm.S} (92%)lib/zinc/Makefile +1 -0
@@ -29,6 +29,7 @@ zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o zinc_curve25519-y := curve25519/curve25519.o +zinc_curve25519-$(CONFIG_ZINC_ARCH_ARM) += curve25519/curve25519-arm.o obj-$(CONFIG_ZINC_CURVE25519) += zinc_curve25519.o quiet_cmd_perlasm = PERLASM $@
lib/zinc/curve25519/curve25519-arm-glue.c new file +43 -0
43 lines not on this page · raw message
lib/zinc/curve25519/curve25519-arm.S renamed from lib/zinc/curve25519/curve25519-arm-supercop.S +154 -195
805 lines not on this page · raw message
lib/zinc/curve25519/curve25519.c +2 -0
@@ -22,6 +22,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "curve25519-x86_64-glue.c" +#elif defined(CONFIG_ZINC_ARCH_ARM ) +#include "curve25519-arm-glue.c" #else static bool *const curve25519_nobs[]__initconst = { };static void __init curve25519_fpu_init (void )
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 18/19 security/keys: rewrite big_key crypto to use Zinc
In reply to Jason A. DonenfeldA while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function. This makes the file considerably more simple; the diffstat alone should justify this commit. It also should be faster, since it no longer requires a mutex around the "aead api object" (nor allocations), allowing us to encrypt multiple items in parallel. We also benefit from being able to pass any type of pointer to Zinc, so we can get rid of the ridiculously complex custom page allocator that big_key really doesn't need.
--- security/keys/Kconfig | 4 +- security/keys/big_key.c | 230 +++++----------------------------------- 2 files changed, 28 insertions(+), 206 deletions(-)
security/keys/Kconfig +1 -3
@@ -45,9 +45,7 @@ config BIG_KEYS bool "Large payload keys" depends on KEYS depends on TMPFS - select CRYPTO - select CRYPTO_AES - select CRYPTO_GCM + select ZINC_CHACHA20POLY1305 help This option provides support for holding large keys within the kernel (for example Kerberos ticket caches). The data may be stored out to
security/keys/big_key.c +27 -203
347 lines not on this page · raw message
Signature · 1 line
2.21.0
Jason A. Donenfeld
maintainerPATCH v9 19/19 net: WireGuard secure network tunnel
In reply to Jason A. DonenfeldWireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:
* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf
This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace have already implemented the API.
This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.
The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:
* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.
* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.
* allowedips.[ch], hashtables.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the specific hashtables we use.
* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.
* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.
* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.
* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.
* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.
* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.
* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.
* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.
* main.c, version.h: Initialization and deinitialization of the module.
* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.
* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.
This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.
We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.
--- MAINTAINERS | 8 + drivers/net/Kconfig | 30 + drivers/net/Makefile | 1 + drivers/net/wireguard/Makefile | 18 + drivers/net/wireguard/allowedips.c | 377 +++++++++ drivers/net/wireguard/allowedips.h | 59 ++ drivers/net/wireguard/cookie.c | 236 ++++++ drivers/net/wireguard/cookie.h | 59 ++ drivers/net/wireguard/device.c | 460 +++++++++++ drivers/net/wireguard/device.h | 65 ++ drivers/net/wireguard/hashtables.c | 221 +++++ drivers/net/wireguard/hashtables.h | 64 ++ drivers/net/wireguard/main.c | 64 ++ drivers/net/wireguard/messages.h | 128 +++ drivers/net/wireguard/netlink.c | 622 ++++++++++++++ drivers/net/wireguard/netlink.h | 12 + drivers/net/wireguard/noise.c | 807 +++++++++++++++++++ drivers/net/wireguard/noise.h | 131 +++ drivers/net/wireguard/peer.c | 239 ++++++ drivers/net/wireguard/peer.h | 83 ++ drivers/net/wireguard/queueing.c | 53 ++ drivers/net/wireguard/queueing.h | 198 +++++ drivers/net/wireguard/ratelimiter.c | 223 +++++ drivers/net/wireguard/ratelimiter.h | 19 + drivers/net/wireguard/receive.c | 604 ++++++++++++++ drivers/net/wireguard/selftest/allowedips.c | 682 ++++++++++++++++ drivers/net/wireguard/selftest/counter.c | 104 +++ drivers/net/wireguard/selftest/ratelimiter.c | 226 ++++++ drivers/net/wireguard/send.c | 431 ++++++++++ drivers/net/wireguard/socket.c | 433 ++++++++++ drivers/net/wireguard/socket.h | 44 + drivers/net/wireguard/timers.c | 241 ++++++ drivers/net/wireguard/timers.h | 31 + drivers/net/wireguard/version.h | 1 + include/uapi/linux/wireguard.h | 190 +++++ tools/testing/selftests/wireguard/netns.sh | 503 ++++++++++++ 36 files changed, 7667 insertions(+) create mode 100644 drivers/net/wireguard/Makefile create mode 100644 drivers/net/wireguard/allowedips.c create mode 100644 drivers/net/wireguard/allowedips.h create mode 100644 drivers/net/wireguard/cookie.c create mode 100644 drivers/net/wireguard/cookie.h create mode 100644 drivers/net/wireguard/device.c create mode 100644 drivers/net/wireguard/device.h create mode 100644 drivers/net/wireguard/hashtables.c create mode 100644 drivers/net/wireguard/hashtables.h create mode 100644 drivers/net/wireguard/main.c create mode 100644 drivers/net/wireguard/messages.h create mode 100644 drivers/net/wireguard/netlink.c create mode 100644 drivers/net/wireguard/netlink.h create mode 100644 drivers/net/wireguard/noise.c create mode 100644 drivers/net/wireguard/noise.h create mode 100644 drivers/net/wireguard/peer.c create mode 100644 drivers/net/wireguard/peer.h create mode 100644 drivers/net/wireguard/queueing.c create mode 100644 drivers/net/wireguard/queueing.h create mode 100644 drivers/net/wireguard/ratelimiter.c create mode 100644 drivers/net/wireguard/ratelimiter.h create mode 100644 drivers/net/wireguard/receive.c create mode 100644 drivers/net/wireguard/selftest/allowedips.c create mode 100644 drivers/net/wireguard/selftest/counter.c create mode 100644 drivers/net/wireguard/selftest/ratelimiter.c create mode 100644 drivers/net/wireguard/send.c create mode 100644 drivers/net/wireguard/socket.c create mode 100644 drivers/net/wireguard/socket.h create mode 100644 drivers/net/wireguard/timers.c create mode 100644 drivers/net/wireguard/timers.h create mode 100644 drivers/net/wireguard/version.h create mode 100644 include/uapi/linux/wireguard.h create mode 100755 tools/testing/selftests/wireguard/netns.sh
MAINTAINERS +8 -0
@@ -16808,6 +16808,14 @@ L: [email protected] S: Maintained F: drivers/gpio/gpio-ws16c48.c +WIREGUARD SECURE NETWORK TUNNEL +M: Jason A. Donenfeld <[email protected]> +S: Maintained +F: drivers/net/wireguard/ +F: tools/testing/selftests/wireguard/ +L: [email protected] +L: [email protected] + WISTRON LAPTOP BUTTON DRIVER M: Miloslav Trmac <[email protected]> S: Maintained
drivers/net/Kconfig +30 -0
@@ -70,6 +70,36 @@ config DUMMY To compile this driver as a module, choose M here: the module will be called dummy. +config WIREGUARD + tristate "WireGuard secure network tunnel" + depends on NET && INET + depends on IPV6 || !IPV6 + select NET_UDP_TUNNEL + select DST_CACHE + select ZINC_CHACHA20POLY1305 + select ZINC_BLAKE2S + select ZINC_CURVE25519 + help + WireGuard is a secure, fast, and easy to use replacement for IPSec + that uses modern cryptography and clever networking tricks. It's + designed to be fairly general purpose and abstract enough to fit most + use cases, while at the same time remaining extremely simple to + configure. See www.wireguard.com for more info. + + It's safe to say Y or M here, as the driver is very lightweight and + is only in use when an administrator chooses to add an interface. + +config WIREGUARD_DEBUG + bool "Debugging checks and verbose messages" + depends on WIREGUARD + help + This will write log messages for handshake and other events + that occur for a WireGuard interface. It will also perform some + extra validation checks and unit tests at various points. This is + only useful for debugging. + + Say N here unless you know what you're doing. + config EQUALIZER tristate "EQL (serial line load balancing) support" ---help---
drivers/net/Makefile +1 -0
@@ -24,6 +24,7 @@ obj-$(CONFIG_RIONET) += rionet.o obj-$(CONFIG_NET_TEAM) += team/ obj-$(CONFIG_TUN) += tun.o obj-$(CONFIG_TAP) += tap.o +obj-$(CONFIG_WIREGUARD) += wireguard/ obj-$(CONFIG_VETH) += veth.o obj-$(CONFIG_VIRTIO_NET) += virtio_net.o obj-$(CONFIG_VXLAN) += vxlan.o
drivers/net/wireguard/Makefile new file +18 -0
@@ -0,0 +1,18 @@ +ccflags-y := -O3 +ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt' +ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG +wireguard-y := main.o +wireguard-y += noise.o +wireguard-y += device.o +wireguard-y += peer.o +wireguard-y += timers.o +wireguard-y += queueing.o +wireguard-y += send.o +wireguard-y += receive.o +wireguard-y += socket.o +wireguard-y += hashtables.o +wireguard-y += allowedips.o +wireguard-y += ratelimiter.o +wireguard-y += cookie.o +wireguard-y += netlink.o +obj-$(CONFIG_WIREGUARD) := wireguard.o
drivers/net/wireguard/allowedips.c new file +377 -0
377 lines not on this page · raw message
drivers/net/wireguard/allowedips.h new file +59 -0
59 lines not on this page · raw message
drivers/net/wireguard/cookie.c new file +236 -0
236 lines not on this page · raw message
drivers/net/wireguard/cookie.h new file +59 -0
59 lines not on this page · raw message
drivers/net/wireguard/device.c new file +460 -0
460 lines not on this page · raw message
drivers/net/wireguard/device.h new file +65 -0
65 lines not on this page · raw message
drivers/net/wireguard/hashtables.c new file +221 -0
221 lines not on this page · raw message
drivers/net/wireguard/hashtables.h new file +64 -0
64 lines not on this page · raw message
drivers/net/wireguard/main.c new file +64 -0
64 lines not on this page · raw message
drivers/net/wireguard/messages.h new file +128 -0
128 lines not on this page · raw message
drivers/net/wireguard/netlink.c new file +622 -0
622 lines not on this page · raw message
drivers/net/wireguard/netlink.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _WG_NETLINK_H +#define _WG_NETLINK_H + +int wg_genetlink_init (void ); +void wg_genetlink_uninit (void ); + +#endif /* _WG_NETLINK_H */
drivers/net/wireguard/noise.c new file +807 -0
558 lines not on this page · raw message
drivers/net/wireguard/noise.h new file +131 -0
131 lines not on this page · raw message
drivers/net/wireguard/peer.c new file +239 -0
239 lines not on this page · raw message
drivers/net/wireguard/peer.h new file +83 -0
83 lines not on this page · raw message
drivers/net/wireguard/queueing.c new file +53 -0
53 lines not on this page · raw message
drivers/net/wireguard/queueing.h new file +198 -0
198 lines not on this page · raw message
drivers/net/wireguard/ratelimiter.c new file +223 -0
223 lines not on this page · raw message
drivers/net/wireguard/ratelimiter.h new file +19 -0
Lines of this file are not shown. View the raw message.
drivers/net/wireguard/receive.c new file +604 -0
604 lines not on this page · raw message
drivers/net/wireguard/selftest/allowedips.c new file +682 -0
682 lines not on this page · raw message
drivers/net/wireguard/selftest/counter.c new file +104 -0
104 lines not on this page · raw message
drivers/net/wireguard/selftest/ratelimiter.c new file +226 -0
226 lines not on this page · raw message
drivers/net/wireguard/send.c new file +431 -0
431 lines not on this page · raw message
drivers/net/wireguard/socket.c new file +433 -0
433 lines not on this page · raw message
drivers/net/wireguard/socket.h new file +44 -0
44 lines not on this page · raw message
drivers/net/wireguard/timers.c new file +241 -0
241 lines not on this page · raw message
drivers/net/wireguard/timers.h new file +31 -0
Lines of this file are not shown. View the raw message.
drivers/net/wireguard/version.h new file +1 -0
Lines of this file are not shown. View the raw message.
include/uapi/linux/wireguard.h new file +190 -0
190 lines not on this page · raw message
tools/testing/selftests/wireguard/netns.sh new file +503 -0
503 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
David Miller
maintainerRe: [PATCH net-next v9 19/19] net: WireGuard secure network tunnel
From: "Jason A. Donenfeld" <[email protected]>
Date: Fri, 22 Mar 2019 01:11:22 -0600
Review
Quoted 2 lines
(quoted diff) +2
-0+static __always_inline void swap_endian(u8 *dst, const u8 *src, u8 bits) +{
Unless you have an absolutely requirement on inlining (if uninlined, the compilation would break), you must not use the __always_inline keyword and you must let the compiler decide what to do.
Said another way: "The code isn't optimal with my compiler on my computer unless I force inline this" is not a valid reason to use __always_inline
And for this reason we never use __inline in foo.c files, always let the compiler decide.
This applies to your entire submission.
drivers/net/wireguard/allowedips.c line 15
Quoted 2 lines
drivers/net/wireguard/allowedips.c new file +2
-0+ ((u64 *)dst)[0 ] =be64_to_cpu (((const __be64 *)src)[0 ]); + ((u64 *)dst)[1 ] =be64_to_cpu (((const __be64 *)src)[1 ]);
Are 'dst' and 'src' both 64-bit aligned? If not you'll get traps on some cpus.
drivers/net/wireguard/device.c line 157
Quoted 3 lines
drivers/net/wireguard/device.c new file +3
-0+__skb_queue_head_init (&packets); +if (!skb_is_gso (skb)) { + skb->next =NULL ;
Why? Direct ->next and ->prev pointer accesses should never be used, along with anything that assumes what the implentation of skb lists looks like.
Always use the helpers instead.
drivers/net/wireguard/hashtables.c
Quoted 5 lines
drivers/net/wireguard/hashtables.c new file +0
-0
No way.
Do not invent your own hashtables, we have several generic versions in tree and in particular rhashtable.
If the generic kernel facilities have a weakness, fix that instead of rolling an entire new hashtable implementation.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v9 19/19] net: WireGuard secure network tunnel
Hey Dave,
Thanks for the comments.
Quoted 2 lines · David Miller
Said another way: "The code isn't optimal with my compiler on my computer unless I force inline this" is not a valid reason to use
__always_inline
Hah, that was my rationale exactly. Okay, no problem, I'll remove those. And if I can't find a way to make the compiler to do the right thing without it, and nothing looks overly insane, then I'll fix gcc bugs separately.
Quoted 4 lines
drivers/net/wireguard/allowedips.c line 15
Quoted 2 lines
drivers/net/wireguard/allowedips.c new file +2
-0+ ((u64 *)dst)[0 ] =be64_to_cpu (((const __be64 *)src)[0 ]); + ((u64 *)dst)[1 ] =be64_to_cpu (((const __be64 *)src)[1 ]);Are 'dst' and 'src' both 64-bit aligned? If not you'll get traps on some cpus.
Yes, they are.
Quoted 10 lines
drivers/net/wireguard/device.c line 157
Quoted 4 lines
drivers/net/wireguard/device.c new file +3
-0+__skb_queue_head_init (&packets); +if (!skb_is_gso (skb)) { + skb->next =NULL ;Why? Direct ->next and ->prev pointer accesses should never be used, along with anything that assumes what the implentation of skb lists looks like.
Always use the helpers instead.
Ack, will do.
Quoted 10 lines
drivers/net/wireguard/hashtables.c
Quoted 2 lines
drivers/net/wireguard/hashtables.c new file +0
-0No way.
Do not invent your own hashtables, we have several generic versions in tree and in particular rhashtable.
If the generic kernel facilities have a weakness, fix that instead of rolling an entire new hashtable implementation.
That file might be poorly named; I'm certainly not implementing my own hashtable, but rather using <linux/hashtable.h>. The file simply has the various accessors for it that does the proper locking and reference counting associated with the things inserted into the vanilla <linux/hashtable.h> hashtable. I'll rename the file to something more descriptive.
Jason
Herbert Xu
maintainerQuoted 6 lines · Jason A. Donenfeld
Changes
v8->v9, along with who suggested it.
--------------------------------------------
- [EVERYBODY] Zinc no longer ships generated assembly code. Rather, we now bundle in the original perlasm generator for it. This is ongoing joint work with Andy Polyakov upstream, so that the same .pl files can live in our tree as well as in the CRYPTOGAMS tree. I personally find that the code required
Quoted 8 lines
to share this in both repositories to be a tiny bit ugly. I think there would be some degree of an advantage to removing that and making the .pl kernel-only, and then carefully tracking Andy's changes (as we already do). Previous opinions on the list, though, were that there's also significant advantage to being able to share the exact same code in both. And I think there's a decent amount of wisdom in that too. Since that appeared to be the prevailing view, and since it also has good reasons arguments, we'll go with that for now.
Thanks for the update.
Quoted 9 lines
- Following the Adiantum merge, the two commits that port the old crypto API over to use Zinc have been removed from this series. We can certainly add them back in at some point, but I thought it'd be favorable to at least begin to receive some sign-offs on the Zinc-specific commits, now that (hopefully all of) the previous feedback has been taken care of. The two commits porting it over are fairly standalone as well, so that shouldn't impact the ability to review this. For now those are living in the jd/with-cryptoapi-port branch of kernel.org's zx2c4/linux.git tree. This also allows us to move this all forward a little bit.
Sorry but adding new implementations of chacha20/poly1305 without removing the existing ones is not acceptable. I really think we ought to separate the zinc interface from these new crypto implementations. They have nothing to do with each other.
As we've been stuck on this point for so long, let's get the ball rolling by first merging just the zinc interface itself with the existing chacha20/poly1305 code. Then we can replace these implementations with your implementations without getting bogged down by all these other discussions.
AFAICS once we resolve Thomas's concerns with regards to the simd patch, then we can merge the zinc interface right away and go from there.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerHey Herbert,
Quoted 11 lines · Herbert Xu
Sorry but adding new implementations of chacha20/poly1305 without removing the existing ones is not acceptable. I really think we ought to separate the zinc interface from these new crypto implementations. They have nothing to do with each other.
As we've been stuck on this point for so long, let's get the ball rolling by first merging just the zinc interface itself with the existing chacha20/poly1305 code. Then we can replace these implementations with your implementations without getting bogged down by all these other discussions.
Quoted 3 lines
AFAICS once we resolve Thomas's concerns with regards to the simd patch, then we can merge the zinc interface right away and go from there.
I think we're slightly closer to being same page, but I haven't followed up here yet because I was waiting for something else first.
Your previous patchset left out a few implementations and wasn't totally complete with regards to the glue code. I'm going to whip something up now that is Zinc, but where there's already an existing assembly implementation in the tree (i.e. Martin's code), it uses that instead of what I've been posting until now. Then, after that lands, I can post stand-alone patches for replacing the various implementations, and we can discuss those separately, alongside all the various discussion of verification and benchmarks and so forth.
So, I'll have something to examine somewhat soon. Working on it now.
Jason
Herbert Xu
maintainerHi Jason:
Quoted 13 lines · Jason A. Donenfeld
I think we're slightly closer to being same page, but I haven't followed up here yet because I was waiting for something else first.
Your previous patchset left out a few implementations and wasn't totally complete with regards to the glue code. I'm going to whip something up now that is Zinc, but where there's already an existing assembly implementation in the tree (i.e. Martin's code), it uses that instead of what I've been posting until now. Then, after that lands, I can post stand-alone patches for replacing the various implementations, and we can discuss those separately, alongside all the various discussion of verification and benchmarks and so forth.
Quoted 1 line
So, I'll have something to examine somewhat soon. Working on it now.
Great, looking forward to seeing it!
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Quoted 30 lines · Jason A. Donenfeld
Hey Herbert,
Quoted 11 lines · Herbert Xu
Sorry but adding new implementations of chacha20/poly1305 without removing the existing ones is not acceptable. I really think we ought to separate the zinc interface from these new crypto implementations. They have nothing to do with each other.
As we've been stuck on this point for so long, let's get the ball rolling by first merging just the zinc interface itself with the existing chacha20/poly1305 code. Then we can replace these implementations with your implementations without getting bogged down by all these other discussions.
Quoted 3 lines
AFAICS once we resolve Thomas's concerns with regards to the simd patch, then we can merge the zinc interface right away and go from there.
I think we're slightly closer to being same page, but I haven't followed up here yet because I was waiting for something else first.
Your previous patchset left out a few implementations and wasn't totally complete with regards to the glue code. I'm going to whip something up now that is Zinc, but where there's already an existing assembly implementation in the tree (i.e. Martin's code), it uses that instead of what I've been posting until now. Then, after that lands, I can post stand-alone patches for replacing the various implementations, and we can discuss those separately, alongside all the various discussion of verification and benchmarks and so forth.
Quoted 3 lines
So, I'll have something to examine somewhat soon. Working on it now.
Jason
FYI, today I started work on updating testmgr to do fuzz tests where it generates random test vectors using the generic implementation of each hash, skcipher, and aead algorithm registered with the crypto API, and tests the other implementations against them.
poly1305-simd is among the failing algorithms because it loses carry bits when handling long "all 0xff bytes" inputs. poly1305-avx2-x86_64.S is definitely broken, and poly1305-sse2-x86_64.S *might* be too. I am working on a patch...
It would be interesting to hear whether the Zinc self-tests detect this too. I couldn't easily tell from testing Herbert's Zinc patch series, because it seemed most of the Zinc self-tests were failing for other reasons.
FWIW, this type of thing is one of the reasons we made the performance-critical part of Adiantum's hash function be NH rather than Poly1305, and for now crypto/adiantum.c only uses the C implementation of Poly1305. Fast NH implementations are much simpler and less error-prone than Poly1305. Of course, proper testing will help a lot...
- Eric
Jason A. Donenfeld
maintainerQuoted 3 lines · Eric Biggers
poly1305-simd is among the failing algorithms because it loses carry bits when handling long "all
0xffbytes" inputs.poly1305-avx2-x86_64.Sis definitely broken, andpoly1305-sse2-x86_64.S*might* be too. I am working on a patch...
Yea.... yikes. I'm kind of souring on this plan of having to deal with that code in Zinc, versus the extensively studied, fuzzed, and scrutinized code from Andy. Subtle carry bugs like that are kind of a testament to my overall plan of preferring formally verified or heavily used implementations to bespoke ones. This stuff is hard to get right.
Jason
Quoted 5 lines · Jason A. Donenfeld
Quoted 3 lines · Eric Biggers
poly1305-simd is among the failing algorithms because it loses carry bits when handling long "all
0xffbytes" inputs.poly1305-avx2-x86_64.Sis definitely broken, andpoly1305-sse2-x86_64.S*might* be too. I am working on a patch...
Quoted 8 lines
Yea.... yikes. I'm kind of souring on this plan of having to deal with that code in Zinc, versus the extensively studied, fuzzed, and scrutinized code from Andy. Subtle carry bugs like that are kind of a testament to my overall plan of preferring formally verified or heavily used implementations to bespoke ones. This stuff is hard to get right.
Jason
I agree that Andy's Poly1305 code is better and we should probably switch to it, but to be fair it's also longer and more complex, and I think you overestimate the extent to which it's actually been "studied, fuzzed, and scrutinized". Andy previously made essentially the same mistake in three of his Poly1305 implementations as well, which he had to fix:
https://mta.openssl.org/pipermail/openssl-commits/2016-April/006639.html
Also OpenSSL's PowerPC implementation of AES-CTR that was incorporated into the kernel was incorrect, as was recently fixed by
commit dcf7b48212c0fab7df69e84fab22d6cb7c8c0fb9
Author: Daniel Axtens <[email protected]>
Date: Fri Mar 15 13:09:01 2019 +1100
crypto: vmx - fix copy-paste error in CTR mode
The original assembly imported from OpenSSL has two copy-paste errors in handling CTR mode. When dealing with a 2 or 3 block tail, the code branches to the CBC decryption exit path, rather than to the CTR exit path.
and by https://github.com/openssl/openssl/pull/8510. This took almost 5 years.
I also still think the extent that you keep emphasizing the phrase "formally verified" when discussing Zinc is somewhat misleading, because the only implementation that is actually formally verified is Curve25519 in C. No other algorithms or implementations are formally verified. E.g. none of the Poly1305 implementations under discussion are formally verified. That needs to be made very clear, and as one consequence of that we really need good tests.
- Eric
David Miller
maintainerFrom: Herbert Xu <[email protected]>
Date: Mon, 25 Mar 2019 19:51:56 +0800
Quoted 3 lines
I really think we ought to separate the zinc interface from these new crypto implementations. They have nothing to do with each other.
I completely agree.