WireGuard: Secure Network Tunnel
Subsystems (30)
Jason A. Donenfeld
maintainerv6 cover WireGuard: Secure Network Tunnel
Changes v5->v6, along with who suggested it.
--------------------------------------------
- [Eric Biggers] Cleaner and more efficient blake2s_final function.
- [Eric Biggers] Remove modulo trick that was working around a gcc 8.1 bug.
- [Eric Biggers] Use crypto_xor_cpy to avoid a memmove in chacha20.
- In the unlikely condition that we transition from SIMD back to scalar instructions for Poly1305, it's important that we also convert back from base 2^26 to 2^64. This is super rare and weird, and it's hard to imagine somebody actually doing this for whatever bad reason, but it is a possibility, so we shouldn't calculate things wrong in that instance.
- [Andrew Lunn] Change BUG_ON to WARN_ON in choice places.
- [Thomas Gleixner] Explicitly dual license files under X where X≠GPL2
to be under X && GPL2.
- [Andy Lutomirski] Use separate symbols for selftests that are of
variable length, so that we neither waste space on disk nor waste space
in memory (via __initconst).
- [Thomas Gleixner] Make SPDX headers a standalone comment, and use the
// commenting style in .h files, per the kernel style guide.
- [Paul Burton] Allow assembler to fill branch delay slots on MIPS32r2,
so that the implementation is more future-proof.
- [Eric Biggers] Use Eric's scalar implementation for ChaCha20 ARM, which
performs very well for Cortex-A7,5 and ARMv6, while using Andy
Polyakov's NEON implementation for other ARMv7.
- [Ard Biesheuvel] Reduce optimization from -O3 to -O2.
- [Arn?d Bi?e(rgmann|shuvel)] Make sure stack frames stay inside 1024
bytes on 32-bit and 1280 bytes on 64-bit. - [Ard Biesheuvel] CRYPTOGAMS-related commit messages now have the corresponding OpenSSL commit written in them.
- [Eric Biggers] Keep HChaCha20 key in native endian, since in some cases it can trivially be passed into the ChaCha20 block later.
- Make constants follow a consistent naming scheme.
- Workaround gcc 8 stack mis-optimization on m68k.
- [Arnd Bergmann] Workaround gcc 8 stack mis-optimization with KASAN.
Many of the above ARM changes are a result of discussions between Ard, Eric, AndyL, AndyP, Samuel, and me. These discussions are ongoing, and so it's possible we'll do another revision with further ARM tweaks. But perhaps this one will be sufficient for merging now, and we can continue to refine later in the cycle.
-----------------------------------------------------------
This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:
* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard
-----------------------------------------------------------
WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.
Ample information, including documentation, installation instructions, and project details, is available at:
* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf
As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:
* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/
The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:
* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf
The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:
* https://www.wireguard.com/formal-verification/
This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.
It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.
Enjoy,
Jason
Jason A. Donenfeld
maintainerPATCH v6 1/23 asm: simd context helper API
Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:
simd_context_t simd_context;
simd_get(&simd_context);
while ((item = get_item_from_queue()) != NULL) {
encrypt_item(item, simd_context);
simd_relax(&simd_context);
}
simd_put(&simd_context);The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.
On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:
void encrypt_item(struct item *item, simd_context_t *simd_context)
{
if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
wild_simd_code(item);
else
boring_scalar_code(item);
}The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.
--- arch/alpha/include/asm/Kbuild | 5 ++- arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/simd.h | 63 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/simd.h | 51 +++++++++++++++++++++--- arch/c6x/include/asm/Kbuild | 3 +- arch/h8300/include/asm/Kbuild | 3 +- arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nds32/include/asm/Kbuild | 7 ++-- arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 7 ++-- arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 3 +- arch/riscv/include/asm/Kbuild | 3 +- arch/s390/include/asm/Kbuild | 3 +- arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 3 +- arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/simd.h | 44 ++++++++++++++++++++- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/simd.h | 20 ++++++++++ include/linux/simd.h | 28 +++++++++++++ 26 files changed, 234 insertions(+), 21 deletions(-) create mode 100644 arch/arm/include/asm/simd.h create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +3 -2
@@ -2,14 +2,15 @@ generic-y += compat.h +generic-y += current.h generic-y += exec.h generic-y += export.h generic-y += fb.h generic-y += irq_work.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += trace_clock.h -generic-y += current.h -generic-y += kprobes.h
arch/arc/include/asm/Kbuild +1 -0
@@ -22,6 +22,7 @@ generic-y += parport.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/arm/include/asm/simd.h new file +63 -0
63 lines not on this page · raw message
arch/arm64/include/asm/simd.h +45 -6
71 lines not on this page · raw message
arch/c6x/include/asm/Kbuild +2 -1
@@ -5,8 +5,8 @@ generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h generic-y += preempt.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += tlbflush.h generic-y += topology.h generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
@@ -8,8 +8,8 @@ generic-y += current.h generic-y += delay.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += spinlock.h generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
@@ -29,6 +29,7 @@ generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += topology.h generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
@@ -4,6 +4,7 @@ generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h +generic-y += simd.h generic-y += trace_clock.h generic-y += vtime.h generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h generic-y += percpu.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
@@ -25,6 +25,7 @@ generic-y += parport.h generic-y += percpu.h generic-y += preempt.h generic-y += serial.h +generic-y += simd.h generic-y += syscalls.h generic-y += topology.h generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
@@ -16,6 +16,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
@@ -7,14 +7,14 @@ generic-y += bug.h generic-y += bugs.h generic-y += checksum.h generic-y += clkdev.h -generic-y += cmpxchg.h generic-y += cmpxchg-local.h +generic-y += cmpxchg.h generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h generic-y += segment.h generic-y += serial.h generic-y += shmbuf.h +generic-y += simd.h generic-y += sizes.h generic-y += stat.h generic-y += switch_to.h generic-y += timex.h generic-y += topology.h generic-y += trace_clock.h -generic-y += xor.h generic-y += unaligned.h generic-y += user.h generic-y += vga.h generic-y += word-at-a-time.h +generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
@@ -33,6 +33,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
@@ -28,12 +28,13 @@ generic-y += module.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += qspinlock_types.h -generic-y += qspinlock.h -generic-y += qrwlock_types.h generic-y += qrwlock.h +generic-y += qrwlock_types.h +generic-y += qspinlock.h +generic-y += qspinlock_types.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += string.h generic-y += switch_to.h generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
@@ -17,6 +17,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += seccomp.h generic-y += segment.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
@@ -4,7 +4,8 @@ generic-y += irq_regs.h generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h +generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += vtime.h -generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
@@ -5,9 +5,9 @@ generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-contiguous.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h generic-y += shmbuf.h generic-y += shmparam.h generic-y += signal.h +generic-y += simd.h generic-y += socket.h generic-y += sockios.h generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h generic-y += asm-offsets.h generic-y += cacheflush.h generic-y += device.h +generic-y += div64.h generic-y += dma-contiguous.h generic-y += dma-mapping.h -generic-y += div64.h generic-y += emergency-restart.h generic-y += export.h generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
@@ -16,6 +16,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += trace_clock.h generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
@@ -17,5 +17,6 @@ generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
@@ -16,15 +16,16 @@ generic-y += io.h generic-y += irq_regs.h generic-y += irq_work.h generic-y += kdebug.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += param.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += switch_to.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.h generic-y += xor.h -generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
@@ -27,6 +27,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += syscalls.h generic-y += topology.h
arch/x86/include/asm/simd.h +43 -1
50 lines not on this page · raw message
arch/xtensa/include/asm/Kbuild +1 -0
@@ -24,6 +24,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += sections.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.h
include/asm-generic/simd.h +20 -0
@@ -1,5 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H +#include <linux/hardirq.h> /*
@@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void) {return !in_interrupt (); } + +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +return false; +} + +#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _SIMD_H +#define _SIMD_H + +typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; + +#include <linux/sched.h> +#include <asm/simd.h> + +static inline void simd_relax (simd_context_t *ctx) +{ +#ifdef CONFIG_PREEMPT +if ((*ctx &HAVE_SIMD_IN_USE ) &&need_resched ()) { +simd_put (ctx); +simd_get (ctx); + } +#endif +} + +#endif /* _SIMD_H */
Signature · 1 line
2.19.0
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 80 lines · Jason A. Donenfeld
Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:
simd_context_t simd_context; simd_get(&simd_context); while ((item = get_item_from_queue()) != NULL) { encrypt_item(item, simd_context); simd_relax(&simd_context); } simd_put(&simd_context);The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.
On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:
void encrypt_item(struct item *item, simd_context_t *simd_context) { if (item->len > LARGE_FOR_SIMD && simd_use(simd_context)) wild_simd_code(item); else boring_scalar_code(item); }The actual XSAVE happens during
simd_use(and only on the first time), so that if the context is never actually used, no performance penalty is hit.Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Greg KH <[email protected]>--- arch/alpha/include/asm/Kbuild | 5 ++- arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/simd.h | 63 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/simd.h | 51 +++++++++++++++++++++--- arch/c6x/include/asm/Kbuild | 3 +- arch/h8300/include/asm/Kbuild | 3 +- arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nds32/include/asm/Kbuild | 7 ++-- arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 7 ++-- arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 3 +- arch/riscv/include/asm/Kbuild | 3 +- arch/s390/include/asm/Kbuild | 3 +- arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 3 +- arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/simd.h | 44 ++++++++++++++++++++- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/simd.h | 20 ++++++++++ include/linux/simd.h | 28 +++++++++++++ 26 files changed, 234 insertions(+), 21 deletions(-) create mode 100644 arch/arm/include/asm/simd.h create mode 100644 include/linux/simd.harch/alpha/include/asm/Kbuild +1
-0@@ -2,14 +2,15 @@ generic-y += compat.h +generic-y += current.h generic-y += exec.h generic-y += export.h generic-y += fb.h
arch/alpha/include/asm/Kbuild line 15
Quoted 10 lines
arch/alpha/include/asm/Kbuild +2
-2generic-y += irq_work.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += trace_clock.h -generic-y += current.h -generic-y += kprobes.h
Given that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.
Quoted 25 lines
arch/arc/include/asm/Kbuild +1
-0@@ -22,6 +22,7 @@ generic-y += parport.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.harch/arm/include/asm/simd.h new file +7
-0@@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <linux/simd.h> +#ifndef _ASM_SIMD_H
arch/arm/include/asm/simd.h line 17
Quoted 10 lines
arch/arm/include/asm/simd.h new file +10
-0+#define _ASM_SIMD_H + +#ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> + +static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +} +
Remember this guy?
https://marc.info/?l=linux-arch&m=149631094625176&w=2
That was never merged, so let's get it right this time.
Quoted 551 lines
arch/arm/include/asm/simd.h new file +46
-0+static inline void simd_get (simd_context_t *ctx) +{ + *ctx =may_use_simd () ?HAVE_FULL_SIMD :HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +if (*ctx &HAVE_SIMD_IN_USE ) +kernel_neon_end (); + *ctx =HAVE_NO_SIMD ; +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +if (!(*ctx &HAVE_FULL_SIMD )) +return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_neon_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +} + +#else + +static __must_check inline bool may_use_simd (void ) +{ +return false; +} + +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +return false; +} +#endif + +#endif /* _ASM_SIMD_H */ arch/arm64/include/asm/simd.h +45
-6@@ -1,11 +1,10 @@ -/* -* Copyright (C) 2017 Linaro Ltd. <[email protected]> +/* SPDX-License-Identifier: GPL-2.0 * -* This program is free software; you can redistribute it and/or modify it -* under the terms of the GNU General Public License version 2 as published -* by the Free Software Foundation. +* Copyright (C) 2017 Linaro Ltd. <[email protected]> +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. */ +#include <linux/simd.h> #ifndef __ASM_SIMD_H#define __ASM_SIMD_H@@ -16,6 +15,8 @@#include <linux/types.h> #ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> +#include <asm/simd.h> DECLARE_PER_CPU (bool, kernel_neon_busy);@@ -40,9 +41,47 @@ static __must_check inline bool may_use_simd(void) !this_cpu_read (kernel_neon_busy); } +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =may_use_simd () ?HAVE_FULL_SIMD :HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +if (*ctx &HAVE_SIMD_IN_USE ) +kernel_neon_end (); + *ctx =HAVE_NO_SIMD ; +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +if (!(*ctx &HAVE_FULL_SIMD )) +return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_neon_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +} +#else /* ! CONFIG_KERNEL_MODE_NEON */ -static __must_check inline bool may_use_simd (void ) { +static __must_check inline bool may_use_simd (void ) +{ +return false; +} + +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{return false; }arch/c6x/include/asm/Kbuild +2
-1@@ -5,8 +5,8 @@ generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h@@ -30,6 +30,7 @@ generic-y += pgalloc.h generic-y += preempt.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += tlbflush.h generic-y += topology.h generic-y += trace_clock.harch/h8300/include/asm/Kbuild +2
-1@@ -8,8 +8,8 @@ generic-y += current.h generic-y += delay.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h@@ -39,6 +39,7 @@ generic-y += preempt.h generic-y += scatterlist.h generic-y += sections.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += spinlock.h generic-y += timex.harch/hexagon/include/asm/Kbuild +1
-0@@ -29,6 +29,7 @@ generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += topology.h generic-y += trace_clock.harch/ia64/include/asm/Kbuild +1
-0@@ -4,6 +4,7 @@ generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h +generic-y += simd.h generic-y += trace_clock.h generic-y += vtime.h generic-y += word-at-a-time.harch/m68k/include/asm/Kbuild +1
-0@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h generic-y += percpu.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.harch/microblaze/include/asm/Kbuild +1
-0@@ -25,6 +25,7 @@ generic-y += parport.h generic-y += percpu.h generic-y += preempt.h generic-y += serial.h +generic-y += simd.h generic-y += syscalls.h generic-y += topology.h generic-y += trace_clock.harch/mips/include/asm/Kbuild +1
-0@@ -16,6 +16,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += user.harch/nds32/include/asm/Kbuild +4
-3@@ -7,14 +7,14 @@ generic-y += bug.h generic-y += bugs.h generic-y += checksum.h generic-y += clkdev.h -generic-y += cmpxchg.h generic-y += cmpxchg-local.h +generic-y += cmpxchg.h generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h@@ -46,14 +46,15 @@ generic-y += sections.h generic-y += segment.h generic-y += serial.h generic-y += shmbuf.h +generic-y += simd.h generic-y += sizes.h generic-y += stat.h generic-y += switch_to.h generic-y += timex.h generic-y += topology.h generic-y += trace_clock.h -generic-y += xor.h generic-y += unaligned.h generic-y += user.h generic-y += vga.h generic-y += word-at-a-time.h +generic-y += xor.harch/nios2/include/asm/Kbuild +1
-0@@ -33,6 +33,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += spinlock.h generic-y += topology.h generic-y += trace_clock.harch/openrisc/include/asm/Kbuild +4
-3@@ -28,12 +28,13 @@ generic-y += module.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += qspinlock_types.h -generic-y += qspinlock.h -generic-y += qrwlock_types.h generic-y += qrwlock.h +generic-y += qrwlock_types.h +generic-y += qspinlock.h +generic-y += qspinlock_types.h generic-y += sections.h generic-y += segment.h +generic-y += simd.h generic-y += string.h generic-y += switch_to.h generic-y += topology.harch/parisc/include/asm/Kbuild +1
-0@@ -17,6 +17,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += seccomp.h generic-y += segment.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.harch/powerpc/include/asm/Kbuild +2
-1@@ -4,7 +4,8 @@ generic-y += irq_regs.h generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h +generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += vtime.h -generic-y += msi.harch/riscv/include/asm/Kbuild +2
-1@@ -5,9 +5,9 @@ generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h -generic-y += dma.h generic-y += dma-contiguous.h generic-y += dma-mapping.h +generic-y += dma.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h@@ -46,6 +46,7 @@ generic-y += setup.h generic-y += shmbuf.h generic-y += shmparam.h generic-y += signal.h +generic-y += simd.h generic-y += socket.h generic-y += sockios.h generic-y += stat.harch/s390/include/asm/Kbuild +2
-1@@ -7,9 +7,9 @@ generated-y += unistd_nr.h generic-y += asm-offsets.h generic-y += cacheflush.h generic-y += device.h +generic-y += div64.h generic-y += dma-contiguous.h generic-y += dma-mapping.h -generic-y += div64.h generic-y += emergency-restart.h generic-y += export.h generic-y += fb.h@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += rwsem.h +generic-y += simd.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.harch/sh/include/asm/Kbuild +1
-0@@ -16,6 +16,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += trace_clock.h generic-y += xor.harch/sparc/include/asm/Kbuild +1
-0@@ -17,5 +17,6 @@ generic-y += msi.h generic-y += preempt.h generic-y += rwsem.h generic-y += serial.h +generic-y += simd.h generic-y += trace_clock.h generic-y += word-at-a-time.harch/um/include/asm/Kbuild +2
-1@@ -16,15 +16,16 @@ generic-y += io.h generic-y += irq_regs.h generic-y += irq_work.h generic-y += kdebug.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += param.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += switch_to.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.h generic-y += xor.h -generic-y += kprobes.harch/unicore32/include/asm/Kbuild +1
-0@@ -27,6 +27,7 @@ generic-y += preempt.h generic-y += sections.h generic-y += segment.h generic-y += serial.h +generic-y += simd.h generic-y += sizes.h generic-y += syscalls.h generic-y += topology.harch/x86/include/asm/simd.h +43
-1@@ -1,4 +1,11 @@ -/* SPDX-License-Identifier: GPL-2.0 */ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H#include <asm/fpu/api.h> @@ -10,3 +17,38 @@ static __must_check inline bool may_use_simd(void) {return irq_fpu_usable (); } + +static inline void simd_get (simd_context_t *ctx) +{ +#if !defined(CONFIG_UML ) + *ctx =may_use_simd () ?HAVE_FULL_SIMD :HAVE_NO_SIMD ; +#else + *ctx =HAVE_NO_SIMD ; +#endif +} + +static inline void simd_put (simd_context_t *ctx) +{ +#if !defined(CONFIG_UML ) +if (*ctx &HAVE_SIMD_IN_USE ) +kernel_fpu_end (); +#endif + *ctx =HAVE_NO_SIMD ; +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +#if !defined(CONFIG_UML ) +if (!(*ctx &HAVE_FULL_SIMD )) +return false; +if (*ctx &HAVE_SIMD_IN_USE ) +return true; +kernel_fpu_begin (); + *ctx |=HAVE_SIMD_IN_USE ; +return true; +#else +return false; +#endif +} + +#endif /* _ASM_SIMD_H */ arch/xtensa/include/asm/Kbuild +1
-0@@ -24,6 +24,7 @@ generic-y += percpu.h generic-y += preempt.h generic-y += rwsem.h generic-y += sections.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += word-at-a-time.hinclude/asm-generic/simd.h +20
-0@@ -1,5 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/simd.h> +#ifndef _ASM_SIMD_H +#define _ASM_SIMD_H +#include <linux/hardirq.h> /* @@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void) {return !in_interrupt (); } + +static inline void simd_get (simd_context_t *ctx) +{ + *ctx =HAVE_NO_SIMD ; +} + +static inline void simd_put (simd_context_t *ctx) +{ +} + +static __must_check inline bool simd_use (simd_context_t *ctx) +{ +return false; +} + +#endif /* _ASM_SIMD_H */ include/linux/simd.h new file +15
-0@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _SIMD_H +#define _SIMD_H + +typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; + +#include <linux/sched.h>
include/linux/simd.h line 25
Quoted 10 lines
include/linux/simd.h new file +10
-0+#include <asm/simd.h> + +static inline void simd_relax (simd_context_t *ctx) +{ +#ifdef CONFIG_PREEMPT +if ((*ctx &HAVE_SIMD_IN_USE ) &&need_resched ()) { +simd_put (ctx); +simd_get (ctx); + } +#endif
Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.
Quoted 6 lines
include/linux/simd.h new file +3
-1+} + +#endif /* _SIMD_H */ --2.19.0
Re: [PATCH net-next v6 01/23] asm: simd context helper API
(+ Joe)
Quoted 138 lines · Ard Biesheuvel
Quoted 80 lines · Jason A. Donenfeld
Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:
simd_context_t simd_context; simd_get(&simd_context); while ((item = get_item_from_queue()) != NULL) { encrypt_item(item, simd_context); simd_relax(&simd_context); } simd_put(&simd_context);The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.
On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:
void encrypt_item(struct item *item, simd_context_t *simd_context) { if (item->len > LARGE_FOR_SIMD && simd_use(simd_context)) wild_simd_code(item); else boring_scalar_code(item); }The actual XSAVE happens during
simd_use(and only on the first time), so that if the context is never actually used, no performance penalty is hit.Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Greg KH <[email protected]>--- arch/alpha/include/asm/Kbuild | 5 ++- arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/simd.h | 63 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/simd.h | 51 +++++++++++++++++++++--- arch/c6x/include/asm/Kbuild | 3 +- arch/h8300/include/asm/Kbuild | 3 +- arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nds32/include/asm/Kbuild | 7 ++-- arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 7 ++-- arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 3 +- arch/riscv/include/asm/Kbuild | 3 +- arch/s390/include/asm/Kbuild | 3 +- arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 3 +- arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/simd.h | 44 ++++++++++++++++++++- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/simd.h | 20 ++++++++++ include/linux/simd.h | 28 +++++++++++++ 26 files changed, 234 insertions(+), 21 deletions(-) create mode 100644 arch/arm/include/asm/simd.h create mode 100644 include/linux/simd.harch/alpha/include/asm/Kbuild +1
-0@@ -2,14 +2,15 @@ generic-y += compat.h +generic-y += current.h generic-y += exec.h generic-y += export.h generic-y += fb.harch/alpha/include/asm/Kbuild line 15
Quoted 10 lines
arch/alpha/include/asm/Kbuild +2
-2generic-y += irq_work.h +generic-y += kprobes.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h generic-y += sections.h +generic-y += simd.h generic-y += trace_clock.h -generic-y += current.h -generic-y += kprobes.hGiven that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.
Quoted 25 lines
arch/arc/include/asm/Kbuild +1
-0@@ -22,6 +22,7 @@ generic-y += parport.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h +generic-y += simd.h generic-y += topology.h generic-y += trace_clock.h generic-y += user.harch/arm/include/asm/simd.h new file +7
-0@@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <linux/simd.h> +#ifndef _ASM_SIMD_Harch/arm/include/asm/simd.h line 17
Quoted 10 lines
arch/arm/include/asm/simd.h new file +10
-0+#define _ASM_SIMD_H + +#ifdef CONFIG_KERNEL_MODE_NEON +#include <asm/neon.h> + +static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +} +Remember this guy?
https://marc.info/?l=linux-arch&m=149631094625176&w=2
That was never merged, so let's get it right this time.
...
Quoted 12 lines
Quoted 12 lines
include/linux/simd.h new file +6
-0@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _SIMD_H
Quoted 8 lines
include/linux/simd.h line 14
Quoted 8 lines
include/linux/simd.h new file +8
-0+#define _SIMD_H + +typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; +
Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
Quoted 23 lines
include/linux/simd.h line 25
Quoted 11 lines
include/linux/simd.h new file +11
-0+#include <linux/sched.h> +#include <asm/simd.h> + +static inline void simd_relax (simd_context_t *ctx) +{ +#ifdef CONFIG_PREEMPT +if ((*ctx &HAVE_SIMD_IN_USE ) &&need_resched ()) { +simd_put (ctx); +simd_get (ctx); + } +#endif Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.
Quoted 6 lines
include/linux/simd.h new file +3
-1+} + +#endif /* _SIMD_H */ --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
include/linux/simd.h line 14
Quoted 6 lines
include/linux/simd.h new file +6
-0+typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
include/linux/simd.h line 14
Quoted 6 lines
include/linux/simd.h new file +6
-0+typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
Quoted 4 lines
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...
Yes that makes sense
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 16 lines · Ard Biesheuvel
Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
include/linux/simd.h line 14
Quoted 6 lines
include/linux/simd.h new file +6
-0+typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
Quoted 6 lines
Quoted 4 lines
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 18 lines · Jason A. Donenfeld
Quoted 16 lines · Ard Biesheuvel
Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
include/linux/simd.h line 14
Quoted 6 lines
include/linux/simd.h new file +6
-0+typedef enum { +HAVE_NO_SIMD =1 <<0 , +HAVE_FULL_SIMD =1 <<1 , +HAVE_SIMD_IN_USE =1 <<31 +}simd_context_t ; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Yes just switch to a struct.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 25 lines · Ard Biesheuvel
Quoted 23 lines · Jason A. Donenfeld
Quoted 21 lines · Ard Biesheuvel
Quoted 18 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+typedef enum { + HAVE_NO_SIMD = 1 << 0, + HAVE_FULL_SIMD = 1 << 1, + HAVE_SIMD_IN_USE = 1 << 31 +} simd_context_t; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Yes just switch to a struct.
Okay. No problem with that, but will wait to hear from Joe first.
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 27 lines · Jason A. Donenfeld
Quoted 25 lines · Ard Biesheuvel
Quoted 23 lines · Jason A. Donenfeld
Quoted 21 lines · Ard Biesheuvel
Quoted 18 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+typedef enum { + HAVE_NO_SIMD = 1 << 0, + HAVE_FULL_SIMD = 1 << 1, + HAVE_SIMD_IN_USE = 1 << 31 +} simd_context_t; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Yes just switch to a struct.
Okay. No problem with that, but will wait to hear from Joe first.
Why do you need to hear from me again?
As far as I know, the only info about typedef avoidance are in Documentation/process/coding-style.rst section 5.
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 36 lines · Joe Perches
Quoted 33 lines · Jason A. Donenfeld
Quoted 29 lines · Ard Biesheuvel
Quoted 28 lines · Jason A. Donenfeld
Quoted 16 lines · Ard Biesheuvel
Quoted 16 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+typedef enum { + HAVE_NO_SIMD = 1 << 0, + HAVE_FULL_SIMD = 1 << 1, + HAVE_SIMD_IN_USE = 1 << 31 +} simd_context_t; +Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
Quoted 6 lines
Quoted 4 lines
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
Yes just switch to a struct.
Okay. No problem with that, but will wait to hear from Joe first.
Why do you need to hear from me again?
As far as I know, the only info about typedef avoidance are in
Documentation/process/coding-style.rstsection 5.
I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 18 lines · Andy Lutomirski
Quoted 18 lines
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Yes just switch to a struct.
Okay. No problem with that, but will wait to hear from Joe first.
Quoted 8 lines
Quoted 6 lines
Why do you need to hear from me again?
As far as I know, the only info about typedef avoidance are in
Documentation/process/coding-style.rstsection 5.I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.
I'll stick with a typedef. Reading the style guide, this clearly falls into 5.a, 5.b, and maybe 5.c. For 5.a, at some point this will possibly contain architecture specific blobs. For 5.b, it is just an enum (integer) right now.
Re: [PATCH net-next v6 01/23] asm: simd context helper API
Quoted 26 lines · Jason A. Donenfeld
Quoted 25 lines · Andy Lutomirski
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?
In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it
struct simd_contextnow with 1 member though...Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes that makes sense
The rationale for it being a typedef or moving to a struct now?
Yes just switch to a struct.
Okay. No problem with that, but will wait to hear from Joe first.
Why do you need to hear from me again?
As far as I know, the only info about typedef avoidance are in
Documentation/process/coding-style.rstsection 5.
Quoted 6 lines
Quoted 1 line
I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.
I'll stick with a typedef. Reading the style guide, this clearly falls into 5.a, 5.b, and maybe 5.c. For 5.a, at some point this will possibly contain architecture specific blobs. For 5.b, it is just an enum (integer) right now.
I can live with that, but other may still object.
In any case, since you are using the enum member as a bitfield, it would be better to typedef it to int instead, and retain the enum definition only for the symbolic constants.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 01/23] asm: simd context helper API
In reply to Ard BiesheuvelQuoted 3 lines · Ard Biesheuvel
Given that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.
Ack. Will retain order for v7.
Quoted 11 lines
arch/arm/include/asm/simd.h line 17
Quoted 5 lines
arch/arm/include/asm/simd.h new file +5
-0+static __must_check inline bool may_use_simd (void ) +{ +return !in_interrupt (); +} +Remember this guy?
https://marc.info/?l=linux-arch&m=149631094625176&w=2
That was never merged, so let's get it right this time.
Wow, nice memory. I had forgotten all about that. Queued for v7.
Quoted 5 lines
Quoted 5 lines
include/linux/simd.h new file +5
-0+static inline void simd_relax (simd_context_t *ctx) +{ +#ifdef CONFIG_PREEMPT +if ((*ctx &HAVE_SIMD_IN_USE ) &&need_resched ()) { +simd_put (ctx);
Quoted 8 lines
include/linux/simd.h line 25
Quoted 3 lines
include/linux/simd.h new file +3
-0+ simd_get(ctx); + } +#endif Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.
Sure, sounds easy enough.
Jason A. Donenfeld
maintainerPATCH v6 2/23 zinc: introduce minimal cryptography library
In reply to Jason A. DonenfeldZinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.
Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:
* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.
* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.
* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.
This last observation has seen itself play out several times over and over again within the kernel:
* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.
* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.
* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.
Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.
It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.
All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].
Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.
SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.
More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.
Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.
Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.
Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.
As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.
[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof
--- MAINTAINERS | 8 ++++++++ lib/Kconfig | 2 ++ lib/Makefile | 2 ++ lib/zinc/Kconfig | 32 ++++++++++++++++++++++++++++++++ lib/zinc/Makefile | 3 +++ 5 files changed, 47 insertions(+) create mode 100644 lib/zinc/Kconfig create mode 100644 lib/zinc/Makefile
MAINTAINERS +8 -0
@@ -16170,6 +16170,14 @@ Q: https://patchwork.linuxtv.org/project/linux-media/list/ S: Maintained F: drivers/media/dvb-frontends/zd1301_demod* +ZINC CRYPTOGRAPHY LIBRARY +M: Jason A. Donenfeld <[email protected]> +M: Samuel Neves <[email protected]> +S: Maintained +F: lib/zinc/ +F: include/zinc/ +L: [email protected] + ZPOOL COMPRESSED PAGE STORAGE API M: Dan Streetman <[email protected]> L: [email protected]
lib/Kconfig +2 -0
@@ -485,6 +485,8 @@ config GLOB_SELFTEST module load) by a small amount, so you're welcome to play with it, but you probably don't need it. +source "lib/zinc/Kconfig" + # # Netlink attribute parsing support is select'ed if needed #
lib/Makefile +2 -0
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o obj-$(CONFIG_ASN1) += asn1_decoder.o +obj-y += zinc/ + obj-$(CONFIG_FONT_SUPPORT) += fonts/ obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +32 -0
@@ -0,0 +1,32 @@ +config ZINC_DEBUG + bool "Zinc cryptography library debugging and self-tests" + help + This builds a series of self-tests for the Zinc crypto library, which + help diagnose any cryptographic algorithm implementation issues that + might be at the root cause of potential bugs. It also adds various + debugging traps. + + Unless you're developing and testing cryptographic routines, or are + especially paranoid about correctness on your hardware, you may say + N here. + +config ZINC_ARCH_ARM + def_bool y + depends on ARM && !64BIT + +config ZINC_ARCH_ARM64 + def_bool y + depends on ARM64 && 64BIT + +config ZINC_ARCH_X86_64 + def_bool y + depends on X86_64 && 64BIT + depends on !UML + +config ZINC_ARCH_MIPS + def_bool y + depends on MIPS && CPU_MIPS32_R2 && !64BIT + +config ZINC_ARCH_MIPS64 + def_bool y + depends on MIPS && 64BIT
lib/zinc/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +ccflags-y := -O2 +ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' +ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
Signature · 1 line
2.19.0
Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 6 lines · Jason A. Donenfeld
Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.
[]
lib/zinc/Makefile line 3
Quoted 9 lines
lib/zinc/Makefile new file +3
-0@@ -0,0 +1,3 @@ +ccflags-y := -O2 +ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' +ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 3 lines · Joe Perches
I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.
There are about 2000 cases in the kernel of the same '#define pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that I pr_err from, why can't I just do this from the makefile, since I want that same pr_fmt to copy the whole directory?
Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 9 lines · Jason A. Donenfeld
Quoted 5 lines · Joe Perches
I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.
Quoted 5 lines
There are about 2000 cases in the kernel of the same '#define
pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that Ipr_errfrom, why can't I just do this from the makefile, since I want that samepr_fmtto copy the whole directory?
Because people like to be able to just read the C file to figure out what it does. Or we could adopt the Makefile approach kernel-wide, since this use of it seems reasonable.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 7 lines · Andy Lutomirski
Quoted 7 lines · Jason A. Donenfeld
Quoted 5 lines · Joe Perches
I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.
Quoted 7 lines
Quoted 5 lines
There are about 2000 cases in the kernel of the same '#define
pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that Ipr_errfrom, why can't I just do this from the makefile, since I want that samepr_fmtto copy the whole directory?Because people like to be able to just read the C file to figure out what it does. Or we could adopt the
Makefileapproach kernel-wide, since this use of it seems reasonable.
It would indeed be nice to see this done tree-wide. I can recall various random bugs over the year where some dmesg messages are missing a prefix because the author forgot to copy and paste the thing to /yet another file/ in the same directory.
Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
In reply to Jason A. DonenfeldQuoted 9 lines · Jason A. Donenfeld
Quoted 3 lines · Joe Perches
I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.There are about 2000 cases in the kernel of the same '#define
pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that Ipr_errfrom,
As far as I can tell, zinc doesn't use pr_err, just pr_info in all the selftest blocks which are only used when DEBUG is #defined.
Perhaps all the pr_info uses should use pr_debug instead as well as dynamic debugging which already optionally adds KBUILD_MODINFO.
Quoted 2 lines
why can't I just do this from the makefile, since
I want that samepr_fmtto copy the whole directory?
Ideally all of those
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
uses will be removed one day soon and the default will change.
https://lore.kernel.org/patchwork/cover/904507/
This slightly odd use might complicate that.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Hi Joe,
Quoted 2 lines · Joe Perches
As far as I can tell, zinc doesn't use
pr_err, justpr_info
Yes, pr_info, not pr_err. Apologies for my imprecision. But the distinction does not matter at all, since they both use pr_fmt in exactly the same way, which is sufficient for the purposes of this discussion.
Quoted 5 lines
Quoted 2 lines
why can't I just do this from the makefile, since
I want that samepr_fmtto copy the whole directory?Ideally all of those
Quoted 8 lines
#define
pr_fmt(fmt)KBUILD_MODNAME": " fmtuses will be removed one day soon and the default
will change.https://lore.kernel.org/patchwork/cover/904507/
This slightly odd use might complicate that.
Oh, that's good to see. In this case, I'm prefixing all of them with
zinc: instead of zinc_chacha20: (the modname), but I wouldn't object to that changing to KBUILD_MODNAME if your patch lands. IOW, once your patch lands, I'm happy to revisit this discussion and fall back to using the sane defaults you're in the process of setting up for everyone.
Jason
Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 23 lines · Jason A. Donenfeld
Hi Joe,
Quoted 2 lines · Joe Perches
As far as I can tell, zinc doesn't use
pr_err, justpr_infoYes,
pr_info, notpr_err. Apologies for my imprecision. But the distinction does not matter at all, since they both usepr_fmtin exactly the same way, which is sufficient for the purposes of this discussion.Quoted 12 lines
Quoted 2 lines
why can't I just do this from the makefile, since
I want that samepr_fmtto copy the whole directory?Ideally all of those
#define
pr_fmt(fmt)KBUILD_MODNAME": " fmtuses will be removed one day soon and the default
will change.
Quoted 8 lines
Quoted 1 line
This slightly odd use might complicate that.
Oh, that's good to see. In this case, I'm prefixing all of them with
zinc: instead ofzinc_chacha20: (the modname), but I wouldn't object to that changing toKBUILD_MODNAMEif your patch lands. IOW, once your patch lands, I'm happy to revisit this discussion and fall back to using the sane defaults you're in the process of setting up for everyone.
I'd still prefer as all these are effectively debugging output that you convert the pr_info uses pr_debug and so avoid using pr_fmt altogether.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Hi Joe,
Quoted 3 lines · Joe Perches
I'd still prefer as all these are effectively debugging output that you convert the
pr_infousespr_debugand so avoid usingpr_fmtaltogether.
I started to write back, "sure no problem, this will be in v7," but then I gave it a closer look, and I think actually I'll be changing these into pr_err/pr_info and not naming this "DEBUG" but rather "SELFTEST", since I think these are actually very good to have beyond mere debugging.
Jason
Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 7 lines · Jason A. Donenfeld
Hi Joe,
Quoted 3 lines · Joe Perches
I'd still prefer as all these are effectively debugging output that you convert the
pr_infousespr_debugand so avoid usingpr_fmtaltogether.
Quoted 5 lines
I started to write back, "sure no problem, this will be in v7," but then I gave it a closer look, and I think actually I'll be changing these into pr_err/pr_info and not naming this "DEBUG" but rather "SELFTEST", since I think these are actually very good to have beyond mere debugging.
Then you should change the CONFIG_ZINC_DEBUG option too.
Dunno what you should do with the
#ifdef DEBUG BUG_ON(...);
#endif
uses though.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library
Quoted 1 line · Joe Perches
Then you should change the
CONFIG_ZINC_DEBUGoption too.
Yes, indeed.
Jason A. Donenfeld
maintainerPATCH v6 3/23 zinc: ChaCha20 generic C implementation and selftest
In reply to Jason A. DonenfeldThis implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
--- include/zinc/chacha20.h | 65 + lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 3 + lib/zinc/chacha20/chacha20.c | 179 +++ lib/zinc/selftest/chacha20.h | 2676 ++++++++++++++++++++++++++++++++++ 5 files changed, 2927 insertions(+) create mode 100644 include/zinc/chacha20.h create mode 100644 lib/zinc/chacha20/chacha20.c create mode 100644 lib/zinc/selftest/chacha20.h
include/zinc/chacha20.h new file +65 -0
@@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20_H +#define _ZINC_CHACHA20_H + +#include <asm/unaligned.h> +#include <linux/simd.h> +#include <linux/kernel.h> +#include <linux/types.h> + +enum { +CHACHA20_NONCE_SIZE =16 , +CHACHA20_KEY_SIZE =32 , +CHACHA20_KEY_WORDS =CHACHA20_KEY_SIZE /sizeof (u32), +CHACHA20_BLOCK_SIZE =64 , +CHACHA20_BLOCK_WORDS =CHACHA20_BLOCK_SIZE /sizeof (u32), +HCHACHA20_NONCE_SIZE =CHACHA20_NONCE_SIZE , +HCHACHA20_KEY_SIZE =CHACHA20_KEY_SIZE +}; + +enum {/* expand 32-byte k */ +CHACHA20_CONSTANT_EXPA =0x61707865U , +CHACHA20_CONSTANT_ND_3 =0x3320646eU , +CHACHA20_CONSTANT_2_BY =0x79622d32U , +CHACHA20_CONSTANT_TE_K =0x6b206574U +}; + +struct chacha20_ctx { +u32 constant[4 ]; +u32 key[8 ]; +u32 counter[4 ]; +}__aligned(32) ; + +static inline void chacha20_init (struct chacha20_ctx *state, +const u8 key[CHACHA20_KEY_SIZE ], +const u64 nonce) +{ + state->constant[0 ] =CHACHA20_CONSTANT_EXPA ; + state->constant[1 ] =CHACHA20_CONSTANT_ND_3 ; + state->constant[2 ] =CHACHA20_CONSTANT_2_BY ; + state->constant[3 ] =CHACHA20_CONSTANT_TE_K ; + state->key[0 ] =get_unaligned_le32 (key +0 ); + state->key[1 ] =get_unaligned_le32 (key +4 ); + state->key[2 ] =get_unaligned_le32 (key +8 ); + state->key[3 ] =get_unaligned_le32 (key +12 ); + state->key[4 ] =get_unaligned_le32 (key +16 ); + state->key[5 ] =get_unaligned_le32 (key +20 ); + state->key[6 ] =get_unaligned_le32 (key +24 ); + state->key[7 ] =get_unaligned_le32 (key +28 ); + state->counter[0 ] =0 ; + state->counter[1 ] =0 ; + state->counter[2 ] = nonce &U32_MAX ; + state->counter[3 ] = nonce >>32 ; +} +void chacha20 (struct chacha20_ctx *state,u8 *dst,const u8 *src,u32 len, +simd_context_t *simd_context); + +void hchacha20 (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t *simd_context); + +#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
@@ -1,3 +1,7 @@ +config ZINC_CHACHA20 + tristate + select CRYPTO_ALGAPI + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" help
lib/zinc/Makefile +3 -0
@@ -1,3 +1,6 @@ ccflags-y := -O2 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc_chacha20-y := chacha20/chacha20.o +obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +179 -0
179 lines not on this page · raw message
lib/zinc/selftest/chacha20.h new file +2676 -0
2,676 lines not on this page · raw message
Signature · 1 line
2.19.0
Re: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest
Quoted 54 lines · Jason A. Donenfeld
This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>--- include/zinc/chacha20.h | 65 + lib/zinc/Kconfig | 4 + lib/zinc/Makefile | 3 + lib/zinc/chacha20/chacha20.c | 179 +++ lib/zinc/selftest/chacha20.h | 2676 ++++++++++++++++++++++++++++++++++ 5 files changed, 2927 insertions(+) create mode 100644 include/zinc/chacha20.h create mode 100644 lib/zinc/chacha20/chacha20.c create mode 100644 lib/zinc/selftest/chacha20.hinclude/zinc/chacha20.h new file +26
-0@@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifndef _ZINC_CHACHA20_H +#define _ZINC_CHACHA20_H + +#include <asm/unaligned.h> +#include <linux/simd.h> +#include <linux/kernel.h> +#include <linux/types.h> + +enum { +CHACHA20_NONCE_SIZE =16 , +CHACHA20_KEY_SIZE =32 , +CHACHA20_KEY_WORDS =CHACHA20_KEY_SIZE /sizeof (u32), +CHACHA20_BLOCK_SIZE =64 , +CHACHA20_BLOCK_WORDS =CHACHA20_BLOCK_SIZE /sizeof (u32), +HCHACHA20_NONCE_SIZE =CHACHA20_NONCE_SIZE , +HCHACHA20_KEY_SIZE =CHACHA20_KEY_SIZE +}; + +enum {/* expand 32-byte k */ +CHACHA20_CONSTANT_EXPA =0x61707865U , + CHACHA20_CONSTANT_ND_3 = 0x3320646eU,
include/zinc/chacha20.h line 36
Quoted 10 lines
include/zinc/chacha20.h new file +10
-0+CHACHA20_CONSTANT_2_BY =0x79622d32U , + CHACHA20_CONSTANT_TE_K = 0x6b206574U +}; + +struct chacha20_ctx { +u32 constant[4 ]; +u32 key[8 ]; +u32 counter[4 ]; +}__aligned(32) ; +
32 *byte* alignment? Is that right? If this is for performance and it actually helps, using __cacheline_aligned is more appropriate,
Quoted 64 lines
include/zinc/chacha20.h new file +29
-0+static inline void chacha20_init (struct chacha20_ctx *state, +const u8 key[CHACHA20_KEY_SIZE ], +const u64 nonce) +{ + state->constant[0 ] =CHACHA20_CONSTANT_EXPA ; + state->constant[1 ] =CHACHA20_CONSTANT_ND_3 ; + state->constant[2 ] =CHACHA20_CONSTANT_2_BY ; + state->constant[3 ] =CHACHA20_CONSTANT_TE_K ; + state->key[0 ] =get_unaligned_le32 (key +0 ); + state->key[1 ] =get_unaligned_le32 (key +4 ); + state->key[2 ] =get_unaligned_le32 (key +8 ); + state->key[3 ] =get_unaligned_le32 (key +12 ); + state->key[4 ] =get_unaligned_le32 (key +16 ); + state->key[5 ] =get_unaligned_le32 (key +20 ); + state->key[6 ] =get_unaligned_le32 (key +24 ); + state->key[7 ] =get_unaligned_le32 (key +28 ); + state->counter[0 ] =0 ; + state->counter[1 ] =0 ; + state->counter[2 ] = nonce &U32_MAX ; + state->counter[3 ] = nonce >>32 ; +} +void chacha20 (struct chacha20_ctx *state,u8 *dst,const u8 *src,u32 len, +simd_context_t *simd_context); + +void hchacha20 (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t *simd_context); + +#endif /* _ZINC_CHACHA20_H */ lib/zinc/Kconfig +4
-0@@ -1,3 +1,7 @@ +config ZINC_CHACHA20 + tristate + select CRYPTO_ALGAPI + config ZINC_DEBUG bool "Zinc cryptography library debugging and self-tests" helplib/zinc/Makefile +3
-0@@ -1,3 +1,6 @@ ccflags-y := -O2 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG + +zinc_chacha20-y := chacha20/chacha20.o +obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.olib/zinc/chacha20/chacha20.c new file +6
-0@@ -0,0 +1,179 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +* +* Implementation of the ChaCha20 stream cipher. +*
lib/zinc/chacha20/chacha20.c line 16
Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10
-0+* Information: https://cr.yp.to/chacha.html +*/ + +#include <zinc/chacha20.h> + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/init.h> +#include <crypto/algapi.h> +
I guess this include is for crypto_xor_cpy() ?
We may want to put a comment here, so we keep track of the interdependencies.
lib/zinc/chacha20/chacha20.c line 17
Quoted 1 line
lib/zinc/chacha20/chacha20.c new file +1
-0+#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION
This #define is never set in subsequent patches, so just drop this #ifndef entirely (for this patch only)
Quoted 138 lines
lib/zinc/chacha20/chacha20.c new file +138
-0+void __init chacha20_fpu_init (void ) +{ +} +static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *out, +const u8 *in,const size_t len, +simd_context_t *simd_context) +{ +return false; +} +static inline bool hchacha20_arch (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ], +simd_context_t *simd_context) +{ +return false; +} +#endif + +#define QUARTER_ROUND (x, a, b, c, d) ( \ + x[a] += x[b], \ + x[d] = rol32((x[d] ^ x[a]), 16), \ + x[c] += x[d], \ + x[b] = rol32((x[b] ^ x[c]), 12), \ + x[a] += x[b], \ + x[d] = rol32((x[d] ^ x[a]), 8), \ + x[c] += x[d], \ + x[b] = rol32((x[b] ^ x[c]), 7) \ +) + +#define C (i, j) (i * 4 + j) + +#define DOUBLE_ROUND (x) ( \ +/* Column Round */ \ + QUARTER_ROUND(x, C(0, 0), C(1, 0), C(2, 0), C(3, 0)), \ + QUARTER_ROUND(x, C(0, 1), C(1, 1), C(2, 1), C(3, 1)), \ + QUARTER_ROUND(x, C(0, 2), C(1, 2), C(2, 2), C(3, 2)), \ + QUARTER_ROUND(x, C(0, 3), C(1, 3), C(2, 3), C(3, 3)), \ +/* Diagonal Round */ \ + QUARTER_ROUND(x, C(0, 0), C(1, 1), C(2, 2), C(3, 3)), \ + QUARTER_ROUND(x, C(0, 1), C(1, 2), C(2, 3), C(3, 0)), \ + QUARTER_ROUND(x, C(0, 2), C(1, 3), C(2, 0), C(3, 1)), \ + QUARTER_ROUND(x, C(0, 3), C(1, 0), C(2, 1), C(3, 2)) \ +) + +#define TWENTY_ROUNDS (x) ( \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x), \ + DOUBLE_ROUND(x) \ +) + +static void chacha20_block_generic (__le32 *stream,u32 *state) +{ +u32 x[CHACHA20_BLOCK_WORDS ]; +int i; + +for (i =0 ; i <ARRAY_SIZE (x); ++i) + x[i] = state[i]; + +TWENTY_ROUNDS (x); + +for (i =0 ; i <ARRAY_SIZE (x); ++i) + stream[i] =cpu_to_le32 (x[i] + state[i]); + + ++state[12 ]; +} + +static void chacha20_generic (struct chacha20_ctx *state,u8 *out,const u8 *in, +u32 len) +{ +__le32 buf[CHACHA20_BLOCK_WORDS ]; + +while (len >=CHACHA20_BLOCK_SIZE ) { +chacha20_block_generic (buf, (u32 *)state); +crypto_xor_cpy (out, in, (u8 *)buf,CHACHA20_BLOCK_SIZE ); + len -=CHACHA20_BLOCK_SIZE ; + out +=CHACHA20_BLOCK_SIZE ; + in +=CHACHA20_BLOCK_SIZE ; + } +if (len) { +chacha20_block_generic (buf, (u32 *)state); +crypto_xor_cpy (out, in, (u8 *)buf, len); + } +} + +void chacha20 (struct chacha20_ctx *state,u8 *dst,const u8 *src,u32 len, +simd_context_t *simd_context) +{ +if (!chacha20_arch (state, dst, src, len, simd_context)) +chacha20_generic (state, dst, src, len); +} +EXPORT_SYMBOL (chacha20); + +static void hchacha20_generic (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ]) +{ +u32 x[] = {CHACHA20_CONSTANT_EXPA , +CHACHA20_CONSTANT_ND_3 , +CHACHA20_CONSTANT_2_BY , +CHACHA20_CONSTANT_TE_K , +get_unaligned_le32 (key +0 ), +get_unaligned_le32 (key +4 ), +get_unaligned_le32 (key +8 ), +get_unaligned_le32 (key +12 ), +get_unaligned_le32 (key +16 ), +get_unaligned_le32 (key +20 ), +get_unaligned_le32 (key +24 ), +get_unaligned_le32 (key +28 ), +get_unaligned_le32 (nonce +0 ), +get_unaligned_le32 (nonce +4 ), +get_unaligned_le32 (nonce +8 ), +get_unaligned_le32 (nonce +12 ) + }; + +TWENTY_ROUNDS (x); + +memcpy (derived_key +0 , x +0 ,sizeof (u32) *4 ); +memcpy (derived_key +4 , x +12 ,sizeof (u32) *4 ); +} + +/* Derived key should be 32-bit aligned */ +void hchacha20 (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ],simd_context_t *simd_context) +{ +if (!hchacha20_arch (derived_key, nonce, key, simd_context)) +hchacha20_generic (derived_key, nonce, key); +} +EXPORT_SYMBOL (hchacha20); + +#include "../selftest/chacha20.h"
lib/zinc/chacha20/chacha20.c line 165
Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10
-0+ +static bool nosimd__initdata = false; + +static int __init mod_init (void ) +{ +if (!nosimd) +chacha20_fpu_init (); +#ifdef DEBUG +if (!chacha20_selftest ()) +return -ENOTRECOVERABLE ;
Return values from initcalls are ignored, and given that chacha20 will be depended upon by random.c, it will never be a module in practice.
Given your previous statement that selftest should *not* be a DEBUG feature (which I wholeheartedly agree with), you could be a bit noisier here imo.
E.g.,
if (WARN_ON(!chach20_selftest())
return ...
Quoted 2,699 lines
lib/zinc/chacha20/chacha20.c new file +14
-0+#endif +return 0 ; +} + +static void __exit mod_exit(void) +{ +} + +module_param (nosimd, bool,0 ); +module_init (mod_init); +module_exit (mod_exit); +MODULE_LICENSE ("GPL v2" ); +MODULE_DESCRIPTION ("ChaCha20 stream cipher" ); +MODULE_AUTHOR ("Jason A. Donenfeld <[email protected]>" );lib/zinc/selftest/chacha20.h new file +2676
-1@@ -0,0 +1,2676 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#ifdef DEBUG + +struct chacha20_testvec { +const u8 *input, *output, *key; +u64 nonce; +size_t ilen; +}; + +struct hchacha20_testvec { +u8 key[HCHACHA20_KEY_SIZE ]; +u8 nonce[HCHACHA20_NONCE_SIZE ]; +u8 output[CHACHA20_KEY_SIZE ]; +}; + +/* These test vectors are generated by reference implementations and are +* designed to check chacha20 implementation block handling, as well as from +* the draft-arciszewski-xchacha-01 document. +*/ + +static const u8 input01[]__initconst = { }; +static const u8 output01[]__initconst = { }; +static const u8 key01[]__initconst = { +0x09 ,0xf4 ,0xe8 ,0x57 ,0x10 ,0xf2 ,0x12 ,0xc3 , +0xc6 ,0x91 ,0xc4 ,0x09 ,0x97 ,0x46 ,0xef ,0xfe , +0x02 ,0x00 ,0xe4 ,0x5c ,0x82 ,0xed ,0x16 ,0xf3 , +0x32 ,0xbe ,0xec ,0x7a ,0xe6 ,0x68 ,0x12 ,0x26 +}; +enum { nonce01 =0x3834e2afca3c66d3ULL }; + +static const u8 input02[]__initconst = { +0x9d +}; +static const u8 output02[]__initconst = { +0x94 +}; +static const u8 key02[]__initconst = { +0x8c ,0x01 ,0xac ,0xaf ,0x62 ,0x63 ,0x56 ,0x7a , +0xad ,0x23 ,0x4c ,0x58 ,0x29 ,0x29 ,0xbe ,0xab , +0xe9 ,0xf8 ,0xdf ,0x6c ,0x8c ,0x74 ,0x4d ,0x7d , +0x13 ,0x94 ,0x10 ,0x02 ,0x3d ,0x8e ,0x9f ,0x94 +}; +enum { nonce02 =0x5d1b3bfdedd9f73aULL }; + +static const u8 input03[]__initconst = { +0x04 ,0x16 +}; +static const u8 output03[]__initconst = { +0x92 ,0x07 +}; +static const u8 key03[]__initconst = { +0x22 ,0x0c ,0x79 ,0x2c ,0x38 ,0x51 ,0xbe ,0x99 , +0xa9 ,0x59 ,0x24 ,0x50 ,0xef ,0x87 ,0x38 ,0xa6 , +0xa0 ,0x97 ,0x20 ,0xcb ,0xb4 ,0x0c ,0x94 ,0x67 , +0x1f ,0x98 ,0xdc ,0xc4 ,0x83 ,0xbc ,0x35 ,0x4d +}; +enum { nonce03 =0x7a3353ad720a3e2eULL }; + +static const u8 input04[]__initconst = { +0xc7 ,0xcc ,0xd0 +}; +static const u8 output04[]__initconst = { +0xd8 ,0x41 ,0x80 +}; +static const u8 key04[]__initconst = { +0x81 ,0x5e ,0x12 ,0x01 ,0xc4 ,0x36 ,0x15 ,0x03 , +0x11 ,0xa0 ,0xe9 ,0x86 ,0xbb ,0x5a ,0xdc ,0x45 , +0x7d ,0x5e ,0x98 ,0xf8 ,0x06 ,0x76 ,0x1c ,0xec , +0xc0 ,0xf7 ,0xca ,0x4e ,0x99 ,0xd9 ,0x42 ,0x38 +}; +enum { nonce04 =0x6816e2fc66176da2ULL }; + +static const u8 input05[]__initconst = { +0x48 ,0xf1 ,0x31 ,0x5f +}; +static const u8 output05[]__initconst = { +0x48 ,0xf7 ,0x13 ,0x67 +}; +static const u8 key05[]__initconst = { +0x3f ,0xd6 ,0xb6 ,0x5e ,0x2f ,0xda ,0x82 ,0x39 , +0x97 ,0x06 ,0xd3 ,0x62 ,0x4f ,0xbd ,0xcb ,0x9b , +0x1d ,0xe6 ,0x4a ,0x76 ,0xab ,0xdd ,0x14 ,0x50 , +0x59 ,0x21 ,0xe3 ,0xb2 ,0xc7 ,0x95 ,0xbc ,0x45 +}; +enum { nonce05 =0xc41a7490e228cc42ULL }; + +static const u8 input06[]__initconst = { +0xae ,0xa2 ,0x85 ,0x1d ,0xc8 +}; +static const u8 output06[]__initconst = { +0xfa ,0xff ,0x45 ,0x6b ,0x6f +}; +static const u8 key06[]__initconst = { +0x04 ,0x8d ,0xea ,0x67 ,0x20 ,0x78 ,0xfb ,0x8f , +0x49 ,0x80 ,0x35 ,0xb5 ,0x7b ,0xe4 ,0x31 ,0x74 , +0x57 ,0x43 ,0x3a ,0x64 ,0x64 ,0xb9 ,0xe6 ,0x23 , +0x4d ,0xfe ,0xb8 ,0x7b ,0x71 ,0x4d ,0x9d ,0x21 +}; +enum { nonce06 =0x251366db50b10903ULL }; + +static const u8 input07[]__initconst = { +0x1a ,0x32 ,0x85 ,0xb6 ,0xe8 ,0x52 +}; +static const u8 output07[]__initconst = { +0xd3 ,0x5f ,0xf0 ,0x07 ,0x69 ,0xec +}; +static const u8 key07[]__initconst = { +0xbf ,0x2d ,0x42 ,0x99 ,0x97 ,0x76 ,0x04 ,0xad , +0xd3 ,0x8f ,0x6e ,0x6a ,0x34 ,0x85 ,0xaf ,0x81 , +0xef ,0x36 ,0x33 ,0xd5 ,0x43 ,0xa2 ,0xaa ,0x08 , +0x0f ,0x77 ,0x42 ,0x83 ,0x58 ,0xc5 ,0x42 ,0x2a +}; +enum { nonce07 =0xe0796da17dba9b58ULL }; + +static const u8 input08[]__initconst = { +0x40 ,0xae ,0xcd ,0xe4 ,0x3d ,0x22 ,0xe0 +}; +static const u8 output08[]__initconst = { +0xfd ,0x8a ,0x9f ,0x3d ,0x05 ,0xc9 ,0xd3 +}; +static const u8 key08[]__initconst = { +0xdc ,0x3f ,0x41 ,0xe3 ,0x23 ,0x2a ,0x8d ,0xf6 , +0x41 ,0x2a ,0xa7 ,0x66 ,0x05 ,0x68 ,0xe4 ,0x7b , +0xc4 ,0x58 ,0xd6 ,0xcc ,0xdf ,0x0d ,0xc6 ,0x25 , +0x1b ,0x61 ,0x32 ,0x12 ,0x4e ,0xf1 ,0xe6 ,0x29 +}; +enum { nonce08 =0xb1d2536d9e159832ULL }; + +static const u8 input09[]__initconst = { +0xba ,0x1d ,0x14 ,0x16 ,0x9f ,0x83 ,0x67 ,0x24 +}; +static const u8 output09[]__initconst = { +0x7c ,0xe3 ,0x78 ,0x1d ,0xa2 ,0xe7 ,0xe9 ,0x39 +}; +static const u8 key09[]__initconst = { +0x17 ,0x55 ,0x90 ,0x52 ,0xa4 ,0xce ,0x12 ,0xae , +0xd4 ,0xfd ,0xd4 ,0xfb ,0xd5 ,0x18 ,0x59 ,0x50 , +0x4e ,0x51 ,0x99 ,0x32 ,0x09 ,0x31 ,0xfc ,0xf7 , +0x27 ,0x10 ,0x8e ,0xa2 ,0x4b ,0xa5 ,0xf5 ,0x62 +}; +enum { nonce09 =0x495fc269536d003ULL }; + +static const u8 input10[]__initconst = { +0x09 ,0xfd ,0x3c ,0x0b ,0x3d ,0x0e ,0xf3 ,0x9d , +0x27 +}; +static const u8 output10[]__initconst = { +0xdc ,0xe4 ,0x33 ,0x60 ,0x0c ,0x07 ,0xcb ,0x51 , +0x6b +}; +static const u8 key10[]__initconst = { +0x4e ,0x00 ,0x72 ,0x37 ,0x0f ,0x52 ,0x4d ,0x6f , +0x37 ,0x50 ,0x3c ,0xb3 ,0x51 ,0x81 ,0x49 ,0x16 , +0x7e ,0xfd ,0xb1 ,0x51 ,0x72 ,0x2e ,0xe4 ,0x16 , +0x68 ,0x5c ,0x5b ,0x8a ,0xc3 ,0x90 ,0x70 ,0x04 +}; +enum { nonce10 =0x1ad9d1114d88cbbdULL }; + +static const u8 input11[]__initconst = { +0x70 ,0x18 ,0x52 ,0x85 ,0xba ,0x66 ,0xff ,0x2c , +0x9a ,0x46 +}; +static const u8 output11[]__initconst = { +0xf5 ,0x2a ,0x7a ,0xfd ,0x31 ,0x7c ,0x91 ,0x41 , +0xb1 ,0xcf +}; +static const u8 key11[]__initconst = { +0x48 ,0xb4 ,0xd0 ,0x7c ,0x88 ,0xd1 ,0x96 ,0x0d , +0x80 ,0x33 ,0xb4 ,0xd5 ,0x31 ,0x9a ,0x88 ,0xca , +0x14 ,0xdc ,0xf0 ,0xa8 ,0xf3 ,0xac ,0xb8 ,0x47 , +0x75 ,0x86 ,0x7c ,0x88 ,0x50 ,0x11 ,0x43 ,0x40 +}; +enum { nonce11 =0x47c35dd1f4f8aa4fULL }; + +static const u8 input12[]__initconst = { +0x9e ,0x8e ,0x3d ,0x2a ,0x05 ,0xfd ,0xe4 ,0x90 , +0x24 ,0x1c ,0xd3 +}; +static const u8 output12[]__initconst = { +0x97 ,0x72 ,0x40 ,0x9f ,0xc0 ,0x6b ,0x05 ,0x33 , +0x42 ,0x7e ,0x28 +}; +static const u8 key12[]__initconst = { +0xee ,0xff ,0x33 ,0x33 ,0xe0 ,0x28 ,0xdf ,0xa2 , +0xb6 ,0x5e ,0x25 ,0x09 ,0x52 ,0xde ,0xa5 ,0x9c , +0x8f ,0x95 ,0xa9 ,0x03 ,0x77 ,0x0f ,0xbe ,0xa1 , +0xd0 ,0x7d ,0x73 ,0x2f ,0xf8 ,0x7e ,0x51 ,0x44 +}; +enum { nonce12 =0xc22d044dc6ea4af3ULL }; + +static const u8 input13[]__initconst = { +0x9c ,0x16 ,0xa2 ,0x22 ,0x4d ,0xbe ,0x04 ,0x9a , +0xb3 ,0xb5 ,0xc6 ,0x58 +}; +static const u8 output13[]__initconst = { +0xf0 ,0x81 ,0xdb ,0x6d ,0xa3 ,0xe9 ,0xb2 ,0xc6 , +0x32 ,0x50 ,0x16 ,0x9f +}; +static const u8 key13[]__initconst = { +0x96 ,0xb3 ,0x01 ,0xd2 ,0x7a ,0x8c ,0x94 ,0x09 , +0x4f ,0x58 ,0xbe ,0x80 ,0xcc ,0xa9 ,0x7e ,0x2d , +0xad ,0x58 ,0x3b ,0x63 ,0xb8 ,0x5c ,0x17 ,0xce , +0xbf ,0x43 ,0x33 ,0x7a ,0x7b ,0x82 ,0x28 ,0x2f +}; +enum { nonce13 =0x2a5d05d88cd7b0daULL }; + +static const u8 input14[]__initconst = { +0x57 ,0x4f ,0xaa ,0x30 ,0xe6 ,0x23 ,0x50 ,0x86 , +0x91 ,0xa5 ,0x60 ,0x96 ,0x2b +}; +static const u8 output14[]__initconst = { +0x6c ,0x1f ,0x3b ,0x42 ,0xb6 ,0x2f ,0xf0 ,0xbd , +0x76 ,0x60 ,0xc7 ,0x7e ,0x8d +}; +static const u8 key14[]__initconst = { +0x22 ,0x85 ,0xaf ,0x8f ,0xa3 ,0x53 ,0xa0 ,0xc4 , +0xb5 ,0x75 ,0xc0 ,0xba ,0x30 ,0x92 ,0xc3 ,0x32 , +0x20 ,0x5a ,0x8f ,0x7e ,0x93 ,0xda ,0x65 ,0x18 , +0xd1 ,0xf6 ,0x9a ,0x9b ,0x8f ,0x85 ,0x30 ,0xe6 +}; +enum { nonce14 =0xf9946c166aa4475fULL }; + +static const u8 input15[]__initconst = { +0x89 ,0x81 ,0xc7 ,0xe2 ,0x00 ,0xac ,0x52 ,0x70 , +0xa4 ,0x79 ,0xab ,0xeb ,0x74 ,0xf7 +}; +static const u8 output15[]__initconst = { +0xb4 ,0xd0 ,0xa9 ,0x9d ,0x15 ,0x5f ,0x48 ,0xd6 , +0x00 ,0x7e ,0x4c ,0x77 ,0x5a ,0x46 +}; +static const u8 key15[]__initconst = { +0x0a ,0x66 ,0x36 ,0xca ,0x5d ,0x82 ,0x23 ,0xb6 , +0xe4 ,0x9b ,0xad ,0x5e ,0xd0 ,0x7f ,0xf6 ,0x7a , +0x7b ,0x03 ,0xa7 ,0x4c ,0xfd ,0xec ,0xd5 ,0xa1 , +0xfc ,0x25 ,0x54 ,0xda ,0x5a ,0x5c ,0xf0 ,0x2c +}; +enum { nonce15 =0x9ab2b87a35e772c8ULL }; + +static const u8 input16[]__initconst = { +0x5f ,0x09 ,0xc0 ,0x8b ,0x1e ,0xde ,0xca ,0xd9 , +0xb7 ,0x5c ,0x23 ,0xc9 ,0x55 ,0x1e ,0xcf +}; +static const u8 output16[]__initconst = { +0x76 ,0x9b ,0x53 ,0xf3 ,0x66 ,0x88 ,0x28 ,0x60 , +0x98 ,0x80 ,0x2c ,0xa8 ,0x80 ,0xa6 ,0x48 +}; +static const u8 key16[]__initconst = { +0x80 ,0xb5 ,0x51 ,0xdf ,0x17 ,0x5b ,0xb0 ,0xef , +0x8b ,0x5b ,0x2e ,0x3e ,0xc5 ,0xe3 ,0xa5 ,0x86 , +0xac ,0x0d ,0x8e ,0x32 ,0x90 ,0x9d ,0x82 ,0x27 , +0xf1 ,0x23 ,0x26 ,0xc3 ,0xea ,0x55 ,0xb6 ,0x63 +}; +enum { nonce16 =0xa82e9d39e4d02ef5ULL }; + +static const u8 input17[]__initconst = { +0x87 ,0x0b ,0x36 ,0x71 ,0x7c ,0xb9 ,0x0b ,0x80 , +0x4d ,0x77 ,0x5c ,0x4f ,0xf5 ,0x51 ,0x0e ,0x1a +}; +static const u8 output17[]__initconst = { +0xf1 ,0x12 ,0x4a ,0x8a ,0xd9 ,0xd0 ,0x08 ,0x67 , +0x66 ,0xd7 ,0x34 ,0xea ,0x32 ,0x3b ,0x54 ,0x0e +}; +static const u8 key17[]__initconst = { +0xfb ,0x71 ,0x5f ,0x3f ,0x7a ,0xc0 ,0x9a ,0xc8 , +0xc8 ,0xcf ,0xe8 ,0xbc ,0xfb ,0x09 ,0xbf ,0x89 , +0x6a ,0xef ,0xd5 ,0xe5 ,0x36 ,0x87 ,0x14 ,0x76 , +0x00 ,0xb9 ,0x32 ,0x28 ,0xb2 ,0x00 ,0x42 ,0x53 +}; +enum { nonce17 =0x229b87e73d557b96ULL }; + +static const u8 input18[]__initconst = { +0x38 ,0x42 ,0xb5 ,0x37 ,0xb4 ,0x3d ,0xfe ,0x59 , +0x38 ,0x68 ,0x88 ,0xfa ,0x89 ,0x8a ,0x5f ,0x90 , +0x3c +}; +static const u8 output18[]__initconst = { +0xac ,0xad ,0x14 ,0xe8 ,0x7e ,0xd7 ,0xce ,0x96 , +0x3d ,0xb3 ,0x78 ,0x85 ,0x22 ,0x5a ,0xcb ,0x39 , +0xd4 +}; +static const u8 key18[]__initconst = { +0xe1 ,0xc1 ,0xa8 ,0xe0 ,0x91 ,0xe7 ,0x38 ,0x66 , +0x80 ,0x17 ,0x12 ,0x3c ,0x5e ,0x2d ,0xbb ,0xea , +0xeb ,0x6c ,0x8b ,0xc8 ,0x1b ,0x6f ,0x7c ,0xea , +0x50 ,0x57 ,0x23 ,0x1e ,0x65 ,0x6f ,0x6d ,0x81 +}; +enum { nonce18 =0xfaf5fcf8f30e57a9ULL }; + +static const u8 input19[]__initconst = { +0x1c ,0x4a ,0x30 ,0x26 ,0xef ,0x9a ,0x32 ,0xa7 , +0x8f ,0xe5 ,0xc0 ,0x0f ,0x30 ,0x3a ,0xbf ,0x38 , +0x54 ,0xba +}; +static const u8 output19[]__initconst = { +0x57 ,0x67 ,0x54 ,0x4f ,0x31 ,0xd6 ,0xef ,0x35 , +0x0b ,0xd9 ,0x52 ,0xa7 ,0x46 ,0x7d ,0x12 ,0x17 , +0x1e ,0xe3 +}; +static const u8 key19[]__initconst = { +0x5a ,0x79 ,0xc1 ,0xea ,0x33 ,0xb3 ,0xc7 ,0x21 , +0xec ,0xf8 ,0xcb ,0xd2 ,0x58 ,0x96 ,0x23 ,0xd6 , +0x4d ,0xed ,0x2f ,0xdf ,0x8a ,0x79 ,0xe6 ,0x8b , +0x38 ,0xa3 ,0xc3 ,0x7a ,0x33 ,0xda ,0x02 ,0xc7 +}; +enum { nonce19 =0x2b23b61840429604ULL }; + +static const u8 input20[]__initconst = { +0xab ,0xe9 ,0x32 ,0xbb ,0x35 ,0x17 ,0xe0 ,0x60 , +0x80 ,0xb1 ,0x27 ,0xdc ,0xe6 ,0x62 ,0x9e ,0x0c , +0x77 ,0xf4 ,0x50 +}; +static const u8 output20[]__initconst = { +0x54 ,0x6d ,0xaa ,0xfc ,0x08 ,0xfb ,0x71 ,0xa8 , +0xd6 ,0x1d ,0x7d ,0xf3 ,0x45 ,0x10 ,0xb5 ,0x4c , +0xcc ,0x4b ,0x45 +}; +static const u8 key20[]__initconst = { +0xa3 ,0xfd ,0x3d ,0xa9 ,0xeb ,0xea ,0x2c ,0x69 , +0xcf ,0x59 ,0x38 ,0x13 ,0x5b ,0xa7 ,0x53 ,0x8f , +0x5e ,0xa2 ,0x33 ,0x86 ,0x4c ,0x75 ,0x26 ,0xaf , +0x35 ,0x12 ,0x09 ,0x71 ,0x81 ,0xea ,0x88 ,0x66 +}; +enum { nonce20 =0x7459667a8fadff58ULL }; + +static const u8 input21[]__initconst = { +0xa6 ,0x82 ,0x21 ,0x23 ,0xad ,0x27 ,0x3f ,0xc6 , +0xd7 ,0x16 ,0x0d ,0x6d ,0x24 ,0x15 ,0x54 ,0xc5 , +0x96 ,0x72 ,0x59 ,0x8a +}; +static const u8 output21[]__initconst = { +0x5f ,0x34 ,0x32 ,0xea ,0x06 ,0xd4 ,0x9e ,0x01 , +0xdc ,0x32 ,0x32 ,0x40 ,0x66 ,0x73 ,0x6d ,0x4a , +0x6b ,0x12 ,0x20 ,0xe8 +}; +static const u8 key21[]__initconst = { +0x96 ,0xfd ,0x13 ,0x23 ,0xa9 ,0x89 ,0x04 ,0xe6 , +0x31 ,0xa5 ,0x2c ,0xc1 ,0x40 ,0xd5 ,0x69 ,0x5c , +0x32 ,0x79 ,0x56 ,0xe0 ,0x29 ,0x93 ,0x8f ,0xe8 , +0x5f ,0x65 ,0x53 ,0x7f ,0xc1 ,0xe9 ,0xaf ,0xaf +}; +enum { nonce21 =0xba8defee9d8e13b5ULL }; + +static const u8 input22[]__initconst = { +0xb8 ,0x32 ,0x1a ,0x81 ,0xd8 ,0x38 ,0x89 ,0x5a , +0xb0 ,0x05 ,0xbe ,0xf4 ,0xd2 ,0x08 ,0xc6 ,0xee , +0x79 ,0x7b ,0x3a ,0x76 ,0x59 +}; +static const u8 output22[]__initconst = { +0xb7 ,0xba ,0xae ,0x80 ,0xe4 ,0x9f ,0x79 ,0x84 , +0x5a ,0x48 ,0x50 ,0x6d ,0xcb ,0xd0 ,0x06 ,0x0c , +0x15 ,0x63 ,0xa7 ,0x5e ,0xbd +}; +static const u8 key22[]__initconst = { +0x0f ,0x35 ,0x3d ,0xeb ,0x5f ,0x0a ,0x82 ,0x0d , +0x24 ,0x59 ,0x71 ,0xd8 ,0xe6 ,0x2d ,0x5f ,0xe1 , +0x7e ,0x0c ,0xae ,0xf6 ,0xdc ,0x2c ,0xc5 ,0x4a , +0x38 ,0x88 ,0xf2 ,0xde ,0xd9 ,0x5f ,0x76 ,0x7c +}; +enum { nonce22 =0xe77f1760e9f5e192ULL }; + +static const u8 input23[]__initconst = { +0x4b ,0x1e ,0x79 ,0x99 ,0xcf ,0xef ,0x64 ,0x4b , +0xb0 ,0x66 ,0xae ,0x99 ,0x2e ,0x68 ,0x97 ,0xf5 , +0x5d ,0x9b ,0x3f ,0x7a ,0xa9 ,0xd9 +}; +static const u8 output23[]__initconst = { +0x5f ,0xa4 ,0x08 ,0x39 ,0xca ,0xfa ,0x2b ,0x83 , +0x5d ,0x95 ,0x70 ,0x7c ,0x2e ,0xd4 ,0xae ,0xfa , +0x45 ,0x4a ,0x77 ,0x7f ,0xa7 ,0x65 +}; +static const u8 key23[]__initconst = { +0x4a ,0x06 ,0x83 ,0x64 ,0xaa ,0xe3 ,0x38 ,0x32 , +0x28 ,0x5d ,0xa4 ,0xb2 ,0x5a ,0xee ,0xcf ,0x8e , +0x19 ,0x67 ,0xf1 ,0x09 ,0xe8 ,0xc9 ,0xf6 ,0x40 , +0x02 ,0x6d ,0x0b ,0xde ,0xfa ,0x81 ,0x03 ,0xb1 +}; +enum { nonce23 =0x9b3f349158709849ULL }; + +static const u8 input24[]__initconst = { +0xc6 ,0xfc ,0x47 ,0x5e ,0xd8 ,0xed ,0xa9 ,0xe5 , +0x4f ,0x82 ,0x79 ,0x35 ,0xee ,0x3e ,0x7e ,0x3e , +0x35 ,0x70 ,0x6e ,0xfa ,0x6d ,0x08 ,0xe8 +}; +static const u8 output24[]__initconst = { +0x3b ,0xc5 ,0xf8 ,0xc2 ,0xbf ,0x2b ,0x90 ,0x33 , +0xa6 ,0xae ,0xf5 ,0x5a ,0x65 ,0xb3 ,0x3d ,0xe1 , +0xcd ,0x5f ,0x55 ,0xfa ,0xe7 ,0xa5 ,0x4a +}; +static const u8 key24[]__initconst = { +0x00 ,0x24 ,0xc3 ,0x65 ,0x5f ,0xe6 ,0x31 ,0xbb , +0x6d ,0xfc ,0x20 ,0x7b ,0x1b ,0xa8 ,0x96 ,0x26 , +0x55 ,0x21 ,0x62 ,0x25 ,0x7e ,0xba ,0x23 ,0x97 , +0xc9 ,0xb8 ,0x53 ,0xa8 ,0xef ,0xab ,0xad ,0x61 +}; +enum { nonce24 =0x13ee0b8f526177c3ULL }; + +static const u8 input25[]__initconst = { +0x33 ,0x07 ,0x16 ,0xb1 ,0x34 ,0x33 ,0x67 ,0x04 , +0x9b ,0x0a ,0xce ,0x1b ,0xe9 ,0xde ,0x1a ,0xec , +0xd0 ,0x55 ,0xfb ,0xc6 ,0x33 ,0xaf ,0x2d ,0xe3 +}; +static const u8 output25[]__initconst = { +0x05 ,0x93 ,0x10 ,0xd1 ,0x58 ,0x6f ,0x68 ,0x62 , +0x45 ,0xdb ,0x91 ,0xae ,0x70 ,0xcf ,0xd4 ,0x5f , +0xee ,0xdf ,0xd5 ,0xba ,0x9e ,0xde ,0x68 ,0xe6 +}; +static const u8 key25[]__initconst = { +0x83 ,0xa9 ,0x4f ,0x5d ,0x74 ,0xd5 ,0x91 ,0xb3 , +0xc9 ,0x97 ,0x19 ,0x15 ,0xdb ,0x0d ,0x0b ,0x4a , +0x3d ,0x55 ,0xcf ,0xab ,0xb2 ,0x05 ,0x21 ,0x35 , +0x45 ,0x50 ,0xeb ,0xf8 ,0xf5 ,0xbf ,0x36 ,0x35 +}; +enum { nonce25 =0x7c6f459e49ebfebcULL }; + +static const u8 input26[]__initconst = { +0xc2 ,0xd4 ,0x7a ,0xa3 ,0x92 ,0xe1 ,0xac ,0x46 , +0x1a ,0x15 ,0x38 ,0xc9 ,0xb5 ,0xfd ,0xdf ,0x84 , +0x38 ,0xbc ,0x6b ,0x1d ,0xb0 ,0x83 ,0x43 ,0x04 , +0x39 +}; +static const u8 output26[]__initconst = { +0x7f ,0xde ,0xd6 ,0x87 ,0xcc ,0x34 ,0xf4 ,0x12 , +0xae ,0x55 ,0xa5 ,0x89 ,0x95 ,0x29 ,0xfc ,0x18 , +0xd8 ,0xc7 ,0x7c ,0xd3 ,0xcb ,0x85 ,0x95 ,0x21 , +0xd2 +}; +static const u8 key26[]__initconst = { +0xe4 ,0xd0 ,0x54 ,0x1d ,0x7d ,0x47 ,0xa8 ,0xc1 , +0x08 ,0xca ,0xe2 ,0x42 ,0x52 ,0x95 ,0x16 ,0x43 , +0xa3 ,0x01 ,0x23 ,0x03 ,0xcc ,0x3b ,0x81 ,0x78 , +0x23 ,0xcc ,0xa7 ,0x36 ,0xd7 ,0xa0 ,0x97 ,0x8d +}; +enum { nonce26 =0x524401012231683ULL }; + +static const u8 input27[]__initconst = { +0x0d ,0xb0 ,0xcf ,0xec ,0xfc ,0x38 ,0x9d ,0x9d , +0x89 ,0x00 ,0x96 ,0xf2 ,0x79 ,0x8a ,0xa1 ,0x8d , +0x32 ,0x5e ,0xc6 ,0x12 ,0x22 ,0xec ,0xf6 ,0x52 , +0xc1 ,0x0b +}; +static const u8 output27[]__initconst = { +0xef ,0xe1 ,0xf2 ,0x67 ,0x8e ,0x2c ,0x00 ,0x9f , +0x1d ,0x4c ,0x66 ,0x1f ,0x94 ,0x58 ,0xdc ,0xbb , +0xb9 ,0x11 ,0x8f ,0x74 ,0xfd ,0x0e ,0x14 ,0x01 , +0xa8 ,0x21 +}; +static const u8 key27[]__initconst = { +0x78 ,0x71 ,0xa4 ,0xe6 ,0xb2 ,0x95 ,0x44 ,0x12 , +0x81 ,0xaa ,0x7e ,0x94 ,0xa7 ,0x8d ,0x44 ,0xea , +0xc4 ,0xbc ,0x01 ,0xb7 ,0x9e ,0xf7 ,0x82 ,0x9e , +0x3b ,0x23 ,0x9f ,0x31 ,0xdd ,0xb8 ,0x0d ,0x18 +}; +enum { nonce27 =0xd58fe0e58fb254d6ULL }; + +static const u8 input28[]__initconst = { +0xaa ,0xb7 ,0xaa ,0xd9 ,0xa8 ,0x91 ,0xd7 ,0x8a , +0x97 ,0x9b ,0xdb ,0x7c ,0x47 ,0x2b ,0xdb ,0xd2 , +0xda ,0x77 ,0xb1 ,0xfa ,0x2d ,0x12 ,0xe3 ,0xe9 , +0xc4 ,0x7f ,0x54 +}; +static const u8 output28[]__initconst = { +0x87 ,0x84 ,0xa9 ,0xa6 ,0xad ,0x8f ,0xe6 ,0x0f , +0x69 ,0xf8 ,0x21 ,0xc3 ,0x54 ,0x95 ,0x0f ,0xb0 , +0x4e ,0xc7 ,0x02 ,0xe4 ,0x04 ,0xb0 ,0x6c ,0x42 , +0x8c ,0x63 ,0xe3 +}; +static const u8 key28[]__initconst = { +0x12 ,0x23 ,0x37 ,0x95 ,0x04 ,0xb4 ,0x21 ,0xe8 , +0xbc ,0x65 ,0x46 ,0x7a ,0xf4 ,0x01 ,0x05 ,0x3f , +0xb1 ,0x34 ,0x73 ,0xd2 ,0x49 ,0xbf ,0x6f ,0x20 , +0xbd ,0x23 ,0x58 ,0x5f ,0xd1 ,0x73 ,0x57 ,0xa6 +}; +enum { nonce28 =0x3a04d51491eb4e07ULL }; + +static const u8 input29[]__initconst = { +0x55 ,0xd0 ,0xd4 ,0x4b ,0x17 ,0xc8 ,0xc4 ,0x2b , +0xc0 ,0x28 ,0xbd ,0x9d ,0x65 ,0x4d ,0xaf ,0x77 , +0x72 ,0x7c ,0x36 ,0x68 ,0xa7 ,0xb6 ,0x87 ,0x4d , +0xb9 ,0x27 ,0x25 ,0x6c +}; +static const u8 output29[]__initconst = { +0x0e ,0xac ,0x4c ,0xf5 ,0x12 ,0xb5 ,0x56 ,0xa5 , +0x00 ,0x9a ,0xd6 ,0xe5 ,0x1a ,0x59 ,0x2c ,0xf6 , +0x42 ,0x22 ,0xcf ,0x23 ,0x98 ,0x34 ,0x29 ,0xac , +0x6e ,0xe3 ,0x37 ,0x6d +}; +static const u8 key29[]__initconst = { +0xda ,0x9d ,0x05 ,0x0c ,0x0c ,0xba ,0x75 ,0xb9 , +0x9e ,0xb1 ,0x8d ,0xd9 ,0x73 ,0x26 ,0x2c ,0xa9 , +0x3a ,0xb5 ,0xcb ,0x19 ,0x49 ,0xa7 ,0x4f ,0xf7 , +0x64 ,0x35 ,0x23 ,0x20 ,0x2a ,0x45 ,0x78 ,0xc7 +}; +enum { nonce29 =0xc25ac9982431cbfULL }; + +static const u8 input30[]__initconst = { +0x4e ,0xd6 ,0x85 ,0xbb ,0xe7 ,0x99 ,0xfa ,0x04 , +0x33 ,0x24 ,0xfd ,0x75 ,0x18 ,0xe3 ,0xd3 ,0x25 , +0xcd ,0xca ,0xae ,0x00 ,0xbe ,0x52 ,0x56 ,0x4a , +0x31 ,0xe9 ,0x4f ,0xae ,0x8a +}; +static const u8 output30[]__initconst = { +0x30 ,0x36 ,0x32 ,0xa2 ,0x3c ,0xb6 ,0xf9 ,0xf9 , +0x76 ,0x70 ,0xad ,0xa6 ,0x10 ,0x41 ,0x00 ,0x4a , +0xfa ,0xce ,0x1b ,0x86 ,0x05 ,0xdb ,0x77 ,0x96 , +0xb3 ,0xb7 ,0x8f ,0x61 ,0x24 +}; +static const u8 key30[]__initconst = { +0x49 ,0x35 ,0x4c ,0x15 ,0x98 ,0xfb ,0xc6 ,0x57 , +0x62 ,0x6d ,0x06 ,0xc3 ,0xd4 ,0x79 ,0x20 ,0x96 , +0x05 ,0x2a ,0x31 ,0x63 ,0xc0 ,0x44 ,0x42 ,0x09 , +0x13 ,0x13 ,0xff ,0x1b ,0xc8 ,0x63 ,0x1f ,0x0b +}; +enum { nonce30 =0x4967f9c08e41568bULL }; + +static const u8 input31[]__initconst = { +0x91 ,0x04 ,0x20 ,0x47 ,0x59 ,0xee ,0xa6 ,0x0f , +0x04 ,0x75 ,0xc8 ,0x18 ,0x95 ,0x44 ,0x01 ,0x28 , +0x20 ,0x6f ,0x73 ,0x68 ,0x66 ,0xb5 ,0x03 ,0xb3 , +0x58 ,0x27 ,0x6e ,0x7a ,0x76 ,0xb8 +}; +static const u8 output31[]__initconst = { +0xe8 ,0x03 ,0x78 ,0x9d ,0x13 ,0x15 ,0x98 ,0xef , +0x64 ,0x68 ,0x12 ,0x41 ,0xb0 ,0x29 ,0x94 ,0x0c , +0x83 ,0x35 ,0x46 ,0xa9 ,0x74 ,0xe1 ,0x75 ,0xf0 , +0xb6 ,0x96 ,0xc3 ,0x6f ,0xd7 ,0x70 +}; +static const u8 key31[]__initconst = { +0xef ,0xcd ,0x5a ,0x4a ,0xf4 ,0x7e ,0x6a ,0x3a , +0x11 ,0x88 ,0x72 ,0x94 ,0xb8 ,0xae ,0x84 ,0xc3 , +0x66 ,0xe0 ,0xde ,0x4b ,0x00 ,0xa5 ,0xd6 ,0x2d , +0x50 ,0xb7 ,0x28 ,0xff ,0x76 ,0x57 ,0x18 ,0x1f +}; +enum { nonce31 =0xcb6f428fa4192e19ULL }; + +static const u8 input32[]__initconst = { +0x90 ,0x06 ,0x50 ,0x4b ,0x98 ,0x14 ,0x30 ,0xf1 , +0xb8 ,0xd7 ,0xf0 ,0xa4 ,0x3e ,0x4e ,0xd8 ,0x00 , +0xea ,0xdb ,0x4f ,0x93 ,0x05 ,0xef ,0x02 ,0x71 , +0x1a ,0xcd ,0xa3 ,0xb1 ,0xae ,0xd3 ,0x18 +}; +static const u8 output32[]__initconst = { +0xcb ,0x4a ,0x37 ,0x3f ,0xea ,0x40 ,0xab ,0x86 , +0xfe ,0xcc ,0x07 ,0xd5 ,0xdc ,0xb2 ,0x25 ,0xb6 , +0xfd ,0x2a ,0x72 ,0xbc ,0x5e ,0xd4 ,0x75 ,0xff , +0x71 ,0xfc ,0xce ,0x1e ,0x6f ,0x22 ,0xc1 +}; +static const u8 key32[]__initconst = { +0xfc ,0x6d ,0xc3 ,0x80 ,0xce ,0xa4 ,0x31 ,0xa1 , +0xcc ,0xfa ,0x9d ,0x10 ,0x0b ,0xc9 ,0x11 ,0x77 , +0x34 ,0xdb ,0xad ,0x1b ,0xc4 ,0xfc ,0xeb ,0x79 , +0x91 ,0xda ,0x59 ,0x3b ,0x0d ,0xb1 ,0x19 ,0x3b +}; +enum { nonce32 =0x88551bf050059467ULL }; + +static const u8 input33[]__initconst = { +0x88 ,0x94 ,0x71 ,0x92 ,0xe8 ,0xd7 ,0xf9 ,0xbd , +0x55 ,0xe3 ,0x22 ,0xdb ,0x99 ,0x51 ,0xfb ,0x50 , +0xbf ,0x82 ,0xb5 ,0x70 ,0x8b ,0x2b ,0x6a ,0x03 , +0x37 ,0xa0 ,0xc6 ,0x19 ,0x5d ,0xc9 ,0xbc ,0xcc +}; +static const u8 output33[]__initconst = { +0xb6 ,0x17 ,0x51 ,0xc8 ,0xea ,0x8a ,0x14 ,0xdc , +0x23 ,0x1b ,0xd4 ,0xed ,0xbf ,0x50 ,0xb9 ,0x38 , +0x00 ,0xc2 ,0x3f ,0x78 ,0x3d ,0xbf ,0xa0 ,0x84 , +0xef ,0x45 ,0xb2 ,0x7d ,0x48 ,0x7b ,0x62 ,0xa7 +}; +static const u8 key33[]__initconst = { +0xb9 ,0x8f ,0x6a ,0xad ,0xb4 ,0x6f ,0xb5 ,0xdc , +0x48 ,0xfa ,0x43 ,0x57 ,0x62 ,0x97 ,0xef ,0x89 , +0x4c ,0x5a ,0x7b ,0x67 ,0xb8 ,0x9d ,0xf0 ,0x42 , +0x2b ,0x8f ,0xf3 ,0x18 ,0x05 ,0x2e ,0x48 ,0xd0 +}; +enum { nonce33 =0x31f16488fe8447f5ULL }; + +static const u8 input34[]__initconst = { +0xda ,0x2b ,0x3d ,0x63 ,0x9e ,0x4f ,0xc2 ,0xb8 , +0x7f ,0xc2 ,0x1a ,0x8b ,0x0d ,0x95 ,0x65 ,0x55 , +0x52 ,0xba ,0x51 ,0x51 ,0xc0 ,0x61 ,0x9f ,0x0a , +0x5d ,0xb0 ,0x59 ,0x8c ,0x64 ,0x6a ,0xab ,0xf5 , +0x57 +}; +static const u8 output34[]__initconst = { +0x5c ,0xf6 ,0x62 ,0x24 ,0x8c ,0x45 ,0xa3 ,0x26 , +0xd0 ,0xe4 ,0x88 ,0x1c ,0xed ,0xc4 ,0x26 ,0x58 , +0xb5 ,0x5d ,0x92 ,0xc4 ,0x17 ,0x44 ,0x1c ,0xb8 , +0x2c ,0xf3 ,0x55 ,0x7e ,0xd6 ,0xe5 ,0xb3 ,0x65 , +0xa8 +}; +static const u8 key34[]__initconst = { +0xde ,0xd1 ,0x27 ,0xb7 ,0x7c ,0xfa ,0xa6 ,0x78 , +0x39 ,0x80 ,0xdf ,0xb7 ,0x46 ,0xac ,0x71 ,0x26 , +0xd0 ,0x2a ,0x56 ,0x79 ,0x12 ,0xeb ,0x26 ,0x37 , +0x01 ,0x0d ,0x30 ,0xe0 ,0xe3 ,0x66 ,0xb2 ,0xf4 +}; +enum { nonce34 =0x92d0d9b252c24149ULL }; + +static const u8 input35[]__initconst = { +0x3a ,0x15 ,0x5b ,0x75 ,0x6e ,0xd0 ,0x52 ,0x20 , +0x6c ,0x82 ,0xfa ,0xce ,0x5b ,0xea ,0xf5 ,0x43 , +0xc1 ,0x81 ,0x7c ,0xb2 ,0xac ,0x16 ,0x3f ,0xd3 , +0x5a ,0xaf ,0x55 ,0x98 ,0xf4 ,0xc6 ,0xba ,0x71 , +0x25 ,0x8b +}; +static const u8 output35[]__initconst = { +0xb3 ,0xaf ,0xac ,0x6d ,0x4d ,0xc7 ,0x68 ,0x56 , +0x50 ,0x5b ,0x69 ,0x2a ,0xe5 ,0x90 ,0xf9 ,0x5f , +0x99 ,0x88 ,0xff ,0x0c ,0xa6 ,0xb1 ,0x83 ,0xd6 , +0x80 ,0xa6 ,0x1b ,0xde ,0x94 ,0xa4 ,0x2c ,0xc3 , +0x74 ,0xfa +}; +static const u8 key35[]__initconst = { +0xd8 ,0x24 ,0xe2 ,0x06 ,0xd7 ,0x7a ,0xce ,0x81 , +0x52 ,0x72 ,0x02 ,0x69 ,0x89 ,0xc4 ,0xe9 ,0x53 , +0x3b ,0x08 ,0x5f ,0x98 ,0x1e ,0x1b ,0x99 ,0x6e , +0x28 ,0x17 ,0x6d ,0xba ,0xc0 ,0x96 ,0xf9 ,0x3c +}; +enum { nonce35 =0x7baf968c4c8e3a37ULL }; + +static const u8 input36[]__initconst = { +0x31 ,0x5d ,0x4f ,0xe3 ,0xac ,0xad ,0x17 ,0xa6 , +0xb5 ,0x01 ,0xe2 ,0xc6 ,0xd4 ,0x7e ,0xc4 ,0x80 , +0xc0 ,0x59 ,0x72 ,0xbb ,0x4b ,0x74 ,0x6a ,0x41 , +0x0f ,0x9c ,0xf6 ,0xca ,0x20 ,0xb3 ,0x73 ,0x07 , +0x6b ,0x02 ,0x2a +}; +static const u8 output36[]__initconst = { +0xf9 ,0x09 ,0x92 ,0x94 ,0x7e ,0x31 ,0xf7 ,0x53 , +0xe8 ,0x8a ,0x5b ,0x20 ,0xef ,0x9b ,0x45 ,0x81 , +0xba ,0x5e ,0x45 ,0x63 ,0xc1 ,0xc7 ,0x9e ,0x06 , +0x0e ,0xd9 ,0x62 ,0x8e ,0x96 ,0xf9 ,0xfa ,0x43 , +0x4d ,0xd4 ,0x28 +}; +static const u8 key36[]__initconst = { +0x13 ,0x30 ,0x4c ,0x06 ,0xae ,0x18 ,0xde ,0x03 , +0x1d ,0x02 ,0x40 ,0xf5 ,0xbb ,0x19 ,0xe3 ,0x88 , +0x41 ,0xb1 ,0x29 ,0x15 ,0x97 ,0xc2 ,0x69 ,0x3f , +0x32 ,0x2a ,0x0c ,0x8b ,0xcf ,0x83 ,0x8b ,0x6c +}; +enum { nonce36 =0x226d251d475075a0ULL }; + +static const u8 input37[]__initconst = { +0x10 ,0x18 ,0xbe ,0xfd ,0x66 ,0xc9 ,0x77 ,0xcc , +0x43 ,0xe5 ,0x46 ,0x0b ,0x08 ,0x8b ,0xae ,0x11 , +0x86 ,0x15 ,0xc2 ,0xf6 ,0x45 ,0xd4 ,0x5f ,0xd6 , +0xb6 ,0x5f ,0x9f ,0x3e ,0x97 ,0xb7 ,0xd4 ,0xad , +0x0b ,0xe8 ,0x31 ,0x94 +}; +static const u8 output37[]__initconst = { +0x03 ,0x2c ,0x1c ,0xee ,0xc6 ,0xdd ,0xed ,0x38 , +0x80 ,0x6d ,0x84 ,0x16 ,0xc3 ,0xc2 ,0x04 ,0x63 , +0xcd ,0xa7 ,0x6e ,0x36 ,0x8b ,0xed ,0x78 ,0x63 , +0x95 ,0xfc ,0x69 ,0x7a ,0x3f ,0x8d ,0x75 ,0x6b , +0x6c ,0x26 ,0x56 ,0x4d +}; +static const u8 key37[]__initconst = { +0xac ,0x84 ,0x4d ,0xa9 ,0x29 ,0x49 ,0x3c ,0x39 , +0x7f ,0xd9 ,0xa6 ,0x01 ,0xf3 ,0x7e ,0xfa ,0x4a , +0x14 ,0x80 ,0x22 ,0x74 ,0xf0 ,0x29 ,0x30 ,0x2d , +0x07 ,0x21 ,0xda ,0xc0 ,0x4d ,0x70 ,0x56 ,0xa2 +}; +enum { nonce37 =0x167823ce3b64925aULL }; + +static const u8 input38[]__initconst = { +0x30 ,0x8f ,0xfa ,0x24 ,0x29 ,0xb1 ,0xfb ,0xce , +0x31 ,0x62 ,0xdc ,0xd0 ,0x46 ,0xab ,0xe1 ,0x31 , +0xd9 ,0xae ,0x60 ,0x0d ,0xca ,0x0a ,0x49 ,0x12 , +0x3d ,0x92 ,0xe9 ,0x91 ,0x67 ,0x12 ,0x62 ,0x18 , +0x89 ,0xe2 ,0xf9 ,0x1c ,0xcc +}; +static const u8 output38[]__initconst = { +0x56 ,0x9c ,0xc8 ,0x7a ,0xc5 ,0x98 ,0xa3 ,0x0f , +0xba ,0xd5 ,0x3e ,0xe1 ,0xc9 ,0x33 ,0x64 ,0x33 , +0xf0 ,0xd5 ,0xf7 ,0x43 ,0x66 ,0x0e ,0x08 ,0x9a , +0x6e ,0x09 ,0xe4 ,0x01 ,0x0d ,0x1e ,0x2f ,0x4b , +0xed ,0x9c ,0x08 ,0x8c ,0x03 +}; +static const u8 key38[]__initconst = { +0x77 ,0x52 ,0x2a ,0x23 ,0xf1 ,0xc5 ,0x96 ,0x2b , +0x89 ,0x4f ,0x3e ,0xf3 ,0xff ,0x0e ,0x94 ,0xce , +0xf1 ,0xbd ,0x53 ,0xf5 ,0x77 ,0xd6 ,0x9e ,0x47 , +0x49 ,0x3d ,0x16 ,0x64 ,0xff ,0x95 ,0x42 ,0x42 +}; +enum { nonce38 =0xff629d7b82cef357ULL }; + +static const u8 input39[]__initconst = { +0x38 ,0x26 ,0x27 ,0xd0 ,0xc2 ,0xf5 ,0x34 ,0xba , +0xda ,0x0f ,0x1c ,0x1c ,0x9a ,0x70 ,0xe5 ,0x8a , +0x78 ,0x2d ,0x8f ,0x9a ,0xbf ,0x89 ,0x6a ,0xfd , +0xd4 ,0x9c ,0x33 ,0xf1 ,0xb6 ,0x89 ,0x16 ,0xe3 , +0x6a ,0x00 ,0xfa ,0x3a ,0x0f ,0x26 +}; +static const u8 output39[]__initconst = { +0x0f ,0xaf ,0x91 ,0x6d ,0x9c ,0x99 ,0xa4 ,0xf7 , +0x3b ,0x9d ,0x9a ,0x98 ,0xca ,0xbb ,0x50 ,0x48 , +0xee ,0xcb ,0x5d ,0xa1 ,0x37 ,0x2d ,0x36 ,0x09 , +0x2a ,0xe2 ,0x1c ,0x3d ,0x98 ,0x40 ,0x1c ,0x16 , +0x56 ,0xa7 ,0x98 ,0xe9 ,0x7d ,0x2b +}; +static const u8 key39[]__initconst = { +0x6e ,0x83 ,0x15 ,0x4d ,0xf8 ,0x78 ,0xa8 ,0x0e , +0x71 ,0x37 ,0xd4 ,0x6e ,0x28 ,0x5c ,0x06 ,0xa1 , +0x2d ,0x6c ,0x72 ,0x7a ,0xfd ,0xf8 ,0x65 ,0x1a , +0xb8 ,0xe6 ,0x29 ,0x7b ,0xe5 ,0xb3 ,0x23 ,0x79 +}; +enum { nonce39 =0xa4d8c491cf093e9dULL }; + +static const u8 input40[]__initconst = { +0x8f ,0x32 ,0x7c ,0x40 ,0x37 ,0x95 ,0x08 ,0x00 , +0x00 ,0xfe ,0x2f ,0x95 ,0x20 ,0x12 ,0x40 ,0x18 , +0x5e ,0x7e ,0x5e ,0x99 ,0xee ,0x8d ,0x91 ,0x7d , +0x50 ,0x7d ,0x21 ,0x45 ,0x27 ,0xe1 ,0x7f ,0xd4 , +0x73 ,0x10 ,0xe1 ,0x33 ,0xbc ,0xf8 ,0xdd +}; +static const u8 output40[]__initconst = { +0x78 ,0x7c ,0xdc ,0x55 ,0x2b ,0xd9 ,0x2b ,0x3a , +0xdd ,0x56 ,0x11 ,0x52 ,0xd3 ,0x2e ,0xe0 ,0x0d , +0x23 ,0x20 ,0x8a ,0xf1 ,0x4f ,0xee ,0xf1 ,0x68 , +0xf6 ,0xdc ,0x53 ,0xcf ,0x17 ,0xd4 ,0xf0 ,0x6c , +0xdc ,0x80 ,0x5f ,0x1c ,0xa4 ,0x91 ,0x05 +}; +static const u8 key40[]__initconst = { +0x0d ,0x86 ,0xbf ,0x8a ,0xba ,0x9e ,0x39 ,0x91 , +0xa8 ,0xe7 ,0x22 ,0xf0 ,0x0c ,0x43 ,0x18 ,0xe4 , +0x1f ,0xb0 ,0xaf ,0x8a ,0x34 ,0x31 ,0xf4 ,0x41 , +0xf0 ,0x89 ,0x85 ,0xca ,0x5d ,0x05 ,0x3b ,0x94 +}; +enum { nonce40 =0xae7acc4f5986439eULL }; + +static const u8 input41[]__initconst = { +0x20 ,0x5f ,0xc1 ,0x83 ,0x36 ,0x02 ,0x76 ,0x96 , +0xf0 ,0xbf ,0x8e ,0x0e ,0x1a ,0xd1 ,0xc7 ,0x88 , +0x18 ,0xc7 ,0x09 ,0xc4 ,0x15 ,0xd9 ,0x4f ,0x5e , +0x1f ,0xb3 ,0xb4 ,0x6d ,0xcb ,0xa0 ,0xd6 ,0x8a , +0x3b ,0x40 ,0x8e ,0x80 ,0xf1 ,0xe8 ,0x8f ,0x5f +}; +static const u8 output41[]__initconst = { +0x0b ,0xd1 ,0x49 ,0x9a ,0x9d ,0xe8 ,0x97 ,0xb8 , +0xd1 ,0xeb ,0x90 ,0x62 ,0x37 ,0xd2 ,0x99 ,0x15 , +0x67 ,0x6d ,0x27 ,0x93 ,0xce ,0x37 ,0x65 ,0xa2 , +0x94 ,0x88 ,0xd6 ,0x17 ,0xbc ,0x1c ,0x6e ,0xa2 , +0xcc ,0xfb ,0x81 ,0x0e ,0x30 ,0x60 ,0x5a ,0x6f +}; +static const u8 key41[]__initconst = { +0x36 ,0x27 ,0x57 ,0x01 ,0x21 ,0x68 ,0x97 ,0xc7 , +0x00 ,0x67 ,0x7b ,0xe9 ,0x0f ,0x55 ,0x49 ,0xbb , +0x92 ,0x18 ,0x98 ,0xf5 ,0x5e ,0xbc ,0xe7 ,0x5a , +0x9d ,0x3d ,0xc7 ,0xbd ,0x59 ,0xec ,0x82 ,0x8e +}; +enum { nonce41 =0x5da05e4c8dfab464ULL }; + +static const u8 input42[]__initconst = { +0xca ,0x30 ,0xcd ,0x63 ,0xf0 ,0x2d ,0xf1 ,0x03 , +0x4d ,0x0d ,0xf2 ,0xf7 ,0x6f ,0xae ,0xd6 ,0x34 , +0xea ,0xf6 ,0x13 ,0xcf ,0x1c ,0xa0 ,0xd0 ,0xe8 , +0xa4 ,0x78 ,0x80 ,0x3b ,0x1e ,0xa5 ,0x32 ,0x4c , +0x73 ,0x12 ,0xd4 ,0x6a ,0x94 ,0xbc ,0xba ,0x80 , +0x5e +}; +static const u8 output42[]__initconst = { +0xec ,0x3f ,0x18 ,0x31 ,0xc0 ,0x7b ,0xb5 ,0xe2 , +0xad ,0xf3 ,0xec ,0xa0 ,0x16 ,0x9d ,0xef ,0xce , +0x05 ,0x65 ,0x59 ,0x9d ,0x5a ,0xca ,0x3e ,0x13 , +0xb9 ,0x5d ,0x5d ,0xb5 ,0xeb ,0xae ,0xc0 ,0x87 , +0xbb ,0xfd ,0xe7 ,0xe4 ,0x89 ,0x5b ,0xd2 ,0x6c , +0x56 +}; +static const u8 key42[]__initconst = { +0x7c ,0x6b ,0x7e ,0x77 ,0xcc ,0x8c ,0x1b ,0x03 , +0x8b ,0x2a ,0xb3 ,0x7c ,0x5a ,0x73 ,0xcc ,0xac , +0xdd ,0x53 ,0x54 ,0x0c ,0x85 ,0xed ,0xcd ,0x47 , +0x24 ,0xc1 ,0xb8 ,0x9b ,0x2e ,0x41 ,0x92 ,0x36 +}; +enum { nonce42 =0xe4d7348b09682c9cULL }; + +static const u8 input43[]__initconst = { +0x52 ,0xf2 ,0x4b ,0x7c ,0xe5 ,0x58 ,0xe8 ,0xd2 , +0xb7 ,0xf3 ,0xa1 ,0x29 ,0x68 ,0xa2 ,0x50 ,0x50 , +0xae ,0x9c ,0x1b ,0xe2 ,0x67 ,0x77 ,0xe2 ,0xdb , +0x85 ,0x55 ,0x7e ,0x84 ,0x8a ,0x12 ,0x3c ,0xb6 , +0x2e ,0xed ,0xd3 ,0xec ,0x47 ,0x68 ,0xfa ,0x52 , +0x46 ,0x9d +}; +static const u8 output43[]__initconst = { +0x1b ,0xf0 ,0x05 ,0xe4 ,0x1c ,0xd8 ,0x74 ,0x9a , +0xf0 ,0xee ,0x00 ,0x54 ,0xce ,0x02 ,0x83 ,0x15 , +0xfb ,0x23 ,0x35 ,0x78 ,0xc3 ,0xda ,0x98 ,0xd8 , +0x9d ,0x1b ,0xb2 ,0x51 ,0x82 ,0xb0 ,0xff ,0xbe , +0x05 ,0xa9 ,0xa4 ,0x04 ,0xba ,0xea ,0x4b ,0x73 , +0x47 ,0x6e +}; +static const u8 key43[]__initconst = { +0xeb ,0xec ,0x0e ,0xa1 ,0x65 ,0xe2 ,0x99 ,0x46 , +0xd8 ,0x54 ,0x8c ,0x4a ,0x93 ,0xdf ,0x6d ,0xbf , +0x93 ,0x34 ,0x94 ,0x57 ,0xc9 ,0x12 ,0x9d ,0x68 , +0x05 ,0xc5 ,0x05 ,0xad ,0x5a ,0xc9 ,0x2a ,0x3b +}; +enum { nonce43 =0xe14f6a902b7827fULL }; + +static const u8 input44[]__initconst = { +0x3e ,0x22 ,0x3e ,0x8e ,0xcd ,0x18 ,0xe2 ,0xa3 , +0x8d ,0x8b ,0x38 ,0xc3 ,0x02 ,0xa3 ,0x31 ,0x48 , +0xc6 ,0x0e ,0xec ,0x99 ,0x51 ,0x11 ,0x6d ,0x8b , +0x32 ,0x35 ,0x3b ,0x08 ,0x58 ,0x76 ,0x25 ,0x30 , +0xe2 ,0xfc ,0xa2 ,0x46 ,0x7d ,0x6e ,0x34 ,0x87 , +0xac ,0x42 ,0xbf +}; +static const u8 output44[]__initconst = { +0x08 ,0x92 ,0x58 ,0x02 ,0x1a ,0xf4 ,0x1f ,0x3d , +0x38 ,0x7b ,0x6b ,0xf6 ,0x84 ,0x07 ,0xa3 ,0x19 , +0x17 ,0x2a ,0xed ,0x57 ,0x1c ,0xf9 ,0x55 ,0x37 , +0x4e ,0xf4 ,0x68 ,0x68 ,0x82 ,0x02 ,0x4f ,0xca , +0x21 ,0x00 ,0xc6 ,0x66 ,0x79 ,0x53 ,0x19 ,0xef , +0x7f ,0xdd ,0x74 +}; +static const u8 key44[]__initconst = { +0x73 ,0xb6 ,0x3e ,0xf4 ,0x57 ,0x52 ,0xa6 ,0x43 , +0x51 ,0xd8 ,0x25 ,0x00 ,0xdb ,0xb4 ,0x52 ,0x69 , +0xd6 ,0x27 ,0x49 ,0xeb ,0x9b ,0xf1 ,0x7b ,0xa0 , +0xd6 ,0x7c ,0x9c ,0xd8 ,0x95 ,0x03 ,0x69 ,0x26 +}; +enum { nonce44 =0xf5e6dc4f35ce24e5ULL }; + +static const u8 input45[]__initconst = { +0x55 ,0x76 ,0xc0 ,0xf1 ,0x74 ,0x03 ,0x7a ,0x6d , +0x14 ,0xd8 ,0x36 ,0x2c ,0x9f ,0x9a ,0x59 ,0x7a , +0x2a ,0xf5 ,0x77 ,0x84 ,0x70 ,0x7c ,0x1d ,0x04 , +0x90 ,0x45 ,0xa4 ,0xc1 ,0x5e ,0xdd ,0x2e ,0x07 , +0x18 ,0x34 ,0xa6 ,0x85 ,0x56 ,0x4f ,0x09 ,0xaf , +0x2f ,0x83 ,0xe1 ,0xc6 +}; +static const u8 output45[]__initconst = { +0x22 ,0x46 ,0xe4 ,0x0b ,0x3a ,0x55 ,0xcc ,0x9b , +0xf0 ,0xc0 ,0x53 ,0xcd ,0x95 ,0xc7 ,0x57 ,0x6c , +0x77 ,0x46 ,0x41 ,0x72 ,0x07 ,0xbf ,0xa8 ,0xe5 , +0x68 ,0x69 ,0xd8 ,0x1e ,0x45 ,0xc1 ,0xa2 ,0x50 , +0xa5 ,0xd1 ,0x62 ,0xc9 ,0x5a ,0x7d ,0x08 ,0x14 , +0xae ,0x44 ,0x16 ,0xb9 +}; +static const u8 key45[]__initconst = { +0x41 ,0xf3 ,0x88 ,0xb2 ,0x51 ,0x25 ,0x47 ,0x02 , +0x39 ,0xe8 ,0x15 ,0x3a ,0x22 ,0x78 ,0x86 ,0x0b , +0xf9 ,0x1e ,0x8d ,0x98 ,0xb2 ,0x22 ,0x82 ,0xac , +0x42 ,0x94 ,0xde ,0x64 ,0xf0 ,0xfd ,0xb3 ,0x6c +}; +enum { nonce45 =0xf51a582daf4aa01aULL }; + +static const u8 input46[]__initconst = { +0xf6 ,0xff ,0x20 ,0xf9 ,0x26 ,0x7e ,0x0f ,0xa8 , +0x6a ,0x45 ,0x5a ,0x91 ,0x73 ,0xc4 ,0x4c ,0x63 , +0xe5 ,0x61 ,0x59 ,0xca ,0xec ,0xc0 ,0x20 ,0x35 , +0xbc ,0x9f ,0x58 ,0x9c ,0x5e ,0xa1 ,0x17 ,0x46 , +0xcc ,0xab ,0x6e ,0xd0 ,0x4f ,0x24 ,0xeb ,0x05 , +0x4d ,0x40 ,0x41 ,0xe0 ,0x9d +}; +static const u8 output46[]__initconst = { +0x31 ,0x6e ,0x63 ,0x3f ,0x9c ,0xe6 ,0xb1 ,0xb7 , +0xef ,0x47 ,0x46 ,0xd7 ,0xb1 ,0x53 ,0x42 ,0x2f , +0x2c ,0xc8 ,0x01 ,0xae ,0x8b ,0xec ,0x42 ,0x2c , +0x6b ,0x2c ,0x9c ,0xb2 ,0xf0 ,0x29 ,0x06 ,0xa5 , +0xcd ,0x7e ,0xc7 ,0x3a ,0x38 ,0x98 ,0x8a ,0xde , +0x03 ,0x29 ,0x14 ,0x8f ,0xf9 +}; +static const u8 key46[]__initconst = { +0xac ,0xa6 ,0x44 ,0x4a ,0x0d ,0x42 ,0x10 ,0xbc , +0xd3 ,0xc9 ,0x8e ,0x9e ,0x71 ,0xa3 ,0x1c ,0x14 , +0x9d ,0x65 ,0x0d ,0x49 ,0x4d ,0x8c ,0xec ,0x46 , +0xe1 ,0x41 ,0xcd ,0xf5 ,0xfc ,0x82 ,0x75 ,0x34 +}; +enum { nonce46 =0x25f85182df84dec5ULL }; + +static const u8 input47[]__initconst = { +0xa1 ,0xd2 ,0xf2 ,0x52 ,0x2f ,0x79 ,0x50 ,0xb2 , +0x42 ,0x29 ,0x5b ,0x44 ,0x20 ,0xf9 ,0xbd ,0x85 , +0xb7 ,0x65 ,0x77 ,0x86 ,0xce ,0x3e ,0x1c ,0xe4 , +0x70 ,0x80 ,0xdd ,0x72 ,0x07 ,0x48 ,0x0f ,0x84 , +0x0d ,0xfd ,0x97 ,0xc0 ,0xb7 ,0x48 ,0x9b ,0xb4 , +0xec ,0xff ,0x73 ,0x14 ,0x99 ,0xe4 +}; +static const u8 output47[]__initconst = { +0xe5 ,0x3c ,0x78 ,0x66 ,0x31 ,0x1e ,0xd6 ,0xc4 , +0x9e ,0x71 ,0xb3 ,0xd7 ,0xd5 ,0xad ,0x84 ,0xf2 , +0x78 ,0x61 ,0x77 ,0xf8 ,0x31 ,0xf0 ,0x13 ,0xad , +0x66 ,0xf5 ,0x31 ,0x7d ,0xeb ,0xdf ,0xaf ,0xcb , +0xac ,0x28 ,0x6c ,0xc2 ,0x9e ,0xe7 ,0x78 ,0xa2 , +0xa2 ,0x58 ,0xce ,0x84 ,0x76 ,0x70 +}; +static const u8 key47[]__initconst = { +0x05 ,0x7f ,0xc0 ,0x7f ,0x37 ,0x20 ,0x71 ,0x02 , +0x3a ,0xe7 ,0x20 ,0x5a ,0x0a ,0x8f ,0x79 ,0x5a , +0xfe ,0xbb ,0x43 ,0x4d ,0x2f ,0xcb ,0xf6 ,0x9e , +0xa2 ,0x97 ,0x00 ,0xad ,0x0d ,0x51 ,0x7e ,0x17 +}; +enum { nonce47 =0xae707c60f54de32bULL }; + +static const u8 input48[]__initconst = { +0x80 ,0x93 ,0x77 ,0x2e ,0x8d ,0xe8 ,0xe6 ,0xc1 , +0x27 ,0xe6 ,0xf2 ,0x89 ,0x5b ,0x33 ,0x62 ,0x18 , +0x80 ,0x6e ,0x17 ,0x22 ,0x8e ,0x83 ,0x31 ,0x40 , +0x8f ,0xc9 ,0x5c ,0x52 ,0x6c ,0x0e ,0xa5 ,0xe9 , +0x6c ,0x7f ,0xd4 ,0x6a ,0x27 ,0x56 ,0x99 ,0xce , +0x8d ,0x37 ,0x59 ,0xaf ,0xc0 ,0x0e ,0xe1 +}; +static const u8 output48[]__initconst = { +0x02 ,0xa4 ,0x2e ,0x33 ,0xb7 ,0x7c ,0x2b ,0x9a , +0x18 ,0x5a ,0xba ,0x53 ,0x38 ,0xaf ,0x00 ,0xeb , +0xd8 ,0x3d ,0x02 ,0x77 ,0x43 ,0x45 ,0x03 ,0x91 , +0xe2 ,0x5e ,0x4e ,0xeb ,0x50 ,0xd5 ,0x5b ,0xe0 , +0xf3 ,0x33 ,0xa7 ,0xa2 ,0xac ,0x07 ,0x6f ,0xeb , +0x3f ,0x6c ,0xcd ,0xf2 ,0x6c ,0x61 ,0x64 +}; +static const u8 key48[]__initconst = { +0xf3 ,0x79 ,0xe7 ,0xf8 ,0x0e ,0x02 ,0x05 ,0x6b , +0x83 ,0x1a ,0xe7 ,0x86 ,0x6b ,0xe6 ,0x8f ,0x3f , +0xd3 ,0xa3 ,0xe4 ,0x6e ,0x29 ,0x06 ,0xad ,0xbc , +0xe8 ,0x33 ,0x56 ,0x39 ,0xdf ,0xb0 ,0xe2 ,0xfe +}; +enum { nonce48 =0xd849b938c6569da0ULL }; + +static const u8 input49[]__initconst = { +0x89 ,0x3b ,0x88 ,0x9e ,0x7b ,0x38 ,0x16 ,0x9f , +0xa1 ,0x28 ,0xf6 ,0xf5 ,0x23 ,0x74 ,0x28 ,0xb0 , +0xdf ,0x6c ,0x9e ,0x8a ,0x71 ,0xaf ,0xed ,0x7a , +0x39 ,0x21 ,0x57 ,0x7d ,0x31 ,0x6c ,0xee ,0x0d , +0x11 ,0x8d ,0x41 ,0x9a ,0x5f ,0xb7 ,0x27 ,0x40 , +0x08 ,0xad ,0xc6 ,0xe0 ,0x00 ,0x43 ,0x9e ,0xae +}; +static const u8 output49[]__initconst = { +0x4d ,0xfd ,0xdb ,0x4c ,0x77 ,0xc1 ,0x05 ,0x07 , +0x4d ,0x6d ,0x32 ,0xcb ,0x2e ,0x0e ,0xff ,0x65 , +0xc9 ,0x27 ,0xeb ,0xa9 ,0x46 ,0x5b ,0xab ,0x06 , +0xe6 ,0xb6 ,0x5a ,0x1e ,0x00 ,0xfb ,0xcf ,0xe4 , +0xb9 ,0x71 ,0x40 ,0x10 ,0xef ,0x12 ,0x39 ,0xf0 , +0xea ,0x40 ,0xb8 ,0x9a ,0xa2 ,0x85 ,0x38 ,0x48 +}; +static const u8 key49[]__initconst = { +0xe7 ,0x10 ,0x40 ,0xd9 ,0x66 ,0xc0 ,0xa8 ,0x6d , +0xa3 ,0xcc ,0x8b ,0xdd ,0x93 ,0xf2 ,0x6e ,0xe0 , +0x90 ,0x7f ,0xd0 ,0xf4 ,0x37 ,0x0c ,0x8b ,0x9b , +0x4c ,0x4d ,0xe6 ,0xf2 ,0x1f ,0xe9 ,0x95 ,0x24 +}; +enum { nonce49 =0xf269817bdae01bc0ULL }; + +static const u8 input50[]__initconst = { +0xda ,0x5b ,0x60 ,0xcd ,0xed ,0x58 ,0x8e ,0x7f , +0xae ,0xdd ,0xc8 ,0x2e ,0x16 ,0x90 ,0xea ,0x4b , +0x0c ,0x74 ,0x14 ,0x35 ,0xeb ,0xee ,0x2c ,0xff , +0x46 ,0x99 ,0x97 ,0x6e ,0xae ,0xa7 ,0x8e ,0x6e , +0x38 ,0xfe ,0x63 ,0xe7 ,0x51 ,0xd9 ,0xaa ,0xce , +0x7b ,0x1e ,0x7e ,0x5d ,0xc0 ,0xe8 ,0x10 ,0x06 , +0x14 +}; +static const u8 output50[]__initconst = { +0xe4 ,0xe5 ,0x86 ,0x1b ,0x66 ,0x19 ,0xac ,0x49 , +0x1c ,0xbd ,0xee ,0x03 ,0xaf ,0x11 ,0xfc ,0x1f , +0x6a ,0xd2 ,0x50 ,0x5c ,0xea ,0x2c ,0xa5 ,0x75 , +0xfd ,0xb7 ,0x0e ,0x80 ,0x8f ,0xed ,0x3f ,0x31 , +0x47 ,0xac ,0x67 ,0x43 ,0xb8 ,0x2e ,0xb4 ,0x81 , +0x6d ,0xe4 ,0x1e ,0xb7 ,0x8b ,0x0c ,0x53 ,0xa9 , +0x26 +}; +static const u8 key50[]__initconst = { +0xd7 ,0xb2 ,0x04 ,0x76 ,0x30 ,0xcc ,0x38 ,0x45 , +0xef ,0xdb ,0xc5 ,0x86 ,0x08 ,0x61 ,0xf0 ,0xee , +0x6d ,0xd8 ,0x22 ,0x04 ,0x8c ,0xfb ,0xcb ,0x37 , +0xa6 ,0xfb ,0x95 ,0x22 ,0xe1 ,0x87 ,0xb7 ,0x6f +}; +enum { nonce50 =0x3b44d09c45607d38ULL }; + +static const u8 input51[]__initconst = { +0xa9 ,0x41 ,0x02 ,0x4b ,0xd7 ,0xd5 ,0xd1 ,0xf1 , +0x21 ,0x55 ,0xb2 ,0x75 ,0x6d ,0x77 ,0x1b ,0x86 , +0xa9 ,0xc8 ,0x90 ,0xfd ,0xed ,0x4a ,0x7b ,0x6c , +0xb2 ,0x5f ,0x9b ,0x5f ,0x16 ,0xa1 ,0x54 ,0xdb , +0xd6 ,0x3f ,0x6a ,0x7f ,0x2e ,0x51 ,0x9d ,0x49 , +0x5b ,0xa5 ,0x0e ,0xf9 ,0xfb ,0x2a ,0x38 ,0xff , +0x20 ,0x8c +}; +static const u8 output51[]__initconst = { +0x18 ,0xf7 ,0x88 ,0xc1 ,0x72 ,0xfd ,0x90 ,0x4b , +0xa9 ,0x2d ,0xdb ,0x47 ,0xb0 ,0xa5 ,0xc4 ,0x37 , +0x01 ,0x95 ,0xc4 ,0xb1 ,0xab ,0xc5 ,0x5b ,0xcd , +0xe1 ,0x97 ,0x78 ,0x13 ,0xde ,0x6a ,0xff ,0x36 , +0xce ,0xa4 ,0x67 ,0xc5 ,0x4a ,0x45 ,0x2b ,0xd9 , +0xff ,0x8f ,0x06 ,0x7c ,0x63 ,0xbb ,0x83 ,0x17 , +0xb4 ,0x6b +}; +static const u8 key51[]__initconst = { +0x82 ,0x1a ,0x79 ,0xab ,0x9a ,0xb5 ,0x49 ,0x6a , +0x30 ,0x6b ,0x99 ,0x19 ,0x11 ,0xc7 ,0xa2 ,0xf4 , +0xca ,0x55 ,0xb9 ,0xdd ,0xe7 ,0x2f ,0xe7 ,0xc1 , +0xdd ,0x27 ,0xad ,0x80 ,0xf2 ,0x56 ,0xad ,0xf3 +}; +enum { nonce51 =0xe93aff94ca71a4a6ULL }; + +static const u8 input52[]__initconst = { +0x89 ,0xdd ,0xf3 ,0xfa ,0xb6 ,0xc1 ,0xaa ,0x9a , +0xc8 ,0xad ,0x6b ,0x00 ,0xa1 ,0x65 ,0xea ,0x14 , +0x55 ,0x54 ,0x31 ,0x8f ,0xf0 ,0x03 ,0x84 ,0x51 , +0x17 ,0x1e ,0x0a ,0x93 ,0x6e ,0x79 ,0x96 ,0xa3 , +0x2a ,0x85 ,0x9c ,0x89 ,0xf8 ,0xd1 ,0xe2 ,0x15 , +0x95 ,0x05 ,0xf4 ,0x43 ,0x4d ,0x6b ,0xf0 ,0x71 , +0x3b ,0x3e ,0xba +}; +static const u8 output52[]__initconst = { +0x0c ,0x42 ,0x6a ,0xb3 ,0x66 ,0x63 ,0x5d ,0x2c , +0x9f ,0x3d ,0xa6 ,0x6e ,0xc7 ,0x5f ,0x79 ,0x2f , +0x50 ,0xe3 ,0xd6 ,0x07 ,0x56 ,0xa4 ,0x2b ,0x2d , +0x8d ,0x10 ,0xc0 ,0x6c ,0xa2 ,0xfc ,0x97 ,0xec , +0x3f ,0x5c ,0x8d ,0x59 ,0xbe ,0x84 ,0xf1 ,0x3e , +0x38 ,0x47 ,0x4f ,0x75 ,0x25 ,0x66 ,0x88 ,0x14 , +0x03 ,0xdd ,0xde +}; +static const u8 key52[]__initconst = { +0x4f ,0xb0 ,0x27 ,0xb6 ,0xdd ,0x24 ,0x0c ,0xdb , +0x6b ,0x71 ,0x2e ,0xac ,0xfc ,0x3f ,0xa6 ,0x48 , +0x5d ,0xd5 ,0xff ,0x53 ,0xb5 ,0x62 ,0xf1 ,0xe0 , +0x93 ,0xfe ,0x39 ,0x4c ,0x9f ,0x03 ,0x11 ,0xa7 +}; +enum { nonce52 =0xed8becec3bdf6f25ULL }; + +static const u8 input53[]__initconst = { +0x68 ,0xd1 ,0xc7 ,0x74 ,0x44 ,0x1c ,0x84 ,0xde , +0x27 ,0x27 ,0x35 ,0xf0 ,0x18 ,0x0b ,0x57 ,0xaa , +0xd0 ,0x1a ,0xd3 ,0x3b ,0x5e ,0x5c ,0x62 ,0x93 , +0xd7 ,0x6b ,0x84 ,0x3b ,0x71 ,0x83 ,0x77 ,0x01 , +0x3e ,0x59 ,0x45 ,0xf4 ,0x77 ,0x6c ,0x6b ,0xcb , +0x88 ,0x45 ,0x09 ,0x1d ,0xc6 ,0x45 ,0x6e ,0xdc , +0x6e ,0x51 ,0xb8 ,0x28 +}; +static const u8 output53[]__initconst = { +0xc5 ,0x90 ,0x96 ,0x78 ,0x02 ,0xf5 ,0xc4 ,0x3c , +0xde ,0xd4 ,0xd4 ,0xc6 ,0xa7 ,0xad ,0x12 ,0x47 , +0x45 ,0xce ,0xcd ,0x8c ,0x35 ,0xcc ,0xa6 ,0x9e , +0x5a ,0xc6 ,0x60 ,0xbb ,0xe3 ,0xed ,0xec ,0x68 , +0x3f ,0x64 ,0xf7 ,0x06 ,0x63 ,0x9c ,0x8c ,0xc8 , +0x05 ,0x3a ,0xad ,0x32 ,0x79 ,0x8b ,0x45 ,0x96 , +0x93 ,0x73 ,0x4c ,0xe0 +}; +static const u8 key53[]__initconst = { +0x42 ,0x4b ,0x20 ,0x81 ,0x49 ,0x50 ,0xe9 ,0xc2 , +0x43 ,0x69 ,0x36 ,0xe7 ,0x68 ,0xae ,0xd5 ,0x7e , +0x42 ,0x1a ,0x1b ,0xb4 ,0x06 ,0x4d ,0xa7 ,0x17 , +0xb5 ,0x31 ,0xd6 ,0x0c ,0xb0 ,0x5c ,0x41 ,0x0b +}; +enum { nonce53 =0xf44ce1931fbda3d7ULL }; + +static const u8 input54[]__initconst = { +0x7b ,0xf6 ,0x8b ,0xae ,0xc0 ,0xcb ,0x10 ,0x8e , +0xe8 ,0xd8 ,0x2e ,0x3b ,0x14 ,0xba ,0xb4 ,0xd2 , +0x58 ,0x6b ,0x2c ,0xec ,0xc1 ,0x81 ,0x71 ,0xb4 , +0xc6 ,0xea ,0x08 ,0xc5 ,0xc9 ,0x78 ,0xdb ,0xa2 , +0xfa ,0x44 ,0x50 ,0x9b ,0xc8 ,0x53 ,0x8d ,0x45 , +0x42 ,0xe7 ,0x09 ,0xc4 ,0x29 ,0xd8 ,0x75 ,0x02 , +0xbb ,0xb2 ,0x78 ,0xcf ,0xe7 +}; +static const u8 output54[]__initconst = { +0xaf ,0x2c ,0x83 ,0x26 ,0x6e ,0x7f ,0xa6 ,0xe9 , +0x03 ,0x75 ,0xfe ,0xfe ,0x87 ,0x58 ,0xcf ,0xb5 , +0xbc ,0x3c ,0x9d ,0xa1 ,0x6e ,0x13 ,0xf1 ,0x0f , +0x9e ,0xbc ,0xe0 ,0x54 ,0x24 ,0x32 ,0xce ,0x95 , +0xe6 ,0xa5 ,0x59 ,0x3d ,0x24 ,0x1d ,0x8f ,0xb1 , +0x74 ,0x6c ,0x56 ,0xe7 ,0x96 ,0xc1 ,0x91 ,0xc8 , +0x2d ,0x0e ,0xb7 ,0x51 ,0x10 +}; +static const u8 key54[]__initconst = { +0x00 ,0x68 ,0x74 ,0xdc ,0x30 ,0x9e ,0xe3 ,0x52 , +0xa9 ,0xae ,0xb6 ,0x7c ,0xa1 ,0xdc ,0x12 ,0x2d , +0x98 ,0x32 ,0x7a ,0x77 ,0xe1 ,0xdd ,0xa3 ,0x76 , +0x72 ,0x34 ,0x83 ,0xd8 ,0xb7 ,0x69 ,0xba ,0x77 +}; +enum { nonce54 =0xbea57d79b798b63aULL }; + +static const u8 input55[]__initconst = { +0xb5 ,0xf4 ,0x2f ,0xc1 ,0x5e ,0x10 ,0xa7 ,0x4e , +0x74 ,0x3d ,0xa3 ,0x96 ,0xc0 ,0x4d ,0x7b ,0x92 , +0x8f ,0xdb ,0x2d ,0x15 ,0x52 ,0x6a ,0x95 ,0x5e , +0x40 ,0x81 ,0x4f ,0x70 ,0x73 ,0xea ,0x84 ,0x65 , +0x3d ,0x9a ,0x4e ,0x03 ,0x95 ,0xf8 ,0x5d ,0x2f , +0x07 ,0x02 ,0x13 ,0x13 ,0xdd ,0x82 ,0xe6 ,0x3b , +0xe1 ,0x5f ,0xb3 ,0x37 ,0x9b ,0x88 +}; +static const u8 output55[]__initconst = { +0xc1 ,0x88 ,0xbd ,0x92 ,0x77 ,0xad ,0x7c ,0x5f , +0xaf ,0xa8 ,0x57 ,0x0e ,0x40 ,0x0a ,0xdc ,0x70 , +0xfb ,0xc6 ,0x71 ,0xfd ,0xc4 ,0x74 ,0x60 ,0xcc , +0xa0 ,0x89 ,0x8e ,0x99 ,0xf0 ,0x06 ,0xa6 ,0x7c , +0x97 ,0x42 ,0x21 ,0x81 ,0x6a ,0x07 ,0xe7 ,0xb3 , +0xf7 ,0xa5 ,0x03 ,0x71 ,0x50 ,0x05 ,0x63 ,0x17 , +0xa9 ,0x46 ,0x0b ,0xff ,0x30 ,0x78 +}; +static const u8 key55[]__initconst = { +0x19 ,0x8f ,0xe7 ,0xd7 ,0x6b ,0x7f ,0x6f ,0x69 , +0x86 ,0x91 ,0x0f ,0xa7 ,0x4a ,0x69 ,0x8e ,0x34 , +0xf3 ,0xdb ,0xde ,0xaf ,0xf2 ,0x66 ,0x1d ,0x64 , +0x97 ,0x0c ,0xcf ,0xfa ,0x33 ,0x84 ,0xfd ,0x0c +}; +enum { nonce55 =0x80aa3d3e2c51ef06ULL }; + +static const u8 input56[]__initconst = { +0x6b ,0xe9 ,0x73 ,0x42 ,0x27 ,0x5e ,0x12 ,0xcd , +0xaa ,0x45 ,0x12 ,0x8b ,0xb3 ,0xe6 ,0x54 ,0x33 , +0x31 ,0x7d ,0xe2 ,0x25 ,0xc6 ,0x86 ,0x47 ,0x67 , +0x86 ,0x83 ,0xe4 ,0x46 ,0xb5 ,0x8f ,0x2c ,0xbb , +0xe4 ,0xb8 ,0x9f ,0xa2 ,0xa4 ,0xe8 ,0x75 ,0x96 , +0x92 ,0x51 ,0x51 ,0xac ,0x8e ,0x2e ,0x6f ,0xfc , +0xbd ,0x0d ,0xa3 ,0x9f ,0x16 ,0x55 ,0x3e +}; +static const u8 output56[]__initconst = { +0x42 ,0x99 ,0x73 ,0x6c ,0xd9 ,0x4b ,0x16 ,0xe5 , +0x18 ,0x63 ,0x1a ,0xd9 ,0x0e ,0xf1 ,0x15 ,0x2e , +0x0f ,0x4b ,0xe4 ,0x5f ,0xa0 ,0x4d ,0xde ,0x9f , +0xa7 ,0x18 ,0xc1 ,0x0c ,0x0b ,0xae ,0x55 ,0xe4 , +0x89 ,0x18 ,0xa4 ,0x78 ,0x9d ,0x25 ,0x0d ,0xd5 , +0x94 ,0x0f ,0xf9 ,0x78 ,0xa3 ,0xa6 ,0xe9 ,0x9e , +0x2c ,0x73 ,0xf0 ,0xf7 ,0x35 ,0xf3 ,0x2b +}; +static const u8 key56[]__initconst = { +0x7d ,0x12 ,0xad ,0x51 ,0xd5 ,0x6f ,0x8f ,0x96 , +0xc0 ,0x5d ,0x9a ,0xd1 ,0x7e ,0x20 ,0x98 ,0x0e , +0x3c ,0x0a ,0x67 ,0x6b ,0x1b ,0x88 ,0x69 ,0xd4 , +0x07 ,0x8c ,0xaf ,0x0f ,0x3a ,0x28 ,0xe4 ,0x5d +}; +enum { nonce56 =0x70f4c372fb8b5984ULL }; + +static const u8 input57[]__initconst = { +0x28 ,0xa3 ,0x06 ,0xe8 ,0xe7 ,0x08 ,0xb9 ,0xef , +0x0d ,0x63 ,0x15 ,0x99 ,0xb2 ,0x78 ,0x7e ,0xaf , +0x30 ,0x50 ,0xcf ,0xea ,0xc9 ,0x91 ,0x41 ,0x2f , +0x3b ,0x38 ,0x70 ,0xc4 ,0x87 ,0xb0 ,0x3a ,0xee , +0x4a ,0xea ,0xe3 ,0x83 ,0x68 ,0x8b ,0xcf ,0xda , +0x04 ,0xa5 ,0xbd ,0xb2 ,0xde ,0x3c ,0x55 ,0x13 , +0xfe ,0x96 ,0xad ,0xc1 ,0x61 ,0x1b ,0x98 ,0xde +}; +static const u8 output57[]__initconst = { +0xf4 ,0x44 ,0xe9 ,0xd2 ,0x6d ,0xc2 ,0x5a ,0xe9 , +0xfd ,0x7e ,0x41 ,0x54 ,0x3f ,0xf4 ,0x12 ,0xd8 , +0x55 ,0x0d ,0x12 ,0x9b ,0xd5 ,0x2e ,0x95 ,0xe5 , +0x77 ,0x42 ,0x3f ,0x2c ,0xfb ,0x28 ,0x9d ,0x72 , +0x6d ,0x89 ,0x82 ,0x27 ,0x64 ,0x6f ,0x0d ,0x57 , +0xa1 ,0x25 ,0xa3 ,0x6b ,0x88 ,0x9a ,0xac ,0x0c , +0x76 ,0x19 ,0x90 ,0xe2 ,0x50 ,0x5a ,0xf8 ,0x12 +}; +static const u8 key57[]__initconst = { +0x08 ,0x26 ,0xb8 ,0xac ,0xf3 ,0xa5 ,0xc6 ,0xa3 , +0x7f ,0x09 ,0x87 ,0xf5 ,0x6c ,0x5a ,0x85 ,0x6c , +0x3d ,0xbd ,0xde ,0xd5 ,0x87 ,0xa3 ,0x98 ,0x7a , +0xaa ,0x40 ,0x3e ,0xf7 ,0xff ,0x44 ,0x5d ,0xee +}; +enum { nonce57 =0xc03a6130bf06b089ULL }; + +static const u8 input58[]__initconst = { +0x82 ,0xa5 ,0x38 ,0x6f ,0xaa ,0xb4 ,0xaf ,0xb2 , +0x42 ,0x01 ,0xa8 ,0x39 ,0x3f ,0x15 ,0x51 ,0xa8 , +0x11 ,0x1b ,0x93 ,0xca ,0x9c ,0xa0 ,0x57 ,0x68 , +0x8f ,0xdb ,0x68 ,0x53 ,0x51 ,0x6d ,0x13 ,0x22 , +0x12 ,0x9b ,0xbd ,0x33 ,0xa8 ,0x52 ,0x40 ,0x57 , +0x80 ,0x9b ,0x98 ,0xef ,0x56 ,0x70 ,0x11 ,0xfa , +0x36 ,0x69 ,0x7d ,0x15 ,0x48 ,0xf9 ,0x3b ,0xeb , +0x42 +}; +static const u8 output58[]__initconst = { +0xff ,0x3a ,0x74 ,0xc3 ,0x3e ,0x44 ,0x64 ,0x4d , +0x0e ,0x5f ,0x9d ,0xa8 ,0xdb ,0xbe ,0x12 ,0xef , +0xba ,0x56 ,0x65 ,0x50 ,0x76 ,0xaf ,0xa4 ,0x4e , +0x01 ,0xc1 ,0xd3 ,0x31 ,0x14 ,0xe2 ,0xbe ,0x7b , +0xa5 ,0x67 ,0xb4 ,0xe3 ,0x68 ,0x40 ,0x9c ,0xb0 , +0xb1 ,0x78 ,0xef ,0x49 ,0x03 ,0x0f ,0x2d ,0x56 , +0xb4 ,0x37 ,0xdb ,0xbc ,0x2d ,0x68 ,0x1c ,0x3c , +0xf1 +}; +static const u8 key58[]__initconst = { +0x7e ,0xf1 ,0x7c ,0x20 ,0x65 ,0xed ,0xcd ,0xd7 , +0x57 ,0xe8 ,0xdb ,0x90 ,0x87 ,0xdb ,0x5f ,0x63 , +0x3d ,0xdd ,0xb8 ,0x2b ,0x75 ,0x8e ,0x04 ,0xb5 , +0xf4 ,0x12 ,0x79 ,0xa9 ,0x4d ,0x42 ,0x16 ,0x7f +}; +enum { nonce58 =0x92838183f80d2f7fULL }; + +static const u8 input59[]__initconst = { +0x37 ,0xf1 ,0x9d ,0xdd ,0xd7 ,0x08 ,0x9f ,0x13 , +0xc5 ,0x21 ,0x82 ,0x75 ,0x08 ,0x9e ,0x25 ,0x16 , +0xb1 ,0xd1 ,0x71 ,0x42 ,0x28 ,0x63 ,0xac ,0x47 , +0x71 ,0x54 ,0xb1 ,0xfc ,0x39 ,0xf0 ,0x61 ,0x4f , +0x7c ,0x6d ,0x4f ,0xc8 ,0x33 ,0xef ,0x7e ,0xc8 , +0xc0 ,0x97 ,0xfc ,0x1a ,0x61 ,0xb4 ,0x87 ,0x6f , +0xdd ,0x5a ,0x15 ,0x7b ,0x1b ,0x95 ,0x50 ,0x94 , +0x1d ,0xba +}; +static const u8 output59[]__initconst = { +0x73 ,0x67 ,0xc5 ,0x07 ,0xbb ,0x57 ,0x79 ,0xd5 , +0xc9 ,0x04 ,0xdd ,0x88 ,0xf3 ,0x86 ,0xe5 ,0x70 , +0x49 ,0x31 ,0xe0 ,0xcc ,0x3b ,0x1d ,0xdf ,0xb0 , +0xaf ,0xf4 ,0x2d ,0xe0 ,0x06 ,0x10 ,0x91 ,0x8d , +0x1c ,0xcf ,0x31 ,0x0b ,0xf6 ,0x73 ,0xda ,0x1c , +0xf0 ,0x17 ,0x52 ,0x9e ,0x20 ,0x2e ,0x9f ,0x8c , +0xb3 ,0x59 ,0xce ,0xd4 ,0xd3 ,0xc1 ,0x81 ,0xe9 , +0x11 ,0x36 +}; +static const u8 key59[]__initconst = { +0xbd ,0x07 ,0xd0 ,0x53 ,0x2c ,0xb3 ,0xcc ,0x3f , +0xc4 ,0x95 ,0xfd ,0xe7 ,0x81 ,0xb3 ,0x29 ,0x99 , +0x05 ,0x45 ,0xd6 ,0x95 ,0x25 ,0x0b ,0x72 ,0xd3 , +0xcd ,0xbb ,0x73 ,0xf8 ,0xfa ,0xc0 ,0x9b ,0x7a +}; +enum { nonce59 =0x4a0db819b0d519e2ULL }; + +static const u8 input60[]__initconst = { +0x58 ,0x4e ,0xdf ,0x94 ,0x3c ,0x76 ,0x0a ,0x79 , +0x47 ,0xf1 ,0xbe ,0x88 ,0xd3 ,0xba ,0x94 ,0xd8 , +0xe2 ,0x8f ,0xe3 ,0x2f ,0x2f ,0x74 ,0x82 ,0x55 , +0xc3 ,0xda ,0xe2 ,0x4e ,0x2c ,0x8c ,0x45 ,0x1d , +0x72 ,0x8f ,0x54 ,0x41 ,0xb5 ,0xb7 ,0x69 ,0xe4 , +0xdc ,0xd2 ,0x36 ,0x21 ,0x5c ,0x28 ,0x52 ,0xf7 , +0x98 ,0x8e ,0x72 ,0xa7 ,0x6d ,0x57 ,0xed ,0xdc , +0x3c ,0xe6 ,0x6a +}; +static const u8 output60[]__initconst = { +0xda ,0xaf ,0xb5 ,0xe3 ,0x30 ,0x65 ,0x5c ,0xb1 , +0x48 ,0x08 ,0x43 ,0x7b ,0x9e ,0xd2 ,0x6a ,0x62 , +0x56 ,0x7c ,0xad ,0xd9 ,0xe5 ,0xf6 ,0x09 ,0x71 , +0xcd ,0xe6 ,0x05 ,0x6b ,0x3f ,0x44 ,0x3a ,0x5c , +0xf6 ,0xf8 ,0xd7 ,0xce ,0x7d ,0xd1 ,0xe0 ,0x4f , +0x88 ,0x15 ,0x04 ,0xd8 ,0x20 ,0xf0 ,0x3e ,0xef , +0xae ,0xa6 ,0x27 ,0xa3 ,0x0e ,0xfc ,0x18 ,0x90 , +0x33 ,0xcd ,0xd3 +}; +static const u8 key60[]__initconst = { +0xbf ,0xfd ,0x25 ,0xb5 ,0xb2 ,0xfc ,0x78 ,0x0c , +0x8e ,0xb9 ,0x57 ,0x2f ,0x26 ,0x4a ,0x7e ,0x71 , +0xcc ,0xf2 ,0xe0 ,0xfd ,0x24 ,0x11 ,0x20 ,0x23 , +0x57 ,0x00 ,0xff ,0x80 ,0x11 ,0x0c ,0x1e ,0xff +}; +enum { nonce60 =0xf18df56fdb7954adULL }; + +static const u8 input61[]__initconst = { +0xb0 ,0xf3 ,0x06 ,0xbc ,0x22 ,0xae ,0x49 ,0x40 , +0xae ,0xff ,0x1b ,0x31 ,0xa7 ,0x98 ,0xab ,0x1d , +0xe7 ,0x40 ,0x23 ,0x18 ,0x4f ,0xab ,0x8e ,0x93 , +0x82 ,0xf4 ,0x56 ,0x61 ,0xfd ,0x2b ,0xcf ,0xa7 , +0xc4 ,0xb4 ,0x0a ,0xf4 ,0xcb ,0xc7 ,0x8c ,0x40 , +0x57 ,0xac ,0x0b ,0x3e ,0x2a ,0x0a ,0x67 ,0x83 , +0x50 ,0xbf ,0xec ,0xb0 ,0xc7 ,0xf1 ,0x32 ,0x26 , +0x98 ,0x80 ,0x33 ,0xb4 +}; +static const u8 output61[]__initconst = { +0x9d ,0x23 ,0x0e ,0xff ,0xcc ,0x7c ,0xd5 ,0xcf , +0x1a ,0xb8 ,0x59 ,0x1e ,0x92 ,0xfd ,0x7f ,0xca , +0xca ,0x3c ,0x18 ,0x81 ,0xde ,0xfa ,0x59 ,0xc8 , +0x6f ,0x9c ,0x24 ,0x3f ,0x3a ,0xe6 ,0x0b ,0xb4 , +0x34 ,0x48 ,0x69 ,0xfc ,0xb6 ,0xea ,0xb2 ,0xde , +0x9f ,0xfd ,0x92 ,0x36 ,0x18 ,0x98 ,0x99 ,0xaa , +0x65 ,0xe2 ,0xea ,0xf4 ,0xb1 ,0x47 ,0x8e ,0xb0 , +0xe7 ,0xd4 ,0x7a ,0x2c +}; +static const u8 key61[]__initconst = { +0xd7 ,0xfd ,0x9b ,0xbd ,0x8f ,0x65 ,0x0d ,0x00 , +0xca ,0xa1 ,0x6c ,0x85 ,0x85 ,0xa4 ,0x6d ,0xf1 , +0xb1 ,0x68 ,0x0c ,0x8b ,0x5d ,0x37 ,0x72 ,0xd0 , +0xd8 ,0xd2 ,0x25 ,0xab ,0x9f ,0x7b ,0x7d ,0x95 +}; +enum { nonce61 =0xd82caf72a9c4864fULL }; + +static const u8 input62[]__initconst = { +0x10 ,0x77 ,0xf3 ,0x2f ,0xc2 ,0x50 ,0xd6 ,0x0c , +0xba ,0xa8 ,0x8d ,0xce ,0x0d ,0x58 ,0x9e ,0x87 , +0xb1 ,0x59 ,0x66 ,0x0a ,0x4a ,0xb3 ,0xd8 ,0xca , +0x0a ,0x6b ,0xf8 ,0xc6 ,0x2b ,0x3f ,0x8e ,0x09 , +0xe0 ,0x0a ,0x15 ,0x85 ,0xfe ,0xaa ,0xc6 ,0xbd , +0x30 ,0xef ,0xe4 ,0x10 ,0x78 ,0x03 ,0xc1 ,0xc7 , +0x8a ,0xd9 ,0xde ,0x0b ,0x51 ,0x07 ,0xc4 ,0x7b , +0xe2 ,0x2e ,0x36 ,0x3a ,0xc2 +}; +static const u8 output62[]__initconst = { +0xa0 ,0x0c ,0xfc ,0xc1 ,0xf6 ,0xaf ,0xc2 ,0xb8 , +0x5c ,0xef ,0x6e ,0xf3 ,0xce ,0x15 ,0x48 ,0x05 , +0xb5 ,0x78 ,0x49 ,0x51 ,0x1f ,0x9d ,0xf4 ,0xbf , +0x2f ,0x53 ,0xa2 ,0xd1 ,0x15 ,0x20 ,0x82 ,0x6b , +0xd2 ,0x22 ,0x6c ,0x4e ,0x14 ,0x87 ,0xe3 ,0xd7 , +0x49 ,0x45 ,0x84 ,0xdb ,0x5f ,0x68 ,0x60 ,0xc4 , +0xb3 ,0xe6 ,0x3f ,0xd1 ,0xfc ,0xa5 ,0x73 ,0xf3 , +0xfc ,0xbb ,0xbe ,0xc8 ,0x9d +}; +static const u8 key62[]__initconst = { +0x6e ,0xc9 ,0xaf ,0xce ,0x35 ,0xb9 ,0x86 ,0xd1 , +0xce ,0x5f ,0xd9 ,0xbb ,0xd5 ,0x1f ,0x7c ,0xcd , +0xfe ,0x19 ,0xaa ,0x3d ,0xea ,0x64 ,0xc1 ,0x28 , +0x40 ,0xba ,0xa1 ,0x28 ,0xcd ,0x40 ,0xb6 ,0xf2 +}; +enum { nonce62 =0xa1c0c265f900cde8ULL }; + +static const u8 input63[]__initconst = { +0x7a ,0x70 ,0x21 ,0x2c ,0xef ,0xa6 ,0x36 ,0xd4 , +0xe0 ,0xab ,0x8c ,0x25 ,0x73 ,0x34 ,0xc8 ,0x94 , +0x6c ,0x81 ,0xcb ,0x19 ,0x8d ,0x5a ,0x49 ,0xaa , +0x6f ,0xba ,0x83 ,0x72 ,0x02 ,0x5e ,0xf5 ,0x89 , +0xce ,0x79 ,0x7e ,0x13 ,0x3d ,0x5b ,0x98 ,0x60 , +0x5d ,0xd9 ,0xfb ,0x15 ,0x93 ,0x4c ,0xf3 ,0x51 , +0x49 ,0x55 ,0xd1 ,0x58 ,0xdd ,0x7e ,0x6d ,0xfe , +0xdd ,0x84 ,0x23 ,0x05 ,0xba ,0xe9 +}; +static const u8 output63[]__initconst = { +0x20 ,0xb3 ,0x5c ,0x03 ,0x03 ,0x78 ,0x17 ,0xfc , +0x3b ,0x35 ,0x30 ,0x9a ,0x00 ,0x18 ,0xf5 ,0xc5 , +0x06 ,0x53 ,0xf5 ,0x04 ,0x24 ,0x9d ,0xd1 ,0xb2 , +0xac ,0x5a ,0xb6 ,0x2a ,0xa5 ,0xda ,0x50 ,0x00 , +0xec ,0xff ,0xa0 ,0x7a ,0x14 ,0x7b ,0xe4 ,0x6b , +0x63 ,0xe8 ,0x66 ,0x86 ,0x34 ,0xfd ,0x74 ,0x44 , +0xa2 ,0x50 ,0x97 ,0x0d ,0xdc ,0xc3 ,0x84 ,0xf8 , +0x71 ,0x02 ,0x31 ,0x95 ,0xed ,0x54 +}; +static const u8 key63[]__initconst = { +0x7d ,0x64 ,0xb4 ,0x12 ,0x81 ,0xe4 ,0xe6 ,0x8f , +0xcc ,0xe7 ,0xd1 ,0x1f ,0x70 ,0x20 ,0xfd ,0xb8 , +0x3a ,0x7d ,0xa6 ,0x53 ,0x65 ,0x30 ,0x5d ,0xe3 , +0x1a ,0x44 ,0xbe ,0x62 ,0xed ,0x90 ,0xc4 ,0xd1 +}; +enum { nonce63 =0xe8e849596c942276ULL }; + +static const u8 input64[]__initconst = { +0x84 ,0xf8 ,0xda ,0x87 ,0x23 ,0x39 ,0x60 ,0xcf , +0xc5 ,0x50 ,0x7e ,0xc5 ,0x47 ,0x29 ,0x7c ,0x05 , +0xc2 ,0xb4 ,0xf4 ,0xb2 ,0xec ,0x5d ,0x48 ,0x36 , +0xbf ,0xfc ,0x06 ,0x8c ,0xf2 ,0x0e ,0x88 ,0xe7 , +0xc9 ,0xc5 ,0xa4 ,0xa2 ,0x83 ,0x20 ,0xa1 ,0x6f , +0x37 ,0xe5 ,0x2d ,0xa1 ,0x72 ,0xa1 ,0x19 ,0xef , +0x05 ,0x42 ,0x08 ,0xf2 ,0x57 ,0x47 ,0x31 ,0x1e , +0x17 ,0x76 ,0x13 ,0xd3 ,0xcc ,0x75 ,0x2c +}; +static const u8 output64[]__initconst = { +0xcb ,0xec ,0x90 ,0x88 ,0xeb ,0x31 ,0x69 ,0x20 , +0xa6 ,0xdc ,0xff ,0x76 ,0x98 ,0xb0 ,0x24 ,0x49 , +0x7b ,0x20 ,0xd9 ,0xd1 ,0x1b ,0xe3 ,0x61 ,0xdc , +0xcf ,0x51 ,0xf6 ,0x70 ,0x72 ,0x33 ,0x28 ,0x94 , +0xac ,0x73 ,0x18 ,0xcf ,0x93 ,0xfd ,0xca ,0x08 , +0x0d ,0xa2 ,0xb9 ,0x57 ,0x1e ,0x51 ,0xb6 ,0x07 , +0x5c ,0xc1 ,0x13 ,0x64 ,0x1d ,0x18 ,0x6f ,0xe6 , +0x0b ,0xb7 ,0x14 ,0x03 ,0x43 ,0xb6 ,0xaf +}; +static const u8 key64[]__initconst = { +0xbf ,0x82 ,0x65 ,0xe4 ,0x50 ,0xf9 ,0x5e ,0xea , +0x28 ,0x91 ,0xd1 ,0xd2 ,0x17 ,0x7c ,0x13 ,0x7e , +0xf5 ,0xd5 ,0x6b ,0x06 ,0x1c ,0x20 ,0xc2 ,0x82 , +0xa1 ,0x7a ,0xa2 ,0x14 ,0xa1 ,0xb0 ,0x54 ,0x58 +}; +enum { nonce64 =0xe57c5095aa5723c9ULL }; + +static const u8 input65[]__initconst = { +0x1c ,0xfb ,0xd3 ,0x3f ,0x85 ,0xd7 ,0xba ,0x7b , +0xae ,0xb1 ,0xa5 ,0xd2 ,0xe5 ,0x40 ,0xce ,0x4d , +0x3e ,0xab ,0x17 ,0x9d ,0x7d ,0x9f ,0x03 ,0x98 , +0x3f ,0x9f ,0xc8 ,0xdd ,0x36 ,0x17 ,0x43 ,0x5c , +0x34 ,0xd1 ,0x23 ,0xe0 ,0x77 ,0xbf ,0x35 ,0x5d , +0x8f ,0xb1 ,0xcb ,0x82 ,0xbb ,0x39 ,0x69 ,0xd8 , +0x90 ,0x45 ,0x37 ,0xfd ,0x98 ,0x25 ,0xf7 ,0x5b , +0xce ,0x06 ,0x43 ,0xba ,0x61 ,0xa8 ,0x47 ,0xb9 +}; +static const u8 output65[]__initconst = { +0x73 ,0xa5 ,0x68 ,0xab ,0x8b ,0xa5 ,0xc3 ,0x7e , +0x74 ,0xf8 ,0x9d ,0xf5 ,0x93 ,0x6e ,0xf2 ,0x71 , +0x6d ,0xde ,0x82 ,0xc5 ,0x40 ,0xa0 ,0x46 ,0xb3 , +0x9a ,0x78 ,0xa8 ,0xf7 ,0xdf ,0xb1 ,0xc3 ,0xdd , +0x8d ,0x90 ,0x00 ,0x68 ,0x21 ,0x48 ,0xe8 ,0xba , +0x56 ,0x9f ,0x8f ,0xe7 ,0xa4 ,0x4d ,0x36 ,0x55 , +0xd0 ,0x34 ,0x99 ,0xa6 ,0x1c ,0x4c ,0xc1 ,0xe2 , +0x65 ,0x98 ,0x14 ,0x8e ,0x6a ,0x05 ,0xb1 ,0x2b +}; +static const u8 key65[]__initconst = { +0xbd ,0x5c ,0x8a ,0xb0 ,0x11 ,0x29 ,0xf3 ,0x00 , +0x7a ,0x78 ,0x32 ,0x63 ,0x34 ,0x00 ,0xe6 ,0x7d , +0x30 ,0x54 ,0xde ,0x37 ,0xda ,0xc2 ,0xc4 ,0x3d , +0x92 ,0x6b ,0x4c ,0xc2 ,0x92 ,0xe9 ,0x9e ,0x2a +}; +enum { nonce65 =0xf654a3031de746f2ULL }; + +static const u8 input66[]__initconst = { +0x4b ,0x27 ,0x30 ,0x8f ,0x28 ,0xd8 ,0x60 ,0x46 , +0x39 ,0x06 ,0x49 ,0xea ,0x1b ,0x71 ,0x26 ,0xe0 , +0x99 ,0x2b ,0xd4 ,0x8f ,0x64 ,0x64 ,0xcd ,0xac , +0x1d ,0x78 ,0x88 ,0x90 ,0xe1 ,0x5c ,0x24 ,0x4b , +0xdc ,0x2d ,0xb7 ,0xee ,0x3a ,0xe6 ,0x86 ,0x2c , +0x21 ,0xe4 ,0x2b ,0xfc ,0xe8 ,0x19 ,0xca ,0x65 , +0xe7 ,0xdd ,0x6f ,0x52 ,0xb3 ,0x11 ,0xe1 ,0xe2 , +0xbf ,0xe8 ,0x70 ,0xe3 ,0x0d ,0x45 ,0xb8 ,0xa5 , +0x20 ,0xb7 ,0xb5 ,0xaf ,0xff ,0x08 ,0xcf ,0x23 , +0x65 ,0xdf ,0x8d ,0xc3 ,0x31 ,0xf3 ,0x1e ,0x6a , +0x58 ,0x8d ,0xcc ,0x45 ,0x16 ,0x86 ,0x1f ,0x31 , +0x5c ,0x27 ,0xcd ,0xc8 ,0x6b ,0x19 ,0x1e ,0xec , +0x44 ,0x75 ,0x63 ,0x97 ,0xfd ,0x79 ,0xf6 ,0x62 , +0xc5 ,0xba ,0x17 ,0xc7 ,0xab ,0x8f ,0xbb ,0xed , +0x85 ,0x2a ,0x98 ,0x79 ,0x21 ,0xec ,0x6e ,0x4d , +0xdc ,0xfa ,0x72 ,0x52 ,0xba ,0xc8 ,0x4c +}; +static const u8 output66[]__initconst = { +0x76 ,0x5b ,0x2c ,0xa7 ,0x62 ,0xb9 ,0x08 ,0x4a , +0xc6 ,0x4a ,0x92 ,0xc3 ,0xbb ,0x10 ,0xb3 ,0xee , +0xff ,0xb9 ,0x07 ,0xc7 ,0x27 ,0xcb ,0x1e ,0xcf , +0x58 ,0x6f ,0xa1 ,0x64 ,0xe8 ,0xf1 ,0x4e ,0xe1 , +0xef ,0x18 ,0x96 ,0xab ,0x97 ,0x28 ,0xd1 ,0x7c , +0x71 ,0x6c ,0xd1 ,0xe2 ,0xfa ,0xd9 ,0x75 ,0xcb , +0xeb ,0xea ,0x0c ,0x86 ,0x82 ,0xd8 ,0xf4 ,0xcc , +0xea ,0xa3 ,0x00 ,0xfa ,0x82 ,0xd2 ,0xcd ,0xcb , +0xdb ,0x63 ,0x28 ,0xe2 ,0x82 ,0xe9 ,0x01 ,0xed , +0x31 ,0xe6 ,0x71 ,0x45 ,0x08 ,0x89 ,0x8a ,0x23 , +0xa8 ,0xb5 ,0xc2 ,0xe2 ,0x9f ,0xe9 ,0xb8 ,0x9a , +0xc4 ,0x79 ,0x6d ,0x71 ,0x52 ,0x61 ,0x74 ,0x6c , +0x1b ,0xd7 ,0x65 ,0x6d ,0x03 ,0xc4 ,0x1a ,0xc0 , +0x50 ,0xba ,0xd6 ,0xc9 ,0x43 ,0x50 ,0xbe ,0x09 , +0x09 ,0x8a ,0xdb ,0xaa ,0x76 ,0x4e ,0x3b ,0x61 , +0x3c ,0x7c ,0x44 ,0xe7 ,0xdb ,0x10 ,0xa7 +}; +static const u8 key66[]__initconst = { +0x88 ,0xdf ,0xca ,0x68 ,0xaf ,0x4f ,0xb3 ,0xfd , +0x6e ,0xa7 ,0x95 ,0x35 ,0x8a ,0xe8 ,0x37 ,0xe8 , +0xc8 ,0x55 ,0xa2 ,0x2a ,0x6d ,0x77 ,0xf8 ,0x93 , +0x7a ,0x41 ,0xf3 ,0x7b ,0x95 ,0xdf ,0x89 ,0xf5 +}; +enum { nonce66 =0x1024b4fdd415cf82ULL }; + +static const u8 input67[]__initconst = { +0xd4 ,0x2e ,0xfa ,0x92 ,0xe9 ,0x29 ,0x68 ,0xb7 , +0x54 ,0x2c ,0xf7 ,0xa4 ,0x2d ,0xb7 ,0x50 ,0xb5 , +0xc5 ,0xb2 ,0x9d ,0x17 ,0x5e ,0x0a ,0xca ,0x37 , +0xbf ,0x60 ,0xae ,0xd2 ,0x98 ,0xe9 ,0xfa ,0x59 , +0x67 ,0x62 ,0xe6 ,0x43 ,0x0c ,0x77 ,0x80 ,0x82 , +0x33 ,0x61 ,0xa3 ,0xff ,0xc1 ,0xa0 ,0x8f ,0x56 , +0xbc ,0xec ,0x65 ,0x43 ,0x88 ,0xa5 ,0xff ,0x51 , +0x64 ,0x30 ,0xee ,0x34 ,0xb7 ,0x5c ,0x28 ,0x68 , +0xc3 ,0x52 ,0xd2 ,0xac ,0x78 ,0x2a ,0xa6 ,0x10 , +0xb8 ,0xb2 ,0x4c ,0x80 ,0x4f ,0x99 ,0xb2 ,0x36 , +0x94 ,0x8f ,0x66 ,0xcb ,0xa1 ,0x91 ,0xed ,0x06 , +0x42 ,0x6d ,0xc1 ,0xae ,0x55 ,0x93 ,0xdd ,0x93 , +0x9e ,0x88 ,0x34 ,0x7f ,0x98 ,0xeb ,0xbe ,0x61 , +0xf9 ,0xa9 ,0x0f ,0xd9 ,0xc4 ,0x87 ,0xd5 ,0xef , +0xcc ,0x71 ,0x8c ,0x0e ,0xce ,0xad ,0x02 ,0xcf , +0xa2 ,0x61 ,0xdf ,0xb1 ,0xfe ,0x3b ,0xdc ,0xc0 , +0x58 ,0xb5 ,0x71 ,0xa1 ,0x83 ,0xc9 ,0xb4 ,0xaf , +0x9d ,0x54 ,0x12 ,0xcd ,0xea ,0x06 ,0xd6 ,0x4e , +0xe5 ,0x27 ,0x0c ,0xc3 ,0xbb ,0xa8 ,0x0a ,0x81 , +0x75 ,0xc3 ,0xc9 ,0xd4 ,0x35 ,0x3e ,0x53 ,0x9f , +0xaa ,0x20 ,0xc0 ,0x68 ,0x39 ,0x2c ,0x96 ,0x39 , +0x53 ,0x81 ,0xda ,0x07 ,0x0f ,0x44 ,0xa5 ,0x47 , +0x0e ,0xb3 ,0x87 ,0x0d ,0x1b ,0xc1 ,0xe5 ,0x41 , +0x35 ,0x12 ,0x58 ,0x96 ,0x69 ,0x8a ,0x1a ,0xa3 , +0x9d ,0x3d ,0xd4 ,0xb1 ,0x8e ,0x1f ,0x96 ,0x87 , +0xda ,0xd3 ,0x19 ,0xe2 ,0xb1 ,0x3a ,0x19 ,0x74 , +0xa0 ,0x00 ,0x9f ,0x4d ,0xbc ,0xcb ,0x0c ,0xe9 , +0xec ,0x10 ,0xdf ,0x2a ,0x88 ,0xdc ,0x30 ,0x51 , +0x46 ,0x56 ,0x53 ,0x98 ,0x6a ,0x26 ,0x14 ,0x05 , +0x54 ,0x81 ,0x55 ,0x0b ,0x3c ,0x85 ,0xdd ,0x33 , +0x81 ,0x11 ,0x29 ,0x82 ,0x46 ,0x35 ,0xe1 ,0xdb , +0x59 ,0x7b +}; +static const u8 output67[]__initconst = { +0x64 ,0x6c ,0xda ,0x7f ,0xd4 ,0xa9 ,0x2a ,0x5e , +0x22 ,0xae ,0x8d ,0x67 ,0xdb ,0xee ,0xfd ,0xd0 , +0x44 ,0x80 ,0x17 ,0xb2 ,0xe3 ,0x87 ,0xad ,0x57 , +0x15 ,0xcb ,0x88 ,0x64 ,0xc0 ,0xf1 ,0x49 ,0x3d , +0xfa ,0xbe ,0xa8 ,0x9f ,0x12 ,0xc3 ,0x57 ,0x56 , +0x70 ,0xa5 ,0xc5 ,0x6b ,0xf1 ,0xab ,0xd5 ,0xde , +0x77 ,0x92 ,0x6a ,0x56 ,0x03 ,0xf5 ,0x21 ,0x0d , +0xb6 ,0xc4 ,0xcc ,0x62 ,0x44 ,0x3f ,0xb1 ,0xc1 , +0x61 ,0x41 ,0x90 ,0xb2 ,0xd5 ,0xb8 ,0xf3 ,0x57 , +0xfb ,0xc2 ,0x6b ,0x25 ,0x58 ,0xc8 ,0x45 ,0x20 , +0x72 ,0x29 ,0x6f ,0x9d ,0xb5 ,0x81 ,0x4d ,0x2b , +0xb2 ,0x89 ,0x9e ,0x91 ,0x53 ,0x97 ,0x1c ,0xd9 , +0x3d ,0x79 ,0xdc ,0x14 ,0xae ,0x01 ,0x73 ,0x75 , +0xf0 ,0xca ,0xd5 ,0xab ,0x62 ,0x5c ,0x7a ,0x7d , +0x3f ,0xfe ,0x22 ,0x7d ,0xee ,0xe2 ,0xcb ,0x76 , +0x55 ,0xec ,0x06 ,0xdd ,0x41 ,0x47 ,0x18 ,0x62 , +0x1d ,0x57 ,0xd0 ,0xd6 ,0xb6 ,0x0f ,0x4b ,0xfc , +0x79 ,0x19 ,0xf4 ,0xd6 ,0x37 ,0x86 ,0x18 ,0x1f , +0x98 ,0x0d ,0x9e ,0x15 ,0x2d ,0xb6 ,0x9a ,0x8a , +0x8c ,0x80 ,0x22 ,0x2f ,0x82 ,0xc4 ,0xc7 ,0x36 , +0xfa ,0xfa ,0x07 ,0xbd ,0xc2 ,0x2a ,0xe2 ,0xea , +0x93 ,0xc8 ,0xb2 ,0x90 ,0x33 ,0xf2 ,0xee ,0x4b , +0x1b ,0xf4 ,0x37 ,0x92 ,0x13 ,0xbb ,0xe2 ,0xce , +0xe3 ,0x03 ,0xcf ,0x07 ,0x94 ,0xab ,0x9a ,0xc9 , +0xff ,0x83 ,0x69 ,0x3a ,0xda ,0x2c ,0xd0 ,0x47 , +0x3d ,0x6c ,0x1a ,0x60 ,0x68 ,0x47 ,0xb9 ,0x36 , +0x52 ,0xdd ,0x16 ,0xef ,0x6c ,0xbf ,0x54 ,0x11 , +0x72 ,0x62 ,0xce ,0x8c ,0x9d ,0x90 ,0xa0 ,0x25 , +0x06 ,0x92 ,0x3e ,0x12 ,0x7e ,0x1a ,0x1d ,0xe5 , +0xa2 ,0x71 ,0xce ,0x1c ,0x4c ,0x6a ,0x7c ,0xdc , +0x3d ,0xe3 ,0x6e ,0x48 ,0x9d ,0xb3 ,0x64 ,0x7d , +0x78 ,0x40 +}; +static const u8 key67[]__initconst = { +0xa9 ,0x20 ,0x75 ,0x89 ,0x7e ,0x37 ,0x85 ,0x48 , +0xa3 ,0xfb ,0x7b ,0xe8 ,0x30 ,0xa7 ,0xe3 ,0x6e , +0xa6 ,0xc1 ,0x71 ,0x17 ,0xc1 ,0x6c ,0x9b ,0xc2 , +0xde ,0xf0 ,0xa7 ,0x19 ,0xec ,0xce ,0xc6 ,0x53 +}; +enum { nonce67 =0x4adc4d1f968c8a10ULL }; + +static const u8 input68[]__initconst = { +0x99 ,0xae ,0x72 ,0xfb ,0x16 ,0xe1 ,0xf1 ,0x59 , +0x43 ,0x15 ,0x4e ,0x33 ,0xa0 ,0x95 ,0xe7 ,0x6c , +0x74 ,0x24 ,0x31 ,0xca ,0x3b ,0x2e ,0xeb ,0xd7 , +0x11 ,0xd8 ,0xe0 ,0x56 ,0x92 ,0x91 ,0x61 ,0x57 , +0xe2 ,0x82 ,0x9f ,0x8f ,0x37 ,0xf5 ,0x3d ,0x24 , +0x92 ,0x9d ,0x87 ,0x00 ,0x8d ,0x89 ,0xe0 ,0x25 , +0x8b ,0xe4 ,0x20 ,0x5b ,0x8a ,0x26 ,0x2c ,0x61 , +0x78 ,0xb0 ,0xa6 ,0x3e ,0x82 ,0x18 ,0xcf ,0xdc , +0x2d ,0x24 ,0xdd ,0x81 ,0x42 ,0xc4 ,0x95 ,0xf0 , +0x48 ,0x60 ,0x71 ,0xe3 ,0xe3 ,0xac ,0xec ,0xbe , +0x98 ,0x6b ,0x0c ,0xb5 ,0x6a ,0xa9 ,0xc8 ,0x79 , +0x23 ,0x2e ,0x38 ,0x0b ,0x72 ,0x88 ,0x8c ,0xe7 , +0x71 ,0x8b ,0x36 ,0xe3 ,0x58 ,0x3d ,0x9c ,0xa0 , +0xa2 ,0xea ,0xcf ,0x0c ,0x6a ,0x6c ,0x64 ,0xdf , +0x97 ,0x21 ,0x8f ,0x93 ,0xfb ,0xba ,0xf3 ,0x5a , +0xd7 ,0x8f ,0xa6 ,0x37 ,0x15 ,0x50 ,0x43 ,0x02 , +0x46 ,0x7f ,0x93 ,0x46 ,0x86 ,0x31 ,0xe2 ,0xaa , +0x24 ,0xa8 ,0x26 ,0xae ,0xe6 ,0xc0 ,0x05 ,0x73 , +0x0b ,0x4f ,0x7e ,0xed ,0x65 ,0xeb ,0x56 ,0x1e , +0xb6 ,0xb3 ,0x0b ,0xc3 ,0x0e ,0x31 ,0x95 ,0xa9 , +0x18 ,0x4d ,0xaf ,0x38 ,0xd7 ,0xec ,0xc6 ,0x44 , +0x72 ,0x77 ,0x4e ,0x25 ,0x4b ,0x25 ,0xdd ,0x1e , +0x8c ,0xa2 ,0xdf ,0xf6 ,0x2a ,0x97 ,0x1a ,0x88 , +0x2c ,0x8a ,0x5d ,0xfe ,0xe8 ,0xfb ,0x35 ,0xe8 , +0x0f ,0x2b ,0x7a ,0x18 ,0x69 ,0x43 ,0x31 ,0x1d , +0x38 ,0x6a ,0x62 ,0x95 ,0x0f ,0x20 ,0x4b ,0xbb , +0x97 ,0x3c ,0xe0 ,0x64 ,0x2f ,0x52 ,0xc9 ,0x2d , +0x4d ,0x9d ,0x54 ,0x04 ,0x3d ,0xc9 ,0xea ,0xeb , +0xd0 ,0x86 ,0x52 ,0xff ,0x42 ,0xe1 ,0x0d ,0x7a , +0xad ,0x88 ,0xf9 ,0x9b ,0x1e ,0x5e ,0x12 ,0x27 , +0x95 ,0x3e ,0x0c ,0x2c ,0x13 ,0x00 ,0x6f ,0x8e , +0x93 ,0x69 ,0x0e ,0x01 ,0x8c ,0xc1 ,0xfd ,0xb3 +}; +static const u8 output68[]__initconst = { +0x26 ,0x3e ,0xf2 ,0xb1 ,0xf5 ,0xef ,0x81 ,0xa4 , +0xb7 ,0x42 ,0xd4 ,0x26 ,0x18 ,0x4b ,0xdd ,0x6a , +0x47 ,0x15 ,0xcb ,0x0e ,0x57 ,0xdb ,0xa7 ,0x29 , +0x7e ,0x7b ,0x3f ,0x47 ,0x89 ,0x57 ,0xab ,0xea , +0x14 ,0x7b ,0xcf ,0x37 ,0xdb ,0x1c ,0xe1 ,0x11 , +0x77 ,0xae ,0x2e ,0x4c ,0xd2 ,0x08 ,0x3f ,0xa6 , +0x62 ,0x86 ,0xa6 ,0xb2 ,0x07 ,0xd5 ,0x3f ,0x9b , +0xdc ,0xc8 ,0x50 ,0x4b ,0x7b ,0xb9 ,0x06 ,0xe6 , +0xeb ,0xac ,0x98 ,0x8c ,0x36 ,0x0c ,0x1e ,0xb2 , +0xc8 ,0xfb ,0x24 ,0x60 ,0x2c ,0x08 ,0x17 ,0x26 , +0x5b ,0xc8 ,0xc2 ,0xdf ,0x9c ,0x73 ,0x67 ,0x4a , +0xdb ,0xcf ,0xd5 ,0x2c ,0x2b ,0xca ,0x24 ,0xcc , +0xdb ,0xc9 ,0xa8 ,0xf2 ,0x5d ,0x67 ,0xdf ,0x5c , +0x62 ,0x0b ,0x58 ,0xc0 ,0x83 ,0xde ,0x8b ,0xf6 , +0x15 ,0x0a ,0xd6 ,0x32 ,0xd8 ,0xf5 ,0xf2 ,0x5f , +0x33 ,0xce ,0x7e ,0xab ,0x76 ,0xcd ,0x14 ,0x91 , +0xd8 ,0x41 ,0x90 ,0x93 ,0xa1 ,0xaf ,0xf3 ,0x45 , +0x6c ,0x1b ,0x25 ,0xbd ,0x48 ,0x51 ,0x6d ,0x15 , +0x47 ,0xe6 ,0x23 ,0x50 ,0x32 ,0x69 ,0x1e ,0xb5 , +0x94 ,0xd3 ,0x97 ,0xba ,0xd7 ,0x37 ,0x4a ,0xba , +0xb9 ,0xcd ,0xfb ,0x96 ,0x9a ,0x90 ,0xe0 ,0x37 , +0xf8 ,0xdf ,0x91 ,0x6c ,0x62 ,0x13 ,0x19 ,0x21 , +0x4b ,0xa9 ,0xf1 ,0x12 ,0x66 ,0xe2 ,0x74 ,0xd7 , +0x81 ,0xa0 ,0x74 ,0x8d ,0x7e ,0x7e ,0xc9 ,0xb1 , +0x69 ,0x8f ,0xed ,0xb3 ,0xf6 ,0x97 ,0xcd ,0x72 , +0x78 ,0x93 ,0xd3 ,0x54 ,0x6b ,0x43 ,0xac ,0x29 , +0xb4 ,0xbc ,0x7d ,0xa4 ,0x26 ,0x4b ,0x7b ,0xab , +0xd6 ,0x67 ,0x22 ,0xff ,0x03 ,0x92 ,0xb6 ,0xd4 , +0x96 ,0x94 ,0x5a ,0xe5 ,0x02 ,0x35 ,0x77 ,0xfa , +0x3f ,0x54 ,0x1d ,0xdd ,0x35 ,0x39 ,0xfe ,0x03 , +0xdd ,0x8e ,0x3c ,0x8c ,0xc2 ,0x69 ,0x2a ,0xb1 , +0xb7 ,0xb3 ,0xa1 ,0x89 ,0x84 ,0xea ,0x16 ,0xe2 +}; +static const u8 key68[]__initconst = { +0xd2 ,0x49 ,0x7f ,0xd7 ,0x49 ,0x66 ,0x0d ,0xb3 , +0x5a ,0x7e ,0x3c ,0xfc ,0x37 ,0x83 ,0x0e ,0xf7 , +0x96 ,0xd8 ,0xd6 ,0x33 ,0x79 ,0x2b ,0x84 ,0x53 , +0x06 ,0xbc ,0x6c ,0x0a ,0x55 ,0x84 ,0xfe ,0xab +}; +enum { nonce68 =0x6a6df7ff0a20de06ULL }; + +static const u8 input69[]__initconst = { +0xf9 ,0x18 ,0x4c ,0xd2 ,0x3f ,0xf7 ,0x22 ,0xd9 , +0x58 ,0xb6 ,0x3b ,0x38 ,0x69 ,0x79 ,0xf4 ,0x71 , +0x5f ,0x38 ,0x52 ,0x1f ,0x17 ,0x6f ,0x6f ,0xd9 , +0x09 ,0x2b ,0xfb ,0x67 ,0xdc ,0xc9 ,0xe8 ,0x4a , +0x70 ,0x9f ,0x2e ,0x3c ,0x06 ,0xe5 ,0x12 ,0x20 , +0x25 ,0x29 ,0xd0 ,0xdc ,0x81 ,0xc5 ,0xc6 ,0x0f , +0xd2 ,0xa8 ,0x81 ,0x15 ,0x98 ,0xb2 ,0x71 ,0x5a , +0x9a ,0xe9 ,0xfb ,0xaf ,0x0e ,0x5f ,0x8a ,0xf3 , +0x16 ,0x4a ,0x47 ,0xf2 ,0x5c ,0xbf ,0xda ,0x52 , +0x9a ,0xa6 ,0x36 ,0xfd ,0xc6 ,0xf7 ,0x66 ,0x00 , +0xcc ,0x6c ,0xd4 ,0xb3 ,0x07 ,0x6d ,0xeb ,0xfe , +0x92 ,0x71 ,0x25 ,0xd0 ,0xcf ,0x9c ,0xe8 ,0x65 , +0x45 ,0x10 ,0xcf ,0x62 ,0x74 ,0x7d ,0xf2 ,0x1b , +0x57 ,0xa0 ,0xf1 ,0x6b ,0xa4 ,0xd5 ,0xfa ,0x12 , +0x27 ,0x5a ,0xf7 ,0x99 ,0xfc ,0xca ,0xf3 ,0xb8 , +0x2c ,0x8b ,0xba ,0x28 ,0x74 ,0xde ,0x8f ,0x78 , +0xa2 ,0x8c ,0xaf ,0x89 ,0x4b ,0x05 ,0xe2 ,0xf3 , +0xf8 ,0xd2 ,0xef ,0xac ,0xa4 ,0xc4 ,0xe2 ,0xe2 , +0x36 ,0xbb ,0x5e ,0xae ,0xe6 ,0x87 ,0x3d ,0x88 , +0x9f ,0xb8 ,0x11 ,0xbb ,0xcf ,0x57 ,0xce ,0xd0 , +0xba ,0x62 ,0xf4 ,0xf8 ,0x9b ,0x95 ,0x04 ,0xc9 , +0xcf ,0x01 ,0xe9 ,0xf1 ,0xc8 ,0xc6 ,0x22 ,0xa4 , +0xf2 ,0x8b ,0x2f ,0x24 ,0x0a ,0xf5 ,0x6e ,0xb7 , +0xd4 ,0x2c ,0xb6 ,0xf7 ,0x5c ,0x97 ,0x61 ,0x0b , +0xd9 ,0xb5 ,0x06 ,0xcd ,0xed ,0x3e ,0x1f ,0xc5 , +0xb2 ,0x6c ,0xa3 ,0xea ,0xb8 ,0xad ,0xa6 ,0x42 , +0x88 ,0x7a ,0x52 ,0xd5 ,0x64 ,0xba ,0xb5 ,0x20 , +0x10 ,0xa0 ,0x0f ,0x0d ,0xea ,0xef ,0x5a ,0x9b , +0x27 ,0xb8 ,0xca ,0x20 ,0x19 ,0x6d ,0xa8 ,0xc4 , +0x46 ,0x04 ,0xb3 ,0xe8 ,0xf8 ,0x66 ,0x1b ,0x0a , +0xce ,0x76 ,0x5d ,0x59 ,0x58 ,0x05 ,0xee ,0x3e , +0x3c ,0x86 ,0x5b ,0x49 ,0x1c ,0x72 ,0x18 ,0x01 , +0x62 ,0x92 ,0x0f ,0x3e ,0xd1 ,0x57 ,0x5e ,0x20 , +0x7b ,0xfb ,0x4d ,0x3c ,0xc5 ,0x35 ,0x43 ,0x2f , +0xb0 ,0xc5 ,0x7c ,0xe4 ,0xa2 ,0x84 ,0x13 ,0x77 +}; +static const u8 output69[]__initconst = { +0xbb ,0x4a ,0x7f ,0x7c ,0xd5 ,0x2f ,0x89 ,0x06 , +0xec ,0x20 ,0xf1 ,0x9a ,0x11 ,0x09 ,0x14 ,0x2e , +0x17 ,0x50 ,0xf9 ,0xd5 ,0xf5 ,0x48 ,0x7c ,0x7a , +0x55 ,0xc0 ,0x57 ,0x03 ,0xe3 ,0xc4 ,0xb2 ,0xb7 , +0x18 ,0x47 ,0x95 ,0xde ,0xaf ,0x80 ,0x06 ,0x3c , +0x5a ,0xf2 ,0xc3 ,0x53 ,0xe3 ,0x29 ,0x92 ,0xf8 , +0xff ,0x64 ,0x85 ,0xb9 ,0xf7 ,0xd3 ,0x80 ,0xd2 , +0x0c ,0x5d ,0x7b ,0x57 ,0x0c ,0x51 ,0x79 ,0x86 , +0xf3 ,0x20 ,0xd2 ,0xb8 ,0x6e ,0x0c ,0x5a ,0xce , +0xeb ,0x88 ,0x02 ,0x8b ,0x82 ,0x1b ,0x7f ,0xf5 , +0xde ,0x7f ,0x48 ,0x48 ,0xdf ,0xa0 ,0x55 ,0xc6 , +0x0c ,0x22 ,0xa1 ,0x80 ,0x8d ,0x3b ,0xcb ,0x40 , +0x2d ,0x3d ,0x0b ,0xf2 ,0xe0 ,0x22 ,0x13 ,0x99 , +0xe1 ,0xa7 ,0x27 ,0x68 ,0x31 ,0xe1 ,0x24 ,0x5d , +0xd2 ,0xee ,0x16 ,0xc1 ,0xd7 ,0xa8 ,0x14 ,0x19 , +0x23 ,0x72 ,0x67 ,0x27 ,0xdc ,0x5e ,0xb9 ,0xc7 , +0xd8 ,0xe3 ,0x55 ,0x50 ,0x40 ,0x98 ,0x7b ,0xe7 , +0x34 ,0x1c ,0x3b ,0x18 ,0x14 ,0xd8 ,0x62 ,0xc1 , +0x93 ,0x84 ,0xf3 ,0x5b ,0xdd ,0x9e ,0x1f ,0x3b , +0x0b ,0xbc ,0x4e ,0x5b ,0x79 ,0xa3 ,0xca ,0x74 , +0x2a ,0x98 ,0xe8 ,0x04 ,0x39 ,0xef ,0xc6 ,0x76 , +0x6d ,0xee ,0x9f ,0x67 ,0x5b ,0x59 ,0x3a ,0xe5 , +0xf2 ,0x3b ,0xca ,0x89 ,0xe8 ,0x9b ,0x03 ,0x3d , +0x11 ,0xd2 ,0x4a ,0x70 ,0xaf ,0x88 ,0xb0 ,0x94 , +0x96 ,0x26 ,0xab ,0x3c ,0xc1 ,0xb8 ,0xe4 ,0xe7 , +0x14 ,0x61 ,0x64 ,0x3a ,0x61 ,0x08 ,0x0f ,0xa9 , +0xce ,0x64 ,0xb2 ,0x40 ,0xf8 ,0x20 ,0x3a ,0xa9 , +0x31 ,0xbd ,0x7e ,0x16 ,0xca ,0xf5 ,0x62 ,0x0f , +0x91 ,0x9f ,0x8e ,0x1d ,0xa4 ,0x77 ,0xf3 ,0x87 , +0x61 ,0xe8 ,0x14 ,0xde ,0x18 ,0x68 ,0x4e ,0x9d , +0x73 ,0xcd ,0x8a ,0xe4 ,0x80 ,0x84 ,0x23 ,0xaa , +0x9d ,0x64 ,0x1c ,0x80 ,0x41 ,0xca ,0x82 ,0x40 , +0x94 ,0x55 ,0xe3 ,0x28 ,0xa1 ,0x97 ,0x71 ,0xba , +0xf2 ,0x2c ,0x39 ,0x62 ,0x29 ,0x56 ,0xd0 ,0xff , +0xb2 ,0x82 ,0x20 ,0x59 ,0x1f ,0xc3 ,0x64 ,0x57 +}; +static const u8 key69[]__initconst = { +0x19 ,0x09 ,0xe9 ,0x7c ,0xd9 ,0x02 ,0x4a ,0x0c , +0x52 ,0x25 ,0xad ,0x5c ,0x2e ,0x8d ,0x86 ,0x10 , +0x85 ,0x2b ,0xba ,0xa4 ,0x44 ,0x5b ,0x39 ,0x3e , +0x18 ,0xaa ,0xce ,0x0e ,0xe2 ,0x69 ,0x3c ,0xcf +}; +enum { nonce69 =0xdb925a1948f0f060ULL }; + +static const u8 input70[]__initconst = { +0x10 ,0xe7 ,0x83 ,0xcf ,0x42 ,0x9f ,0xf2 ,0x41 , +0xc7 ,0xe4 ,0xdb ,0xf9 ,0xa3 ,0x02 ,0x1d ,0x8d , +0x50 ,0x81 ,0x2c ,0x6b ,0x92 ,0xe0 ,0x4e ,0xea , +0x26 ,0x83 ,0x2a ,0xd0 ,0x31 ,0xf1 ,0x23 ,0xf3 , +0x0e ,0x88 ,0x14 ,0x31 ,0xf9 ,0x01 ,0x63 ,0x59 , +0x21 ,0xd1 ,0x8b ,0xdd ,0x06 ,0xd0 ,0xc6 ,0xab , +0x91 ,0x71 ,0x82 ,0x4d ,0xd4 ,0x62 ,0x37 ,0x17 , +0xf9 ,0x50 ,0xf9 ,0xb5 ,0x74 ,0xce ,0x39 ,0x80 , +0x80 ,0x78 ,0xf8 ,0xdc ,0x1c ,0xdb ,0x7c ,0x3d , +0xd4 ,0x86 ,0x31 ,0x00 ,0x75 ,0x7b ,0xd1 ,0x42 , +0x9f ,0x1b ,0x97 ,0x88 ,0x0e ,0x14 ,0x0e ,0x1e , +0x7d ,0x7b ,0xc4 ,0xd2 ,0xf3 ,0xc1 ,0x6d ,0x17 , +0x5d ,0xc4 ,0x75 ,0x54 ,0x0f ,0x38 ,0x65 ,0x89 , +0xd8 ,0x7d ,0xab ,0xc9 ,0xa7 ,0x0a ,0x21 ,0x0b , +0x37 ,0x12 ,0x05 ,0x07 ,0xb5 ,0x68 ,0x32 ,0x32 , +0xb9 ,0xf8 ,0x97 ,0x17 ,0x03 ,0xed ,0x51 ,0x8f , +0x3d ,0x5a ,0xd0 ,0x12 ,0x01 ,0x6e ,0x2e ,0x91 , +0x1c ,0xbe ,0x6b ,0xa3 ,0xcc ,0x75 ,0x62 ,0x06 , +0x8e ,0x65 ,0xbb ,0xe2 ,0x29 ,0x71 ,0x4b ,0x89 , +0x6a ,0x9d ,0x85 ,0x8c ,0x8c ,0xdf ,0x94 ,0x95 , +0x23 ,0x66 ,0xf8 ,0x92 ,0xee ,0x56 ,0xeb ,0xb3 , +0xeb ,0xd2 ,0x4a ,0x3b ,0x77 ,0x8a ,0x6e ,0xf6 , +0xca ,0xd2 ,0x34 ,0x00 ,0xde ,0xbe ,0x1d ,0x7a , +0x73 ,0xef ,0x2b ,0x80 ,0x56 ,0x16 ,0x29 ,0xbf , +0x6e ,0x33 ,0xed ,0x0d ,0xe2 ,0x02 ,0x60 ,0x74 , +0xe9 ,0x0a ,0xbc ,0xd1 ,0xc5 ,0xe8 ,0x53 ,0x02 , +0x79 ,0x0f ,0x25 ,0x0c ,0xef ,0xab ,0xd3 ,0xbc , +0xb7 ,0xfc ,0xf3 ,0xb0 ,0x34 ,0xd1 ,0x07 ,0xd2 , +0x5a ,0x31 ,0x1f ,0xec ,0x1f ,0x87 ,0xed ,0xdd , +0x6a ,0xc1 ,0xe8 ,0xb3 ,0x25 ,0x4c ,0xc6 ,0x9b , +0x91 ,0x73 ,0xec ,0x06 ,0x73 ,0x9e ,0x57 ,0x65 , +0x32 ,0x75 ,0x11 ,0x74 ,0x6e ,0xa4 ,0x7d ,0x0d , +0x74 ,0x9f ,0x51 ,0x10 ,0x10 ,0x47 ,0xc9 ,0x71 , +0x6e ,0x97 ,0xae ,0x44 ,0x41 ,0xef ,0x98 ,0x78 , +0xf4 ,0xc5 ,0xbd ,0x5e ,0x00 ,0xe5 ,0xfd ,0xe2 , +0xbe ,0x8c ,0xc2 ,0xae ,0xc2 ,0xee ,0x59 ,0xf6 , +0xcb ,0x20 ,0x54 ,0x84 ,0xc3 ,0x31 ,0x7e ,0x67 , +0x71 ,0xb6 ,0x76 ,0xbe ,0x81 ,0x8f ,0x82 ,0xad , +0x01 ,0x8f ,0xc4 ,0x00 ,0x04 ,0x3d ,0x8d ,0x34 , +0xaa ,0xea ,0xc0 ,0xea ,0x91 ,0x42 ,0xb6 ,0xb8 , +0x43 ,0xf3 ,0x17 ,0xb2 ,0x73 ,0x64 ,0x82 ,0x97 , +0xd5 ,0xc9 ,0x07 ,0x77 ,0xb1 ,0x26 ,0xe2 ,0x00 , +0x6a ,0xae ,0x70 ,0x0b ,0xbe ,0xe6 ,0xb8 ,0x42 , +0x81 ,0x55 ,0xf7 ,0xb8 ,0x96 ,0x41 ,0x9d ,0xd4 , +0x2c ,0x27 ,0x00 ,0xcc ,0x91 ,0x28 ,0x22 ,0xa4 , +0x7b ,0x42 ,0x51 ,0x9e ,0xd6 ,0xec ,0xf3 ,0x6b , +0x00 ,0xff ,0x5c ,0xa2 ,0xac ,0x47 ,0x33 ,0x2d , +0xf8 ,0x11 ,0x65 ,0x5f ,0x4d ,0x79 ,0x8b ,0x4f , +0xad ,0xf0 ,0x9d ,0xcd ,0xb9 ,0x7b ,0x08 ,0xf7 , +0x32 ,0x51 ,0xfa ,0x39 ,0xaa ,0x78 ,0x05 ,0xb1 , +0xf3 ,0x5d ,0xe8 ,0x7c ,0x8e ,0x4f ,0xa2 ,0xe0 , +0x98 ,0x0c ,0xb2 ,0xa7 ,0xf0 ,0x35 ,0x8e ,0x70 , +0x7c ,0x82 ,0xf3 ,0x1b ,0x26 ,0x28 ,0x12 ,0xe5 , +0x23 ,0x57 ,0xe4 ,0xb4 ,0x9b ,0x00 ,0x39 ,0x97 , +0xef ,0x7c ,0x46 ,0x9b ,0x34 ,0x6b ,0xe7 ,0x0e , +0xa3 ,0x2a ,0x18 ,0x11 ,0x64 ,0xc6 ,0x7c ,0x8b , +0x06 ,0x02 ,0xf5 ,0x69 ,0x76 ,0xf9 ,0xaa ,0x09 , +0x5f ,0x68 ,0xf8 ,0x4a ,0x79 ,0x58 ,0xec ,0x37 , +0xcf ,0x3a ,0xcc ,0x97 ,0x70 ,0x1d ,0x3e ,0x52 , +0x18 ,0x0a ,0xad ,0x28 ,0x5b ,0x3b ,0xe9 ,0x03 , +0x84 ,0xe9 ,0x68 ,0x50 ,0xce ,0xc4 ,0xbc ,0x3e , +0x21 ,0xad ,0x63 ,0xfe ,0xc6 ,0xfd ,0x6e ,0x69 , +0x84 ,0xa9 ,0x30 ,0xb1 ,0x7a ,0xc4 ,0x31 ,0x10 , +0xc1 ,0x1f ,0x6e ,0xeb ,0xa5 ,0xa6 ,0x01 +}; +static const u8 output70[]__initconst = { +0x0f ,0x93 ,0x2a ,0x20 ,0xb3 ,0x87 ,0x2d ,0xce , +0xd1 ,0x3b ,0x30 ,0xfd ,0x06 ,0x6d ,0x0a ,0xaa , +0x3e ,0xc4 ,0x29 ,0x02 ,0x8a ,0xde ,0xa6 ,0x4b , +0x45 ,0x1b ,0x4f ,0x25 ,0x59 ,0xd5 ,0x56 ,0x6a , +0x3b ,0x37 ,0xbd ,0x3e ,0x47 ,0x12 ,0x2c ,0x4e , +0x60 ,0x5f ,0x05 ,0x75 ,0x61 ,0x23 ,0x05 ,0x74 , +0xcb ,0xfc ,0x5a ,0xb3 ,0xac ,0x5c ,0x3d ,0xab , +0x52 ,0x5f ,0x05 ,0xbc ,0x57 ,0xc0 ,0x7e ,0xcf , +0x34 ,0x5d ,0x7f ,0x41 ,0xa3 ,0x17 ,0x78 ,0xd5 , +0x9f ,0xec ,0x0f ,0x1e ,0xf9 ,0xfe ,0xa3 ,0xbd , +0x28 ,0xb0 ,0xba ,0x4d ,0x84 ,0xdb ,0xae ,0x8f , +0x1d ,0x98 ,0xb7 ,0xdc ,0xf9 ,0xad ,0x55 ,0x9c , +0x89 ,0xfe ,0x9b ,0x9c ,0xa9 ,0x89 ,0xf6 ,0x97 , +0x9c ,0x3f ,0x09 ,0x3e ,0xc6 ,0x02 ,0xc2 ,0x55 , +0x58 ,0x09 ,0x54 ,0x66 ,0xe4 ,0x36 ,0x81 ,0x35 , +0xca ,0x88 ,0x17 ,0x89 ,0x80 ,0x24 ,0x2b ,0x21 , +0x89 ,0xee ,0x45 ,0x5a ,0xe7 ,0x1f ,0xd5 ,0xa5 , +0x16 ,0xa4 ,0xda ,0x70 ,0x7e ,0xe9 ,0x4f ,0x24 , +0x61 ,0x97 ,0xab ,0xa0 ,0xe0 ,0xe7 ,0xb8 ,0x5c , +0x0f ,0x25 ,0x17 ,0x37 ,0x75 ,0x12 ,0xb5 ,0x40 , +0xde ,0x1c ,0x0d ,0x8a ,0x77 ,0x62 ,0x3c ,0x86 , +0xd9 ,0x70 ,0x2e ,0x96 ,0x30 ,0xd2 ,0x55 ,0xb3 , +0x6b ,0xc3 ,0xf2 ,0x9c ,0x47 ,0xf3 ,0x3a ,0x24 , +0x52 ,0xc6 ,0x38 ,0xd8 ,0x22 ,0xb3 ,0x0c ,0xfd , +0x2f ,0xa3 ,0x3c ,0xb5 ,0xe8 ,0x26 ,0xe1 ,0xa3 , +0xad ,0xb0 ,0x82 ,0x17 ,0xc1 ,0x53 ,0xb8 ,0x34 , +0x48 ,0xee ,0x39 ,0xae ,0x51 ,0x43 ,0xec ,0x82 , +0xce ,0x87 ,0xc6 ,0x76 ,0xb9 ,0x76 ,0xd3 ,0x53 , +0xfe ,0x49 ,0x24 ,0x7d ,0x02 ,0x42 ,0x2b ,0x72 , +0xfb ,0xcb ,0xd8 ,0x96 ,0x02 ,0xc6 ,0x9a ,0x20 , +0xf3 ,0x5a ,0x67 ,0xe8 ,0x13 ,0xf8 ,0xb2 ,0xcb , +0xa2 ,0xec ,0x18 ,0x20 ,0x4a ,0xb0 ,0x73 ,0x53 , +0x21 ,0xb0 ,0x77 ,0x53 ,0xd8 ,0x76 ,0xa1 ,0x30 , +0x17 ,0x72 ,0x2e ,0x33 ,0x5f ,0x33 ,0x6b ,0x28 , +0xfb ,0xb0 ,0xf4 ,0xec ,0x8e ,0xed ,0x20 ,0x7d , +0x57 ,0x8c ,0x74 ,0x28 ,0x64 ,0x8b ,0xeb ,0x59 , +0x38 ,0x3f ,0xe7 ,0x83 ,0x2e ,0xe5 ,0x64 ,0x4d , +0x5c ,0x1f ,0xe1 ,0x3b ,0xd9 ,0x84 ,0xdb ,0xc9 , +0xec ,0xd8 ,0xc1 ,0x7c ,0x1f ,0x1b ,0x68 ,0x35 , +0xc6 ,0x34 ,0x10 ,0xef ,0x19 ,0xc9 ,0x0a ,0xd6 , +0x43 ,0x7f ,0xa6 ,0xcb ,0x9d ,0xf4 ,0xf0 ,0x16 , +0xb1 ,0xb1 ,0x96 ,0x64 ,0xec ,0x8d ,0x22 ,0x4c , +0x4b ,0xe8 ,0x1a ,0xba ,0x6f ,0xb7 ,0xfc ,0xa5 , +0x69 ,0x3e ,0xad ,0x78 ,0x79 ,0x19 ,0xb5 ,0x04 , +0x69 ,0xe5 ,0x3f ,0xff ,0x60 ,0x8c ,0xda ,0x0b , +0x7b ,0xf7 ,0xe7 ,0xe6 ,0x29 ,0x3a ,0x85 ,0xba , +0xb5 ,0xb0 ,0x35 ,0xbd ,0x38 ,0xce ,0x34 ,0x5e , +0xf2 ,0xdc ,0xd1 ,0x8f ,0xc3 ,0x03 ,0x24 ,0xa2 , +0x03 ,0xf7 ,0x4e ,0x49 ,0x5b ,0xcf ,0x6d ,0xb0 , +0xeb ,0xe3 ,0x30 ,0x28 ,0xd5 ,0x5b ,0x82 ,0x5f , +0xe4 ,0x7c ,0x1e ,0xec ,0xd2 ,0x39 ,0xf9 ,0x6f , +0x2e ,0xb3 ,0xcd ,0x01 ,0xb1 ,0x67 ,0xaa ,0xea , +0xaa ,0xb3 ,0x63 ,0xaf ,0xd9 ,0xb2 ,0x1f ,0xba , +0x05 ,0x20 ,0xeb ,0x19 ,0x32 ,0xf0 ,0x6c ,0x3f , +0x40 ,0xcc ,0x93 ,0xb3 ,0xd8 ,0x25 ,0xa6 ,0xe4 , +0xce ,0xd7 ,0x7e ,0x48 ,0x99 ,0x65 ,0x7f ,0x86 , +0xc5 ,0xd4 ,0x79 ,0x6b ,0xab ,0x43 ,0xb8 ,0x6b , +0xf1 ,0x2f ,0xea ,0x4c ,0x5e ,0xf0 ,0x3b ,0xb4 , +0xb8 ,0xb0 ,0x94 ,0x0c ,0x6b ,0xe7 ,0x22 ,0x93 , +0xaa ,0x01 ,0xcb ,0xf1 ,0x11 ,0x60 ,0xf6 ,0x69 , +0xcf ,0x14 ,0xde ,0xfb ,0x90 ,0x05 ,0x27 ,0x0c , +0x1a ,0x9e ,0xf0 ,0xb4 ,0xc6 ,0xa1 ,0xe8 ,0xdd , +0xd0 ,0x4c ,0x25 ,0x4f ,0x9c ,0xb7 ,0xb1 ,0xb0 , +0x21 ,0xdb ,0x87 ,0x09 ,0x03 ,0xf2 ,0xb3 +}; +static const u8 key70[]__initconst = { +0x3b ,0x5b ,0x59 ,0x36 ,0x44 ,0xd1 ,0xba ,0x71 , +0x55 ,0x87 ,0x4d ,0x62 ,0x3d ,0xc2 ,0xfc ,0xaa , +0x3f ,0x4e ,0x1a ,0xe4 ,0xca ,0x09 ,0xfc ,0x6a , +0xb2 ,0xd6 ,0x5d ,0x79 ,0xf9 ,0x1a ,0x91 ,0xa7 +}; +enum { nonce70 =0x3fd6786dd147a85ULL }; + +static const u8 input71[]__initconst = { +0x18 ,0x78 ,0xd6 ,0x79 ,0xe4 ,0x9a ,0x6c ,0x73 , +0x17 ,0xd4 ,0x05 ,0x0f ,0x1e ,0x9f ,0xd9 ,0x2b , +0x86 ,0x48 ,0x7d ,0xf4 ,0xd9 ,0x1c ,0x76 ,0xfc , +0x8e ,0x22 ,0x34 ,0xe1 ,0x48 ,0x4a ,0x8d ,0x79 , +0xb7 ,0xbb ,0x88 ,0xab ,0x90 ,0xde ,0xc5 ,0xb4 , +0xb4 ,0xe7 ,0x85 ,0x49 ,0xda ,0x57 ,0xeb ,0xc9 , +0xcd ,0x21 ,0xfc ,0x45 ,0x6e ,0x32 ,0x67 ,0xf2 , +0x4f ,0xa6 ,0x54 ,0xe5 ,0x20 ,0xed ,0xcf ,0xc6 , +0x62 ,0x25 ,0x8e ,0x00 ,0xf8 ,0x6b ,0xa2 ,0x80 , +0xac ,0x88 ,0xa6 ,0x59 ,0x27 ,0x83 ,0x95 ,0x11 , +0x3f ,0x70 ,0x5e ,0x3f ,0x11 ,0xfb ,0x26 ,0xbf , +0xe1 ,0x48 ,0x75 ,0xf9 ,0x86 ,0xbf ,0xa6 ,0x5d , +0x15 ,0x61 ,0x66 ,0xbf ,0x78 ,0x8f ,0x6b ,0x9b , +0xda ,0x98 ,0xb7 ,0x19 ,0xe2 ,0xf2 ,0xa3 ,0x9c , +0x7c ,0x6a ,0x9a ,0xd8 ,0x3d ,0x4c ,0x2c ,0xe1 , +0x09 ,0xb4 ,0x28 ,0x82 ,0x4e ,0xab ,0x0c ,0x75 , +0x63 ,0xeb ,0xbc ,0xd0 ,0x71 ,0xa2 ,0x73 ,0x85 , +0xed ,0x53 ,0x7a ,0x3f ,0x68 ,0x9f ,0xd0 ,0xa9 , +0x00 ,0x5a ,0x9e ,0x80 ,0x55 ,0x00 ,0xe6 ,0xae , +0x0c ,0x03 ,0x40 ,0xed ,0xfc ,0x68 ,0x4a ,0xb7 , +0x1e ,0x09 ,0x65 ,0x30 ,0x5a ,0x3d ,0x97 ,0x4d , +0x5e ,0x51 ,0x8e ,0xda ,0xc3 ,0x55 ,0x8c ,0xfb , +0xcf ,0x83 ,0x05 ,0x35 ,0x0d ,0x08 ,0x1b ,0xf3 , +0x3a ,0x57 ,0x96 ,0xac ,0x58 ,0x8b ,0xfa ,0x00 , +0x49 ,0x15 ,0x78 ,0xd2 ,0x4b ,0xed ,0xb8 ,0x59 , +0x78 ,0x9b ,0x7f ,0xaa ,0xfc ,0xe7 ,0x46 ,0xdc , +0x7b ,0x34 ,0xd0 ,0x34 ,0xe5 ,0x10 ,0xff ,0x4d , +0x5a ,0x4d ,0x60 ,0xa7 ,0x16 ,0x54 ,0xc4 ,0xfd , +0xca ,0x5d ,0x68 ,0xc7 ,0x4a ,0x01 ,0x8d ,0x7f , +0x74 ,0x5d ,0xff ,0xb8 ,0x37 ,0x15 ,0x62 ,0xfa , +0x44 ,0x45 ,0xcf ,0x77 ,0x3b ,0x1d ,0xb2 ,0xd2 , +0x0d ,0x42 ,0x00 ,0x39 ,0x68 ,0x1f ,0xcc ,0x89 , +0x73 ,0x5d ,0xa9 ,0x2e ,0xfd ,0x58 ,0x62 ,0xca , +0x35 ,0x8e ,0x70 ,0x70 ,0xaa ,0x6e ,0x14 ,0xe9 , +0xa4 ,0xe2 ,0x10 ,0x66 ,0x71 ,0xdc ,0x4c ,0xfc , +0xa9 ,0xdc ,0x8f ,0x57 ,0x4d ,0xc5 ,0xac ,0xd7 , +0xa9 ,0xf3 ,0xf3 ,0xa1 ,0xff ,0x62 ,0xa0 ,0x8f , +0xe4 ,0x96 ,0x3e ,0xcb ,0x9f ,0x76 ,0x42 ,0x39 , +0x1f ,0x24 ,0xfd ,0xfd ,0x79 ,0xe8 ,0x27 ,0xdf , +0xa8 ,0xf6 ,0x33 ,0x8b ,0x31 ,0x59 ,0x69 ,0xcf , +0x6a ,0xef ,0x89 ,0x4d ,0xa7 ,0xf6 ,0x7e ,0x97 , +0x14 ,0xbd ,0xda ,0xdd ,0xb4 ,0x84 ,0x04 ,0x24 , +0xe0 ,0x17 ,0xe1 ,0x0f ,0x1f ,0x8a ,0x6a ,0x71 , +0x74 ,0x41 ,0xdc ,0x59 ,0x5c ,0x8f ,0x01 ,0x25 , +0x92 ,0xf0 ,0x2e ,0x15 ,0x62 ,0x71 ,0x9a ,0x9f , +0x87 ,0xdf ,0x62 ,0x49 ,0x7f ,0x86 ,0x62 ,0xfc , +0x20 ,0x84 ,0xd7 ,0xe3 ,0x3a ,0xd9 ,0x37 ,0x85 , +0xb7 ,0x84 ,0x5a ,0xf9 ,0xed ,0x21 ,0x32 ,0x94 , +0x3e ,0x04 ,0xe7 ,0x8c ,0x46 ,0x76 ,0x21 ,0x67 , +0xf6 ,0x95 ,0x64 ,0x92 ,0xb7 ,0x15 ,0xf6 ,0xe3 , +0x41 ,0x27 ,0x9d ,0xd7 ,0xe3 ,0x79 ,0x75 ,0x92 , +0xd0 ,0xc1 ,0xf3 ,0x40 ,0x92 ,0x08 ,0xde ,0x90 , +0x22 ,0x82 ,0xb2 ,0x69 ,0xae ,0x1a ,0x35 ,0x11 , +0x89 ,0xc8 ,0x06 ,0x82 ,0x95 ,0x23 ,0x44 ,0x08 , +0x22 ,0xf2 ,0x71 ,0x73 ,0x1b ,0x88 ,0x11 ,0xcf , +0x1c ,0x7e ,0x8a ,0x2e ,0xdc ,0x79 ,0x57 ,0xce , +0x1f ,0xe7 ,0x6c ,0x07 ,0xd8 ,0x06 ,0xbe ,0xec , +0xa3 ,0xcf ,0xf9 ,0x68 ,0xa5 ,0xb8 ,0xf0 ,0xe3 , +0x3f ,0x01 ,0x92 ,0xda ,0xf1 ,0xa0 ,0x2d ,0x7b , +0xab ,0x57 ,0x58 ,0x2a ,0xaf ,0xab ,0xbd ,0xf2 , +0xe5 ,0xaf ,0x7e ,0x1f ,0x46 ,0x24 ,0x9e ,0x20 , +0x22 ,0x0f ,0x84 ,0x4c ,0xb7 ,0xd8 ,0x03 ,0xe8 , +0x09 ,0x73 ,0x6c ,0xc6 ,0x9b ,0x90 ,0xe0 ,0xdb , +0xf2 ,0x71 ,0xba ,0xad ,0xb3 ,0xec ,0xda ,0x7a +}; +static const u8 output71[]__initconst = { +0x28 ,0xc5 ,0x9b ,0x92 ,0xf9 ,0x21 ,0x4f ,0xbb , +0xef ,0x3b ,0xf0 ,0xf5 ,0x3a ,0x6d ,0x7f ,0xd6 , +0x6a ,0x8d ,0xa1 ,0x01 ,0x5c ,0x62 ,0x20 ,0x8b , +0x5b ,0x39 ,0xd5 ,0xd3 ,0xc2 ,0xf6 ,0x9d ,0x5e , +0xcc ,0xe1 ,0xa2 ,0x61 ,0x16 ,0xe2 ,0xce ,0xe9 , +0x86 ,0xd0 ,0xfc ,0xce ,0x9a ,0x28 ,0x27 ,0xc4 , +0x0c ,0xb9 ,0xaa ,0x8d ,0x48 ,0xdb ,0xbf ,0x82 , +0x7d ,0xd0 ,0x35 ,0xc4 ,0x06 ,0x34 ,0xb4 ,0x19 , +0x51 ,0x73 ,0xf4 ,0x7a ,0xf4 ,0xfd ,0xe9 ,0x1d , +0xdc ,0x0f ,0x7e ,0xf7 ,0x96 ,0x03 ,0xe3 ,0xb1 , +0x2e ,0x22 ,0x59 ,0xb7 ,0x6d ,0x1c ,0x97 ,0x8c , +0xd7 ,0x31 ,0x08 ,0x26 ,0x4c ,0x6d ,0xc6 ,0x14 , +0xa5 ,0xeb ,0x45 ,0x6a ,0x88 ,0xa3 ,0xa2 ,0x36 , +0xc4 ,0x35 ,0xb1 ,0x5a ,0xa0 ,0xad ,0xf7 ,0x06 , +0x9b ,0x5d ,0xc1 ,0x15 ,0xc1 ,0xce ,0x0a ,0xb0 , +0x57 ,0x2e ,0x3f ,0x6f ,0x0d ,0x10 ,0xd9 ,0x11 , +0x2c ,0x9c ,0xad ,0x2d ,0xa5 ,0x81 ,0xfb ,0x4e , +0x8f ,0xd5 ,0x32 ,0x4e ,0xaf ,0x5c ,0xc1 ,0x86 , +0xde ,0x56 ,0x5a ,0x33 ,0x29 ,0xf7 ,0x67 ,0xc6 , +0x37 ,0x6f ,0xb2 ,0x37 ,0x4e ,0xd4 ,0x69 ,0x79 , +0xaf ,0xd5 ,0x17 ,0x79 ,0xe0 ,0xba ,0x62 ,0xa3 , +0x68 ,0xa4 ,0x87 ,0x93 ,0x8d ,0x7e ,0x8f ,0xa3 , +0x9c ,0xef ,0xda ,0xe3 ,0xa5 ,0x1f ,0xcd ,0x30 , +0xa6 ,0x55 ,0xac ,0x4c ,0x69 ,0x74 ,0x02 ,0xc7 , +0x5d ,0x95 ,0x81 ,0x4a ,0x68 ,0x11 ,0xd3 ,0xa9 , +0x98 ,0xb1 ,0x0b ,0x0d ,0xae ,0x40 ,0x86 ,0x65 , +0xbf ,0xcc ,0x2d ,0xef ,0x57 ,0xca ,0x1f ,0xe4 , +0x34 ,0x4e ,0xa6 ,0x5e ,0x82 ,0x6e ,0x61 ,0xad , +0x0b ,0x3c ,0xf8 ,0xeb ,0x01 ,0x43 ,0x7f ,0x87 , +0xa2 ,0xa7 ,0x6a ,0xe9 ,0x62 ,0x23 ,0x24 ,0x61 , +0xf1 ,0xf7 ,0x36 ,0xdb ,0x10 ,0xe5 ,0x57 ,0x72 , +0x3a ,0xc2 ,0xae ,0xcc ,0x75 ,0xc7 ,0x80 ,0x05 , +0x0a ,0x5c ,0x4c ,0x95 ,0xda ,0x02 ,0x01 ,0x14 , +0x06 ,0x6b ,0x5c ,0x65 ,0xc2 ,0xb8 ,0x4a ,0xd6 , +0xd3 ,0xb4 ,0xd8 ,0x12 ,0x52 ,0xb5 ,0x60 ,0xd3 , +0x8e ,0x5f ,0x5c ,0x76 ,0x33 ,0x7a ,0x05 ,0xe5 , +0xcb ,0xef ,0x4f ,0x89 ,0xf1 ,0xba ,0x32 ,0x6f , +0x33 ,0xcd ,0x15 ,0x8d ,0xa3 ,0x0c ,0x3f ,0x63 , +0x11 ,0xe7 ,0x0e ,0xe0 ,0x00 ,0x01 ,0xe9 ,0xe8 , +0x8e ,0x36 ,0x34 ,0x8d ,0x96 ,0xb5 ,0x03 ,0xcf , +0x55 ,0x62 ,0x49 ,0x7a ,0x34 ,0x44 ,0xa5 ,0xee , +0x8c ,0x46 ,0x06 ,0x22 ,0xab ,0x1d ,0x53 ,0x9c , +0xa1 ,0xf9 ,0x67 ,0x18 ,0x57 ,0x89 ,0xf9 ,0xc2 , +0xd1 ,0x7e ,0xbe ,0x36 ,0x40 ,0xcb ,0xe9 ,0x04 , +0xde ,0xb1 ,0x3b ,0x29 ,0x52 ,0xc5 ,0x9a ,0xb5 , +0xa2 ,0x7c ,0x7b ,0xfe ,0xe5 ,0x92 ,0x73 ,0xea , +0xea ,0x7b ,0xba ,0x0a ,0x8c ,0x88 ,0x15 ,0xe6 , +0x53 ,0xbf ,0x1c ,0x33 ,0xf4 ,0x9b ,0x9a ,0x5e , +0x8d ,0xae ,0x60 ,0xdc ,0xcb ,0x5d ,0xfa ,0xbe , +0x06 ,0xc3 ,0x3f ,0x06 ,0xe7 ,0x00 ,0x40 ,0x7b , +0xaa ,0x94 ,0xfa ,0x6d ,0x1f ,0xe4 ,0xc5 ,0xa9 , +0x1b ,0x5f ,0x36 ,0xea ,0x5a ,0xdd ,0xa5 ,0x48 , +0x6a ,0x55 ,0xd2 ,0x47 ,0x28 ,0xbf ,0x96 ,0xf1 , +0x9f ,0xb6 ,0x11 ,0x4b ,0xd3 ,0x44 ,0x7d ,0x48 , +0x41 ,0x61 ,0xdb ,0x12 ,0xd4 ,0xc2 ,0x59 ,0x82 , +0x4c ,0x47 ,0x5c ,0x04 ,0xf6 ,0x7b ,0xd3 ,0x92 , +0x2e ,0xe8 ,0x40 ,0xef ,0x15 ,0x32 ,0x97 ,0xdc , +0x35 ,0x4c ,0x6e ,0xa4 ,0x97 ,0xe9 ,0x24 ,0xde , +0x63 ,0x8b ,0xb1 ,0x6b ,0x48 ,0xbb ,0x46 ,0x1f , +0x84 ,0xd6 ,0x17 ,0xb0 ,0x5a ,0x4a ,0x4e ,0xd5 , +0x31 ,0xd7 ,0xcf ,0xa0 ,0x39 ,0xc6 ,0x2e ,0xfc , +0xa6 ,0xa3 ,0xd3 ,0x0f ,0xa4 ,0x28 ,0xac ,0xb2 , +0xf4 ,0x48 ,0x8d ,0x50 ,0xa5 ,0x1c ,0x44 ,0x5d , +0x6e ,0x38 ,0xb7 ,0x2b ,0x8a ,0x45 ,0xa7 ,0x3d +}; +static const u8 key71[]__initconst = { +0x8b ,0x68 ,0xc4 ,0xb7 ,0x0d ,0x81 ,0xef ,0x52 , +0x1e ,0x05 ,0x96 ,0x72 ,0x62 ,0x89 ,0x27 ,0x83 , +0xd0 ,0xc7 ,0x33 ,0x6d ,0xf2 ,0xcc ,0x69 ,0xf9 , +0x23 ,0xae ,0x99 ,0xb1 ,0xd1 ,0x05 ,0x4e ,0x54 +}; +enum { nonce71 =0x983f03656d64b5f6ULL }; + +static const u8 input72[]__initconst = { +0x6b ,0x09 ,0xc9 ,0x57 ,0x3d ,0x79 ,0x04 ,0x8c , +0x65 ,0xad ,0x4a ,0x0f ,0xa1 ,0x31 ,0x3a ,0xdd , +0x14 ,0x8e ,0xe8 ,0xfe ,0xbf ,0x42 ,0x87 ,0x98 , +0x2e ,0x8d ,0x83 ,0xa3 ,0xf8 ,0x55 ,0x3d ,0x84 , +0x1e ,0x0e ,0x05 ,0x4a ,0x38 ,0x9e ,0xe7 ,0xfe , +0xd0 ,0x4d ,0x79 ,0x74 ,0x3a ,0x0b ,0x9b ,0xe1 , +0xfd ,0x51 ,0x84 ,0x4e ,0xb2 ,0x25 ,0xe4 ,0x64 , +0x4c ,0xda ,0xcf ,0x46 ,0xec ,0xba ,0x12 ,0xeb , +0x5a ,0x33 ,0x09 ,0x6e ,0x78 ,0x77 ,0x8f ,0x30 , +0xb1 ,0x7d ,0x3f ,0x60 ,0x8c ,0xf2 ,0x1d ,0x8e , +0xb4 ,0x70 ,0xa2 ,0x90 ,0x7c ,0x79 ,0x1a ,0x2c , +0xf6 ,0x28 ,0x79 ,0x7c ,0x53 ,0xc5 ,0xfa ,0xcc , +0x65 ,0x9b ,0xe1 ,0x51 ,0xd1 ,0x7f ,0x1d ,0xc4 , +0xdb ,0xd4 ,0xd9 ,0x04 ,0x61 ,0x7d ,0xbe ,0x12 , +0xfc ,0xcd ,0xaf ,0xe4 ,0x0f ,0x9c ,0x20 ,0xb5 , +0x22 ,0x40 ,0x18 ,0xda ,0xe4 ,0xda ,0x8c ,0x2d , +0x84 ,0xe3 ,0x5f ,0x53 ,0x17 ,0xed ,0x78 ,0xdc , +0x2f ,0xe8 ,0x31 ,0xc7 ,0xe6 ,0x39 ,0x71 ,0x40 , +0xb4 ,0x0f ,0xc9 ,0xa9 ,0x7e ,0x78 ,0x87 ,0xc1 , +0x05 ,0x78 ,0xbb ,0x01 ,0xf2 ,0x8f ,0x33 ,0xb0 , +0x6e ,0x84 ,0xcd ,0x36 ,0x33 ,0x5c ,0x5b ,0x8e , +0xf1 ,0xac ,0x30 ,0xfe ,0x33 ,0xec ,0x08 ,0xf3 , +0x7e ,0xf2 ,0xf0 ,0x4c ,0xf2 ,0xad ,0xd8 ,0xc1 , +0xd4 ,0x4e ,0x87 ,0x06 ,0xd4 ,0x75 ,0xe7 ,0xe3 , +0x09 ,0xd3 ,0x4d ,0xe3 ,0x21 ,0x32 ,0xba ,0xb4 , +0x68 ,0x68 ,0xcb ,0x4c ,0xa3 ,0x1e ,0xb3 ,0x87 , +0x7b ,0xd3 ,0x0c ,0x63 ,0x37 ,0x71 ,0x79 ,0xfb , +0x58 ,0x36 ,0x57 ,0x0f ,0x34 ,0x1d ,0xc1 ,0x42 , +0x02 ,0x17 ,0xe7 ,0xed ,0xe8 ,0xe7 ,0x76 ,0xcb , +0x42 ,0xc4 ,0x4b ,0xe2 ,0xb2 ,0x5e ,0x42 ,0xd5 , +0xec ,0x9d ,0xc1 ,0x32 ,0x71 ,0xe4 ,0xeb ,0x10 , +0x68 ,0x1a ,0x6e ,0x99 ,0x8e ,0x73 ,0x12 ,0x1f , +0x97 ,0x0c ,0x9e ,0xcd ,0x02 ,0x3e ,0x4c ,0xa0 , +0xf2 ,0x8d ,0xe5 ,0x44 ,0xca ,0x6d ,0xfe ,0x07 , +0xe3 ,0xe8 ,0x9b ,0x76 ,0xc1 ,0x6d ,0xb7 ,0x6e , +0x0d ,0x14 ,0x00 ,0x6f ,0x8a ,0xfd ,0x43 ,0xc6 , +0x43 ,0xa5 ,0x9c ,0x02 ,0x47 ,0x10 ,0xd4 ,0xb4 , +0x9b ,0x55 ,0x67 ,0xc8 ,0x7f ,0xc1 ,0x8a ,0x1f , +0x1e ,0xd1 ,0xbc ,0x99 ,0x5d ,0x50 ,0x4f ,0x89 , +0xf1 ,0xe6 ,0x5d ,0x91 ,0x40 ,0xdc ,0x20 ,0x67 , +0x56 ,0xc2 ,0xef ,0xbd ,0x2c ,0xa2 ,0x99 ,0x38 , +0xe0 ,0x45 ,0xec ,0x44 ,0x05 ,0x52 ,0x65 ,0x11 , +0xfc ,0x3b ,0x19 ,0xcb ,0x71 ,0xc2 ,0x8e ,0x0e , +0x03 ,0x2a ,0x03 ,0x3b ,0x63 ,0x06 ,0x31 ,0x9a , +0xac ,0x53 ,0x04 ,0x14 ,0xd4 ,0x80 ,0x9d ,0x6b , +0x42 ,0x7e ,0x7e ,0x4e ,0xdc ,0xc7 ,0x01 ,0x49 , +0x9f ,0xf5 ,0x19 ,0x86 ,0x13 ,0x28 ,0x2b ,0xa6 , +0xa6 ,0xbe ,0xa1 ,0x7e ,0x71 ,0x05 ,0x00 ,0xff , +0x59 ,0x2d ,0xb6 ,0x63 ,0xf0 ,0x1e ,0x2e ,0x69 , +0x9b ,0x85 ,0xf1 ,0x1e ,0x8a ,0x64 ,0x39 ,0xab , +0x00 ,0x12 ,0xe4 ,0x33 ,0x4b ,0xb5 ,0xd8 ,0xb3 , +0x6b ,0x5b ,0x8b ,0x5c ,0xd7 ,0x6f ,0x23 ,0xcf , +0x3f ,0x2e ,0x5e ,0x47 ,0xb9 ,0xb8 ,0x1f ,0xf0 , +0x1d ,0xda ,0xe7 ,0x4f ,0x6e ,0xab ,0xc3 ,0x36 , +0xb4 ,0x74 ,0x6b ,0xeb ,0xc7 ,0x5d ,0x91 ,0xe5 , +0xda ,0xf2 ,0xc2 ,0x11 ,0x17 ,0x48 ,0xf8 ,0x9c , +0xc9 ,0x8b ,0xc1 ,0xa2 ,0xf4 ,0xcd ,0x16 ,0xf8 , +0x27 ,0xd9 ,0x6c ,0x6f ,0xb5 ,0x8f ,0x77 ,0xca , +0x1b ,0xd8 ,0xef ,0x84 ,0x68 ,0x71 ,0x53 ,0xc1 , +0x43 ,0x0f ,0x9f ,0x98 ,0xae ,0x7e ,0x31 ,0xd2 , +0x98 ,0xfb ,0x20 ,0xa2 ,0xad ,0x00 ,0x10 ,0x83 , +0x00 ,0x8b ,0xeb ,0x56 ,0xd2 ,0xc4 ,0xcc ,0x7f , +0x2f ,0x4e ,0xfa ,0x88 ,0x13 ,0xa4 ,0x2c ,0xde , +0x6b ,0x77 ,0x86 ,0x10 ,0x6a ,0xab ,0x43 ,0x0a , +0x02 +}; +static const u8 output72[]__initconst = { +0x42 ,0x89 ,0xa4 ,0x80 ,0xd2 ,0xcb ,0x5f ,0x7f , +0x2a ,0x1a ,0x23 ,0x00 ,0xa5 ,0x6a ,0x95 ,0xa3 , +0x9a ,0x41 ,0xa1 ,0xd0 ,0x2d ,0x1e ,0xd6 ,0x13 , +0x34 ,0x40 ,0x4e ,0x7f ,0x1a ,0xbe ,0xa0 ,0x3d , +0x33 ,0x9c ,0x56 ,0x2e ,0x89 ,0x25 ,0x45 ,0xf9 , +0xf0 ,0xba ,0x9c ,0x6d ,0xd1 ,0xd1 ,0xde ,0x51 , +0x47 ,0x63 ,0xc9 ,0xbd ,0xfa ,0xa2 ,0x9e ,0xad , +0x6a ,0x7b ,0x21 ,0x1a ,0x6c ,0x3e ,0xff ,0x46 , +0xbe ,0xf3 ,0x35 ,0x7a ,0x6e ,0xb3 ,0xb9 ,0xf7 , +0xda ,0x5e ,0xf0 ,0x14 ,0xb5 ,0x70 ,0xa4 ,0x2b , +0xdb ,0xbb ,0xc7 ,0x31 ,0x4b ,0x69 ,0x5a ,0x83 , +0x70 ,0xd9 ,0x58 ,0xd4 ,0x33 ,0x84 ,0x23 ,0xf0 , +0xae ,0xbb ,0x6d ,0x26 ,0x7c ,0xc8 ,0x30 ,0xf7 , +0x24 ,0xad ,0xbd ,0xe4 ,0x2c ,0x38 ,0x38 ,0xac , +0xe1 ,0x4a ,0x9b ,0xac ,0x33 ,0x0e ,0x4a ,0xf4 , +0x93 ,0xed ,0x07 ,0x82 ,0x81 ,0x4f ,0x8f ,0xb1 , +0xdd ,0x73 ,0xd5 ,0x50 ,0x6d ,0x44 ,0x1e ,0xbe , +0xa7 ,0xcd ,0x17 ,0x57 ,0xd5 ,0x3b ,0x62 ,0x36 , +0xcf ,0x7d ,0xc8 ,0xd8 ,0xd1 ,0x78 ,0xd7 ,0x85 , +0x46 ,0x76 ,0x5d ,0xcc ,0xfe ,0xe8 ,0x94 ,0xc5 , +0xad ,0xbc ,0x5e ,0xbc ,0x8d ,0x1d ,0xdf ,0x03 , +0xc9 ,0x6b ,0x1b ,0x81 ,0xd1 ,0xb6 ,0x5a ,0x24 , +0xe3 ,0xdc ,0x3f ,0x20 ,0xc9 ,0x07 ,0x73 ,0x4c , +0x43 ,0x13 ,0x87 ,0x58 ,0x34 ,0x0d ,0x14 ,0x63 , +0x0f ,0x6f ,0xad ,0x8d ,0xac ,0x7c ,0x67 ,0x68 , +0xa3 ,0x9d ,0x7f ,0x00 ,0xdf ,0x28 ,0xee ,0x67 , +0xf4 ,0x5c ,0x26 ,0xcb ,0xef ,0x56 ,0x71 ,0xc8 , +0xc6 ,0x67 ,0x5f ,0x38 ,0xbb ,0xa0 ,0xb1 ,0x5c , +0x1f ,0xb3 ,0x08 ,0xd9 ,0x38 ,0xcf ,0x74 ,0x54 , +0xc6 ,0xa4 ,0xc4 ,0xc0 ,0x9f ,0xb3 ,0xd0 ,0xda , +0x62 ,0x67 ,0x8b ,0x81 ,0x33 ,0xf0 ,0xa9 ,0x73 , +0xa4 ,0xd1 ,0x46 ,0x88 ,0x8d ,0x85 ,0x12 ,0x40 , +0xba ,0x1a ,0xcd ,0x82 ,0xd8 ,0x8d ,0xc4 ,0x52 , +0xe7 ,0x01 ,0x94 ,0x2e ,0x0e ,0xd0 ,0xaf ,0xe7 , +0x2d ,0x3f ,0x3c ,0xaa ,0xf4 ,0xf5 ,0xa7 ,0x01 , +0x4c ,0x14 ,0xe2 ,0xc2 ,0x96 ,0x76 ,0xbe ,0x05 , +0xaa ,0x19 ,0xb1 ,0xbd ,0x95 ,0xbb ,0x5a ,0xf9 , +0xa5 ,0xa7 ,0xe6 ,0x16 ,0x38 ,0x34 ,0xf7 ,0x9d , +0x19 ,0x66 ,0x16 ,0x8e ,0x7f ,0x2b ,0x5a ,0xfb , +0xb5 ,0x29 ,0x79 ,0xbf ,0x52 ,0xae ,0x30 ,0x95 , +0x3f ,0x31 ,0x33 ,0x28 ,0xde ,0xc5 ,0x0d ,0x55 , +0x89 ,0xec ,0x21 ,0x11 ,0x0f ,0x8b ,0xfe ,0x63 , +0x3a ,0xf1 ,0x95 ,0x5c ,0xcd ,0x50 ,0xe4 ,0x5d , +0x8f ,0xa7 ,0xc8 ,0xca ,0x93 ,0xa0 ,0x67 ,0x82 , +0x63 ,0x5c ,0xd0 ,0xed ,0xe7 ,0x08 ,0xc5 ,0x60 , +0xf8 ,0xb4 ,0x47 ,0xf0 ,0x1a ,0x65 ,0x4e ,0xa3 , +0x51 ,0x68 ,0xc7 ,0x14 ,0xa1 ,0xd9 ,0x39 ,0x72 , +0xa8 ,0x6f ,0x7c ,0x7e ,0xf6 ,0x03 ,0x0b ,0x25 , +0x9b ,0xf2 ,0xca ,0x49 ,0xae ,0x5b ,0xf8 ,0x0f , +0x71 ,0x51 ,0x01 ,0xa6 ,0x23 ,0xa9 ,0xdf ,0xd0 , +0x7a ,0x39 ,0x19 ,0xf5 ,0xc5 ,0x26 ,0x44 ,0x7b , +0x0a ,0x4a ,0x41 ,0xbf ,0xf2 ,0x8e ,0x83 ,0x50 , +0x91 ,0x96 ,0x72 ,0x02 ,0xf6 ,0x80 ,0xbf ,0x95 , +0x41 ,0xac ,0xda ,0xb0 ,0xba ,0xe3 ,0x76 ,0xb1 , +0x9d ,0xff ,0x1f ,0x33 ,0x02 ,0x85 ,0xfc ,0x2a , +0x29 ,0xe6 ,0xe3 ,0x9d ,0xd0 ,0xef ,0xc2 ,0xd6 , +0x9c ,0x4a ,0x62 ,0xac ,0xcb ,0xea ,0x8b ,0xc3 , +0x08 ,0x6e ,0x49 ,0x09 ,0x26 ,0x19 ,0xc1 ,0x30 , +0xcc ,0x27 ,0xaa ,0xc6 ,0x45 ,0x88 ,0xbd ,0xae , +0xd6 ,0x79 ,0xff ,0x4e ,0xfc ,0x66 ,0x4d ,0x02 , +0xa5 ,0xee ,0x8e ,0xa5 ,0xb6 ,0x15 ,0x72 ,0x24 , +0xb1 ,0xbf ,0xbf ,0x64 ,0xcf ,0xcc ,0x93 ,0xe9 , +0xb6 ,0xfd ,0xb4 ,0xb6 ,0x21 ,0xb5 ,0x48 ,0x08 , +0x0f ,0x11 ,0x65 ,0xe1 ,0x47 ,0xee ,0x93 ,0x29 , +0xad +}; +static const u8 key72[]__initconst = { +0xb9 ,0xa2 ,0xfc ,0x59 ,0x06 ,0x3f ,0x77 ,0xa5 , +0x66 ,0xd0 ,0x2b ,0x22 ,0x74 ,0x22 ,0x4c ,0x1e , +0x6a ,0x39 ,0xdf ,0xe1 ,0x0d ,0x4c ,0x64 ,0x99 , +0x54 ,0x8a ,0xba ,0x1d ,0x2c ,0x21 ,0x5f ,0xc3 +}; +enum { nonce72 =0x3d069308fa3db04bULL }; + +static const u8 input73[]__initconst = { +0xe4 ,0xdd ,0x36 ,0xd4 ,0xf5 ,0x70 ,0x51 ,0x73 , +0x97 ,0x1d ,0x45 ,0x05 ,0x92 ,0xe7 ,0xeb ,0xb7 , +0x09 ,0x82 ,0x6e ,0x25 ,0x6c ,0x50 ,0xf5 ,0x40 , +0x19 ,0xba ,0xbc ,0xf4 ,0x39 ,0x14 ,0xc5 ,0x15 , +0x83 ,0x40 ,0xbd ,0x26 ,0xe0 ,0xff ,0x3b ,0x22 , +0x7c ,0x7c ,0xd7 ,0x0b ,0xe9 ,0x25 ,0x0c ,0x3d , +0x92 ,0x38 ,0xbe ,0xe4 ,0x22 ,0x75 ,0x65 ,0xf1 , +0x03 ,0x85 ,0x34 ,0x09 ,0xb8 ,0x77 ,0xfb ,0x48 , +0xb1 ,0x2e ,0x21 ,0x67 ,0x9b ,0x9d ,0xad ,0x18 , +0x82 ,0x0d ,0x6b ,0xc3 ,0xcf ,0x00 ,0x61 ,0x6e , +0xda ,0xdc ,0xa7 ,0x0b ,0x5c ,0x02 ,0x1d ,0xa6 , +0x4e ,0x0d ,0x7f ,0x37 ,0x01 ,0x5a ,0x37 ,0xf3 , +0x2b ,0xbf ,0xba ,0xe2 ,0x1c ,0xb3 ,0xa3 ,0xbc , +0x1c ,0x93 ,0x1a ,0xb1 ,0x71 ,0xaf ,0xe2 ,0xdd , +0x17 ,0xee ,0x53 ,0xfa ,0xfb ,0x02 ,0x40 ,0x3e , +0x03 ,0xca ,0xe7 ,0xc3 ,0x51 ,0x81 ,0xcc ,0x8c , +0xca ,0xcf ,0x4e ,0xc5 ,0x78 ,0x99 ,0xfd ,0xbf , +0xea ,0xab ,0x38 ,0x81 ,0xfc ,0xd1 ,0x9e ,0x41 , +0x0b ,0x84 ,0x25 ,0xf1 ,0x6b ,0x3c ,0xf5 ,0x40 , +0x0d ,0xc4 ,0x3e ,0xb3 ,0x6a ,0xec ,0x6e ,0x75 , +0xdc ,0x9b ,0xdf ,0x08 ,0x21 ,0x16 ,0xfb ,0x7a , +0x8e ,0x19 ,0x13 ,0x02 ,0xa7 ,0xfc ,0x58 ,0x21 , +0xc3 ,0xb3 ,0x59 ,0x5a ,0x9c ,0xef ,0x38 ,0xbd , +0x87 ,0x55 ,0xd7 ,0x0d ,0x1f ,0x84 ,0xdc ,0x98 , +0x22 ,0xca ,0x87 ,0x96 ,0x71 ,0x6d ,0x68 ,0x00 , +0xcb ,0x4f ,0x2f ,0xc4 ,0x64 ,0x0c ,0xc1 ,0x53 , +0x0c ,0x90 ,0xe7 ,0x3c ,0x88 ,0xca ,0xc5 ,0x85 , +0xa3 ,0x2a ,0x96 ,0x7c ,0x82 ,0x6d ,0x45 ,0xf5 , +0xb7 ,0x8d ,0x17 ,0x69 ,0xd6 ,0xcd ,0x3c ,0xd3 , +0xe7 ,0x1c ,0xce ,0x93 ,0x50 ,0xd4 ,0x59 ,0xa2 , +0xd8 ,0x8b ,0x72 ,0x60 ,0x5b ,0x25 ,0x14 ,0xcd , +0x5a ,0xe8 ,0x8c ,0xdb ,0x23 ,0x8d ,0x2b ,0x59 , +0x12 ,0x13 ,0x10 ,0x47 ,0xa4 ,0xc8 ,0x3c ,0xc1 , +0x81 ,0x89 ,0x6c ,0x98 ,0xec ,0x8f ,0x7b ,0x32 , +0xf2 ,0x87 ,0xd9 ,0xa2 ,0x0d ,0xc2 ,0x08 ,0xf9 , +0xd5 ,0xf3 ,0x91 ,0xe7 ,0xb3 ,0x87 ,0xa7 ,0x0b , +0x64 ,0x8f ,0xb9 ,0x55 ,0x1c ,0x81 ,0x96 ,0x6c , +0xa1 ,0xc9 ,0x6e ,0x3b ,0xcd ,0x17 ,0x1b ,0xfc , +0xa6 ,0x05 ,0xba ,0x4a ,0x7d ,0x03 ,0x3c ,0x59 , +0xc8 ,0xee ,0x50 ,0xb2 ,0x5b ,0xe1 ,0x4d ,0x6a , +0x1f ,0x09 ,0xdc ,0xa2 ,0x51 ,0xd1 ,0x93 ,0x3a , +0x5f ,0x72 ,0x1d ,0x26 ,0x14 ,0x62 ,0xa2 ,0x41 , +0x3d ,0x08 ,0x70 ,0x7b ,0x27 ,0x3d ,0xbc ,0xdf , +0x15 ,0xfa ,0xb9 ,0x5f ,0xb5 ,0x38 ,0x84 ,0x0b , +0x58 ,0x3d ,0xee ,0x3f ,0x32 ,0x65 ,0x6d ,0xd7 , +0xce ,0x97 ,0x3c ,0x8d ,0xfb ,0x63 ,0xb9 ,0xb0 , +0xa8 ,0x4a ,0x72 ,0x99 ,0x97 ,0x58 ,0xc8 ,0xa7 , +0xf9 ,0x4c ,0xae ,0xc1 ,0x63 ,0xb9 ,0x57 ,0x18 , +0x8a ,0xfa ,0xab ,0xe9 ,0xf3 ,0x67 ,0xe6 ,0xfd , +0xd2 ,0x9d ,0x5c ,0xa9 ,0x8e ,0x11 ,0x0a ,0xf4 , +0x4b ,0xf1 ,0xec ,0x1a ,0xaf ,0x50 ,0x5d ,0x16 , +0x13 ,0x69 ,0x2e ,0xbd ,0x0d ,0xe6 ,0xf0 ,0xb2 , +0xed ,0xb4 ,0x4c ,0x59 ,0x77 ,0x37 ,0x00 ,0x0b , +0xc7 ,0xa7 ,0x9e ,0x37 ,0xf3 ,0x60 ,0x70 ,0xef , +0xf3 ,0xc1 ,0x74 ,0x52 ,0x87 ,0xc6 ,0xa1 ,0x81 , +0xbd ,0x0a ,0x2c ,0x5d ,0x2c ,0x0c ,0x6a ,0x81 , +0xa1 ,0xfe ,0x26 ,0x78 ,0x6c ,0x03 ,0x06 ,0x07 , +0x34 ,0xaa ,0xd1 ,0x1b ,0x40 ,0x03 ,0x39 ,0x56 , +0xcf ,0x2a ,0x92 ,0xc1 ,0x4e ,0xdf ,0x29 ,0x24 , +0x83 ,0x22 ,0x7a ,0xea ,0x67 ,0x1e ,0xe7 ,0x54 , +0x64 ,0xd3 ,0xbd ,0x3a ,0x5d ,0xae ,0xca ,0xf0 , +0x9c ,0xd6 ,0x5a ,0x9a ,0x62 ,0xc8 ,0xc7 ,0x83 , +0xf9 ,0x89 ,0xde ,0x2d ,0x53 ,0x64 ,0x61 ,0xf7 , +0xa3 ,0xa7 ,0x31 ,0x38 ,0xc6 ,0x22 ,0x9c ,0xb4 , +0x87 ,0xe0 +}; +static const u8 output73[]__initconst = { +0x34 ,0xed ,0x05 ,0xb0 ,0x14 ,0xbc ,0x8c ,0xcc , +0x95 ,0xbd ,0x99 ,0x0f ,0xb1 ,0x98 ,0x17 ,0x10 , +0xae ,0xe0 ,0x08 ,0x53 ,0xa3 ,0x69 ,0xd2 ,0xed , +0x66 ,0xdb ,0x2a ,0x34 ,0x8d ,0x0c ,0x6e ,0xce , +0x63 ,0x69 ,0xc9 ,0xe4 ,0x57 ,0xc3 ,0x0c ,0x8b , +0xa6 ,0x2c ,0xa7 ,0xd2 ,0x08 ,0xff ,0x4f ,0xec , +0x61 ,0x8c ,0xee ,0x0d ,0xfa ,0x6b ,0xe0 ,0xe8 , +0x71 ,0xbc ,0x41 ,0x46 ,0xd7 ,0x33 ,0x1d ,0xc0 , +0xfd ,0xad ,0xca ,0x8b ,0x34 ,0x56 ,0xa4 ,0x86 , +0x71 ,0x62 ,0xae ,0x5e ,0x3d ,0x2b ,0x66 ,0x3e , +0xae ,0xd8 ,0xc0 ,0xe1 ,0x21 ,0x3b ,0xca ,0xd2 , +0x6b ,0xa2 ,0xb8 ,0xc7 ,0x98 ,0x4a ,0xf3 ,0xcf , +0xb8 ,0x62 ,0xd8 ,0x33 ,0xe6 ,0x80 ,0xdb ,0x2f , +0x0a ,0xaf ,0x90 ,0x3c ,0xe1 ,0xec ,0xe9 ,0x21 , +0x29 ,0x42 ,0x9e ,0xa5 ,0x50 ,0xe9 ,0x93 ,0xd3 , +0x53 ,0x1f ,0xac ,0x2a ,0x24 ,0x07 ,0xb8 ,0xed , +0xed ,0x38 ,0x2c ,0xc4 ,0xa1 ,0x2b ,0x31 ,0x5d , +0x9c ,0x24 ,0x7b ,0xbf ,0xd9 ,0xbb ,0x4e ,0x87 , +0x8f ,0x32 ,0x30 ,0xf1 ,0x11 ,0x29 ,0x54 ,0x94 , +0x00 ,0x95 ,0x1d ,0x1d ,0x24 ,0xc0 ,0xd4 ,0x34 , +0x49 ,0x1d ,0xd5 ,0xe3 ,0xa6 ,0xde ,0x8b ,0xbf , +0x5a ,0x9f ,0x58 ,0x5a ,0x9b ,0x70 ,0xe5 ,0x9b , +0xb3 ,0xdb ,0xe8 ,0xb8 ,0xca ,0x1b ,0x43 ,0xe3 , +0xc6 ,0x6f ,0x0a ,0xd6 ,0x32 ,0x11 ,0xd4 ,0x04 , +0xef ,0xa3 ,0xe4 ,0x3f ,0x12 ,0xd8 ,0xc1 ,0x73 , +0x51 ,0x87 ,0x03 ,0xbd ,0xba ,0x60 ,0x79 ,0xee , +0x08 ,0xcc ,0xf7 ,0xc0 ,0xaa ,0x4c ,0x33 ,0xc4 , +0xc7 ,0x09 ,0xf5 ,0x91 ,0xcb ,0x74 ,0x57 ,0x08 , +0x1b ,0x90 ,0xa9 ,0x1b ,0x60 ,0x02 ,0xd2 ,0x3f , +0x7a ,0xbb ,0xfd ,0x78 ,0xf0 ,0x15 ,0xf9 ,0x29 , +0x82 ,0x8f ,0xc4 ,0xb2 ,0x88 ,0x1f ,0xbc ,0xcc , +0x53 ,0x27 ,0x8b ,0x07 ,0x5f ,0xfc ,0x91 ,0x29 , +0x82 ,0x80 ,0x59 ,0x0a ,0x3c ,0xea ,0xc4 ,0x7e , +0xad ,0xd2 ,0x70 ,0x46 ,0xbd ,0x9e ,0x3b ,0x1c , +0x8a ,0x62 ,0xea ,0x69 ,0xbd ,0xf6 ,0x96 ,0x15 , +0xb5 ,0x57 ,0xe8 ,0x63 ,0x5f ,0x65 ,0x46 ,0x84 , +0x58 ,0x50 ,0x87 ,0x4b ,0x0e ,0x5b ,0x52 ,0x90 , +0xb0 ,0xae ,0x37 ,0x0f ,0xdd ,0x7e ,0xa2 ,0xa0 , +0x8b ,0x78 ,0xc8 ,0x5a ,0x1f ,0x53 ,0xdb ,0xc5 , +0xbf ,0x73 ,0x20 ,0xa9 ,0x44 ,0xfb ,0x1e ,0xc7 , +0x97 ,0xb2 ,0x3a ,0x5a ,0x17 ,0xe6 ,0x8b ,0x9b , +0xe8 ,0xf8 ,0x2a ,0x01 ,0x27 ,0xa3 ,0x71 ,0x28 , +0xe3 ,0x19 ,0xc6 ,0xaf ,0xf5 ,0x3a ,0x26 ,0xc0 , +0x5c ,0x69 ,0x30 ,0x78 ,0x75 ,0x27 ,0xf2 ,0x0c , +0x22 ,0x71 ,0x65 ,0xc6 ,0x8e ,0x7b ,0x47 ,0xe3 , +0x31 ,0xaf ,0x7b ,0xc6 ,0xc2 ,0x55 ,0x68 ,0x81 , +0xaa ,0x1b ,0x21 ,0x65 ,0xfb ,0x18 ,0x35 ,0x45 , +0x36 ,0x9a ,0x44 ,0xba ,0x5c ,0xff ,0x06 ,0xde , +0x3a ,0xc8 ,0x44 ,0x0b ,0xaa ,0x8e ,0x34 ,0xe2 , +0x84 ,0xac ,0x18 ,0xfe ,0x9b ,0xe1 ,0x4f ,0xaa , +0xb6 ,0x90 ,0x0b ,0x1c ,0x2c ,0xd9 ,0x9a ,0x10 , +0x18 ,0xf9 ,0x49 ,0x41 ,0x42 ,0x1b ,0xb5 ,0xe1 , +0x26 ,0xac ,0x2d ,0x38 ,0x00 ,0x00 ,0xe4 ,0xb4 , +0x50 ,0x6f ,0x14 ,0x18 ,0xd6 ,0x3d ,0x00 ,0x59 , +0x3c ,0x45 ,0xf3 ,0x42 ,0x13 ,0x44 ,0xb8 ,0x57 , +0xd4 ,0x43 ,0x5c ,0x8a ,0x2a ,0xb4 ,0xfc ,0x0a , +0x25 ,0x5a ,0xdc ,0x8f ,0x11 ,0x0b ,0x11 ,0x44 , +0xc7 ,0x0e ,0x54 ,0x8b ,0x22 ,0x01 ,0x7e ,0x67 , +0x2e ,0x15 ,0x3a ,0xb9 ,0xee ,0x84 ,0x10 ,0xd4 , +0x80 ,0x57 ,0xd7 ,0x75 ,0xcf ,0x8b ,0xcb ,0x03 , +0xc9 ,0x92 ,0x2b ,0x69 ,0xd8 ,0x5a ,0x9b ,0x06 , +0x85 ,0x47 ,0xaa ,0x4c ,0x28 ,0xde ,0x49 ,0x58 , +0xe6 ,0x11 ,0x1e ,0x5e ,0x64 ,0x8e ,0x3b ,0xe0 , +0x40 ,0x2e ,0xac ,0x96 ,0x97 ,0x15 ,0x37 ,0x1e , +0x30 ,0xdd +}; +static const u8 key73[]__initconst = { +0x96 ,0x06 ,0x1e ,0xc1 ,0x6d ,0xba ,0x49 ,0x5b , +0x65 ,0x80 ,0x79 ,0xdd ,0xf3 ,0x67 ,0xa8 ,0x6e , +0x2d ,0x9c ,0x54 ,0x46 ,0xd8 ,0x4a ,0xeb ,0x7e , +0x23 ,0x86 ,0x51 ,0xd8 ,0x49 ,0x49 ,0x56 ,0xe0 +}; +enum { nonce73 =0xbefb83cb67e11ffdULL }; + +static const u8 input74[]__initconst = { +0x47 ,0x22 ,0x70 ,0xe5 ,0x2f ,0x41 ,0x18 ,0x45 , +0x07 ,0xd3 ,0x6d ,0x32 ,0x0d ,0x43 ,0x92 ,0x2b , +0x9b ,0x65 ,0x73 ,0x13 ,0x1a ,0x4f ,0x49 ,0x8f , +0xff ,0xf8 ,0xcc ,0xae ,0x15 ,0xab ,0x9d ,0x7d , +0xee ,0x22 ,0x5d ,0x8b ,0xde ,0x81 ,0x5b ,0x81 , +0x83 ,0x49 ,0x35 ,0x9b ,0xb4 ,0xbc ,0x4e ,0x01 , +0xc2 ,0x29 ,0xa7 ,0xf1 ,0xca ,0x3a ,0xce ,0x3f , +0xf5 ,0x31 ,0x93 ,0xa8 ,0xe2 ,0xc9 ,0x7d ,0x03 , +0x26 ,0xa4 ,0xbc ,0xa8 ,0x9c ,0xb9 ,0x68 ,0xf3 , +0xb3 ,0x91 ,0xe8 ,0xe6 ,0xc7 ,0x2b ,0x1a ,0xce , +0xd2 ,0x41 ,0x53 ,0xbd ,0xa3 ,0x2c ,0x54 ,0x94 , +0x21 ,0xa1 ,0x40 ,0xae ,0xc9 ,0x0c ,0x11 ,0x92 , +0xfd ,0x91 ,0xa9 ,0x40 ,0xca ,0xde ,0x21 ,0x4e , +0x1e ,0x3d ,0xcc ,0x2c ,0x87 ,0x11 ,0xef ,0x46 , +0xed ,0x52 ,0x03 ,0x11 ,0x19 ,0x43 ,0x25 ,0xc7 , +0x0d ,0xc3 ,0x37 ,0x5f ,0xd3 ,0x6f ,0x0c ,0x6a , +0x45 ,0x30 ,0x88 ,0xec ,0xf0 ,0x21 ,0xef ,0x1d , +0x7b ,0x38 ,0x63 ,0x4b ,0x49 ,0x0c ,0x72 ,0xf6 , +0x4c ,0x40 ,0xc3 ,0xcc ,0x03 ,0xa7 ,0xae ,0xa8 , +0x8c ,0x37 ,0x03 ,0x1c ,0x11 ,0xae ,0x0d ,0x1b , +0x62 ,0x97 ,0x27 ,0xfc ,0x56 ,0x4b ,0xb7 ,0xfd , +0xbc ,0xfb ,0x0e ,0xfc ,0x61 ,0xad ,0xc6 ,0xb5 , +0x9c ,0x8c ,0xc6 ,0x38 ,0x27 ,0x91 ,0x29 ,0x3d , +0x29 ,0xc8 ,0x37 ,0xc9 ,0x96 ,0x69 ,0xe3 ,0xdc , +0x3e ,0x61 ,0x35 ,0x9b ,0x99 ,0x4f ,0xb9 ,0x4e , +0x5a ,0x29 ,0x1c ,0x2e ,0xcf ,0x16 ,0xcb ,0x69 , +0x87 ,0xe4 ,0x1a ,0xc4 ,0x6e ,0x78 ,0x43 ,0x00 , +0x03 ,0xb2 ,0x8b ,0x03 ,0xd0 ,0xb4 ,0xf1 ,0xd2 , +0x7d ,0x2d ,0x7e ,0xfc ,0x19 ,0x66 ,0x5b ,0xa3 , +0x60 ,0x3f ,0x9d ,0xbd ,0xfa ,0x3e ,0xca ,0x7b , +0x26 ,0x08 ,0x19 ,0x16 ,0x93 ,0x5d ,0x83 ,0xfd , +0xf9 ,0x21 ,0xc6 ,0x31 ,0x34 ,0x6f ,0x0c ,0xaa , +0x28 ,0xf9 ,0x18 ,0xa2 ,0xc4 ,0x78 ,0x3b ,0x56 , +0xc0 ,0x88 ,0x16 ,0xba ,0x22 ,0x2c ,0x07 ,0x2f , +0x70 ,0xd0 ,0xb0 ,0x46 ,0x35 ,0xc7 ,0x14 ,0xdc , +0xbb ,0x56 ,0x23 ,0x1e ,0x36 ,0x36 ,0x2d ,0x73 , +0x78 ,0xc7 ,0xce ,0xf3 ,0x58 ,0xf7 ,0x58 ,0xb5 , +0x51 ,0xff ,0x33 ,0x86 ,0x0e ,0x3b ,0x39 ,0xfb , +0x1a ,0xfd ,0xf8 ,0x8b ,0x09 ,0x33 ,0x1b ,0x83 , +0xf2 ,0xe6 ,0x38 ,0x37 ,0xef ,0x47 ,0x84 ,0xd9 , +0x82 ,0x77 ,0x2b ,0x82 ,0xcc ,0xf9 ,0xee ,0x94 , +0x71 ,0x78 ,0x81 ,0xc8 ,0x4d ,0x91 ,0xd7 ,0x35 , +0x29 ,0x31 ,0x30 ,0x5c ,0x4a ,0x23 ,0x23 ,0xb1 , +0x38 ,0x6b ,0xac ,0x22 ,0x3f ,0x80 ,0xc7 ,0xe0 , +0x7d ,0xfa ,0x76 ,0x47 ,0xd4 ,0x6f ,0x93 ,0xa0 , +0xa0 ,0x93 ,0x5d ,0x68 ,0xf7 ,0x43 ,0x25 ,0x8f , +0x1b ,0xc7 ,0x87 ,0xea ,0x59 ,0x0c ,0xa2 ,0xfa , +0xdb ,0x2f ,0x72 ,0x43 ,0xcf ,0x90 ,0xf1 ,0xd6 , +0x58 ,0xf3 ,0x17 ,0x6a ,0xdf ,0xb3 ,0x4e ,0x0e , +0x38 ,0x24 ,0x48 ,0x1f ,0xb7 ,0x01 ,0xec ,0x81 , +0xb1 ,0x87 ,0x5b ,0xec ,0x9c ,0x11 ,0x1a ,0xff , +0xa5 ,0xca ,0x5a ,0x63 ,0x31 ,0xb2 ,0xe4 ,0xc6 , +0x3c ,0x1d ,0xaf ,0x27 ,0xb2 ,0xd4 ,0x19 ,0xa2 , +0xcc ,0x04 ,0x92 ,0x42 ,0xd2 ,0xc1 ,0x8c ,0x3b , +0xce ,0xf5 ,0x74 ,0xc1 ,0x81 ,0xf8 ,0x20 ,0x23 , +0x6f ,0x20 ,0x6d ,0x78 ,0x36 ,0x72 ,0x2c ,0x52 , +0xdf ,0x5e ,0xe8 ,0x75 ,0xce ,0x1c ,0x49 ,0x9d , +0x93 ,0x6f ,0x65 ,0xeb ,0xb1 ,0xbd ,0x8e ,0x5e , +0xe5 ,0x89 ,0xc4 ,0x8a ,0x81 ,0x3d ,0x9a ,0xa7 , +0x11 ,0x82 ,0x8e ,0x38 ,0x5b ,0x5b ,0xca ,0x7d , +0x4b ,0x72 ,0xc2 ,0x9c ,0x30 ,0x5e ,0x7f ,0xc0 , +0x6f ,0x91 ,0xd5 ,0x67 ,0x8c ,0x3e ,0xae ,0xda , +0x2b ,0x3c ,0x53 ,0xcc ,0x50 ,0x97 ,0x36 ,0x0b , +0x79 ,0xd6 ,0x73 ,0x6e ,0x7d ,0x42 ,0x56 ,0xe1 , +0xaa ,0xfc ,0xb3 ,0xa7 ,0xc8 ,0x01 ,0xaa ,0xc1 , +0xfc ,0x5c ,0x72 ,0x8e ,0x63 ,0xa8 ,0x46 ,0x18 , +0xee ,0x11 ,0xe7 ,0x30 ,0x09 ,0x83 ,0x6c ,0xd9 , +0xf4 ,0x7a ,0x7b ,0xb5 ,0x1f ,0x6d ,0xc7 ,0xbc , +0xcb ,0x55 ,0xea ,0x40 ,0x58 ,0x7a ,0x00 ,0x00 , +0x90 ,0x60 ,0xc5 ,0x64 ,0x69 ,0x05 ,0x99 ,0xd2 , +0x49 ,0x62 ,0x4f ,0xcb ,0x97 ,0xdf ,0xdd ,0x6b , +0x60 ,0x75 ,0xe2 ,0xe0 ,0x6f ,0x76 ,0xd0 ,0x37 , +0x67 ,0x0a ,0xcf ,0xff ,0xc8 ,0x61 ,0x84 ,0x14 , +0x80 ,0x7c ,0x1d ,0x31 ,0x8d ,0x90 ,0xde ,0x0b , +0x1c ,0x74 ,0x9f ,0x82 ,0x96 ,0x80 ,0xda ,0xaf , +0x8d ,0x99 ,0x86 ,0x9f ,0x24 ,0x99 ,0x28 ,0x3e , +0xe0 ,0xa3 ,0xc3 ,0x90 ,0x2d ,0x14 ,0x65 ,0x1e , +0x3b ,0xb9 ,0xba ,0x13 ,0xa5 ,0x77 ,0x73 ,0x63 , +0x9a ,0x06 ,0x3d ,0xa9 ,0x28 ,0x9b ,0xba ,0x25 , +0x61 ,0xc9 ,0xcd ,0xcf ,0x7a ,0x4d ,0x96 ,0x09 , +0xcb ,0xca ,0x03 ,0x9c ,0x54 ,0x34 ,0x31 ,0x85 , +0xa0 ,0x3d ,0xe5 ,0xbc ,0xa5 ,0x5f ,0x1b ,0xd3 , +0x10 ,0x63 ,0x74 ,0x9d ,0x01 ,0x92 ,0x88 ,0xf0 , +0x27 ,0x9c ,0x28 ,0xd9 ,0xfd ,0xe2 ,0x4e ,0x01 , +0x8d ,0x61 ,0x79 ,0x60 ,0x61 ,0x5b ,0x76 ,0xab , +0x06 ,0xd3 ,0x44 ,0x87 ,0x43 ,0x52 ,0xcd ,0x06 , +0x68 ,0x1e ,0x2d ,0xc5 ,0xb0 ,0x07 ,0x25 ,0xdf , +0x0a ,0x50 ,0xd7 ,0xd9 ,0x08 ,0x53 ,0x65 ,0xf1 , +0x0c ,0x2c ,0xde ,0x3f ,0x9d ,0x03 ,0x1f ,0xe1 , +0x49 ,0x43 ,0x3c ,0x83 ,0x81 ,0x37 ,0xf8 ,0xa2 , +0x0b ,0xf9 ,0x61 ,0x1c ,0xc1 ,0xdb ,0x79 ,0xbc , +0x64 ,0xce ,0x06 ,0x4e ,0x87 ,0x89 ,0x62 ,0x73 , +0x51 ,0xbc ,0xa4 ,0x32 ,0xd4 ,0x18 ,0x62 ,0xab , +0x65 ,0x7e ,0xad ,0x1e ,0x91 ,0xa3 ,0xfa ,0x2d , +0x58 ,0x9e ,0x2a ,0xe9 ,0x74 ,0x44 ,0x64 ,0x11 , +0xe6 ,0xb6 ,0xb3 ,0x00 ,0x7e ,0xa3 ,0x16 ,0xef , +0x72 +}; +static const u8 output74[]__initconst = { +0xf5 ,0xca ,0x45 ,0x65 ,0x50 ,0x35 ,0x47 ,0x67 , +0x6f ,0x4f ,0x67 ,0xff ,0x34 ,0xd9 ,0xc3 ,0x37 , +0x2a ,0x26 ,0xb0 ,0x4f ,0x08 ,0x1e ,0x45 ,0x13 , +0xc7 ,0x2c ,0x14 ,0x75 ,0x33 ,0xd8 ,0x8e ,0x1e , +0x1b ,0x11 ,0x0d ,0x97 ,0x04 ,0x33 ,0x8a ,0xe4 , +0xd8 ,0x8d ,0x0e ,0x12 ,0x8d ,0xdb ,0x6e ,0x02 , +0xfa ,0xe5 ,0xbd ,0x3a ,0xb5 ,0x28 ,0x07 ,0x7d , +0x20 ,0xf0 ,0x12 ,0x64 ,0x83 ,0x2f ,0x59 ,0x79 , +0x17 ,0x88 ,0x3c ,0x2d ,0x08 ,0x2f ,0x55 ,0xda , +0xcc ,0x02 ,0x3a ,0x82 ,0xcd ,0x03 ,0x94 ,0xdf , +0xdf ,0xab ,0x8a ,0x13 ,0xf5 ,0xe6 ,0x74 ,0xdf , +0x7b ,0xe2 ,0xab ,0x34 ,0xbc ,0x00 ,0x85 ,0xbf , +0x5a ,0x48 ,0xc8 ,0xff ,0x8d ,0x6c ,0x27 ,0x48 , +0x19 ,0x2d ,0x08 ,0xfa ,0x82 ,0x62 ,0x39 ,0x55 , +0x32 ,0x11 ,0xa8 ,0xd7 ,0xb9 ,0x08 ,0x2c ,0xd6 , +0x7a ,0xd9 ,0x83 ,0x9f ,0x9b ,0xfb ,0xec ,0x3a , +0xd1 ,0x08 ,0xc7 ,0xad ,0xdc ,0x98 ,0x4c ,0xbc , +0x98 ,0xeb ,0x36 ,0xb0 ,0x39 ,0xf4 ,0x3a ,0xd6 , +0x53 ,0x02 ,0xa0 ,0xa9 ,0x73 ,0xa1 ,0xca ,0xef , +0xd8 ,0xd2 ,0xec ,0x0e ,0xf8 ,0xf5 ,0xac ,0x8d , +0x34 ,0x41 ,0x06 ,0xa8 ,0xc6 ,0xc3 ,0x31 ,0xbc , +0xe5 ,0xcc ,0x7e ,0x72 ,0x63 ,0x59 ,0x3e ,0x63 , +0xc2 ,0x8d ,0x2b ,0xd5 ,0xb9 ,0xfd ,0x1e ,0x31 , +0x69 ,0x32 ,0x05 ,0xd6 ,0xde ,0xc9 ,0xe6 ,0x4c , +0xac ,0x68 ,0xf7 ,0x1f ,0x9d ,0xcd ,0x0e ,0xa2 , +0x15 ,0x3d ,0xd6 ,0x47 ,0x99 ,0xab ,0x08 ,0x5f , +0x28 ,0xc3 ,0x4c ,0xc2 ,0xd5 ,0xdd ,0x10 ,0xb7 , +0xbd ,0xdb ,0x9b ,0xcf ,0x85 ,0x27 ,0x29 ,0x76 , +0x98 ,0xeb ,0xad ,0x31 ,0x64 ,0xe7 ,0xfb ,0x61 , +0xe0 ,0xd8 ,0x1a ,0xa6 ,0xe2 ,0xe7 ,0x43 ,0x42 , +0x77 ,0xc9 ,0x82 ,0x00 ,0xac ,0x85 ,0xe0 ,0xa2 , +0xd4 ,0x62 ,0xe3 ,0xb7 ,0x17 ,0x6e ,0xb2 ,0x9e , +0x21 ,0x58 ,0x73 ,0xa9 ,0x53 ,0x2d ,0x3c ,0xe1 , +0xdd ,0xd6 ,0x6e ,0x92 ,0xf2 ,0x1d ,0xc2 ,0x22 , +0x5f ,0x9a ,0x7e ,0xd0 ,0x52 ,0xbf ,0x54 ,0x19 , +0xd7 ,0x80 ,0x63 ,0x3e ,0xd0 ,0x08 ,0x2d ,0x37 , +0x0c ,0x15 ,0xf7 ,0xde ,0xab ,0x2b ,0xe3 ,0x16 , +0x21 ,0x3a ,0xee ,0xa5 ,0xdc ,0xdf ,0xde ,0xa3 , +0x69 ,0xcb ,0xfd ,0x92 ,0x89 ,0x75 ,0xcf ,0xc9 , +0x8a ,0xa4 ,0xc8 ,0xdd ,0xcc ,0x21 ,0xe6 ,0xfe , +0x9e ,0x43 ,0x76 ,0xb2 ,0x45 ,0x22 ,0xb9 ,0xb5 , +0xac ,0x7e ,0x3d ,0x26 ,0xb0 ,0x53 ,0xc8 ,0xab , +0xfd ,0xea ,0x2c ,0xd1 ,0x44 ,0xc5 ,0x60 ,0x1b , +0x8a ,0x99 ,0x0d ,0xa5 ,0x0e ,0x67 ,0x6e ,0x3a , +0x96 ,0x55 ,0xec ,0xe8 ,0xcc ,0xbe ,0x49 ,0xd9 , +0xf2 ,0x72 ,0x9f ,0x30 ,0x21 ,0x97 ,0x57 ,0x19 , +0xbe ,0x5e ,0x33 ,0x0c ,0xee ,0xc0 ,0x72 ,0x0d , +0x2e ,0xd1 ,0xe1 ,0x52 ,0xc2 ,0xea ,0x41 ,0xbb , +0xe1 ,0x6d ,0xd4 ,0x17 ,0xa9 ,0x8d ,0x89 ,0xa9 , +0xd6 ,0x4b ,0xc6 ,0x4c ,0xf2 ,0x88 ,0x97 ,0x54 , +0x3f ,0x4f ,0x57 ,0xb7 ,0x37 ,0xf0 ,0x2c ,0x11 , +0x15 ,0x56 ,0xdb ,0x28 ,0xb5 ,0x16 ,0x84 ,0x66 , +0xce ,0x45 ,0x3f ,0x61 ,0x75 ,0xb6 ,0xbe ,0x00 , +0xd1 ,0xe4 ,0xf5 ,0x27 ,0x54 ,0x7f ,0xc2 ,0xf1 , +0xb3 ,0x32 ,0x9a ,0xe8 ,0x07 ,0x02 ,0xf3 ,0xdb , +0xa9 ,0xd1 ,0xc2 ,0xdf ,0xee ,0xad ,0xe5 ,0x8a , +0x3c ,0xfa ,0x67 ,0xec ,0x6b ,0xa4 ,0x08 ,0xfe , +0xba ,0x5a ,0x58 ,0x0b ,0x78 ,0x11 ,0x91 ,0x76 , +0xe3 ,0x1a ,0x28 ,0x54 ,0x5e ,0xbd ,0x71 ,0x1b , +0x8b ,0xdc ,0x6c ,0xf4 ,0x6f ,0xd7 ,0xf4 ,0xf3 , +0xe1 ,0x03 ,0xa4 ,0x3c ,0x8d ,0x91 ,0x2e ,0xba , +0x5f ,0x7f ,0x8c ,0xaf ,0x69 ,0x89 ,0x29 ,0x0a , +0x5b ,0x25 ,0x13 ,0xc4 ,0x2e ,0x16 ,0xc2 ,0x15 , +0x07 ,0x5d ,0x58 ,0x33 ,0x7c ,0xe0 ,0xf0 ,0x55 , +0x5f ,0xbf ,0x5e ,0xf0 ,0x71 ,0x48 ,0x8f ,0xf7 , +0x48 ,0xb3 ,0xf7 ,0x0d ,0xa1 ,0xd0 ,0x63 ,0xb1 , +0xad ,0xae ,0xb5 ,0xb0 ,0x5f ,0x71 ,0xaf ,0x24 , +0x8b ,0xb9 ,0x1c ,0x44 ,0xd2 ,0x1a ,0x53 ,0xd1 , +0xd5 ,0xb4 ,0xa9 ,0xff ,0x88 ,0x73 ,0xb5 ,0xaa , +0x15 ,0x32 ,0x5f ,0x59 ,0x9d ,0x2e ,0xb5 ,0xcb , +0xde ,0x21 ,0x2e ,0xe9 ,0x35 ,0xed ,0xfd ,0x0f , +0xb6 ,0xbb ,0xe6 ,0x4b ,0x16 ,0xf1 ,0x45 ,0x1e , +0xb4 ,0x84 ,0xe9 ,0x58 ,0x1c ,0x0c ,0x95 ,0xc0 , +0xcf ,0x49 ,0x8b ,0x59 ,0xa1 ,0x78 ,0xe6 ,0x80 , +0x12 ,0x49 ,0x7a ,0xd4 ,0x66 ,0x62 ,0xdf ,0x9c , +0x18 ,0xc8 ,0x8c ,0xda ,0xc1 ,0xa6 ,0xbc ,0x65 , +0x28 ,0xd2 ,0xa4 ,0xe8 ,0xf1 ,0x35 ,0xdb ,0x5a , +0x75 ,0x1f ,0x73 ,0x60 ,0xec ,0xa8 ,0xda ,0x5a , +0x43 ,0x15 ,0x83 ,0x9b ,0xe7 ,0xb1 ,0xa6 ,0x81 , +0xbb ,0xef ,0xf3 ,0x8f ,0x0f ,0xd3 ,0x79 ,0xa2 , +0xe5 ,0xaa ,0x42 ,0xef ,0xa0 ,0x13 ,0x4e ,0x91 , +0x2d ,0xcb ,0x61 ,0x7a ,0x9a ,0x33 ,0x14 ,0x50 , +0x77 ,0x4a ,0xd0 ,0x91 ,0x48 ,0xe0 ,0x0c ,0xe0 , +0x11 ,0xcb ,0xdf ,0xb0 ,0xce ,0x06 ,0xd2 ,0x79 , +0x4d ,0x69 ,0xb9 ,0xc9 ,0x36 ,0x74 ,0x8f ,0x81 , +0x72 ,0x73 ,0xf3 ,0x17 ,0xb7 ,0x13 ,0xcb ,0x5b , +0xd2 ,0x5c ,0x33 ,0x61 ,0xb7 ,0x61 ,0x79 ,0xb0 , +0xc0 ,0x4d ,0xa1 ,0xc7 ,0x5d ,0x98 ,0xc9 ,0xe1 , +0x98 ,0xbd ,0x78 ,0x5a ,0x2c ,0x64 ,0x53 ,0xaf , +0xaf ,0x66 ,0x51 ,0x47 ,0xe4 ,0x48 ,0x66 ,0x8b , +0x07 ,0x52 ,0xa3 ,0x03 ,0x93 ,0x28 ,0xad ,0xcc , +0xa3 ,0x86 ,0xad ,0x63 ,0x04 ,0x35 ,0x6c ,0x49 , +0xd5 ,0x28 ,0x0e ,0x00 ,0x47 ,0xf4 ,0xd4 ,0x32 , +0x27 ,0x19 ,0xb3 ,0x29 ,0xe7 ,0xbc ,0xbb ,0xce , +0x3e ,0x3e ,0xd5 ,0x67 ,0x20 ,0xe4 ,0x0b ,0x75 , +0x95 ,0x24 ,0xe0 ,0x6c ,0xb6 ,0x29 ,0x0c ,0x14 , +0xfd +}; +static const u8 key74[]__initconst = { +0xf0 ,0x41 ,0x5b ,0x00 ,0x56 ,0xc4 ,0xac ,0xf6 , +0xa2 ,0x4c ,0x33 ,0x41 ,0x16 ,0x09 ,0x1b ,0x8e , +0x4d ,0xe8 ,0x8c ,0xd9 ,0x48 ,0xab ,0x3e ,0x60 , +0xcb ,0x49 ,0x3e ,0xaf ,0x2b ,0x8b ,0xc8 ,0xf0 +}; +enum { nonce74 =0xcbdb0ffd0e923384ULL }; + +static const struct chacha20_testvec chacha20_testvecs[]__initconst = { + { input01, output01, key01, nonce01,sizeof (input01) }, + { input02, output02, key02, nonce02,sizeof (input02) }, + { input03, output03, key03, nonce03,sizeof (input03) }, + { input04, output04, key04, nonce04,sizeof (input04) }, + { input05, output05, key05, nonce05,sizeof (input05) }, + { input06, output06, key06, nonce06,sizeof (input06) }, + { input07, output07, key07, nonce07,sizeof (input07) }, + { input08, output08, key08, nonce08,sizeof (input08) }, + { input09, output09, key09, nonce09,sizeof (input09) }, + { input10, output10, key10, nonce10,sizeof (input10) }, + { input11, output11, key11, nonce11,sizeof (input11) }, + { input12, output12, key12, nonce12,sizeof (input12) }, + { input13, output13, key13, nonce13,sizeof (input13) }, + { input14, output14, key14, nonce14,sizeof (input14) }, + { input15, output15, key15, nonce15,sizeof (input15) }, + { input16, output16, key16, nonce16,sizeof (input16) }, + { input17, output17, key17, nonce17,sizeof (input17) }, + { input18, output18, key18, nonce18,sizeof (input18) }, + { input19, output19, key19, nonce19,sizeof (input19) }, + { input20, output20, key20, nonce20,sizeof (input20) }, + { input21, output21, key21, nonce21,sizeof (input21) }, + { input22, output22, key22, nonce22,sizeof (input22) }, + { input23, output23, key23, nonce23,sizeof (input23) }, + { input24, output24, key24, nonce24,sizeof (input24) }, + { input25, output25, key25, nonce25,sizeof (input25) }, + { input26, output26, key26, nonce26,sizeof (input26) }, + { input27, output27, key27, nonce27,sizeof (input27) }, + { input28, output28, key28, nonce28,sizeof (input28) }, + { input29, output29, key29, nonce29,sizeof (input29) }, + { input30, output30, key30, nonce30,sizeof (input30) }, + { input31, output31, key31, nonce31,sizeof (input31) }, + { input32, output32, key32, nonce32,sizeof (input32) }, + { input33, output33, key33, nonce33,sizeof (input33) }, + { input34, output34, key34, nonce34,sizeof (input34) }, + { input35, output35, key35, nonce35,sizeof (input35) }, + { input36, output36, key36, nonce36,sizeof (input36) }, + { input37, output37, key37, nonce37,sizeof (input37) }, + { input38, output38, key38, nonce38,sizeof (input38) }, + { input39, output39, key39, nonce39,sizeof (input39) }, + { input40, output40, key40, nonce40,sizeof (input40) }, + { input41, output41, key41, nonce41,sizeof (input41) }, + { input42, output42, key42, nonce42,sizeof (input42) }, + { input43, output43, key43, nonce43,sizeof (input43) }, + { input44, output44, key44, nonce44,sizeof (input44) }, + { input45, output45, key45, nonce45,sizeof (input45) }, + { input46, output46, key46, nonce46,sizeof (input46) }, + { input47, output47, key47, nonce47,sizeof (input47) }, + { input48, output48, key48, nonce48,sizeof (input48) }, + { input49, output49, key49, nonce49,sizeof (input49) }, + { input50, output50, key50, nonce50,sizeof (input50) }, + { input51, output51, key51, nonce51,sizeof (input51) }, + { input52, output52, key52, nonce52,sizeof (input52) }, + { input53, output53, key53, nonce53,sizeof (input53) }, + { input54, output54, key54, nonce54,sizeof (input54) }, + { input55, output55, key55, nonce55,sizeof (input55) }, + { input56, output56, key56, nonce56,sizeof (input56) }, + { input57, output57, key57, nonce57,sizeof (input57) }, + { input58, output58, key58, nonce58,sizeof (input58) }, + { input59, output59, key59, nonce59,sizeof (input59) }, + { input60, output60, key60, nonce60,sizeof (input60) }, + { input61, output61, key61, nonce61,sizeof (input61) }, + { input62, output62, key62, nonce62,sizeof (input62) }, + { input63, output63, key63, nonce63,sizeof (input63) }, + { input64, output64, key64, nonce64,sizeof (input64) }, + { input65, output65, key65, nonce65,sizeof (input65) }, + { input66, output66, key66, nonce66,sizeof (input66) }, + { input67, output67, key67, nonce67,sizeof (input67) }, + { input68, output68, key68, nonce68,sizeof (input68) }, + { input69, output69, key69, nonce69,sizeof (input69) }, + { input70, output70, key70, nonce70,sizeof (input70) }, + { input71, output71, key71, nonce71,sizeof (input71) }, + { input72, output72, key72, nonce72,sizeof (input72) }, + { input73, output73, key73, nonce73,sizeof (input73) }, + { input74, output74, key74, nonce74,sizeof (input74) } +}; + +static const struct hchacha20_testvec hchacha20_testvecs[]__initconst = {{ + .key = {0x00 ,0x01 ,0x02 ,0x03 ,0x04 ,0x05 ,0x06 ,0x07 , +0x08 ,0x09 ,0x0a ,0x0b ,0x0c ,0x0d ,0x0e ,0x0f , +0x10 ,0x11 ,0x12 ,0x13 ,0x14 ,0x15 ,0x16 ,0x17 , +0x18 ,0x19 ,0x1a ,0x1b ,0x1c ,0x1d ,0x1e ,0x1f }, + .nonce = {0x00 ,0x00 ,0x00 ,0x09 ,0x00 ,0x00 ,0x00 ,0x4a , +0x00 ,0x00 ,0x00 ,0x00 ,0x31 ,0x41 ,0x59 ,0x27 }, + .output = {0x82 ,0x41 ,0x3b ,0x42 ,0x27 ,0xb2 ,0x7b ,0xfe , +0xd3 ,0x0e ,0x42 ,0x50 ,0x8a ,0x87 ,0x7d ,0x73 , +0xa0 ,0xf9 ,0xe4 ,0xd5 ,0x8a ,0x74 ,0xa8 ,0x53 , +0xc1 ,0x2e ,0xc4 ,0x13 ,0x26 ,0xd3 ,0xec ,0xdc } +}}; + +static bool __init chacha20_selftest (void ) +{ +enum {MAXIMUM_TEST_BUFFER_LEN =1UL <<10 }; +size_t i, j, k; +u32 derived_key[CHACHA20_KEY_WORDS ]; +u8 *offset_input =NULL , *computed_output =NULL ; +u8 offset_key[CHACHA20_KEY_SIZE +1 ] +__aligned(__alignof__(unsigned long)) ; +struct chacha20_ctx state; +bool success = true; +simd_context_t simd_context; + + offset_input =kmalloc (MAXIMUM_TEST_BUFFER_LEN +1 ,GFP_KERNEL ); + computed_output =kmalloc (MAXIMUM_TEST_BUFFER_LEN +1 ,GFP_KERNEL ); +if (!computed_output || !offset_input) { +pr_info ("chacha20 self-test malloc: FAIL\n" ); + success = false; + goto out; + } + +simd_get (&simd_context); +for (i =0 ; i <ARRAY_SIZE (chacha20_testvecs); ++i) { +/* Boring case */ +memset (computed_output,0 ,MAXIMUM_TEST_BUFFER_LEN +1 ); +memset (&state,0 ,sizeof (state)); +chacha20_init (&state, chacha20_testvecs[i].key, + chacha20_testvecs[i].nonce); +chacha20 (&state, computed_output, chacha20_testvecs[i].input, + chacha20_testvecs[i].ilen, &simd_context); +if (memcmp (computed_output, chacha20_testvecs[i].output, + chacha20_testvecs[i].ilen)) { +pr_info ("chacha20 self-test %zu: FAIL\n" , i +1 ); + success = false; + } +for (k = chacha20_testvecs[i].ilen; + k <MAXIMUM_TEST_BUFFER_LEN +1 ; ++k) { +if (computed_output[k]) { +pr_info ("chacha20 self-test %zu (zero check): FAIL\n" , + i +1 ); + success = false; + } + } + +/* Unaligned case */ +memset (computed_output,0 ,MAXIMUM_TEST_BUFFER_LEN +1 ); +memset (&state,0 ,sizeof (state)); +memcpy (offset_input +1 , chacha20_testvecs[i].input, + chacha20_testvecs[i].ilen); +memcpy (offset_key +1 , chacha20_testvecs[i].key, +CHACHA20_KEY_SIZE ); +chacha20_init (&state, offset_key +1 , chacha20_testvecs[i].nonce); +chacha20 (&state, computed_output +1 , offset_input +1 , + chacha20_testvecs[i].ilen, &simd_context); +if (memcmp (computed_output +1 , chacha20_testvecs[i].output, + chacha20_testvecs[i].ilen)) { +pr_info ("chacha20 self-test %zu (unaligned): FAIL\n" , + i +1 ); + success = false; + } +if (computed_output[0 ]) { +pr_info ("chacha20 self-test %zu (unaligned, zero check): FAIL\n" , + i +1 ); + success = false; + } +for (k = chacha20_testvecs[i].ilen +1 ; + k <MAXIMUM_TEST_BUFFER_LEN +1 ; ++k) { +if (computed_output[k]) { +pr_info ("chacha20 self-test %zu (unaligned, zero check): FAIL\n" , + i +1 ); + success = false; + } + } + +/* Chunked case */ +if (chacha20_testvecs[i].ilen <=CHACHA20_BLOCK_SIZE ) + goto next_test; +memset (computed_output,0 ,MAXIMUM_TEST_BUFFER_LEN +1 ); +memset (&state,0 ,sizeof (state)); +chacha20_init (&state, chacha20_testvecs[i].key, + chacha20_testvecs[i].nonce); +chacha20 (&state, computed_output, chacha20_testvecs[i].input, +CHACHA20_BLOCK_SIZE , &simd_context); +chacha20 (&state, computed_output +CHACHA20_BLOCK_SIZE , + chacha20_testvecs[i].input +CHACHA20_BLOCK_SIZE , + chacha20_testvecs[i].ilen -CHACHA20_BLOCK_SIZE , + &simd_context); +if (memcmp (computed_output, chacha20_testvecs[i].output, + chacha20_testvecs[i].ilen)) { +pr_info ("chacha20 self-test %zu (chunked): FAIL\n" , + i +1 ); + success = false; + } +for (k = chacha20_testvecs[i].ilen; + k <MAXIMUM_TEST_BUFFER_LEN +1 ; ++k) { +if (computed_output[k]) { +pr_info ("chacha20 self-test %zu (chunked, zero check): FAIL\n" , + i +1 ); + success = false; + } + } + +next_test: +/* Sliding unaligned case */ +if (chacha20_testvecs[i].ilen >CHACHA20_BLOCK_SIZE +1 || + !chacha20_testvecs[i].ilen) +continue ; +for (j =1 ; j <CHACHA20_BLOCK_SIZE ; ++j) { +memset (computed_output,0 ,MAXIMUM_TEST_BUFFER_LEN +1 ); +memset (&state,0 ,sizeof (state)); +memcpy (offset_input + j, chacha20_testvecs[i].input, + chacha20_testvecs[i].ilen); +chacha20_init (&state, chacha20_testvecs[i].key, + chacha20_testvecs[i].nonce); +chacha20 (&state, computed_output + j, offset_input + j, + chacha20_testvecs[i].ilen, &simd_context); +if (memcmp (computed_output + j, + chacha20_testvecs[i].output, + chacha20_testvecs[i].ilen)) { +pr_info ("chacha20 self-test %zu (unaligned, slide %zu): FAIL\n" , + i +1 , j); + success = false; + } +for (k = j; k < j; ++k) { +if (computed_output[k]) { +pr_info ("chacha20 self-test %zu (unaligned, slide %zu, zero check): FAIL\n" , + i +1 , j); + success = false; + } + } +for (k = chacha20_testvecs[i].ilen + j; + k <MAXIMUM_TEST_BUFFER_LEN +1 ; ++k) { +if (computed_output[k]) { +pr_info ("chacha20 self-test %zu (unaligned, slide %zu, zero check): FAIL\n" , + i +1 , j); + success = false; + } + } + } + } +for (i =0 ; i <ARRAY_SIZE (hchacha20_testvecs); ++i) { +memset (&derived_key,0 ,sizeof (derived_key)); +hchacha20 (derived_key, hchacha20_testvecs[i].nonce, + hchacha20_testvecs[i].key, &simd_context); +cpu_to_le32_array (derived_key,ARRAY_SIZE (derived_key)); +if (memcmp (derived_key, hchacha20_testvecs[i].output, +CHACHA20_KEY_SIZE )) { +pr_info ("hchacha20 self-test %zu: FAIL\n" , i +1 ); + success = false; + } + } +simd_put (&simd_context); +if (success) +pr_info ("chacha20 self-tests: pass\n" ); + +out: +kfree (offset_input); +kfree (computed_output); +return success; +} +#endif --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest
Hi Ard,
Quoted 9 lines · Ard Biesheuvel
include/zinc/chacha20.h line 36
Quoted 6 lines
include/zinc/chacha20.h new file +6
-0+struct chacha20_ctx { +u32 constant[4 ]; +u32 key[8 ]; +u32 counter[4 ]; +}__aligned(32) ; +32 *byte* alignment? Is that right? If this is for performance and it actually helps, using
__cacheline_alignedis more appropriate,
It was originally this way, I believe, for vmovdqa, which requires 32-byte alignment in vex.256 encoding, and I never removed it after changing some things. But I'll spend some time ensuring this is so and if it doesn't make sense anymore it'll be gone by v7. On the other hand, your suggestion of __cacheline_aligned may actually be something worth considering, especially on MIPS.
Quoted 3 lines
I guess this include is for
crypto_xor_cpy()?We may want to put a comment here, so we keep track of the interdependencies.
Right, it's for crypto_xor_cpy. Good idea, I'll add the comment.
Quoted 4 lines
Quoted 1 line
+#ifndef
HAVE_CHACHA20_ARCH_IMPLEMENTATIONThis #define is never set in subsequent patches, so just drop this #ifndef entirely (for this patch only)
Okay. It's also there for the other primitives too; I'll nix it for all of them.
Quoted 10 lines
Return values from initcalls are ignored, and given that chacha20 will be depended upon by
random.c, it will never be a module in practice.Given your previous statement that selftest should *not* be a DEBUG feature (which I wholeheartedly agree with), you could be a bit noisier here imo.
E.g.,if (WARN_ON(!chach20_selftest())
return ...
That's an excellent idea. We could bloat the whole thing with something like:
#ifdef MODULE
#define WARN_ON_IF_MODULE(x) (x)
#else
#define WARN_ON_IF_MODULE(x) WARN_ON(x)
#endif
But given that kind of thing would probably need to touch more files in the tree, and hence involve more drawn-out conversation, I'll keep it as the simple WARN_ON for now. Besides, being noisy no matter what might actually be the best strategy for receiving bug reports on what is potentially a pretty catastrophic error.
Thanks for the review.
Jason
Herbert Xu
maintainerRe: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest
In reply to Jason A. DonenfeldQuoted 11 lines · Jason A. Donenfeld
This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.
Information: https://cr.yp.to/chacha.html
Signed-off-by Jason A. Donenfeld <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>
I don't have any problems with this interface.
However, I'm very unhappy about the fact that this adds another copy of the chacha20 implementation to the kernel. Either this is better than the one that we already have or it isn't.
So if it's better please replace the existing chacha20 C implementation with this or otherwise just adapt the existing implementation to this interface.
If we're replacing the existing one then you can either replace it directly within this patch or in a subsequent patch but please do it in the same series because otherwise we could wind up with two implementations for years to come.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest
Hi Herbert,
Quoted 6 lines · Herbert Xu
I don't have any problems with this interface.
However, I'm very unhappy about the fact that this adds another copy of the chacha20 implementation to the kernel. Either this is better than the one that we already have or it isn't.
Quoted 8 lines
So if it's better please replace the existing chacha20 C implementation with this or otherwise just adapt the existing implementation to this interface.
If we're replacing the existing one then you can either replace it directly within this patch or in a subsequent patch but please do it in the same series because otherwise we could wind up with two implementations for years to come.
Indeed this series takes care of removing the other implementation later on.
In case it helps to see things visualized, here's the layout of the whole series:
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard
Note that comments on this are probably a bit more useful than on the version posted to the mailing list, since I've already started incorporating quite a bit of feedback from v6, and so v7 will be posted sometime later containing these.
Jason
Jason A. Donenfeld
maintainerPATCH v6 4/23 zinc: ChaCha20 x86_64 implementation
In reply to Jason A. DonenfeldThis provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.
- When aligning the stack, the original code was saving %rsp to %r9.
To keep objtool happy, we use instead the DRAP idiom to save %rsp
to %r10:
leaq 8(%rsp),%r10
... code here ...
leaq -8(%r10),%rsp - The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.
- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da
Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:
size old new
---- ---- ----
0 62 52
16 414 376
32 410 400
48 414 422
64 362 356
80 714 666
96 714 700
112 712 718
128 692 646
144 1042 674
160 1042 694
176 1042 726
192 1018 650
208 1366 686
224 1366 696
240 1366 722
256 640 656
272 988 1246
288 988 1276
304 992 1296
320 972 1222
336 1318 1256
352 1318 1276
368 1316 1294
384 1294 1218
400 1642 1258
416 1642 1282
432 1642 1302
448 1628 1224
464 1970 1258
480 1970 1280
496 1970 1300
512 656 676
528 1010 1290
544 1010 1306
560 1010 1332
576 986 1254
592 1340 1284
608 1334 1310
624 1340 1334
640 1314 1254
656 1664 1282
672 1674 1306
688 1662 1336
704 1638 1250
720 1992 1292
736 1994 1308
752 1988 1334
768 1252 1254
784 1596 1290
800 1596 1314
816 1596 1330
832 1576 1256
848 1922 1286
864 1922 1314
880 1926 1338
896 1898 1258
912 2248 1288
928 2248 1320
944 2248 1338
960 2226 1268
976 2574 1288
992 2576 1312
1008 2574 1340
Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:
size old new
---- ---- ----
0 64 54
16 386 372
32 388 396
48 388 420
64 366 350
80 708 666
96 708 692
112 706 736
128 692 648
144 1036 682
160 1036 708
176 1036 730
192 1016 658
208 1360 684
224 1362 708
240 1360 732
256 644 500
272 990 526
288 988 556
304 988 576
320 972 500
336 1314 532
352 1316 558
368 1318 578
384 1308 506
400 1644 532
416 1644 556
432 1644 594
448 1624 508
464 1970 534
480 1970 556
496 1968 582
512 660 624
528 1016 682
544 1016 702
560 1018 728
576 998 654
592 1344 680
608 1344 708
624 1344 730
640 1326 654
656 1670 686
672 1670 708
688 1670 732
704 1652 658
720 1998 682
736 1998 710
752 1996 734
768 1256 662
784 1606 688
800 1606 714
816 1606 736
832 1584 660
848 1948 688
864 1950 714
880 1948 736
896 1912 688
912 2258 718
928 2258 744
944 2256 768
960 2238 692
976 2584 718
992 2584 744
1008 2584 770
Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: [email protected]
--- lib/zinc/Makefile | 1 + lib/zinc/chacha20/chacha20-x86_64-glue.h | 105 + lib/zinc/chacha20/chacha20-x86_64.S | 2632 ++++++++++++++++++++++ lib/zinc/chacha20/chacha20.c | 4 +- 4 files changed, 2741 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +1 -0
@@ -3,4 +3,5 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +105 -0
105 lines not on this page · raw message
lib/zinc/chacha20/chacha20-x86_64.S new file +2632 -0
2,632 lines not on this page · raw message
lib/zinc/chacha20/chacha20.c +3 -1
@@ -14,7 +14,9 @@#include <linux/init.h> #include <crypto/algapi.h> -#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION +#if defined(CONFIG_ZINC_ARCH_X86_64 ) +#include "chacha20-x86_64-glue.h" +#else void __init chacha20_fpu_init (void ) { }
Signature · 1 line
2.19.0
Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Quoted 30 lines · Jason A. Donenfeld
This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.
- When aligning the stack, the original code was saving %rsp to %r9.
To keep objtool happy, we use instead the DRAP idiom to save %rsp to %r10: leaq 8(%rsp),%r10 ... code here ... leaq -8(%r10),%rsp- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rspwas added in the prolog of a few functions.
Quoted 9 lines
- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
cded951378069a478391843f5f8653c1eb5128da
I'd still prefer the kernel side changes to be presented as a separate followup patch (preferably based on the .pl but I understand that is more difficult with x86 than with ARM code)
Quoted 138 lines
Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:
size old new ---- ---- ---- 0 62 52 16 414 376 32 410 400 48 414 422 64 362 356 80 714 666 96 714 700 112 712 718 128 692 646 144 1042 674 160 1042 694 176 1042 726 192 1018 650 208 1366 686 224 1366 696 240 1366 722 256 640 656 272 988 1246 288 988 1276 304 992 1296 320 972 1222 336 1318 1256 352 1318 1276 368 1316 1294 384 1294 1218 400 1642 1258 416 1642 1282 432 1642 1302 448 1628 1224 464 1970 1258 480 1970 1280 496 1970 1300 512 656 676 528 1010 1290 544 1010 1306 560 1010 1332 576 986 1254 592 1340 1284 608 1334 1310 624 1340 1334 640 1314 1254 656 1664 1282 672 1674 1306 688 1662 1336 704 1638 1250 720 1992 1292 736 1994 1308 752 1988 1334 768 1252 1254 784 1596 1290 800 1596 1314 816 1596 1330 832 1576 1256 848 1922 1286 864 1922 1314 880 1926 1338 896 1898 1258 912 2248 1288 928 2248 1320 944 2248 1338 960 2226 1268 976 2574 1288 992 2576 1312 1008 2574 1340Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:
size old new ---- ---- ---- 0 64 54 16 386 372 32 388 396 48 388 420 64 366 350 80 708 666 96 708 692 112 706 736 128 692 648 144 1036 682 160 1036 708 176 1036 730 192 1016 658 208 1360 684 224 1362 708 240 1360 732 256 644 500 272 990 526 288 988 556 304 988 576 320 972 500 336 1314 532 352 1316 558 368 1318 578 384 1308 506 400 1644 532 416 1644 556 432 1644 594 448 1624 508 464 1970 534 480 1970 556 496 1968 582 512 660 624 528 1016 682 544 1016 702 560 1018 728 576 998 654 592 1344 680 608 1344 708 624 1344 730 640 1326 654 656 1670 686 672 1670 708 688 1670 732 704 1652 658 720 1998 682 736 1998 710 752 1996 734 768 1256 662 784 1606 688 800 1606 714 816 1606 736 832 1584 660 848 1948 688 864 1950 714 880 1948 736 896 1912 688 912 2258 718 928 2258 744 944 2256 768 960 2238 692 976 2584 718 992 2584 744 1008 2584 770
Quoted 2 lines
Signed-off-by Jason A. Donenfeld <[email protected]>Signed-off-by Samuel Neves <[email protected]>
Please drop this SOB line: SOB is not about (co-)authorship but about who handled the patch on its way into mainline. You are sending the patch so your SOB should come last.
Quoted 51 lines
Co-developed-by Samuel Neves <[email protected]>Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: [email protected]--- lib/zinc/Makefile | 1 + lib/zinc/chacha20/chacha20-x86_64-glue.h | 105 + lib/zinc/chacha20/chacha20-x86_64.S | 2632 ++++++++++++++++++++++ lib/zinc/chacha20/chacha20.c | 4 +- 4 files changed, 2741 insertions(+), 1 deletion(-) create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h create mode 100644 lib/zinc/chacha20/chacha20-x86_64.Slib/zinc/Makefile +1
-0@@ -3,4 +3,5 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt' ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.olib/zinc/chacha20/chacha20-x86_64-glue.h new file +17
-0@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <asm/fpu/api.h> +#include <asm/cpufeature.h> +#include <asm/processor.h> +#include <asm/intel-family.h> + +#ifdef CONFIG_AS_SSSE3 +asmlinkage void hchacha20_ssse3(u32 *derived_key, const u8 *nonce, +const u8 *key); +asmlinkage void chacha20_ssse3(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#endif +#ifdef CONFIG_AS_AVX2
lib/zinc/chacha20/chacha20-x86_64-glue.h line 27
Quoted 10 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +10
-0+asmlinkage void chacha20_avx2(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#endif +#ifdef CONFIG_AS_AVX512 +asmlinkage void chacha20_avx512(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +asmlinkage void chacha20_avx512vl(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#endif +
You can drop the #ifdefs above ...
Quoted 28 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +28
-0+static bool chacha20_use_ssse3__ro_after_init ; +static bool chacha20_use_avx2__ro_after_init ; +static bool chacha20_use_avx512__ro_after_init ; +static bool chacha20_use_avx512vl__ro_after_init ; + +static void __init chacha20_fpu_init (void ) +{ + chacha20_use_ssse3 =boot_cpu_has (X86_FEATURE_SSSE3 ); + chacha20_use_avx2 = +boot_cpu_has (X86_FEATURE_AVX ) && +boot_cpu_has (X86_FEATURE_AVX2 ) && +cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ,NULL ); + chacha20_use_avx512 = +boot_cpu_has (X86_FEATURE_AVX ) && +boot_cpu_has (X86_FEATURE_AVX2 ) && +boot_cpu_has (X86_FEATURE_AVX512F ) && +cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM | +XFEATURE_MASK_AVX512 ,NULL ) && +/* Skylake downclocks unacceptably much when using zmm. */ + boot_cpu_data.x86_model !=INTEL_FAM6_SKYLAKE_X ; + chacha20_use_avx512vl = +boot_cpu_has (X86_FEATURE_AVX ) && +boot_cpu_has (X86_FEATURE_AVX2 ) && +boot_cpu_has (X86_FEATURE_AVX512F ) && +boot_cpu_has (X86_FEATURE_AVX512VL ) && +cpu_has_xfeatures (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM | +XFEATURE_MASK_AVX512 ,NULL ); +}
lib/zinc/chacha20/chacha20-x86_64-glue.h line 65
Quoted 10 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +10
-0+ +static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,const size_t len, +simd_context_t *simd_context) +{ +if (!chacha20_use_ssse3 || len <=CHACHA20_BLOCK_SIZE || + !simd_use (simd_context)) +return false; + +#ifdef CONFIG_AS_AVX512
... and use IS_ENABLED(CONFIG_AS_AVX512) here inside the if().
Quoted 40 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +40
-0+if (chacha20_use_avx512 && len >=CHACHA20_BLOCK_SIZE *8 ) { +chacha20_avx512 (dst, src, len, state->key, state->counter); + goto success; + } +if (chacha20_use_avx512vl && len >=CHACHA20_BLOCK_SIZE *4 ) { +chacha20_avx512vl (dst, src, len, state->key, state->counter); + goto success; + } +#endif +#ifdef CONFIG_AS_AVX2 +if (chacha20_use_avx2 && len >=CHACHA20_BLOCK_SIZE *4 ) { +chacha20_avx2 (dst, src, len, state->key, state->counter); + goto success; + } +#endif +#ifdef CONFIG_AS_SSSE3 +if (chacha20_use_ssse3) { +chacha20_ssse3 (dst, src, len, state->key, state->counter); + goto success; + } +#endif +return false; +success: + state->counter[0 ] += (len +63 ) /64 ; +return true; +} + +static inline bool hchacha20_arch (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ], +simd_context_t *simd_context) +{ +#if defined(CONFIG_AS_SSSE3 ) +if (chacha20_use_ssse3 &&simd_use (simd_context)) { +hchacha20_ssse3 (derived_key, nonce, key); +return true; + } +#endif +return false; +}
...
lib/zinc/chacha20/chacha20.c line 17
Quoted 9 lines
lib/zinc/chacha20/chacha20.c +0
-1@@ -14,7 +14,9 @@#include <linux/init.h> #include <crypto/algapi.h> -#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION
As I mentioned in reply to the previous patch, please get rid of this CPP symbol ^^^
Quoted 9 lines
lib/zinc/chacha20/chacha20.c +3
-1+#if defined(CONFIG_ZINC_ARCH_X86_64 ) +#include "chacha20-x86_64-glue.h" +#else void __init chacha20_fpu_init (void ) { } --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Hi Ard,
Quoted 2 lines · Ard Biesheuvel
Please drop this SOB line: SOB is not about (co-)authorship but about who handled the patch on its way into mainline.
The s-o-b line is about the DCoO, which is given here by both of us, with the code starting in a shared tree that only had that s-o-b. I can remove it though if you think it's strange to have two s-o-bs.
Quoted 2 lines
You are sending the
patch so your SOB should come last.
I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.
Quoted 1 line
You can drop the #ifdefs above ...
Yep. All of those ifdefs in this and in the other primitives are already gone, and everything is fully IS_ENABLEDenabled now.
Quoted 3 lines
Quoted 1 line
+#ifdef
CONFIG_AS_AVX512... and use
IS_ENABLED(CONFIG_AS_AVX512)here inside theif().
Already done, per your suggestion a few days ago.
Quoted 4 lines
Quoted 1 line
-#ifndef
HAVE_CHACHA20_ARCH_IMPLEMENTATIONAs I mentioned in reply to the previous patch, please get rid of this CPP symbol ^^^
Ack.
Jason
Borislav Petkov
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Quoted 3 lines · Jason A. Donenfeld
I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.
Documentation/process/submitting-patches.rst
section 12, first sentence.
The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Quoted 5 lines · Borislav Petkov
Quoted 3 lines · Jason A. Donenfeld
I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.
Quoted 7 lines
Documentation/process/submitting-patches.rstsection 12, first sentence.
The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.
Note that this is the author of the *patch* not necessarily the author of the code.
Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.
Borislav Petkov
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Quoted 8 lines · Ard Biesheuvel
Note that this is the author of the *patch* not necessarily the author of the code.
Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.
... and the actual author could be named with Originally-by or Co-Developed-by:, and even in free text in the commit message.
But the correct SOB chain denoting who handled the patch on its way up, is important.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Quoted 4 lines · Borislav Petkov
Quoted 3 lines · Ard Biesheuvel
Note that this is the author of the *patch* not necessarily the author of the code.
Quoted 8 lines
Quoted 5 lines
Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.
... and the actual author could be named with Originally-by or Co-Developed-by:, and even in free text in the commit message.
Or you could just put the original author as the first S-o-b and then another (second) S-o-b line where you put yourself as a submitter.
AFAIK, that's how people usually do it.
Quoted 9 lines
But the correct SOB chain denoting who handled the patch on its way up, is important.
Signature · 4 lines
Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
In reply to Borislav PetkovHi Borislav,
Quoted 6 lines · Borislav Petkov
Documentation/process/submitting-patches.rstsection 12, first sentence.
The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.
Thanks for the clarification.
I've received the SOB directly from the individual in question as it traveled through an intermediate tree on its way to the tree being submitted here. Therefore I will retain that SOB so that it shows the full path.
Jason
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
In reply to Ard BiesheuvelHi Ard,
Quoted 6 lines · Ard Biesheuvel
Quoted 3 lines
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
cded951378069a478391843f5f8653c1eb5128daI'd still prefer the kernel side changes to be presented as a separate followup patch
I've restructured things for the MIPS64 and x86_64 and it will be as you've asked for v7.
Jason
Herbert Xu
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
In reply to Jason A. DonenfeldQuoted 36 lines · Jason A. Donenfeld
This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:
- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.
- When aligning the stack, the original code was saving %rsp to %r9. To keep objtool happy, we use instead the DRAP idiom to save %rsp to %r10:
leaq 8(%rsp),%r10 ... code here ... leaq -8(%r10),%rsp- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rspwas added in the prolog of a few functions.- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.
Quoted 8 lines
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
cded951378069a478391843f5f8653c1eb5128daCycle counts on a Core i7 6700HQ using the AVX-2 codepath:
size old new ---- ---- ---- 0 62 52
What is the old column? Is it the existing x86-64 implementation in the kernel or something else? This needs to be made clear in the patch description.
The same comment from the previous patch also stands here. Please ensure that we only have one copy of chacha x86-64 code in the kernel. Either replace the existing one right here or in a follow-up patch in the same series.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation
Hi Herbert,
Quoted 7 lines · Herbert Xu
Quoted 3 lines
size old new ---- ---- ---- 0 62 52What is the old column? Is it the existing x86-64 implementation in the kernel or something else? This needs to be made clear in the patch description.
Good idea. I'll make that more clear.
Quoted 4 lines
The same comment from the previous patch also stands here. Please ensure that we only have one copy of chacha x86-64 code in the kernel. Either replace the existing one right here or in a follow-up patch in the same series.
Yes indeed this happens within this same series.
Jason
Jason A. Donenfeld
maintainerPATCH v6 5/23 zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2
Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org
--- lib/zinc/chacha20/chacha20-arm-cryptogams.S | 1440 ++++++++++++ lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++ 2 files changed, 3413 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.S
lib/zinc/chacha20/chacha20-arm-cryptogams.S new file +1440 -0
@@ -0,0 +1,1440 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* + * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + */ + +#include "arm_arch.h" + +.text +#if defined(__thumb2__) || defined(__clang__) +.syntax unified +#endif +#if defined(__thumb2__) +.thumb +#else +.code 32 +#endif + +#if defined(__thumb2__) || defined(__clang__) +#define ldrhsb ldrbhs +#endif + +.align 5 +.Lsigma: +.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral +.Lone: +.long 1,0,0,0 +.Lrot8: +.long 0x02010003,0x06050407 +#if __ARM_MAX_ARCH__>=7 +.LOPENSSL_armcap: +.word OPENSSL_armcap_P-.LChaCha20_ctr32 +#else +.word -1 +#endif + +.globl ChaCha20_ctr32 +.type ChaCha20_ctr32,%function +.align 5 +ChaCha20_ctr32: +.LChaCha20_ctr32: + ldr r12,[sp,#0] @ pull pointer to counter and nonce + stmdb sp!,{r0-r2,r4-r11,lr} +#if __ARM_ARCH__<7 && !defined(__thumb2__) + sub r14,pc,#16 @ ChaCha20_ctr32 +#else + adr r14,.LChaCha20_ctr32 +#endif + cmp r2,#0 @ len==0? +#ifdef __thumb2__ + itt eq +#endif + addeq sp,sp,#4*3 + beq .Lno_data +#if __ARM_MAX_ARCH__>=7 + cmp r2,#192 @ test len + bls .Lshort + ldr r4,[r14,#-24] + ldr r4,[r14,r4] +# ifdef __APPLE__ + ldr r4,[r4] +# endif + tst r4,#ARMV7_NEON + bne .LChaCha20_neon +.Lshort: +#endif + ldmia r12,{r4-r7} @ load counter and nonce + sub sp,sp,#4*(16) @ off-load area + sub r14,r14,#64 @ .Lsigma + stmdb sp!,{r4-r7} @ copy counter and nonce + ldmia r3,{r4-r11} @ load key + ldmia r14,{r0-r3} @ load sigma + stmdb sp!,{r4-r11} @ copy key + stmdb sp!,{r0-r3} @ copy sigma + str r10,[sp,#4*(16+10)] @ off-load "rx" + str r11,[sp,#4*(16+11)] @ off-load "rx" + b .Loop_outer_enter + +.align 4 +.Loop_outer: + ldmia sp,{r0-r9} @ load key material + str r11,[sp,#4*(32+2)] @ save len + str r12, [sp,#4*(32+1)] @ save inp + str r14, [sp,#4*(32+0)] @ save out +.Loop_outer_enter: + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(16+15)] + mov r11,#10 + b .Loop + +.align 4 +.Loop: + subs r11,r11,#1 + add r0,r0,r4,ror#13 + add r1,r1,r5,ror#13 + eor r12,r0,r12,ror#24 + eor r10,r1,r10,ror#24 + add r8,r8,r12,ror#16 + add r9,r9,r10,ror#16 + eor r4,r8,r4,ror#13 + eor r5,r9,r5,ror#13 + add r0,r0,r4,ror#20 + add r1,r1,r5,ror#20 + eor r12,r0,r12,ror#16 + eor r10,r1,r10,ror#16 + add r8,r8,r12,ror#24 + str r10,[sp,#4*(16+13)] + add r9,r9,r10,ror#24 + ldr r10,[sp,#4*(16+15)] + str r8,[sp,#4*(16+8)] + eor r4,r4,r8,ror#12 + str r9,[sp,#4*(16+9)] + eor r5,r5,r9,ror#12 + ldr r8,[sp,#4*(16+10)] + add r2,r2,r6,ror#13 + ldr r9,[sp,#4*(16+11)] + add r3,r3,r7,ror#13 + eor r14,r2,r14,ror#24 + eor r10,r3,r10,ror#24 + add r8,r8,r14,ror#16 + add r9,r9,r10,ror#16 + eor r6,r8,r6,ror#13 + eor r7,r9,r7,ror#13 + add r2,r2,r6,ror#20 + add r3,r3,r7,ror#20 + eor r14,r2,r14,ror#16 + eor r10,r3,r10,ror#16 + add r8,r8,r14,ror#24 + add r9,r9,r10,ror#24 + eor r6,r6,r8,ror#12 + eor r7,r7,r9,ror#12 + add r0,r0,r5,ror#13 + add r1,r1,r6,ror#13 + eor r10,r0,r10,ror#24 + eor r12,r1,r12,ror#24 + add r8,r8,r10,ror#16 + add r9,r9,r12,ror#16 + eor r5,r8,r5,ror#13 + eor r6,r9,r6,ror#13 + add r0,r0,r5,ror#20 + add r1,r1,r6,ror#20 + eor r10,r0,r10,ror#16 + eor r12,r1,r12,ror#16 + str r10,[sp,#4*(16+15)] + add r8,r8,r10,ror#24 + ldr r10,[sp,#4*(16+13)] + add r9,r9,r12,ror#24 + str r8,[sp,#4*(16+10)] + eor r5,r5,r8,ror#12 + str r9,[sp,#4*(16+11)] + eor r6,r6,r9,ror#12 + ldr r8,[sp,#4*(16+8)] + add r2,r2,r7,ror#13 + ldr r9,[sp,#4*(16+9)] + add r3,r3,r4,ror#13 + eor r10,r2,r10,ror#24 + eor r14,r3,r14,ror#24 + add r8,r8,r10,ror#16 + add r9,r9,r14,ror#16 + eor r7,r8,r7,ror#13 + eor r4,r9,r4,ror#13 + add r2,r2,r7,ror#20 + add r3,r3,r4,ror#20 + eor r10,r2,r10,ror#16 + eor r14,r3,r14,ror#16 + add r8,r8,r10,ror#24 + add r9,r9,r14,ror#24 + eor r7,r7,r8,ror#12 + eor r4,r4,r9,ror#12 + bne .Loop + + ldr r11,[sp,#4*(32+2)] @ load len + + str r8, [sp,#4*(16+8)] @ modulo-scheduled store + str r9, [sp,#4*(16+9)] + str r12,[sp,#4*(16+12)] + str r10, [sp,#4*(16+13)] + str r14,[sp,#4*(16+14)] + + @ at this point we have first half of 512-bit result in + @ rx and second half at sp+4*(16+8) + + cmp r11,#64 @ done yet? +#ifdef __thumb2__ + itete lo +#endif + addlo r12,sp,#4*(0) @ shortcut or ... + ldrhs r12,[sp,#4*(32+1)] @ ... load inp + addlo r14,sp,#4*(0) @ shortcut or ... + ldrhs r14,[sp,#4*(32+0)] @ ... load out + + ldr r8,[sp,#4*(0)] @ load key material + ldr r9,[sp,#4*(1)] + +#if __ARM_ARCH__>=6 || !defined(__ARMEB__) +# if __ARM_ARCH__<7 + orr r10,r12,r14 + tst r10,#3 @ are input and output aligned? + ldr r10,[sp,#4*(2)] + bne .Lunaligned + cmp r11,#64 @ restore flags +# else + ldr r10,[sp,#4*(2)] +# endif + ldr r11,[sp,#4*(3)] + + add r0,r0,r8 @ accumulate key material + add r1,r1,r9 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + + add r2,r2,r10 + add r3,r3,r11 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r0,r0,r8 @ xor with input + eorhs r1,r1,r9 + add r8,sp,#4*(4) + str r0,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r2,r2,r10 + eorhs r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r1,[r14,#-12] + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#13 @ accumulate key material + add r5,r9,r5,ror#13 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + add r6,r10,r6,ror#13 + add r7,r11,r7,ror#13 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r4,r4,r8 + eorhs r5,r5,r9 + add r8,sp,#4*(8) + str r4,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r6,r6,r10 + eorhs r7,r7,r11 + str r5,[r14,#-12] + ldmia r8,{r8-r11} @ load key material + str r6,[r14,#-8] + add r0,sp,#4*(16+8) + str r7,[r14,#-4] + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + add r1,r1,r9 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] +# ifdef __thumb2__ + itt hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it + add r2,r2,r10 + add r3,r3,r11 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r0,r0,r8 + eorhs r1,r1,r9 + add r8,sp,#4*(12) + str r0,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r2,r2,r10 + eorhs r3,r3,r11 + str r1,[r14,#-12] + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#24 @ accumulate key material + add r5,r9,r5,ror#24 +# ifdef __thumb2__ + itt hi +# endif + addhi r8,r8,#1 @ next counter value + strhi r8,[sp,#4*(12)] @ save next counter value +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + add r6,r10,r6,ror#24 + add r7,r11,r7,ror#24 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r4,r4,r8 + eorhs r5,r5,r9 +# ifdef __thumb2__ + it ne +# endif + ldrne r8,[sp,#4*(32+2)] @ re-load len +# ifdef __thumb2__ + itt hs +# endif + eorhs r6,r6,r10 + eorhs r7,r7,r11 + str r4,[r14],#16 @ store output + str r5,[r14,#-12] +# ifdef __thumb2__ + it hs +# endif + subhs r11,r8,#64 @ len-=64 + str r6,[r14,#-8] + str r7,[r14,#-4] + bhi .Loop_outer + + beq .Ldone +# if __ARM_ARCH__<7 + b .Ltail + +.align 4 +.Lunaligned: @ unaligned endian-neutral path + cmp r11,#64 @ restore flags +# endif +#endif +#if __ARM_ARCH__<7 + ldr r11,[sp,#4*(3)] + add r0,r8,r0 @ accumulate key material + add r1,r9,r1 + add r2,r10,r2 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r3,r11,r3 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r0,r8,r0 @ xor with input (or zero) + eor r1,r9,r1 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r2,r10,r2 + strb r0,[r14],#16 @ store output + eor r3,r11,r3 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r1,[r14,#-12] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-8] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r3,[r14,#-4] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-15] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r1,[r14,#-11] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-7] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r3,[r14,#-3] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-14] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r1,[r14,#-10] + strb r2,[r14,#-6] + eor r0,r8,r0,lsr#8 + strb r3,[r14,#-2] + eor r1,r9,r1,lsr#8 + strb r0,[r14,#-13] + eor r2,r10,r2,lsr#8 + strb r1,[r14,#-9] + eor r3,r11,r3,lsr#8 + strb r2,[r14,#-5] + strb r3,[r14,#-1] + add r8,sp,#4*(4+0) + ldmia r8,{r8-r11} @ load key material + add r0,sp,#4*(16+8) + add r4,r8,r4,ror#13 @ accumulate key material + add r5,r9,r5,ror#13 + add r6,r10,r6,ror#13 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r7,r11,r7,ror#13 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r4,r8,r4 @ xor with input (or zero) + eor r5,r9,r5 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r6,r10,r6 + strb r4,[r14],#16 @ store output + eor r7,r11,r7 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r5,[r14,#-12] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-8] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r7,[r14,#-4] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-15] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r5,[r14,#-11] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-7] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r7,[r14,#-3] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-14] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r5,[r14,#-10] + strb r6,[r14,#-6] + eor r4,r8,r4,lsr#8 + strb r7,[r14,#-2] + eor r5,r9,r5,lsr#8 + strb r4,[r14,#-13] + eor r6,r10,r6,lsr#8 + strb r5,[r14,#-9] + eor r7,r11,r7,lsr#8 + strb r6,[r14,#-5] + strb r7,[r14,#-1] + add r8,sp,#4*(4+4) + ldmia r8,{r8-r11} @ load key material + ldmia r0,{r0-r7} @ load second half +# ifdef __thumb2__ + itt hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" + strhi r11,[sp,#4*(16+11)] @ copy "rx" + add r0,r8,r0 @ accumulate key material + add r1,r9,r1 + add r2,r10,r2 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r3,r11,r3 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r0,r8,r0 @ xor with input (or zero) + eor r1,r9,r1 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r2,r10,r2 + strb r0,[r14],#16 @ store output + eor r3,r11,r3 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r1,[r14,#-12] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-8] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r3,[r14,#-4] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-15] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r1,[r14,#-11] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-7] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r3,[r14,#-3] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-14] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r1,[r14,#-10] + strb r2,[r14,#-6] + eor r0,r8,r0,lsr#8 + strb r3,[r14,#-2] + eor r1,r9,r1,lsr#8 + strb r0,[r14,#-13] + eor r2,r10,r2,lsr#8 + strb r1,[r14,#-9] + eor r3,r11,r3,lsr#8 + strb r2,[r14,#-5] + strb r3,[r14,#-1] + add r8,sp,#4*(4+8) + ldmia r8,{r8-r11} @ load key material + add r4,r8,r4,ror#24 @ accumulate key material +# ifdef __thumb2__ + itt hi +# endif + addhi r8,r8,#1 @ next counter value + strhi r8,[sp,#4*(12)] @ save next counter value + add r5,r9,r5,ror#24 + add r6,r10,r6,ror#24 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r7,r11,r7,ror#24 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r4,r8,r4 @ xor with input (or zero) + eor r5,r9,r5 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r6,r10,r6 + strb r4,[r14],#16 @ store output + eor r7,r11,r7 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r5,[r14,#-12] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-8] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r7,[r14,#-4] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-15] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r5,[r14,#-11] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-7] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r7,[r14,#-3] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-14] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r5,[r14,#-10] + strb r6,[r14,#-6] + eor r4,r8,r4,lsr#8 + strb r7,[r14,#-2] + eor r5,r9,r5,lsr#8 + strb r4,[r14,#-13] + eor r6,r10,r6,lsr#8 + strb r5,[r14,#-9] + eor r7,r11,r7,lsr#8 + strb r6,[r14,#-5] + strb r7,[r14,#-1] +# ifdef __thumb2__ + it ne +# endif + ldrne r8,[sp,#4*(32+2)] @ re-load len +# ifdef __thumb2__ + it hs +# endif + subhs r11,r8,#64 @ len-=64 + bhi .Loop_outer + + beq .Ldone +#endif + +.Ltail: + ldr r12,[sp,#4*(32+1)] @ load inp + add r9,sp,#4*(0) + ldr r14,[sp,#4*(32+0)] @ load out + +.Loop_tail: + ldrb r10,[r9],#1 @ read buffer on stack + ldrb r11,[r12],#1 @ read input + subs r8,r8,#1 + eor r11,r11,r10 + strb r11,[r14],#1 @ store output + bne .Loop_tail + +.Ldone: + add sp,sp,#4*(32+3) +.Lno_data: + ldmia sp!,{r4-r11,pc} +.size ChaCha20_ctr32,.-ChaCha20_ctr32 +#if __ARM_MAX_ARCH__>=7 +.arch armv7-a +.fpu neon + +.type ChaCha20_neon,%function +.align 5 +ChaCha20_neon: + ldr r12,[sp,#0] @ pull pointer to counter and nonce + stmdb sp!,{r0-r2,r4-r11,lr} +.LChaCha20_neon: + adr r14,.Lsigma + vstmdb sp!,{d8-d15} @ ABI spec says so + stmdb sp!,{r0-r3} + + vld1.32 {q1-q2},[r3] @ load key + ldmia r3,{r4-r11} @ load key + + sub sp,sp,#4*(16+16) + vld1.32 {q3},[r12] @ load counter and nonce + add r12,sp,#4*8 + ldmia r14,{r0-r3} @ load sigma + vld1.32 {q0},[r14]! @ load sigma + vld1.32 {q12},[r14]! @ one + @ vld1.32 {d30},[r14] @ rot8 + vst1.32 {q2-q3},[r12] @ copy 1/2key|counter|nonce + vst1.32 {q0-q1},[sp] @ copy sigma|1/2key + + str r10,[sp,#4*(16+10)] @ off-load "rx" + str r11,[sp,#4*(16+11)] @ off-load "rx" + vshl.i32 d26,d24,#1 @ two + vstr d24,[sp,#4*(16+0)] + vshl.i32 d28,d24,#2 @ four + vstr d26,[sp,#4*(16+2)] + vmov q4,q0 + vstr d28,[sp,#4*(16+4)] + vmov q8,q0 + @ vstr d30,[sp,#4*(16+6)] + vmov q5,q1 + vmov q9,q1 + b .Loop_neon_enter + +.align 4 +.Loop_neon_outer: + ldmia sp,{r0-r9} @ load key material + cmp r11,#64*2 @ if len<=64*2 + bls .Lbreak_neon @ switch to integer-only + @ vldr d30,[sp,#4*(16+6)] @ rot8 + vmov q4,q0 + str r11,[sp,#4*(32+2)] @ save len + vmov q8,q0 + str r12, [sp,#4*(32+1)] @ save inp + vmov q5,q1 + str r14, [sp,#4*(32+0)] @ save out + vmov q9,q1 +.Loop_neon_enter: + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + vadd.i32 q7,q3,q12 @ counter+1 + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + vmov q6,q2 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + vmov q10,q2 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + vadd.i32 q11,q7,q12 @ counter+2 + add r12,r12,#3 @ counter+3 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(16+15)] + mov r11,#10 + b .Loop_neon + +.align 4 +.Loop_neon: + subs r11,r11,#1 + vadd.i32 q0,q0,q1 + add r0,r0,r4,ror#13 + vadd.i32 q4,q4,q5 + add r1,r1,r5,ror#13 + vadd.i32 q8,q8,q9 + eor r12,r0,r12,ror#24 + veor q3,q3,q0 + eor r10,r1,r10,ror#24 + veor q7,q7,q4 + add r8,r8,r12,ror#16 + veor q11,q11,q8 + add r9,r9,r10,ror#16 + vrev32.16 q3,q3 + eor r4,r8,r4,ror#13 + vrev32.16 q7,q7 + eor r5,r9,r5,ror#13 + vrev32.16 q11,q11 + add r0,r0,r4,ror#20 + vadd.i32 q2,q2,q3 + add r1,r1,r5,ror#20 + vadd.i32 q6,q6,q7 + eor r12,r0,r12,ror#16 + vadd.i32 q10,q10,q11 + eor r10,r1,r10,ror#16 + veor q12,q1,q2 + add r8,r8,r12,ror#24 + veor q13,q5,q6 + str r10,[sp,#4*(16+13)] + veor q14,q9,q10 + add r9,r9,r10,ror#24 + vshr.u32 q1,q12,#20 + ldr r10,[sp,#4*(16+15)] + vshr.u32 q5,q13,#20 + str r8,[sp,#4*(16+8)] + vshr.u32 q9,q14,#20 + eor r4,r4,r8,ror#12 + vsli.32 q1,q12,#12 + str r9,[sp,#4*(16+9)] + vsli.32 q5,q13,#12 + eor r5,r5,r9,ror#12 + vsli.32 q9,q14,#12 + ldr r8,[sp,#4*(16+10)] + vadd.i32 q0,q0,q1 + add r2,r2,r6,ror#13 + vadd.i32 q4,q4,q5 + ldr r9,[sp,#4*(16+11)] + vadd.i32 q8,q8,q9 + add r3,r3,r7,ror#13 + veor q12,q3,q0 + eor r14,r2,r14,ror#24 + veor q13,q7,q4 + eor r10,r3,r10,ror#24 + veor q14,q11,q8 + add r8,r8,r14,ror#16 + vshr.u32 q3,q12,#24 + add r9,r9,r10,ror#16 + vshr.u32 q7,q13,#24 + eor r6,r8,r6,ror#13 + vshr.u32 q11,q14,#24 + eor r7,r9,r7,ror#13 + vsli.32 q3,q12,#8 + add r2,r2,r6,ror#20 + vsli.32 q7,q13,#8 + add r3,r3,r7,ror#20 + vsli.32 q11,q14,#8 + eor r14,r2,r14,ror#16 + vadd.i32 q2,q2,q3 + eor r10,r3,r10,ror#16 + vadd.i32 q6,q6,q7 + add r8,r8,r14,ror#24 + vadd.i32 q10,q10,q11 + add r9,r9,r10,ror#24 + veor q12,q1,q2 + eor r6,r6,r8,ror#12 + veor q13,q5,q6 + eor r7,r7,r9,ror#12 + veor q14,q9,q10 + vshr.u32 q1,q12,#25 + vshr.u32 q5,q13,#25 + vshr.u32 q9,q14,#25 + vsli.32 q1,q12,#7 + vsli.32 q5,q13,#7 + vsli.32 q9,q14,#7 + vext.8 q2,q2,q2,#8 + vext.8 q6,q6,q6,#8 + vext.8 q10,q10,q10,#8 + vext.8 q1,q1,q1,#4 + vext.8 q5,q5,q5,#4 + vext.8 q9,q9,q9,#4 + vext.8 q3,q3,q3,#12 + vext.8 q7,q7,q7,#12 + vext.8 q11,q11,q11,#12 + vadd.i32 q0,q0,q1 + add r0,r0,r5,ror#13 + vadd.i32 q4,q4,q5 + add r1,r1,r6,ror#13 + vadd.i32 q8,q8,q9 + eor r10,r0,r10,ror#24 + veor q3,q3,q0 + eor r12,r1,r12,ror#24 + veor q7,q7,q4 + add r8,r8,r10,ror#16 + veor q11,q11,q8 + add r9,r9,r12,ror#16 + vrev32.16 q3,q3 + eor r5,r8,r5,ror#13 + vrev32.16 q7,q7 + eor r6,r9,r6,ror#13 + vrev32.16 q11,q11 + add r0,r0,r5,ror#20 + vadd.i32 q2,q2,q3 + add r1,r1,r6,ror#20 + vadd.i32 q6,q6,q7 + eor r10,r0,r10,ror#16 + vadd.i32 q10,q10,q11 + eor r12,r1,r12,ror#16 + veor q12,q1,q2 + str r10,[sp,#4*(16+15)] + veor q13,q5,q6 + add r8,r8,r10,ror#24 + veor q14,q9,q10 + ldr r10,[sp,#4*(16+13)] + vshr.u32 q1,q12,#20 + add r9,r9,r12,ror#24 + vshr.u32 q5,q13,#20 + str r8,[sp,#4*(16+10)] + vshr.u32 q9,q14,#20 + eor r5,r5,r8,ror#12 + vsli.32 q1,q12,#12 + str r9,[sp,#4*(16+11)] + vsli.32 q5,q13,#12 + eor r6,r6,r9,ror#12 + vsli.32 q9,q14,#12 + ldr r8,[sp,#4*(16+8)] + vadd.i32 q0,q0,q1 + add r2,r2,r7,ror#13 + vadd.i32 q4,q4,q5 + ldr r9,[sp,#4*(16+9)] + vadd.i32 q8,q8,q9 + add r3,r3,r4,ror#13 + veor q12,q3,q0 + eor r10,r2,r10,ror#24 + veor q13,q7,q4 + eor r14,r3,r14,ror#24 + veor q14,q11,q8 + add r8,r8,r10,ror#16 + vshr.u32 q3,q12,#24 + add r9,r9,r14,ror#16 + vshr.u32 q7,q13,#24 + eor r7,r8,r7,ror#13 + vshr.u32 q11,q14,#24 + eor r4,r9,r4,ror#13 + vsli.32 q3,q12,#8 + add r2,r2,r7,ror#20 + vsli.32 q7,q13,#8 + add r3,r3,r4,ror#20 + vsli.32 q11,q14,#8 + eor r10,r2,r10,ror#16 + vadd.i32 q2,q2,q3 + eor r14,r3,r14,ror#16 + vadd.i32 q6,q6,q7 + add r8,r8,r10,ror#24 + vadd.i32 q10,q10,q11 + add r9,r9,r14,ror#24 + veor q12,q1,q2 + eor r7,r7,r8,ror#12 + veor q13,q5,q6 + eor r4,r4,r9,ror#12 + veor q14,q9,q10 + vshr.u32 q1,q12,#25 + vshr.u32 q5,q13,#25 + vshr.u32 q9,q14,#25 + vsli.32 q1,q12,#7 + vsli.32 q5,q13,#7 + vsli.32 q9,q14,#7 + vext.8 q2,q2,q2,#8 + vext.8 q6,q6,q6,#8 + vext.8 q10,q10,q10,#8 + vext.8 q1,q1,q1,#12 + vext.8 q5,q5,q5,#12 + vext.8 q9,q9,q9,#12 + vext.8 q3,q3,q3,#4 + vext.8 q7,q7,q7,#4 + vext.8 q11,q11,q11,#4 + bne .Loop_neon + + add r11,sp,#32 + vld1.32 {q12-q13},[sp] @ load key material + vld1.32 {q14-q15},[r11] + + ldr r11,[sp,#4*(32+2)] @ load len + + str r8, [sp,#4*(16+8)] @ modulo-scheduled store + str r9, [sp,#4*(16+9)] + str r12,[sp,#4*(16+12)] + str r10, [sp,#4*(16+13)] + str r14,[sp,#4*(16+14)] + + @ at this point we have first half of 512-bit result in + @ rx and second half at sp+4*(16+8) + + ldr r12,[sp,#4*(32+1)] @ load inp + ldr r14,[sp,#4*(32+0)] @ load out + + vadd.i32 q0,q0,q12 @ accumulate key material + vadd.i32 q4,q4,q12 + vadd.i32 q8,q8,q12 + vldr d24,[sp,#4*(16+0)] @ one + + vadd.i32 q1,q1,q13 + vadd.i32 q5,q5,q13 + vadd.i32 q9,q9,q13 + vldr d26,[sp,#4*(16+2)] @ two + + vadd.i32 q2,q2,q14 + vadd.i32 q6,q6,q14 + vadd.i32 q10,q10,q14 + vadd.i32 d14,d14,d24 @ counter+1 + vadd.i32 d22,d22,d26 @ counter+2 + + vadd.i32 q3,q3,q15 + vadd.i32 q7,q7,q15 + vadd.i32 q11,q11,q15 + + cmp r11,#64*4 + blo .Ltail_neon + + vld1.8 {q12-q13},[r12]! @ load input + mov r11,sp + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 @ xor with input + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + vst1.8 {q0-q1},[r14]! @ store output + veor q5,q5,q13 + vld1.8 {q12-q13},[r12]! + veor q6,q6,q14 + vst1.8 {q2-q3},[r14]! + veor q7,q7,q15 + vld1.8 {q14-q15},[r12]! + + veor q8,q8,q12 + vld1.32 {q0-q1},[r11]! @ load for next iteration + veor d25,d25,d25 + vldr d24,[sp,#4*(16+4)] @ four + veor q9,q9,q13 + vld1.32 {q2-q3},[r11] + veor q10,q10,q14 + vst1.8 {q4-q5},[r14]! + veor q11,q11,q15 + vst1.8 {q6-q7},[r14]! + + vadd.i32 d6,d6,d24 @ next counter value + vldr d24,[sp,#4*(16+0)] @ one + + ldmia sp,{r8-r11} @ load key material + add r0,r0,r8 @ accumulate key material + ldr r8,[r12],#16 @ load input + vst1.8 {q8-q9},[r14]! + add r1,r1,r9 + ldr r9,[r12,#-12] + vst1.8 {q10-q11},[r14]! + add r2,r2,r10 + ldr r10,[r12,#-8] + add r3,r3,r11 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif + eor r0,r0,r8 @ xor with input + add r8,sp,#4*(4) + eor r1,r1,r9 + str r0,[r14],#16 @ store output + eor r2,r2,r10 + str r1,[r14,#-12] + eor r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#13 @ accumulate key material + ldr r8,[r12],#16 @ load input + add r5,r9,r5,ror#13 + ldr r9,[r12,#-12] + add r6,r10,r6,ror#13 + ldr r10,[r12,#-8] + add r7,r11,r7,ror#13 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + eor r4,r4,r8 + add r8,sp,#4*(8) + eor r5,r5,r9 + str r4,[r14],#16 @ store output + eor r6,r6,r10 + str r5,[r14,#-12] + eor r7,r7,r11 + ldmia r8,{r8-r11} @ load key material + str r6,[r14,#-8] + add r0,sp,#4*(16+8) + str r7,[r14,#-4] + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + ldr r8,[r12],#16 @ load input + add r1,r1,r9 + ldr r9,[r12,#-12] +# ifdef __thumb2__ + it hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it + add r2,r2,r10 + ldr r10,[r12,#-8] +# ifdef __thumb2__ + it hi +# endif + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it + add r3,r3,r11 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif + eor r0,r0,r8 + add r8,sp,#4*(12) + eor r1,r1,r9 + str r0,[r14],#16 @ store output + eor r2,r2,r10 + str r1,[r14,#-12] + eor r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#24 @ accumulate key material + add r8,r8,#4 @ next counter value + add r5,r9,r5,ror#24 + str r8,[sp,#4*(12)] @ save next counter value + ldr r8,[r12],#16 @ load input + add r6,r10,r6,ror#24 + add r4,r4,#3 @ counter+3 + ldr r9,[r12,#-12] + add r7,r11,r7,ror#24 + ldr r10,[r12,#-8] + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + eor r4,r4,r8 +# ifdef __thumb2__ + it hi +# endif + ldrhi r8,[sp,#4*(32+2)] @ re-load len + eor r5,r5,r9 + eor r6,r6,r10 + str r4,[r14],#16 @ store output + eor r7,r7,r11 + str r5,[r14,#-12] + sub r11,r8,#64*4 @ len-=64*4 + str r6,[r14,#-8] + str r7,[r14,#-4] + bhi .Loop_neon_outer + + b .Ldone_neon + +.align 4 +.Lbreak_neon: + @ harmonize NEON and integer-only stack frames: load data + @ from NEON frame, but save to integer-only one; distance + @ between the two is 4*(32+4+16-32)=4*(20). + + str r11, [sp,#4*(20+32+2)] @ save len + add r11,sp,#4*(32+4) + str r12, [sp,#4*(20+32+1)] @ save inp + str r14, [sp,#4*(20+32+0)] @ save out + + ldr r12,[sp,#4*(16+10)] + ldr r14,[sp,#4*(16+11)] + vldmia r11,{d8-d15} @ fulfill ABI requirement + str r12,[sp,#4*(20+16+10)] @ copy "rx" + str r14,[sp,#4*(20+16+11)] @ copy "rx" + + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(20+16+15)] + add r11,sp,#4*(20) + vst1.32 {q0-q1},[r11]! @ copy key + add sp,sp,#4*(20) @ switch frame + vst1.32 {q2-q3},[r11] + mov r11,#10 + b .Loop @ go integer-only + +.align 4 +.Ltail_neon: + cmp r11,#64*3 + bhs .L192_or_more_neon + cmp r11,#64*2 + bhs .L128_or_more_neon + cmp r11,#64*1 + bhs .L64_or_more_neon + + add r8,sp,#4*(8) + vst1.8 {q0-q1},[sp] + add r10,sp,#4*(0) + vst1.8 {q2-q3},[r8] + b .Loop_tail_neon + +.align 4 +.L64_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + veor q2,q2,q14 + veor q3,q3,q15 + vst1.8 {q0-q1},[r14]! + vst1.8 {q2-q3},[r14]! + + beq .Ldone_neon + + add r8,sp,#4*(8) + vst1.8 {q4-q5},[sp] + add r10,sp,#4*(0) + vst1.8 {q6-q7},[r8] + sub r11,r11,#64*1 @ len-=64*1 + b .Loop_tail_neon + +.align 4 +.L128_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + veor q5,q5,q13 + vst1.8 {q0-q1},[r14]! + veor q6,q6,q14 + vst1.8 {q2-q3},[r14]! + veor q7,q7,q15 + vst1.8 {q4-q5},[r14]! + vst1.8 {q6-q7},[r14]! + + beq .Ldone_neon + + add r8,sp,#4*(8) + vst1.8 {q8-q9},[sp] + add r10,sp,#4*(0) + vst1.8 {q10-q11},[r8] + sub r11,r11,#64*2 @ len-=64*2 + b .Loop_tail_neon + +.align 4 +.L192_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + veor q5,q5,q13 + vld1.8 {q12-q13},[r12]! + veor q6,q6,q14 + vst1.8 {q0-q1},[r14]! + veor q7,q7,q15 + vld1.8 {q14-q15},[r12]! + + veor q8,q8,q12 + vst1.8 {q2-q3},[r14]! + veor q9,q9,q13 + vst1.8 {q4-q5},[r14]! + veor q10,q10,q14 + vst1.8 {q6-q7},[r14]! + veor q11,q11,q15 + vst1.8 {q8-q9},[r14]! + vst1.8 {q10-q11},[r14]! + + beq .Ldone_neon + + ldmia sp,{r8-r11} @ load key material + add r0,r0,r8 @ accumulate key material + add r8,sp,#4*(4) + add r1,r1,r9 + add r2,r2,r10 + add r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + + add r4,r8,r4,ror#13 @ accumulate key material + add r8,sp,#4*(8) + add r5,r9,r5,ror#13 + add r6,r10,r6,ror#13 + add r7,r11,r7,ror#13 + ldmia r8,{r8-r11} @ load key material +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + stmia sp,{r0-r7} + add r0,sp,#4*(16+8) + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + add r8,sp,#4*(12) + add r1,r1,r9 + add r2,r2,r10 + add r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + + add r4,r8,r4,ror#24 @ accumulate key material + add r8,sp,#4*(8) + add r5,r9,r5,ror#24 + add r4,r4,#3 @ counter+3 + add r6,r10,r6,ror#24 + add r7,r11,r7,ror#24 + ldr r11,[sp,#4*(32+2)] @ re-load len +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + stmia r8,{r0-r7} + add r10,sp,#4*(0) + sub r11,r11,#64*3 @ len-=64*3 + +.Loop_tail_neon: + ldrb r8,[r10],#1 @ read buffer on stack + ldrb r9,[r12],#1 @ read input + subs r11,r11,#1 + eor r8,r8,r9 + strb r8,[r14],#1 @ store output + bne .Loop_tail_neon + +.Ldone_neon: + add sp,sp,#4*(32+4) + vldmia sp,{d8-d15} + add sp,sp,#4*(16+3) + ldmia sp!,{r4-r11,pc} +.size ChaCha20_neon,.-ChaCha20_neon +.comm OPENSSL_armcap_P,4,4 +#endif
lib/zinc/chacha20/chacha20-arm64-cryptogams.S new file +1973 -0
1,560 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
Quoted 10 lines · Jason A. Donenfeld
These NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
87cc649f30aaf69b351701875b9dac07c29ce8a2Signed-off-by Jason A. Donenfeld <[email protected]>
Quoted 8 lines
Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: [email protected]
As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.
Quoted 3,435 lines
--- lib/zinc/chacha20/chacha20-arm-cryptogams.S | 1440 ++++++++++++ lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++ 2 files changed, 3413 insertions(+) create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.Slib/zinc/chacha20/chacha20-arm-cryptogams.S new file +1440
-0@@ -0,0 +1,1440 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* + * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + */ + +#include "arm_arch.h" + +.text +#if defined(__thumb2__) || defined(__clang__) +.syntax unified +#endif +#if defined(__thumb2__) +.thumb +#else +.code 32 +#endif + +#if defined(__thumb2__) || defined(__clang__) +#define ldrhsb ldrbhs +#endif + +.align 5 +.Lsigma: +.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral +.Lone: +.long 1,0,0,0 +.Lrot8: +.long 0x02010003,0x06050407 +#if __ARM_MAX_ARCH__>=7 +.LOPENSSL_armcap: +.word OPENSSL_armcap_P-.LChaCha20_ctr32 +#else +.word -1 +#endif + +.globl ChaCha20_ctr32 +.type ChaCha20_ctr32,%function +.align 5 +ChaCha20_ctr32: +.LChaCha20_ctr32: + ldr r12,[sp,#0] @ pull pointer to counter and nonce + stmdb sp!,{r0-r2,r4-r11,lr} +#if __ARM_ARCH__<7 && !defined(__thumb2__) + sub r14,pc,#16 @ ChaCha20_ctr32 +#else + adr r14,.LChaCha20_ctr32 +#endif + cmp r2,#0 @ len==0? +#ifdef __thumb2__ + itt eq +#endif + addeq sp,sp,#4*3 + beq .Lno_data +#if __ARM_MAX_ARCH__>=7 + cmp r2,#192 @ test len + bls .Lshort + ldr r4,[r14,#-24] + ldr r4,[r14,r4] +# ifdef __APPLE__ + ldr r4,[r4] +# endif + tst r4,#ARMV7_NEON + bne .LChaCha20_neon +.Lshort: +#endif + ldmia r12,{r4-r7} @ load counter and nonce + sub sp,sp,#4*(16) @ off-load area + sub r14,r14,#64 @ .Lsigma + stmdb sp!,{r4-r7} @ copy counter and nonce + ldmia r3,{r4-r11} @ load key + ldmia r14,{r0-r3} @ load sigma + stmdb sp!,{r4-r11} @ copy key + stmdb sp!,{r0-r3} @ copy sigma + str r10,[sp,#4*(16+10)] @ off-load "rx" + str r11,[sp,#4*(16+11)] @ off-load "rx" + b .Loop_outer_enter + +.align 4 +.Loop_outer: + ldmia sp,{r0-r9} @ load key material + str r11,[sp,#4*(32+2)] @ save len + str r12, [sp,#4*(32+1)] @ save inp + str r14, [sp,#4*(32+0)] @ save out +.Loop_outer_enter: + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(16+15)] + mov r11,#10 + b .Loop + +.align 4 +.Loop: + subs r11,r11,#1 + add r0,r0,r4,ror#13 + add r1,r1,r5,ror#13 + eor r12,r0,r12,ror#24 + eor r10,r1,r10,ror#24 + add r8,r8,r12,ror#16 + add r9,r9,r10,ror#16 + eor r4,r8,r4,ror#13 + eor r5,r9,r5,ror#13 + add r0,r0,r4,ror#20 + add r1,r1,r5,ror#20 + eor r12,r0,r12,ror#16 + eor r10,r1,r10,ror#16 + add r8,r8,r12,ror#24 + str r10,[sp,#4*(16+13)] + add r9,r9,r10,ror#24 + ldr r10,[sp,#4*(16+15)] + str r8,[sp,#4*(16+8)] + eor r4,r4,r8,ror#12 + str r9,[sp,#4*(16+9)] + eor r5,r5,r9,ror#12 + ldr r8,[sp,#4*(16+10)] + add r2,r2,r6,ror#13 + ldr r9,[sp,#4*(16+11)] + add r3,r3,r7,ror#13 + eor r14,r2,r14,ror#24 + eor r10,r3,r10,ror#24 + add r8,r8,r14,ror#16 + add r9,r9,r10,ror#16 + eor r6,r8,r6,ror#13 + eor r7,r9,r7,ror#13 + add r2,r2,r6,ror#20 + add r3,r3,r7,ror#20 + eor r14,r2,r14,ror#16 + eor r10,r3,r10,ror#16 + add r8,r8,r14,ror#24 + add r9,r9,r10,ror#24 + eor r6,r6,r8,ror#12 + eor r7,r7,r9,ror#12 + add r0,r0,r5,ror#13 + add r1,r1,r6,ror#13 + eor r10,r0,r10,ror#24 + eor r12,r1,r12,ror#24 + add r8,r8,r10,ror#16 + add r9,r9,r12,ror#16 + eor r5,r8,r5,ror#13 + eor r6,r9,r6,ror#13 + add r0,r0,r5,ror#20 + add r1,r1,r6,ror#20 + eor r10,r0,r10,ror#16 + eor r12,r1,r12,ror#16 + str r10,[sp,#4*(16+15)] + add r8,r8,r10,ror#24 + ldr r10,[sp,#4*(16+13)] + add r9,r9,r12,ror#24 + str r8,[sp,#4*(16+10)] + eor r5,r5,r8,ror#12 + str r9,[sp,#4*(16+11)] + eor r6,r6,r9,ror#12 + ldr r8,[sp,#4*(16+8)] + add r2,r2,r7,ror#13 + ldr r9,[sp,#4*(16+9)] + add r3,r3,r4,ror#13 + eor r10,r2,r10,ror#24 + eor r14,r3,r14,ror#24 + add r8,r8,r10,ror#16 + add r9,r9,r14,ror#16 + eor r7,r8,r7,ror#13 + eor r4,r9,r4,ror#13 + add r2,r2,r7,ror#20 + add r3,r3,r4,ror#20 + eor r10,r2,r10,ror#16 + eor r14,r3,r14,ror#16 + add r8,r8,r10,ror#24 + add r9,r9,r14,ror#24 + eor r7,r7,r8,ror#12 + eor r4,r4,r9,ror#12 + bne .Loop + + ldr r11,[sp,#4*(32+2)] @ load len + + str r8, [sp,#4*(16+8)] @ modulo-scheduled store + str r9, [sp,#4*(16+9)] + str r12,[sp,#4*(16+12)] + str r10, [sp,#4*(16+13)] + str r14,[sp,#4*(16+14)] + + @ at this point we have first half of 512-bit result in + @ rx and second half at sp+4*(16+8) + + cmp r11,#64 @ done yet? +#ifdef __thumb2__ + itete lo +#endif + addlo r12,sp,#4*(0) @ shortcut or ... + ldrhs r12,[sp,#4*(32+1)] @ ... load inp + addlo r14,sp,#4*(0) @ shortcut or ... + ldrhs r14,[sp,#4*(32+0)] @ ... load out + + ldr r8,[sp,#4*(0)] @ load key material + ldr r9,[sp,#4*(1)] + +#if __ARM_ARCH__>=6 || !defined(__ARMEB__) +# if __ARM_ARCH__<7 + orr r10,r12,r14 + tst r10,#3 @ are input and output aligned? + ldr r10,[sp,#4*(2)] + bne .Lunaligned + cmp r11,#64 @ restore flags +# else + ldr r10,[sp,#4*(2)] +# endif + ldr r11,[sp,#4*(3)] + + add r0,r0,r8 @ accumulate key material + add r1,r1,r9 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + + add r2,r2,r10 + add r3,r3,r11 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r0,r0,r8 @ xor with input + eorhs r1,r1,r9 + add r8,sp,#4*(4) + str r0,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r2,r2,r10 + eorhs r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r1,[r14,#-12] + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#13 @ accumulate key material + add r5,r9,r5,ror#13 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + add r6,r10,r6,ror#13 + add r7,r11,r7,ror#13 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r4,r4,r8 + eorhs r5,r5,r9 + add r8,sp,#4*(8) + str r4,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r6,r6,r10 + eorhs r7,r7,r11 + str r5,[r14,#-12] + ldmia r8,{r8-r11} @ load key material + str r6,[r14,#-8] + add r0,sp,#4*(16+8) + str r7,[r14,#-4] + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + add r1,r1,r9 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] +# ifdef __thumb2__ + itt hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it + add r2,r2,r10 + add r3,r3,r11 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r0,r0,r8 + eorhs r1,r1,r9 + add r8,sp,#4*(12) + str r0,[r14],#16 @ store output +# ifdef __thumb2__ + itt hs +# endif + eorhs r2,r2,r10 + eorhs r3,r3,r11 + str r1,[r14,#-12] + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#24 @ accumulate key material + add r5,r9,r5,ror#24 +# ifdef __thumb2__ + itt hi +# endif + addhi r8,r8,#1 @ next counter value + strhi r8,[sp,#4*(12)] @ save next counter value +# ifdef __thumb2__ + itt hs +# endif + ldrhs r8,[r12],#16 @ load input + ldrhs r9,[r12,#-12] + add r6,r10,r6,ror#24 + add r7,r11,r7,ror#24 +# ifdef __thumb2__ + itt hs +# endif + ldrhs r10,[r12,#-8] + ldrhs r11,[r12,#-4] +# if __ARM_ARCH__>=6 && defined(__ARMEB__) + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif +# ifdef __thumb2__ + itt hs +# endif + eorhs r4,r4,r8 + eorhs r5,r5,r9 +# ifdef __thumb2__ + it ne +# endif + ldrne r8,[sp,#4*(32+2)] @ re-load len +# ifdef __thumb2__ + itt hs +# endif + eorhs r6,r6,r10 + eorhs r7,r7,r11 + str r4,[r14],#16 @ store output + str r5,[r14,#-12] +# ifdef __thumb2__ + it hs +# endif + subhs r11,r8,#64 @ len-=64 + str r6,[r14,#-8] + str r7,[r14,#-4] + bhi .Loop_outer + + beq .Ldone +# if __ARM_ARCH__<7 + b .Ltail + +.align 4 +.Lunaligned: @ unaligned endian-neutral path + cmp r11,#64 @ restore flags +# endif +#endif +#if __ARM_ARCH__<7 + ldr r11,[sp,#4*(3)] + add r0,r8,r0 @ accumulate key material + add r1,r9,r1 + add r2,r10,r2 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r3,r11,r3 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r0,r8,r0 @ xor with input (or zero) + eor r1,r9,r1 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r2,r10,r2 + strb r0,[r14],#16 @ store output + eor r3,r11,r3 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r1,[r14,#-12] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-8] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r3,[r14,#-4] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-15] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r1,[r14,#-11] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-7] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r3,[r14,#-3] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-14] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r1,[r14,#-10] + strb r2,[r14,#-6] + eor r0,r8,r0,lsr#8 + strb r3,[r14,#-2] + eor r1,r9,r1,lsr#8 + strb r0,[r14,#-13] + eor r2,r10,r2,lsr#8 + strb r1,[r14,#-9] + eor r3,r11,r3,lsr#8 + strb r2,[r14,#-5] + strb r3,[r14,#-1] + add r8,sp,#4*(4+0) + ldmia r8,{r8-r11} @ load key material + add r0,sp,#4*(16+8) + add r4,r8,r4,ror#13 @ accumulate key material + add r5,r9,r5,ror#13 + add r6,r10,r6,ror#13 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r7,r11,r7,ror#13 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r4,r8,r4 @ xor with input (or zero) + eor r5,r9,r5 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r6,r10,r6 + strb r4,[r14],#16 @ store output + eor r7,r11,r7 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r5,[r14,#-12] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-8] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r7,[r14,#-4] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-15] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r5,[r14,#-11] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-7] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r7,[r14,#-3] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-14] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r5,[r14,#-10] + strb r6,[r14,#-6] + eor r4,r8,r4,lsr#8 + strb r7,[r14,#-2] + eor r5,r9,r5,lsr#8 + strb r4,[r14,#-13] + eor r6,r10,r6,lsr#8 + strb r5,[r14,#-9] + eor r7,r11,r7,lsr#8 + strb r6,[r14,#-5] + strb r7,[r14,#-1] + add r8,sp,#4*(4+4) + ldmia r8,{r8-r11} @ load key material + ldmia r0,{r0-r7} @ load second half +# ifdef __thumb2__ + itt hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" + strhi r11,[sp,#4*(16+11)] @ copy "rx" + add r0,r8,r0 @ accumulate key material + add r1,r9,r1 + add r2,r10,r2 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r3,r11,r3 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r0,r8,r0 @ xor with input (or zero) + eor r1,r9,r1 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r2,r10,r2 + strb r0,[r14],#16 @ store output + eor r3,r11,r3 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r1,[r14,#-12] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-8] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r3,[r14,#-4] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-15] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r1,[r14,#-11] + eor r0,r8,r0,lsr#8 + strb r2,[r14,#-7] + eor r1,r9,r1,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r3,[r14,#-3] + eor r2,r10,r2,lsr#8 + strb r0,[r14,#-14] + eor r3,r11,r3,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r1,[r14,#-10] + strb r2,[r14,#-6] + eor r0,r8,r0,lsr#8 + strb r3,[r14,#-2] + eor r1,r9,r1,lsr#8 + strb r0,[r14,#-13] + eor r2,r10,r2,lsr#8 + strb r1,[r14,#-9] + eor r3,r11,r3,lsr#8 + strb r2,[r14,#-5] + strb r3,[r14,#-1] + add r8,sp,#4*(4+8) + ldmia r8,{r8-r11} @ load key material + add r4,r8,r4,ror#24 @ accumulate key material +# ifdef __thumb2__ + itt hi +# endif + addhi r8,r8,#1 @ next counter value + strhi r8,[sp,#4*(12)] @ save next counter value + add r5,r9,r5,ror#24 + add r6,r10,r6,ror#24 +# ifdef __thumb2__ + itete lo +# endif + eorlo r8,r8,r8 @ zero or ... + ldrhsb r8,[r12],#16 @ ... load input + eorlo r9,r9,r9 + ldrhsb r9,[r12,#-12] + + add r7,r11,r7,ror#24 +# ifdef __thumb2__ + itete lo +# endif + eorlo r10,r10,r10 + ldrhsb r10,[r12,#-8] + eorlo r11,r11,r11 + ldrhsb r11,[r12,#-4] + + eor r4,r8,r4 @ xor with input (or zero) + eor r5,r9,r5 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-15] @ load more input + ldrhsb r9,[r12,#-11] + eor r6,r10,r6 + strb r4,[r14],#16 @ store output + eor r7,r11,r7 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-7] + ldrhsb r11,[r12,#-3] + strb r5,[r14,#-12] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-8] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-14] @ load more input + ldrhsb r9,[r12,#-10] + strb r7,[r14,#-4] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-15] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-6] + ldrhsb r11,[r12,#-2] + strb r5,[r14,#-11] + eor r4,r8,r4,lsr#8 + strb r6,[r14,#-7] + eor r5,r9,r5,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r8,[r12,#-13] @ load more input + ldrhsb r9,[r12,#-9] + strb r7,[r14,#-3] + eor r6,r10,r6,lsr#8 + strb r4,[r14,#-14] + eor r7,r11,r7,lsr#8 +# ifdef __thumb2__ + itt hs +# endif + ldrhsb r10,[r12,#-5] + ldrhsb r11,[r12,#-1] + strb r5,[r14,#-10] + strb r6,[r14,#-6] + eor r4,r8,r4,lsr#8 + strb r7,[r14,#-2] + eor r5,r9,r5,lsr#8 + strb r4,[r14,#-13] + eor r6,r10,r6,lsr#8 + strb r5,[r14,#-9] + eor r7,r11,r7,lsr#8 + strb r6,[r14,#-5] + strb r7,[r14,#-1] +# ifdef __thumb2__ + it ne +# endif + ldrne r8,[sp,#4*(32+2)] @ re-load len +# ifdef __thumb2__ + it hs +# endif + subhs r11,r8,#64 @ len-=64 + bhi .Loop_outer + + beq .Ldone +#endif + +.Ltail: + ldr r12,[sp,#4*(32+1)] @ load inp + add r9,sp,#4*(0) + ldr r14,[sp,#4*(32+0)] @ load out + +.Loop_tail: + ldrb r10,[r9],#1 @ read buffer on stack + ldrb r11,[r12],#1 @ read input + subs r8,r8,#1 + eor r11,r11,r10 + strb r11,[r14],#1 @ store output + bne .Loop_tail + +.Ldone: + add sp,sp,#4*(32+3) +.Lno_data: + ldmia sp!,{r4-r11,pc} +.size ChaCha20_ctr32,.-ChaCha20_ctr32 +#if __ARM_MAX_ARCH__>=7 +.arch armv7-a +.fpu neon + +.type ChaCha20_neon,%function +.align 5 +ChaCha20_neon: + ldr r12,[sp,#0] @ pull pointer to counter and nonce + stmdb sp!,{r0-r2,r4-r11,lr} +.LChaCha20_neon: + adr r14,.Lsigma + vstmdb sp!,{d8-d15} @ ABI spec says so + stmdb sp!,{r0-r3} + + vld1.32 {q1-q2},[r3] @ load key + ldmia r3,{r4-r11} @ load key + + sub sp,sp,#4*(16+16) + vld1.32 {q3},[r12] @ load counter and nonce + add r12,sp,#4*8 + ldmia r14,{r0-r3} @ load sigma + vld1.32 {q0},[r14]! @ load sigma + vld1.32 {q12},[r14]! @ one + @ vld1.32 {d30},[r14] @ rot8 + vst1.32 {q2-q3},[r12] @ copy 1/2key|counter|nonce + vst1.32 {q0-q1},[sp] @ copy sigma|1/2key + + str r10,[sp,#4*(16+10)] @ off-load "rx" + str r11,[sp,#4*(16+11)] @ off-load "rx" + vshl.i32 d26,d24,#1 @ two + vstr d24,[sp,#4*(16+0)] + vshl.i32 d28,d24,#2 @ four + vstr d26,[sp,#4*(16+2)] + vmov q4,q0 + vstr d28,[sp,#4*(16+4)] + vmov q8,q0 + @ vstr d30,[sp,#4*(16+6)] + vmov q5,q1 + vmov q9,q1 + b .Loop_neon_enter + +.align 4 +.Loop_neon_outer: + ldmia sp,{r0-r9} @ load key material + cmp r11,#64*2 @ if len<=64*2 + bls .Lbreak_neon @ switch to integer-only + @ vldr d30,[sp,#4*(16+6)] @ rot8 + vmov q4,q0 + str r11,[sp,#4*(32+2)] @ save len + vmov q8,q0 + str r12, [sp,#4*(32+1)] @ save inp + vmov q5,q1 + str r14, [sp,#4*(32+0)] @ save out + vmov q9,q1 +.Loop_neon_enter: + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + vadd.i32 q7,q3,q12 @ counter+1 + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + vmov q6,q2 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + vmov q10,q2 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + vadd.i32 q11,q7,q12 @ counter+2 + add r12,r12,#3 @ counter+3 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(16+15)] + mov r11,#10 + b .Loop_neon + +.align 4 +.Loop_neon: + subs r11,r11,#1 + vadd.i32 q0,q0,q1 + add r0,r0,r4,ror#13 + vadd.i32 q4,q4,q5 + add r1,r1,r5,ror#13 + vadd.i32 q8,q8,q9 + eor r12,r0,r12,ror#24 + veor q3,q3,q0 + eor r10,r1,r10,ror#24 + veor q7,q7,q4 + add r8,r8,r12,ror#16 + veor q11,q11,q8 + add r9,r9,r10,ror#16 + vrev32.16 q3,q3 + eor r4,r8,r4,ror#13 + vrev32.16 q7,q7 + eor r5,r9,r5,ror#13 + vrev32.16 q11,q11 + add r0,r0,r4,ror#20 + vadd.i32 q2,q2,q3 + add r1,r1,r5,ror#20 + vadd.i32 q6,q6,q7 + eor r12,r0,r12,ror#16 + vadd.i32 q10,q10,q11 + eor r10,r1,r10,ror#16 + veor q12,q1,q2 + add r8,r8,r12,ror#24 + veor q13,q5,q6 + str r10,[sp,#4*(16+13)] + veor q14,q9,q10 + add r9,r9,r10,ror#24 + vshr.u32 q1,q12,#20 + ldr r10,[sp,#4*(16+15)] + vshr.u32 q5,q13,#20 + str r8,[sp,#4*(16+8)] + vshr.u32 q9,q14,#20 + eor r4,r4,r8,ror#12 + vsli.32 q1,q12,#12 + str r9,[sp,#4*(16+9)] + vsli.32 q5,q13,#12 + eor r5,r5,r9,ror#12 + vsli.32 q9,q14,#12 + ldr r8,[sp,#4*(16+10)] + vadd.i32 q0,q0,q1 + add r2,r2,r6,ror#13 + vadd.i32 q4,q4,q5 + ldr r9,[sp,#4*(16+11)] + vadd.i32 q8,q8,q9 + add r3,r3,r7,ror#13 + veor q12,q3,q0 + eor r14,r2,r14,ror#24 + veor q13,q7,q4 + eor r10,r3,r10,ror#24 + veor q14,q11,q8 + add r8,r8,r14,ror#16 + vshr.u32 q3,q12,#24 + add r9,r9,r10,ror#16 + vshr.u32 q7,q13,#24 + eor r6,r8,r6,ror#13 + vshr.u32 q11,q14,#24 + eor r7,r9,r7,ror#13 + vsli.32 q3,q12,#8 + add r2,r2,r6,ror#20 + vsli.32 q7,q13,#8 + add r3,r3,r7,ror#20 + vsli.32 q11,q14,#8 + eor r14,r2,r14,ror#16 + vadd.i32 q2,q2,q3 + eor r10,r3,r10,ror#16 + vadd.i32 q6,q6,q7 + add r8,r8,r14,ror#24 + vadd.i32 q10,q10,q11 + add r9,r9,r10,ror#24 + veor q12,q1,q2 + eor r6,r6,r8,ror#12 + veor q13,q5,q6 + eor r7,r7,r9,ror#12 + veor q14,q9,q10 + vshr.u32 q1,q12,#25 + vshr.u32 q5,q13,#25 + vshr.u32 q9,q14,#25 + vsli.32 q1,q12,#7 + vsli.32 q5,q13,#7 + vsli.32 q9,q14,#7 + vext.8 q2,q2,q2,#8 + vext.8 q6,q6,q6,#8 + vext.8 q10,q10,q10,#8 + vext.8 q1,q1,q1,#4 + vext.8 q5,q5,q5,#4 + vext.8 q9,q9,q9,#4 + vext.8 q3,q3,q3,#12 + vext.8 q7,q7,q7,#12 + vext.8 q11,q11,q11,#12 + vadd.i32 q0,q0,q1 + add r0,r0,r5,ror#13 + vadd.i32 q4,q4,q5 + add r1,r1,r6,ror#13 + vadd.i32 q8,q8,q9 + eor r10,r0,r10,ror#24 + veor q3,q3,q0 + eor r12,r1,r12,ror#24 + veor q7,q7,q4 + add r8,r8,r10,ror#16 + veor q11,q11,q8 + add r9,r9,r12,ror#16 + vrev32.16 q3,q3 + eor r5,r8,r5,ror#13 + vrev32.16 q7,q7 + eor r6,r9,r6,ror#13 + vrev32.16 q11,q11 + add r0,r0,r5,ror#20 + vadd.i32 q2,q2,q3 + add r1,r1,r6,ror#20 + vadd.i32 q6,q6,q7 + eor r10,r0,r10,ror#16 + vadd.i32 q10,q10,q11 + eor r12,r1,r12,ror#16 + veor q12,q1,q2 + str r10,[sp,#4*(16+15)] + veor q13,q5,q6 + add r8,r8,r10,ror#24 + veor q14,q9,q10 + ldr r10,[sp,#4*(16+13)] + vshr.u32 q1,q12,#20 + add r9,r9,r12,ror#24 + vshr.u32 q5,q13,#20 + str r8,[sp,#4*(16+10)] + vshr.u32 q9,q14,#20 + eor r5,r5,r8,ror#12 + vsli.32 q1,q12,#12 + str r9,[sp,#4*(16+11)] + vsli.32 q5,q13,#12 + eor r6,r6,r9,ror#12 + vsli.32 q9,q14,#12 + ldr r8,[sp,#4*(16+8)] + vadd.i32 q0,q0,q1 + add r2,r2,r7,ror#13 + vadd.i32 q4,q4,q5 + ldr r9,[sp,#4*(16+9)] + vadd.i32 q8,q8,q9 + add r3,r3,r4,ror#13 + veor q12,q3,q0 + eor r10,r2,r10,ror#24 + veor q13,q7,q4 + eor r14,r3,r14,ror#24 + veor q14,q11,q8 + add r8,r8,r10,ror#16 + vshr.u32 q3,q12,#24 + add r9,r9,r14,ror#16 + vshr.u32 q7,q13,#24 + eor r7,r8,r7,ror#13 + vshr.u32 q11,q14,#24 + eor r4,r9,r4,ror#13 + vsli.32 q3,q12,#8 + add r2,r2,r7,ror#20 + vsli.32 q7,q13,#8 + add r3,r3,r4,ror#20 + vsli.32 q11,q14,#8 + eor r10,r2,r10,ror#16 + vadd.i32 q2,q2,q3 + eor r14,r3,r14,ror#16 + vadd.i32 q6,q6,q7 + add r8,r8,r10,ror#24 + vadd.i32 q10,q10,q11 + add r9,r9,r14,ror#24 + veor q12,q1,q2 + eor r7,r7,r8,ror#12 + veor q13,q5,q6 + eor r4,r4,r9,ror#12 + veor q14,q9,q10 + vshr.u32 q1,q12,#25 + vshr.u32 q5,q13,#25 + vshr.u32 q9,q14,#25 + vsli.32 q1,q12,#7 + vsli.32 q5,q13,#7 + vsli.32 q9,q14,#7 + vext.8 q2,q2,q2,#8 + vext.8 q6,q6,q6,#8 + vext.8 q10,q10,q10,#8 + vext.8 q1,q1,q1,#12 + vext.8 q5,q5,q5,#12 + vext.8 q9,q9,q9,#12 + vext.8 q3,q3,q3,#4 + vext.8 q7,q7,q7,#4 + vext.8 q11,q11,q11,#4 + bne .Loop_neon + + add r11,sp,#32 + vld1.32 {q12-q13},[sp] @ load key material + vld1.32 {q14-q15},[r11] + + ldr r11,[sp,#4*(32+2)] @ load len + + str r8, [sp,#4*(16+8)] @ modulo-scheduled store + str r9, [sp,#4*(16+9)] + str r12,[sp,#4*(16+12)] + str r10, [sp,#4*(16+13)] + str r14,[sp,#4*(16+14)] + + @ at this point we have first half of 512-bit result in + @ rx and second half at sp+4*(16+8) + + ldr r12,[sp,#4*(32+1)] @ load inp + ldr r14,[sp,#4*(32+0)] @ load out + + vadd.i32 q0,q0,q12 @ accumulate key material + vadd.i32 q4,q4,q12 + vadd.i32 q8,q8,q12 + vldr d24,[sp,#4*(16+0)] @ one + + vadd.i32 q1,q1,q13 + vadd.i32 q5,q5,q13 + vadd.i32 q9,q9,q13 + vldr d26,[sp,#4*(16+2)] @ two + + vadd.i32 q2,q2,q14 + vadd.i32 q6,q6,q14 + vadd.i32 q10,q10,q14 + vadd.i32 d14,d14,d24 @ counter+1 + vadd.i32 d22,d22,d26 @ counter+2 + + vadd.i32 q3,q3,q15 + vadd.i32 q7,q7,q15 + vadd.i32 q11,q11,q15 + + cmp r11,#64*4 + blo .Ltail_neon + + vld1.8 {q12-q13},[r12]! @ load input + mov r11,sp + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 @ xor with input + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + vst1.8 {q0-q1},[r14]! @ store output + veor q5,q5,q13 + vld1.8 {q12-q13},[r12]! + veor q6,q6,q14 + vst1.8 {q2-q3},[r14]! + veor q7,q7,q15 + vld1.8 {q14-q15},[r12]! + + veor q8,q8,q12 + vld1.32 {q0-q1},[r11]! @ load for next iteration + veor d25,d25,d25 + vldr d24,[sp,#4*(16+4)] @ four + veor q9,q9,q13 + vld1.32 {q2-q3},[r11] + veor q10,q10,q14 + vst1.8 {q4-q5},[r14]! + veor q11,q11,q15 + vst1.8 {q6-q7},[r14]! + + vadd.i32 d6,d6,d24 @ next counter value + vldr d24,[sp,#4*(16+0)] @ one + + ldmia sp,{r8-r11} @ load key material + add r0,r0,r8 @ accumulate key material + ldr r8,[r12],#16 @ load input + vst1.8 {q8-q9},[r14]! + add r1,r1,r9 + ldr r9,[r12,#-12] + vst1.8 {q10-q11},[r14]! + add r2,r2,r10 + ldr r10,[r12,#-8] + add r3,r3,r11 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif + eor r0,r0,r8 @ xor with input + add r8,sp,#4*(4) + eor r1,r1,r9 + str r0,[r14],#16 @ store output + eor r2,r2,r10 + str r1,[r14,#-12] + eor r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#13 @ accumulate key material + ldr r8,[r12],#16 @ load input + add r5,r9,r5,ror#13 + ldr r9,[r12,#-12] + add r6,r10,r6,ror#13 + ldr r10,[r12,#-8] + add r7,r11,r7,ror#13 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + eor r4,r4,r8 + add r8,sp,#4*(8) + eor r5,r5,r9 + str r4,[r14],#16 @ store output + eor r6,r6,r10 + str r5,[r14,#-12] + eor r7,r7,r11 + ldmia r8,{r8-r11} @ load key material + str r6,[r14,#-8] + add r0,sp,#4*(16+8) + str r7,[r14,#-4] + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + ldr r8,[r12],#16 @ load input + add r1,r1,r9 + ldr r9,[r12,#-12] +# ifdef __thumb2__ + it hi +# endif + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it + add r2,r2,r10 + ldr r10,[r12,#-8] +# ifdef __thumb2__ + it hi +# endif + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it + add r3,r3,r11 + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 +# endif + eor r0,r0,r8 + add r8,sp,#4*(12) + eor r1,r1,r9 + str r0,[r14],#16 @ store output + eor r2,r2,r10 + str r1,[r14,#-12] + eor r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + str r2,[r14,#-8] + str r3,[r14,#-4] + + add r4,r8,r4,ror#24 @ accumulate key material + add r8,r8,#4 @ next counter value + add r5,r9,r5,ror#24 + str r8,[sp,#4*(12)] @ save next counter value + ldr r8,[r12],#16 @ load input + add r6,r10,r6,ror#24 + add r4,r4,#3 @ counter+3 + ldr r9,[r12,#-12] + add r7,r11,r7,ror#24 + ldr r10,[r12,#-8] + ldr r11,[r12,#-4] +# ifdef __ARMEB__ + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + eor r4,r4,r8 +# ifdef __thumb2__ + it hi +# endif + ldrhi r8,[sp,#4*(32+2)] @ re-load len + eor r5,r5,r9 + eor r6,r6,r10 + str r4,[r14],#16 @ store output + eor r7,r7,r11 + str r5,[r14,#-12] + sub r11,r8,#64*4 @ len-=64*4 + str r6,[r14,#-8] + str r7,[r14,#-4] + bhi .Loop_neon_outer + + b .Ldone_neon + +.align 4 +.Lbreak_neon: + @ harmonize NEON and integer-only stack frames: load data + @ from NEON frame, but save to integer-only one; distance + @ between the two is 4*(32+4+16-32)=4*(20). + + str r11, [sp,#4*(20+32+2)] @ save len + add r11,sp,#4*(32+4) + str r12, [sp,#4*(20+32+1)] @ save inp + str r14, [sp,#4*(20+32+0)] @ save out + + ldr r12,[sp,#4*(16+10)] + ldr r14,[sp,#4*(16+11)] + vldmia r11,{d8-d15} @ fulfill ABI requirement + str r12,[sp,#4*(20+16+10)] @ copy "rx" + str r14,[sp,#4*(20+16+11)] @ copy "rx" + + ldr r11, [sp,#4*(15)] + mov r4,r4,ror#19 @ twist b[0..3] + ldr r12,[sp,#4*(12)] @ modulo-scheduled load + mov r5,r5,ror#19 + ldr r10, [sp,#4*(13)] + mov r6,r6,ror#19 + ldr r14,[sp,#4*(14)] + mov r7,r7,ror#19 + mov r11,r11,ror#8 @ twist d[0..3] + mov r12,r12,ror#8 + mov r10,r10,ror#8 + mov r14,r14,ror#8 + str r11, [sp,#4*(20+16+15)] + add r11,sp,#4*(20) + vst1.32 {q0-q1},[r11]! @ copy key + add sp,sp,#4*(20) @ switch frame + vst1.32 {q2-q3},[r11] + mov r11,#10 + b .Loop @ go integer-only + +.align 4 +.Ltail_neon: + cmp r11,#64*3 + bhs .L192_or_more_neon + cmp r11,#64*2 + bhs .L128_or_more_neon + cmp r11,#64*1 + bhs .L64_or_more_neon + + add r8,sp,#4*(8) + vst1.8 {q0-q1},[sp] + add r10,sp,#4*(0) + vst1.8 {q2-q3},[r8] + b .Loop_tail_neon + +.align 4 +.L64_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + veor q2,q2,q14 + veor q3,q3,q15 + vst1.8 {q0-q1},[r14]! + vst1.8 {q2-q3},[r14]! + + beq .Ldone_neon + + add r8,sp,#4*(8) + vst1.8 {q4-q5},[sp] + add r10,sp,#4*(0) + vst1.8 {q6-q7},[r8] + sub r11,r11,#64*1 @ len-=64*1 + b .Loop_tail_neon + +.align 4 +.L128_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + veor q5,q5,q13 + vst1.8 {q0-q1},[r14]! + veor q6,q6,q14 + vst1.8 {q2-q3},[r14]! + veor q7,q7,q15 + vst1.8 {q4-q5},[r14]! + vst1.8 {q6-q7},[r14]! + + beq .Ldone_neon + + add r8,sp,#4*(8) + vst1.8 {q8-q9},[sp] + add r10,sp,#4*(0) + vst1.8 {q10-q11},[r8] + sub r11,r11,#64*2 @ len-=64*2 + b .Loop_tail_neon + +.align 4 +.L192_or_more_neon: + vld1.8 {q12-q13},[r12]! + vld1.8 {q14-q15},[r12]! + veor q0,q0,q12 + veor q1,q1,q13 + vld1.8 {q12-q13},[r12]! + veor q2,q2,q14 + veor q3,q3,q15 + vld1.8 {q14-q15},[r12]! + + veor q4,q4,q12 + veor q5,q5,q13 + vld1.8 {q12-q13},[r12]! + veor q6,q6,q14 + vst1.8 {q0-q1},[r14]! + veor q7,q7,q15 + vld1.8 {q14-q15},[r12]! + + veor q8,q8,q12 + vst1.8 {q2-q3},[r14]! + veor q9,q9,q13 + vst1.8 {q4-q5},[r14]! + veor q10,q10,q14 + vst1.8 {q6-q7},[r14]! + veor q11,q11,q15 + vst1.8 {q8-q9},[r14]! + vst1.8 {q10-q11},[r14]! + + beq .Ldone_neon + + ldmia sp,{r8-r11} @ load key material + add r0,r0,r8 @ accumulate key material + add r8,sp,#4*(4) + add r1,r1,r9 + add r2,r2,r10 + add r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + + add r4,r8,r4,ror#13 @ accumulate key material + add r8,sp,#4*(8) + add r5,r9,r5,ror#13 + add r6,r10,r6,ror#13 + add r7,r11,r7,ror#13 + ldmia r8,{r8-r11} @ load key material +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + stmia sp,{r0-r7} + add r0,sp,#4*(16+8) + + ldmia r0,{r0-r7} @ load second half + + add r0,r0,r8 @ accumulate key material + add r8,sp,#4*(12) + add r1,r1,r9 + add r2,r2,r10 + add r3,r3,r11 + ldmia r8,{r8-r11} @ load key material + + add r4,r8,r4,ror#24 @ accumulate key material + add r8,sp,#4*(8) + add r5,r9,r5,ror#24 + add r4,r4,#3 @ counter+3 + add r6,r10,r6,ror#24 + add r7,r11,r7,ror#24 + ldr r11,[sp,#4*(32+2)] @ re-load len +# ifdef __ARMEB__ + rev r0,r0 + rev r1,r1 + rev r2,r2 + rev r3,r3 + rev r4,r4 + rev r5,r5 + rev r6,r6 + rev r7,r7 +# endif + stmia r8,{r0-r7} + add r10,sp,#4*(0) + sub r11,r11,#64*3 @ len-=64*3 + +.Loop_tail_neon: + ldrb r8,[r10],#1 @ read buffer on stack + ldrb r9,[r12],#1 @ read input + subs r11,r11,#1 + eor r8,r8,r9 + strb r8,[r14],#1 @ store output + bne .Loop_tail_neon + +.Ldone_neon: + add sp,sp,#4*(32+4) + vldmia sp,{d8-d15} + add sp,sp,#4*(16+3) + ldmia sp!,{r4-r11,pc} +.size ChaCha20_neon,.-ChaCha20_neon +.comm OPENSSL_armcap_P,4,4 +#endiflib/zinc/chacha20/chacha20-arm64-cryptogams.S new file +1973
-1@@ -0,0 +1,1973 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* + * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + */ + +#include "arm_arch.h" + +.text + + + +.align 5 +.Lsigma: +.quad 0x3320646e61707865,0x6b20657479622d32 // endian-neutral +.Lone: +.long 1,0,0,0 +.LOPENSSL_armcap_P: +#ifdef __ILP32__ +.long OPENSSL_armcap_P-. +#else +.quad OPENSSL_armcap_P-. +#endif +.byte 67,104,97,67,104,97,50,48,32,102,111,114,32,65,82,77,118,56,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0 +.align 2 + +.globl ChaCha20_ctr32 +.type ChaCha20_ctr32,%function +.align 5 +ChaCha20_ctr32: + cbz x2,.Labort + adr x5,.LOPENSSL_armcap_P + cmp x2,#192 + b.lo .Lshort +#ifdef __ILP32__ + ldrsw x6,[x5] +#else + ldr x6,[x5] +#endif + ldr w17,[x6,x5] + tst w17,#ARMV7_NEON + b.ne ChaCha20_neon + +.Lshort: + stp x29,x30,[sp,#-96]! + add x29,sp,#0 + + adr x5,.Lsigma + stp x19,x20,[sp,#16] + stp x21,x22,[sp,#32] + stp x23,x24,[sp,#48] + stp x25,x26,[sp,#64] + stp x27,x28,[sp,#80] + sub sp,sp,#64 + + ldp x22,x23,[x5] // load sigma + ldp x24,x25,[x3] // load key + ldp x26,x27,[x3,#16] + ldp x28,x30,[x4] // load counter +#ifdef __ARMEB__ + ror x24,x24,#32 + ror x25,x25,#32 + ror x26,x26,#32 + ror x27,x27,#32 + ror x28,x28,#32 + ror x30,x30,#32 +#endif + +.Loop_outer: + mov w5,w22 // unpack key block + lsr x6,x22,#32 + mov w7,w23 + lsr x8,x23,#32 + mov w9,w24 + lsr x10,x24,#32 + mov w11,w25 + lsr x12,x25,#32 + mov w13,w26 + lsr x14,x26,#32 + mov w15,w27 + lsr x16,x27,#32 + mov w17,w28 + lsr x19,x28,#32 + mov w20,w30 + lsr x21,x30,#32 + + mov x4,#10 + subs x2,x2,#64 +.Loop: + sub x4,x4,#1 + add w5,w5,w9 + add w6,w6,w10 + add w7,w7,w11 + add w8,w8,w12 + eor w17,w17,w5 + eor w19,w19,w6 + eor w20,w20,w7 + eor w21,w21,w8 + ror w17,w17,#16 + ror w19,w19,#16 + ror w20,w20,#16 + ror w21,w21,#16 + add w13,w13,w17 + add w14,w14,w19 + add w15,w15,w20 + add w16,w16,w21 + eor w9,w9,w13 + eor w10,w10,w14 + eor w11,w11,w15 + eor w12,w12,w16 + ror w9,w9,#20 + ror w10,w10,#20 + ror w11,w11,#20 + ror w12,w12,#20 + add w5,w5,w9 + add w6,w6,w10 + add w7,w7,w11 + add w8,w8,w12 + eor w17,w17,w5 + eor w19,w19,w6 + eor w20,w20,w7 + eor w21,w21,w8 + ror w17,w17,#24 + ror w19,w19,#24 + ror w20,w20,#24 + ror w21,w21,#24 + add w13,w13,w17 + add w14,w14,w19 + add w15,w15,w20 + add w16,w16,w21 + eor w9,w9,w13 + eor w10,w10,w14 + eor w11,w11,w15 + eor w12,w12,w16 + ror w9,w9,#25 + ror w10,w10,#25 + ror w11,w11,#25 + ror w12,w12,#25 + add w5,w5,w10 + add w6,w6,w11 + add w7,w7,w12 + add w8,w8,w9 + eor w21,w21,w5 + eor w17,w17,w6 + eor w19,w19,w7 + eor w20,w20,w8 + ror w21,w21,#16 + ror w17,w17,#16 + ror w19,w19,#16 + ror w20,w20,#16 + add w15,w15,w21 + add w16,w16,w17 + add w13,w13,w19 + add w14,w14,w20 + eor w10,w10,w15 + eor w11,w11,w16 + eor w12,w12,w13 + eor w9,w9,w14 + ror w10,w10,#20 + ror w11,w11,#20 + ror w12,w12,#20 + ror w9,w9,#20 + add w5,w5,w10 + add w6,w6,w11 + add w7,w7,w12 + add w8,w8,w9 + eor w21,w21,w5 + eor w17,w17,w6 + eor w19,w19,w7 + eor w20,w20,w8 + ror w21,w21,#24 + ror w17,w17,#24 + ror w19,w19,#24 + ror w20,w20,#24 + add w15,w15,w21 + add w16,w16,w17 + add w13,w13,w19 + add w14,w14,w20 + eor w10,w10,w15 + eor w11,w11,w16 + eor w12,w12,w13 + eor w9,w9,w14 + ror w10,w10,#25 + ror w11,w11,#25 + ror w12,w12,#25 + ror w9,w9,#25 + cbnz x4,.Loop + + add w5,w5,w22 // accumulate key block + add x6,x6,x22,lsr#32 + add w7,w7,w23 + add x8,x8,x23,lsr#32 + add w9,w9,w24 + add x10,x10,x24,lsr#32 + add w11,w11,w25 + add x12,x12,x25,lsr#32 + add w13,w13,w26 + add x14,x14,x26,lsr#32 + add w15,w15,w27 + add x16,x16,x27,lsr#32 + add w17,w17,w28 + add x19,x19,x28,lsr#32 + add w20,w20,w30 + add x21,x21,x30,lsr#32 + + b.lo .Ltail + + add x5,x5,x6,lsl#32 // pack + add x7,x7,x8,lsl#32 + ldp x6,x8,[x1,#0] // load input + add x9,x9,x10,lsl#32 + add x11,x11,x12,lsl#32 + ldp x10,x12,[x1,#16] + add x13,x13,x14,lsl#32 + add x15,x15,x16,lsl#32 + ldp x14,x16,[x1,#32] + add x17,x17,x19,lsl#32 + add x20,x20,x21,lsl#32 + ldp x19,x21,[x1,#48] + add x1,x1,#64 +#ifdef __ARMEB__ + rev x5,x5 + rev x7,x7 + rev x9,x9 + rev x11,x11 + rev x13,x13 + rev x15,x15 + rev x17,x17 + rev x20,x20 +#endif + eor x5,x5,x6 + eor x7,x7,x8 + eor x9,x9,x10 + eor x11,x11,x12 + eor x13,x13,x14 + eor x15,x15,x16 + eor x17,x17,x19 + eor x20,x20,x21 + + stp x5,x7,[x0,#0] // store output + add x28,x28,#1 // increment counter + stp x9,x11,[x0,#16] + stp x13,x15,[x0,#32] + stp x17,x20,[x0,#48] + add x0,x0,#64 + + b.hi .Loop_outer + + ldp x19,x20,[x29,#16] + add sp,sp,#64 + ldp x21,x22,[x29,#32] + ldp x23,x24,[x29,#48] + ldp x25,x26,[x29,#64] + ldp x27,x28,[x29,#80] + ldp x29,x30,[sp],#96 +.Labort: + ret + +.align 4 +.Ltail: + add x2,x2,#64 +.Less_than_64: + sub x0,x0,#1 + add x1,x1,x2 + add x0,x0,x2 + add x4,sp,x2 + neg x2,x2 + + add x5,x5,x6,lsl#32 // pack + add x7,x7,x8,lsl#32 + add x9,x9,x10,lsl#32 + add x11,x11,x12,lsl#32 + add x13,x13,x14,lsl#32 + add x15,x15,x16,lsl#32 + add x17,x17,x19,lsl#32 + add x20,x20,x21,lsl#32 +#ifdef __ARMEB__ + rev x5,x5 + rev x7,x7 + rev x9,x9 + rev x11,x11 + rev x13,x13 + rev x15,x15 + rev x17,x17 + rev x20,x20 +#endif + stp x5,x7,[sp,#0] + stp x9,x11,[sp,#16] + stp x13,x15,[sp,#32] + stp x17,x20,[sp,#48] + +.Loop_tail: + ldrb w10,[x1,x2] + ldrb w11,[x4,x2] + add x2,x2,#1 + eor w10,w10,w11 + strb w10,[x0,x2] + cbnz x2,.Loop_tail + + stp xzr,xzr,[sp,#0] + stp xzr,xzr,[sp,#16] + stp xzr,xzr,[sp,#32] + stp xzr,xzr,[sp,#48] + + ldp x19,x20,[x29,#16] + add sp,sp,#64 + ldp x21,x22,[x29,#32] + ldp x23,x24,[x29,#48] + ldp x25,x26,[x29,#64] + ldp x27,x28,[x29,#80] + ldp x29,x30,[sp],#96 + ret +.size ChaCha20_ctr32,.-ChaCha20_ctr32 + +.type ChaCha20_neon,%function +.align 5 +ChaCha20_neon: + stp x29,x30,[sp,#-96]! + add x29,sp,#0 + + adr x5,.Lsigma + stp x19,x20,[sp,#16] + stp x21,x22,[sp,#32] + stp x23,x24,[sp,#48] + stp x25,x26,[sp,#64] + stp x27,x28,[sp,#80] + cmp x2,#512 + b.hs .L512_or_more_neon + + sub sp,sp,#64 + + ldp x22,x23,[x5] // load sigma + ld1 {v24.4s},[x5],#16 + ldp x24,x25,[x3] // load key + ldp x26,x27,[x3,#16] + ld1 {v25.4s,v26.4s},[x3] + ldp x28,x30,[x4] // load counter + ld1 {v27.4s},[x4] + ld1 {v31.4s},[x5] +#ifdef __ARMEB__ + rev64 v24.4s,v24.4s + ror x24,x24,#32 + ror x25,x25,#32 + ror x26,x26,#32 + ror x27,x27,#32 + ror x28,x28,#32 + ror x30,x30,#32 +#endif + add v27.4s,v27.4s,v31.4s // += 1 + add v28.4s,v27.4s,v31.4s + add v29.4s,v28.4s,v31.4s + shl v31.4s,v31.4s,#2 // 1 -> 4 + +.Loop_outer_neon: + mov w5,w22 // unpack key block + lsr x6,x22,#32 + mov v0.16b,v24.16b + mov w7,w23 + lsr x8,x23,#32 + mov v4.16b,v24.16b + mov w9,w24 + lsr x10,x24,#32 + mov v16.16b,v24.16b + mov w11,w25 + mov v1.16b,v25.16b + lsr x12,x25,#32 + mov v5.16b,v25.16b + mov w13,w26 + mov v17.16b,v25.16b + lsr x14,x26,#32 + mov v3.16b,v27.16b + mov w15,w27 + mov v7.16b,v28.16b + lsr x16,x27,#32 + mov v19.16b,v29.16b + mov w17,w28 + mov v2.16b,v26.16b + lsr x19,x28,#32 + mov v6.16b,v26.16b + mov w20,w30 + mov v18.16b,v26.16b + lsr x21,x30,#32 + + mov x4,#10 + subs x2,x2,#256 +.Loop_neon: + sub x4,x4,#1 + add v0.4s,v0.4s,v1.4s + add w5,w5,w9 + add v4.4s,v4.4s,v5.4s + add w6,w6,w10 + add v16.4s,v16.4s,v17.4s + add w7,w7,w11 + eor v3.16b,v3.16b,v0.16b + add w8,w8,w12 + eor v7.16b,v7.16b,v4.16b + eor w17,w17,w5 + eor v19.16b,v19.16b,v16.16b + eor w19,w19,w6 + rev32 v3.8h,v3.8h + eor w20,w20,w7 + rev32 v7.8h,v7.8h + eor w21,w21,w8 + rev32 v19.8h,v19.8h + ror w17,w17,#16 + add v2.4s,v2.4s,v3.4s + ror w19,w19,#16 + add v6.4s,v6.4s,v7.4s + ror w20,w20,#16 + add v18.4s,v18.4s,v19.4s + ror w21,w21,#16 + eor v20.16b,v1.16b,v2.16b + add w13,w13,w17 + eor v21.16b,v5.16b,v6.16b + add w14,w14,w19 + eor v22.16b,v17.16b,v18.16b + add w15,w15,w20 + ushr v1.4s,v20.4s,#20 + add w16,w16,w21 + ushr v5.4s,v21.4s,#20 + eor w9,w9,w13 + ushr v17.4s,v22.4s,#20 + eor w10,w10,w14 + sli v1.4s,v20.4s,#12 + eor w11,w11,w15 + sli v5.4s,v21.4s,#12 + eor w12,w12,w16 + sli v17.4s,v22.4s,#12 + ror w9,w9,#20 + add v0.4s,v0.4s,v1.4s + ror w10,w10,#20 + add v4.4s,v4.4s,v5.4s + ror w11,w11,#20 + add v16.4s,v16.4s,v17.4s + ror w12,w12,#20 + eor v20.16b,v3.16b,v0.16b + add w5,w5,w9 + eor v21.16b,v7.16b,v4.16b + add w6,w6,w10 + eor v22.16b,v19.16b,v16.16b + add w7,w7,w11 + ushr v3.4s,v20.4s,#24 + add w8,w8,w12 + ushr v7.4s,v21.4s,#24 + eor w17,w17,w5 + ushr v19.4s,v22.4s,#24 + eor w19,w19,w6 + sli v3.4s,v20.4s,#8 + eor w20,w20,w7 + sli v7.4s,v21.4s,#8 + eor w21,w21,w8 + sli v19.4s,v22.4s,#8 + ror w17,w17,#24 + add v2.4s,v2.4s,v3.4s + ror w19,w19,#24 + add v6.4s,v6.4s,v7.4s + ror w20,w20,#24 + add v18.4s,v18.4s,v19.4s + ror w21,w21,#24 + eor v20.16b,v1.16b,v2.16b + add w13,w13,w17 + eor v21.16b,v5.16b,v6.16b + add w14,w14,w19 + eor v22.16b,v17.16b,v18.16b + add w15,w15,w20 + ushr v1.4s,v20.4s,#25 + add w16,w16,w21 + ushr v5.4s,v21.4s,#25 + eor w9,w9,w13 + ushr v17.4s,v22.4s,#25 + eor w10,w10,w14 + sli v1.4s,v20.4s,#7 + eor w11,w11,w15 + sli v5.4s,v21.4s,#7 + eor w12,w12,w16 + sli v17.4s,v22.4s,#7 + ror w9,w9,#25 + ext v2.16b,v2.16b,v2.16b,#8 + ror w10,w10,#25 + ext v6.16b,v6.16b,v6.16b,#8 + ror w11,w11,#25 + ext v18.16b,v18.16b,v18.16b,#8 + ror w12,w12,#25 + ext v3.16b,v3.16b,v3.16b,#12 + ext v7.16b,v7.16b,v7.16b,#12 + ext v19.16b,v19.16b,v19.16b,#12 + ext v1.16b,v1.16b,v1.16b,#4 + ext v5.16b,v5.16b,v5.16b,#4 + ext v17.16b,v17.16b,v17.16b,#4 + add v0.4s,v0.4s,v1.4s + add w5,w5,w10 + add v4.4s,v4.4s,v5.4s + add w6,w6,w11 + add v16.4s,v16.4s,v17.4s + add w7,w7,w12 + eor v3.16b,v3.16b,v0.16b + add w8,w8,w9 + eor v7.16b,v7.16b,v4.16b + eor w21,w21,w5 + eor v19.16b,v19.16b,v16.16b + eor w17,w17,w6 + rev32 v3.8h,v3.8h + eor w19,w19,w7 + rev32 v7.8h,v7.8h + eor w20,w20,w8 + rev32 v19.8h,v19.8h + ror w21,w21,#16 + add v2.4s,v2.4s,v3.4s + ror w17,w17,#16 + add v6.4s,v6.4s,v7.4s + ror w19,w19,#16 + add v18.4s,v18.4s,v19.4s + ror w20,w20,#16 + eor v20.16b,v1.16b,v2.16b + add w15,w15,w21 + eor v21.16b,v5.16b,v6.16b + add w16,w16,w17 + eor v22.16b,v17.16b,v18.16b + add w13,w13,w19 + ushr v1.4s,v20.4s,#20 + add w14,w14,w20 + ushr v5.4s,v21.4s,#20 + eor w10,w10,w15 + ushr v17.4s,v22.4s,#20 + eor w11,w11,w16 + sli v1.4s,v20.4s,#12 + eor w12,w12,w13 + sli v5.4s,v21.4s,#12 + eor w9,w9,w14 + sli v17.4s,v22.4s,#12 + ror w10,w10,#20 + add v0.4s,v0.4s,v1.4s + ror w11,w11,#20 + add v4.4s,v4.4s,v5.4s + ror w12,w12,#20 + add v16.4s,v16.4s,v17.4s + ror w9,w9,#20 + eor v20.16b,v3.16b,v0.16b + add w5,w5,w10 + eor v21.16b,v7.16b,v4.16b + add w6,w6,w11 + eor v22.16b,v19.16b,v16.16b + add w7,w7,w12 + ushr v3.4s,v20.4s,#24 + add w8,w8,w9 + ushr v7.4s,v21.4s,#24 + eor w21,w21,w5 + ushr v19.4s,v22.4s,#24 + eor w17,w17,w6 + sli v3.4s,v20.4s,#8 + eor w19,w19,w7 + sli v7.4s,v21.4s,#8 + eor w20,w20,w8 + sli v19.4s,v22.4s,#8 + ror w21,w21,#24 + add v2.4s,v2.4s,v3.4s + ror w17,w17,#24 + add v6.4s,v6.4s,v7.4s + ror w19,w19,#24 + add v18.4s,v18.4s,v19.4s + ror w20,w20,#24 + eor v20.16b,v1.16b,v2.16b + add w15,w15,w21 + eor v21.16b,v5.16b,v6.16b + add w16,w16,w17 + eor v22.16b,v17.16b,v18.16b + add w13,w13,w19 + ushr v1.4s,v20.4s,#25 + add w14,w14,w20 + ushr v5.4s,v21.4s,#25 + eor w10,w10,w15 + ushr v17.4s,v22.4s,#25 + eor w11,w11,w16 + sli v1.4s,v20.4s,#7 + eor w12,w12,w13 + sli v5.4s,v21.4s,#7 + eor w9,w9,w14 + sli v17.4s,v22.4s,#7 + ror w10,w10,#25 + ext v2.16b,v2.16b,v2.16b,#8 + ror w11,w11,#25 + ext v6.16b,v6.16b,v6.16b,#8 + ror w12,w12,#25 + ext v18.16b,v18.16b,v18.16b,#8 + ror w9,w9,#25 + ext v3.16b,v3.16b,v3.16b,#4 + ext v7.16b,v7.16b,v7.16b,#4 + ext v19.16b,v19.16b,v19.16b,#4 + ext v1.16b,v1.16b,v1.16b,#12 + ext v5.16b,v5.16b,v5.16b,#12 + ext v17.16b,v17.16b,v17.16b,#12 + cbnz x4,.Loop_neon + + add w5,w5,w22 // accumulate key block + add v0.4s,v0.4s,v24.4s + add x6,x6,x22,lsr#32 + add v4.4s,v4.4s,v24.4s + add w7,w7,w23 + add v16.4s,v16.4s,v24.4s + add x8,x8,x23,lsr#32 + add v2.4s,v2.4s,v26.4s + add w9,w9,w24 + add v6.4s,v6.4s,v26.4s + add x10,x10,x24,lsr#32 + add v18.4s,v18.4s,v26.4s + add w11,w11,w25 + add v3.4s,v3.4s,v27.4s + add x12,x12,x25,lsr#32 + add w13,w13,w26 + add v7.4s,v7.4s,v28.4s + add x14,x14,x26,lsr#32 + add w15,w15,w27 + add v19.4s,v19.4s,v29.4s + add x16,x16,x27,lsr#32 + add w17,w17,w28 + add v1.4s,v1.4s,v25.4s + add x19,x19,x28,lsr#32 + add w20,w20,w30 + add v5.4s,v5.4s,v25.4s + add x21,x21,x30,lsr#32 + add v17.4s,v17.4s,v25.4s + + b.lo .Ltail_neon + + add x5,x5,x6,lsl#32 // pack + add x7,x7,x8,lsl#32 + ldp x6,x8,[x1,#0] // load input + add x9,x9,x10,lsl#32 + add x11,x11,x12,lsl#32 + ldp x10,x12,[x1,#16] + add x13,x13,x14,lsl#32 + add x15,x15,x16,lsl#32 + ldp x14,x16,[x1,#32] + add x17,x17,x19,lsl#32 + add x20,x20,x21,lsl#32 + ldp x19,x21,[x1,#48] + add x1,x1,#64 +#ifdef __ARMEB__ + rev x5,x5 + rev x7,x7 + rev x9,x9 + rev x11,x11 + rev x13,x13 + rev x15,x15 + rev x17,x17 + rev x20,x20 +#endif + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64 + eor x5,x5,x6 + eor x7,x7,x8 + eor x9,x9,x10 + eor x11,x11,x12 + eor x13,x13,x14 + eor v0.16b,v0.16b,v20.16b + eor x15,x15,x16 + eor v1.16b,v1.16b,v21.16b + eor x17,x17,x19 + eor v2.16b,v2.16b,v22.16b + eor x20,x20,x21 + eor v3.16b,v3.16b,v23.16b + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64 + + stp x5,x7,[x0,#0] // store output + add x28,x28,#4 // increment counter + stp x9,x11,[x0,#16] + add v27.4s,v27.4s,v31.4s // += 4 + stp x13,x15,[x0,#32] + add v28.4s,v28.4s,v31.4s + stp x17,x20,[x0,#48] + add v29.4s,v29.4s,v31.4s + add x0,x0,#64 + + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64 + ld1 {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64 + + eor v4.16b,v4.16b,v20.16b + eor v5.16b,v5.16b,v21.16b + eor v6.16b,v6.16b,v22.16b + eor v7.16b,v7.16b,v23.16b + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64 + + eor v16.16b,v16.16b,v0.16b + eor v17.16b,v17.16b,v1.16b + eor v18.16b,v18.16b,v2.16b + eor v19.16b,v19.16b,v3.16b + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64 + + b.hi .Loop_outer_neon + + ldp x19,x20,[x29,#16] + add sp,sp,#64 + ldp x21,x22,[x29,#32] + ldp x23,x24,[x29,#48] + ldp x25,x26,[x29,#64] + ldp x27,x28,[x29,#80] + ldp x29,x30,[sp],#96 + ret + +.Ltail_neon: + add x2,x2,#256 + cmp x2,#64 + b.lo .Less_than_64 + + add x5,x5,x6,lsl#32 // pack + add x7,x7,x8,lsl#32 + ldp x6,x8,[x1,#0] // load input + add x9,x9,x10,lsl#32 + add x11,x11,x12,lsl#32 + ldp x10,x12,[x1,#16] + add x13,x13,x14,lsl#32 + add x15,x15,x16,lsl#32 + ldp x14,x16,[x1,#32] + add x17,x17,x19,lsl#32 + add x20,x20,x21,lsl#32 + ldp x19,x21,[x1,#48] + add x1,x1,#64 +#ifdef __ARMEB__ + rev x5,x5 + rev x7,x7 + rev x9,x9 + rev x11,x11 + rev x13,x13 + rev x15,x15 + rev x17,x17 + rev x20,x20 +#endif + eor x5,x5,x6 + eor x7,x7,x8 + eor x9,x9,x10 + eor x11,x11,x12 + eor x13,x13,x14 + eor x15,x15,x16 + eor x17,x17,x19 + eor x20,x20,x21 + + stp x5,x7,[x0,#0] // store output + add x28,x28,#4 // increment counter + stp x9,x11,[x0,#16] + stp x13,x15,[x0,#32] + stp x17,x20,[x0,#48] + add x0,x0,#64 + b.eq .Ldone_neon + sub x2,x2,#64 + cmp x2,#64 + b.lo .Less_than_128 + + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64 + eor v0.16b,v0.16b,v20.16b + eor v1.16b,v1.16b,v21.16b + eor v2.16b,v2.16b,v22.16b + eor v3.16b,v3.16b,v23.16b + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64 + b.eq .Ldone_neon + sub x2,x2,#64 + cmp x2,#64 + b.lo .Less_than_192 + + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64 + eor v4.16b,v4.16b,v20.16b + eor v5.16b,v5.16b,v21.16b + eor v6.16b,v6.16b,v22.16b + eor v7.16b,v7.16b,v23.16b + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64 + b.eq .Ldone_neon + sub x2,x2,#64 + + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[sp] + b .Last_neon + +.Less_than_128: + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[sp] + b .Last_neon +.Less_than_192: + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[sp] + b .Last_neon + +.align 4 +.Last_neon: + sub x0,x0,#1 + add x1,x1,x2 + add x0,x0,x2 + add x4,sp,x2 + neg x2,x2 + +.Loop_tail_neon: + ldrb w10,[x1,x2] + ldrb w11,[x4,x2] + add x2,x2,#1 + eor w10,w10,w11 + strb w10,[x0,x2] + cbnz x2,.Loop_tail_neon + + stp xzr,xzr,[sp,#0] + stp xzr,xzr,[sp,#16] + stp xzr,xzr,[sp,#32] + stp xzr,xzr,[sp,#48] + +.Ldone_neon: + ldp x19,x20,[x29,#16] + add sp,sp,#64 + ldp x21,x22,[x29,#32] + ldp x23,x24,[x29,#48] + ldp x25,x26,[x29,#64] + ldp x27,x28,[x29,#80] + ldp x29,x30,[sp],#96 + ret +.size ChaCha20_neon,.-ChaCha20_neon +.type ChaCha20_512_neon,%function +.align 5 +ChaCha20_512_neon: + stp x29,x30,[sp,#-96]! + add x29,sp,#0 + + adr x5,.Lsigma + stp x19,x20,[sp,#16] + stp x21,x22,[sp,#32] + stp x23,x24,[sp,#48] + stp x25,x26,[sp,#64] + stp x27,x28,[sp,#80] + +.L512_or_more_neon: + sub sp,sp,#128+64 + + ldp x22,x23,[x5] // load sigma + ld1 {v24.4s},[x5],#16 + ldp x24,x25,[x3] // load key + ldp x26,x27,[x3,#16] + ld1 {v25.4s,v26.4s},[x3] + ldp x28,x30,[x4] // load counter + ld1 {v27.4s},[x4] + ld1 {v31.4s},[x5] +#ifdef __ARMEB__ + rev64 v24.4s,v24.4s + ror x24,x24,#32 + ror x25,x25,#32 + ror x26,x26,#32 + ror x27,x27,#32 + ror x28,x28,#32 + ror x30,x30,#32 +#endif + add v27.4s,v27.4s,v31.4s // += 1 + stp q24,q25,[sp,#0] // off-load key block, invariant part + add v27.4s,v27.4s,v31.4s // not typo + str q26,[sp,#32] + add v28.4s,v27.4s,v31.4s + add v29.4s,v28.4s,v31.4s + add v30.4s,v29.4s,v31.4s + shl v31.4s,v31.4s,#2 // 1 -> 4 + + stp d8,d9,[sp,#128+0] // meet ABI requirements + stp d10,d11,[sp,#128+16] + stp d12,d13,[sp,#128+32] + stp d14,d15,[sp,#128+48] + + sub x2,x2,#512 // not typo + +.Loop_outer_512_neon: + mov v0.16b,v24.16b + mov v4.16b,v24.16b + mov v8.16b,v24.16b + mov v12.16b,v24.16b + mov v16.16b,v24.16b + mov v20.16b,v24.16b + mov v1.16b,v25.16b + mov w5,w22 // unpack key block + mov v5.16b,v25.16b + lsr x6,x22,#32 + mov v9.16b,v25.16b + mov w7,w23 + mov v13.16b,v25.16b + lsr x8,x23,#32 + mov v17.16b,v25.16b + mov w9,w24 + mov v21.16b,v25.16b + lsr x10,x24,#32 + mov v3.16b,v27.16b + mov w11,w25 + mov v7.16b,v28.16b + lsr x12,x25,#32 + mov v11.16b,v29.16b + mov w13,w26 + mov v15.16b,v30.16b + lsr x14,x26,#32 + mov v2.16b,v26.16b + mov w15,w27 + mov v6.16b,v26.16b + lsr x16,x27,#32 + add v19.4s,v3.4s,v31.4s // +4 + mov w17,w28 + add v23.4s,v7.4s,v31.4s // +4 + lsr x19,x28,#32 + mov v10.16b,v26.16b + mov w20,w30 + mov v14.16b,v26.16b + lsr x21,x30,#32 + mov v18.16b,v26.16b + stp q27,q28,[sp,#48] // off-load key block, variable part + mov v22.16b,v26.16b + str q29,[sp,#80] + + mov x4,#5 + subs x2,x2,#512 +.Loop_upper_neon: + sub x4,x4,#1 + add v0.4s,v0.4s,v1.4s + add w5,w5,w9 + add v4.4s,v4.4s,v5.4s + add w6,w6,w10 + add v8.4s,v8.4s,v9.4s + add w7,w7,w11 + add v12.4s,v12.4s,v13.4s + add w8,w8,w12 + add v16.4s,v16.4s,v17.4s + eor w17,w17,w5 + add v20.4s,v20.4s,v21.4s + eor w19,w19,w6 + eor v3.16b,v3.16b,v0.16b + eor w20,w20,w7 + eor v7.16b,v7.16b,v4.16b + eor w21,w21,w8 + eor v11.16b,v11.16b,v8.16b + ror w17,w17,#16 + eor v15.16b,v15.16b,v12.16b + ror w19,w19,#16 + eor v19.16b,v19.16b,v16.16b + ror w20,w20,#16 + eor v23.16b,v23.16b,v20.16b + ror w21,w21,#16 + rev32 v3.8h,v3.8h + add w13,w13,w17 + rev32 v7.8h,v7.8h + add w14,w14,w19 + rev32 v11.8h,v11.8h + add w15,w15,w20 + rev32 v15.8h,v15.8h + add w16,w16,w21 + rev32 v19.8h,v19.8h + eor w9,w9,w13 + rev32 v23.8h,v23.8h + eor w10,w10,w14 + add v2.4s,v2.4s,v3.4s + eor w11,w11,w15 + add v6.4s,v6.4s,v7.4s + eor w12,w12,w16 + add v10.4s,v10.4s,v11.4s + ror w9,w9,#20 + add v14.4s,v14.4s,v15.4s + ror w10,w10,#20 + add v18.4s,v18.4s,v19.4s + ror w11,w11,#20 + add v22.4s,v22.4s,v23.4s + ror w12,w12,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w9 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w10 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w11 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w12 + eor v28.16b,v17.16b,v18.16b + eor w17,w17,w5 + eor v29.16b,v21.16b,v22.16b + eor w19,w19,w6 + ushr v1.4s,v24.4s,#20 + eor w20,w20,w7 + ushr v5.4s,v25.4s,#20 + eor w21,w21,w8 + ushr v9.4s,v26.4s,#20 + ror w17,w17,#24 + ushr v13.4s,v27.4s,#20 + ror w19,w19,#24 + ushr v17.4s,v28.4s,#20 + ror w20,w20,#24 + ushr v21.4s,v29.4s,#20 + ror w21,w21,#24 + sli v1.4s,v24.4s,#12 + add w13,w13,w17 + sli v5.4s,v25.4s,#12 + add w14,w14,w19 + sli v9.4s,v26.4s,#12 + add w15,w15,w20 + sli v13.4s,v27.4s,#12 + add w16,w16,w21 + sli v17.4s,v28.4s,#12 + eor w9,w9,w13 + sli v21.4s,v29.4s,#12 + eor w10,w10,w14 + add v0.4s,v0.4s,v1.4s + eor w11,w11,w15 + add v4.4s,v4.4s,v5.4s + eor w12,w12,w16 + add v8.4s,v8.4s,v9.4s + ror w9,w9,#25 + add v12.4s,v12.4s,v13.4s + ror w10,w10,#25 + add v16.4s,v16.4s,v17.4s + ror w11,w11,#25 + add v20.4s,v20.4s,v21.4s + ror w12,w12,#25 + eor v24.16b,v3.16b,v0.16b + add w5,w5,w10 + eor v25.16b,v7.16b,v4.16b + add w6,w6,w11 + eor v26.16b,v11.16b,v8.16b + add w7,w7,w12 + eor v27.16b,v15.16b,v12.16b + add w8,w8,w9 + eor v28.16b,v19.16b,v16.16b + eor w21,w21,w5 + eor v29.16b,v23.16b,v20.16b + eor w17,w17,w6 + ushr v3.4s,v24.4s,#24 + eor w19,w19,w7 + ushr v7.4s,v25.4s,#24 + eor w20,w20,w8 + ushr v11.4s,v26.4s,#24 + ror w21,w21,#16 + ushr v15.4s,v27.4s,#24 + ror w17,w17,#16 + ushr v19.4s,v28.4s,#24 + ror w19,w19,#16 + ushr v23.4s,v29.4s,#24 + ror w20,w20,#16 + sli v3.4s,v24.4s,#8 + add w15,w15,w21 + sli v7.4s,v25.4s,#8 + add w16,w16,w17 + sli v11.4s,v26.4s,#8 + add w13,w13,w19 + sli v15.4s,v27.4s,#8 + add w14,w14,w20 + sli v19.4s,v28.4s,#8 + eor w10,w10,w15 + sli v23.4s,v29.4s,#8 + eor w11,w11,w16 + add v2.4s,v2.4s,v3.4s + eor w12,w12,w13 + add v6.4s,v6.4s,v7.4s + eor w9,w9,w14 + add v10.4s,v10.4s,v11.4s + ror w10,w10,#20 + add v14.4s,v14.4s,v15.4s + ror w11,w11,#20 + add v18.4s,v18.4s,v19.4s + ror w12,w12,#20 + add v22.4s,v22.4s,v23.4s + ror w9,w9,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w10 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w11 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w12 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w9 + eor v28.16b,v17.16b,v18.16b + eor w21,w21,w5 + eor v29.16b,v21.16b,v22.16b + eor w17,w17,w6 + ushr v1.4s,v24.4s,#25 + eor w19,w19,w7 + ushr v5.4s,v25.4s,#25 + eor w20,w20,w8 + ushr v9.4s,v26.4s,#25 + ror w21,w21,#24 + ushr v13.4s,v27.4s,#25 + ror w17,w17,#24 + ushr v17.4s,v28.4s,#25 + ror w19,w19,#24 + ushr v21.4s,v29.4s,#25 + ror w20,w20,#24 + sli v1.4s,v24.4s,#7 + add w15,w15,w21 + sli v5.4s,v25.4s,#7 + add w16,w16,w17 + sli v9.4s,v26.4s,#7 + add w13,w13,w19 + sli v13.4s,v27.4s,#7 + add w14,w14,w20 + sli v17.4s,v28.4s,#7 + eor w10,w10,w15 + sli v21.4s,v29.4s,#7 + eor w11,w11,w16 + ext v2.16b,v2.16b,v2.16b,#8 + eor w12,w12,w13 + ext v6.16b,v6.16b,v6.16b,#8 + eor w9,w9,w14 + ext v10.16b,v10.16b,v10.16b,#8 + ror w10,w10,#25 + ext v14.16b,v14.16b,v14.16b,#8 + ror w11,w11,#25 + ext v18.16b,v18.16b,v18.16b,#8 + ror w12,w12,#25 + ext v22.16b,v22.16b,v22.16b,#8 + ror w9,w9,#25 + ext v3.16b,v3.16b,v3.16b,#12 + ext v7.16b,v7.16b,v7.16b,#12 + ext v11.16b,v11.16b,v11.16b,#12 + ext v15.16b,v15.16b,v15.16b,#12 + ext v19.16b,v19.16b,v19.16b,#12 + ext v23.16b,v23.16b,v23.16b,#12 + ext v1.16b,v1.16b,v1.16b,#4 + ext v5.16b,v5.16b,v5.16b,#4 + ext v9.16b,v9.16b,v9.16b,#4 + ext v13.16b,v13.16b,v13.16b,#4 + ext v17.16b,v17.16b,v17.16b,#4 + ext v21.16b,v21.16b,v21.16b,#4 + add v0.4s,v0.4s,v1.4s + add w5,w5,w9 + add v4.4s,v4.4s,v5.4s + add w6,w6,w10 + add v8.4s,v8.4s,v9.4s + add w7,w7,w11 + add v12.4s,v12.4s,v13.4s + add w8,w8,w12 + add v16.4s,v16.4s,v17.4s + eor w17,w17,w5 + add v20.4s,v20.4s,v21.4s + eor w19,w19,w6 + eor v3.16b,v3.16b,v0.16b + eor w20,w20,w7 + eor v7.16b,v7.16b,v4.16b + eor w21,w21,w8 + eor v11.16b,v11.16b,v8.16b + ror w17,w17,#16 + eor v15.16b,v15.16b,v12.16b + ror w19,w19,#16 + eor v19.16b,v19.16b,v16.16b + ror w20,w20,#16 + eor v23.16b,v23.16b,v20.16b + ror w21,w21,#16 + rev32 v3.8h,v3.8h + add w13,w13,w17 + rev32 v7.8h,v7.8h + add w14,w14,w19 + rev32 v11.8h,v11.8h + add w15,w15,w20 + rev32 v15.8h,v15.8h + add w16,w16,w21 + rev32 v19.8h,v19.8h + eor w9,w9,w13 + rev32 v23.8h,v23.8h + eor w10,w10,w14 + add v2.4s,v2.4s,v3.4s + eor w11,w11,w15 + add v6.4s,v6.4s,v7.4s + eor w12,w12,w16 + add v10.4s,v10.4s,v11.4s + ror w9,w9,#20 + add v14.4s,v14.4s,v15.4s + ror w10,w10,#20 + add v18.4s,v18.4s,v19.4s + ror w11,w11,#20 + add v22.4s,v22.4s,v23.4s + ror w12,w12,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w9 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w10 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w11 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w12 + eor v28.16b,v17.16b,v18.16b + eor w17,w17,w5 + eor v29.16b,v21.16b,v22.16b + eor w19,w19,w6 + ushr v1.4s,v24.4s,#20 + eor w20,w20,w7 + ushr v5.4s,v25.4s,#20 + eor w21,w21,w8 + ushr v9.4s,v26.4s,#20 + ror w17,w17,#24 + ushr v13.4s,v27.4s,#20 + ror w19,w19,#24 + ushr v17.4s,v28.4s,#20 + ror w20,w20,#24 + ushr v21.4s,v29.4s,#20 + ror w21,w21,#24 + sli v1.4s,v24.4s,#12 + add w13,w13,w17 + sli v5.4s,v25.4s,#12 + add w14,w14,w19 + sli v9.4s,v26.4s,#12 + add w15,w15,w20 + sli v13.4s,v27.4s,#12 + add w16,w16,w21 + sli v17.4s,v28.4s,#12 + eor w9,w9,w13 + sli v21.4s,v29.4s,#12 + eor w10,w10,w14 + add v0.4s,v0.4s,v1.4s + eor w11,w11,w15 + add v4.4s,v4.4s,v5.4s + eor w12,w12,w16 + add v8.4s,v8.4s,v9.4s + ror w9,w9,#25 + add v12.4s,v12.4s,v13.4s + ror w10,w10,#25 + add v16.4s,v16.4s,v17.4s + ror w11,w11,#25 + add v20.4s,v20.4s,v21.4s + ror w12,w12,#25 + eor v24.16b,v3.16b,v0.16b + add w5,w5,w10 + eor v25.16b,v7.16b,v4.16b + add w6,w6,w11 + eor v26.16b,v11.16b,v8.16b + add w7,w7,w12 + eor v27.16b,v15.16b,v12.16b + add w8,w8,w9 + eor v28.16b,v19.16b,v16.16b + eor w21,w21,w5 + eor v29.16b,v23.16b,v20.16b + eor w17,w17,w6 + ushr v3.4s,v24.4s,#24 + eor w19,w19,w7 + ushr v7.4s,v25.4s,#24 + eor w20,w20,w8 + ushr v11.4s,v26.4s,#24 + ror w21,w21,#16 + ushr v15.4s,v27.4s,#24 + ror w17,w17,#16 + ushr v19.4s,v28.4s,#24 + ror w19,w19,#16 + ushr v23.4s,v29.4s,#24 + ror w20,w20,#16 + sli v3.4s,v24.4s,#8 + add w15,w15,w21 + sli v7.4s,v25.4s,#8 + add w16,w16,w17 + sli v11.4s,v26.4s,#8 + add w13,w13,w19 + sli v15.4s,v27.4s,#8 + add w14,w14,w20 + sli v19.4s,v28.4s,#8 + eor w10,w10,w15 + sli v23.4s,v29.4s,#8 + eor w11,w11,w16 + add v2.4s,v2.4s,v3.4s + eor w12,w12,w13 + add v6.4s,v6.4s,v7.4s + eor w9,w9,w14 + add v10.4s,v10.4s,v11.4s + ror w10,w10,#20 + add v14.4s,v14.4s,v15.4s + ror w11,w11,#20 + add v18.4s,v18.4s,v19.4s + ror w12,w12,#20 + add v22.4s,v22.4s,v23.4s + ror w9,w9,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w10 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w11 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w12 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w9 + eor v28.16b,v17.16b,v18.16b + eor w21,w21,w5 + eor v29.16b,v21.16b,v22.16b + eor w17,w17,w6 + ushr v1.4s,v24.4s,#25 + eor w19,w19,w7 + ushr v5.4s,v25.4s,#25 + eor w20,w20,w8 + ushr v9.4s,v26.4s,#25 + ror w21,w21,#24 + ushr v13.4s,v27.4s,#25 + ror w17,w17,#24 + ushr v17.4s,v28.4s,#25 + ror w19,w19,#24 + ushr v21.4s,v29.4s,#25 + ror w20,w20,#24 + sli v1.4s,v24.4s,#7 + add w15,w15,w21 + sli v5.4s,v25.4s,#7 + add w16,w16,w17 + sli v9.4s,v26.4s,#7 + add w13,w13,w19 + sli v13.4s,v27.4s,#7 + add w14,w14,w20 + sli v17.4s,v28.4s,#7 + eor w10,w10,w15 + sli v21.4s,v29.4s,#7 + eor w11,w11,w16 + ext v2.16b,v2.16b,v2.16b,#8 + eor w12,w12,w13 + ext v6.16b,v6.16b,v6.16b,#8 + eor w9,w9,w14 + ext v10.16b,v10.16b,v10.16b,#8 + ror w10,w10,#25 + ext v14.16b,v14.16b,v14.16b,#8 + ror w11,w11,#25 + ext v18.16b,v18.16b,v18.16b,#8 + ror w12,w12,#25 + ext v22.16b,v22.16b,v22.16b,#8 + ror w9,w9,#25 + ext v3.16b,v3.16b,v3.16b,#4 + ext v7.16b,v7.16b,v7.16b,#4 + ext v11.16b,v11.16b,v11.16b,#4 + ext v15.16b,v15.16b,v15.16b,#4 + ext v19.16b,v19.16b,v19.16b,#4 + ext v23.16b,v23.16b,v23.16b,#4 + ext v1.16b,v1.16b,v1.16b,#12 + ext v5.16b,v5.16b,v5.16b,#12 + ext v9.16b,v9.16b,v9.16b,#12 + ext v13.16b,v13.16b,v13.16b,#12 + ext v17.16b,v17.16b,v17.16b,#12 + ext v21.16b,v21.16b,v21.16b,#12 + cbnz x4,.Loop_upper_neon + + add w5,w5,w22 // accumulate key block + add x6,x6,x22,lsr#32 + add w7,w7,w23 + add x8,x8,x23,lsr#32 + add w9,w9,w24 + add x10,x10,x24,lsr#32 + add w11,w11,w25 + add x12,x12,x25,lsr#32 + add w13,w13,w26 + add x14,x14,x26,lsr#32 + add w15,w15,w27 + add x16,x16,x27,lsr#32 + add w17,w17,w28 + add x19,x19,x28,lsr#32 + add w20,w20,w30 + add x21,x21,x30,lsr#32 + + add x5,x5,x6,lsl#32 // pack + add x7,x7,x8,lsl#32 + ldp x6,x8,[x1,#0] // load input + add x9,x9,x10,lsl#32 + add x11,x11,x12,lsl#32 + ldp x10,x12,[x1,#16] + add x13,x13,x14,lsl#32 + add x15,x15,x16,lsl#32 + ldp x14,x16,[x1,#32] + add x17,x17,x19,lsl#32 + add x20,x20,x21,lsl#32 + ldp x19,x21,[x1,#48] + add x1,x1,#64 +#ifdef __ARMEB__ + rev x5,x5 + rev x7,x7 + rev x9,x9 + rev x11,x11 + rev x13,x13 + rev x15,x15 + rev x17,x17 + rev x20,x20 +#endif + eor x5,x5,x6 + eor x7,x7,x8 + eor x9,x9,x10 + eor x11,x11,x12 + eor x13,x13,x14 + eor x15,x15,x16 + eor x17,x17,x19 + eor x20,x20,x21 + + stp x5,x7,[x0,#0] // store output + add x28,x28,#1 // increment counter + mov w5,w22 // unpack key block + lsr x6,x22,#32 + stp x9,x11,[x0,#16] + mov w7,w23 + lsr x8,x23,#32 + stp x13,x15,[x0,#32] + mov w9,w24 + lsr x10,x24,#32 + stp x17,x20,[x0,#48] + add x0,x0,#64 + mov w11,w25 + lsr x12,x25,#32 + mov w13,w26 + lsr x14,x26,#32 + mov w15,w27 + lsr x16,x27,#32 + mov w17,w28 + lsr x19,x28,#32 + mov w20,w30 + lsr x21,x30,#32 + + mov x4,#5 +.Loop_lower_neon: + sub x4,x4,#1 + add v0.4s,v0.4s,v1.4s + add w5,w5,w9 + add v4.4s,v4.4s,v5.4s + add w6,w6,w10 + add v8.4s,v8.4s,v9.4s + add w7,w7,w11 + add v12.4s,v12.4s,v13.4s + add w8,w8,w12 + add v16.4s,v16.4s,v17.4s + eor w17,w17,w5 + add v20.4s,v20.4s,v21.4s + eor w19,w19,w6 + eor v3.16b,v3.16b,v0.16b + eor w20,w20,w7 + eor v7.16b,v7.16b,v4.16b + eor w21,w21,w8 + eor v11.16b,v11.16b,v8.16b + ror w17,w17,#16 + eor v15.16b,v15.16b,v12.16b + ror w19,w19,#16 + eor v19.16b,v19.16b,v16.16b + ror w20,w20,#16 + eor v23.16b,v23.16b,v20.16b + ror w21,w21,#16 + rev32 v3.8h,v3.8h + add w13,w13,w17 + rev32 v7.8h,v7.8h + add w14,w14,w19 + rev32 v11.8h,v11.8h + add w15,w15,w20 + rev32 v15.8h,v15.8h + add w16,w16,w21 + rev32 v19.8h,v19.8h + eor w9,w9,w13 + rev32 v23.8h,v23.8h + eor w10,w10,w14 + add v2.4s,v2.4s,v3.4s + eor w11,w11,w15 + add v6.4s,v6.4s,v7.4s + eor w12,w12,w16 + add v10.4s,v10.4s,v11.4s + ror w9,w9,#20 + add v14.4s,v14.4s,v15.4s + ror w10,w10,#20 + add v18.4s,v18.4s,v19.4s + ror w11,w11,#20 + add v22.4s,v22.4s,v23.4s + ror w12,w12,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w9 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w10 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w11 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w12 + eor v28.16b,v17.16b,v18.16b + eor w17,w17,w5 + eor v29.16b,v21.16b,v22.16b + eor w19,w19,w6 + ushr v1.4s,v24.4s,#20 + eor w20,w20,w7 + ushr v5.4s,v25.4s,#20 + eor w21,w21,w8 + ushr v9.4s,v26.4s,#20 + ror w17,w17,#24 + ushr v13.4s,v27.4s,#20 + ror w19,w19,#24 + ushr v17.4s,v28.4s,#20 + ror w20,w20,#24 + ushr v21.4s,v29.4s,#20 + ror w21,w21,#24 + sli v1.4s,v24.4s,#12 + add w13,w13,w17 + sli v5.4s,v25.4s,#12 + add w14,w14,w19 + sli v9.4s,v26.4s,#12 + add w15,w15,w20 + sli v13.4s,v27.4s,#12 + add w16,w16,w21 + sli v17.4s,v28.4s,#12 + eor w9,w9,w13 + sli v21.4s,v29.4s,#12 + eor w10,w10,w14 + add v0.4s,v0.4s,v1.4s + eor w11,w11,w15 + add v4.4s,v4.4s,v5.4s + eor w12,w12,w16 + add v8.4s,v8.4s,v9.4s + ror w9,w9,#25 + add v12.4s,v12.4s,v13.4s + ror w10,w10,#25 + add v16.4s,v16.4s,v17.4s + ror w11,w11,#25 + add v20.4s,v20.4s,v21.4s + ror w12,w12,#25 + eor v24.16b,v3.16b,v0.16b + add w5,w5,w10 + eor v25.16b,v7.16b,v4.16b + add w6,w6,w11 + eor v26.16b,v11.16b,v8.16b + add w7,w7,w12 + eor v27.16b,v15.16b,v12.16b + add w8,w8,w9 + eor v28.16b,v19.16b,v16.16b + eor w21,w21,w5 + eor v29.16b,v23.16b,v20.16b + eor w17,w17,w6 + ushr v3.4s,v24.4s,#24 + eor w19,w19,w7 + ushr v7.4s,v25.4s,#24 + eor w20,w20,w8 + ushr v11.4s,v26.4s,#24 + ror w21,w21,#16 + ushr v15.4s,v27.4s,#24 + ror w17,w17,#16 + ushr v19.4s,v28.4s,#24 + ror w19,w19,#16 + ushr v23.4s,v29.4s,#24 + ror w20,w20,#16 + sli v3.4s,v24.4s,#8 + add w15,w15,w21 + sli v7.4s,v25.4s,#8 + add w16,w16,w17 + sli v11.4s,v26.4s,#8 + add w13,w13,w19 + sli v15.4s,v27.4s,#8 + add w14,w14,w20 + sli v19.4s,v28.4s,#8 + eor w10,w10,w15 + sli v23.4s,v29.4s,#8 + eor w11,w11,w16 + add v2.4s,v2.4s,v3.4s + eor w12,w12,w13 + add v6.4s,v6.4s,v7.4s + eor w9,w9,w14 + add v10.4s,v10.4s,v11.4s + ror w10,w10,#20 + add v14.4s,v14.4s,v15.4s + ror w11,w11,#20 + add v18.4s,v18.4s,v19.4s + ror w12,w12,#20 + add v22.4s,v22.4s,v23.4s + ror w9,w9,#20 + eor v24.16b,v1.16b,v2.16b + add w5,w5,w10 + eor v25.16b,v5.16b,v6.16b + add w6,w6,w11 + eor v26.16b,v9.16b,v10.16b + add w7,w7,w12 + eor v27.16b,v13.16b,v14.16b + add w8,w8,w9 + eor v28.16b,v17.16b,v18.16b + eor w21,w21,w5 + eor v29.16b,v21.16b,v22.16b + eor w17,w17,w6 + ushr v1.4s,v24.4s,#25 + eor w19,w19,w7 + ushr v5.4s,v25.4s,#25 + eor w20,w20,w8 + ushr v9.4s,v26.4s,#25 + ror w21,w21,#24 + ushr v13.4s,v27.4s,#25 + ror w17,w17,#24 + ushr v17.4s,v28.4s,#25 + ror w19,w19,#24 + ushr v21.4s,v29.4s,#25 + ror w20,w20,#24 + sli v1.4s,v24.4s,#7 + add w15,w15,w21 + sli v5.4s,v25.4s,#7 + add w16,w16,w17Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2.19.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
Quoted 20 lines · Ard Biesheuvel
Quoted 10 lines · Jason A. Donenfeld
These NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.
While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit
87cc649f30aaf69b351701875b9dac07c29ce8a2Signed-off-by Jason A. Donenfeld <[email protected]>Quoted 8 lines
Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org
Quoted 4 lines
As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.
Note that I am getting bounces from LAKML because the patch is too big.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
In reply to Ard BiesheuvelHi Ard,
Quoted 3 lines · Ard Biesheuvel
As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.
We're working on that, actually. It's not obvious when it'll be ready to ship -- perhaps after the initial merge, but perhaps way sooner -- but that is something we're trying to do for arm/arm64/mips64.
Jason
Jason A. Donenfeld
maintainerPATCH v6 6/23 zinc: port Andy Polyakov's ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese port and prepare Andy Polyakov's implementations for the kernel, but don't actually wire up any of the code yet. The wiring will be done in a subsequent commit, since we'll need to merge these implementations with another one. We make a few small changes to the assembly:
- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed (prefixed with .L) to fit kernel conventions.
- Constants have been rearranged so that they are closer to the code that is using them. [ARM only]
- The neon code can jump to the scalar code when it makes sense to do
so.
- The neon_512 function as a separate function has been removed, leaving the decision up to the main neon entry point. [ARM64 only]
--- lib/zinc/chacha20/chacha20-arm-cryptogams.S | 367 +++++++++--------- lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 57 +-- 2 files changed, 193 insertions(+), 231 deletions(-)
lib/zinc/chacha20/chacha20-arm-cryptogams.S +180 -187
@@ -1,9 +1,12 @@ /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ /* + * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. + * + * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS. */ -#include "arm_arch.h" +#include <linux/linkage.h> .text #if defined(__thumb2__) || defined(__clang__)
@@ -24,48 +27,25 @@ .long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral .Lone: .long 1,0,0,0 -.Lrot8: -.long 0x02010003,0x06050407 -#if __ARM_MAX_ARCH__>=7 -.LOPENSSL_armcap: -.word OPENSSL_armcap_P-.LChaCha20_ctr32 -#else .word -1 -#endif -.globl ChaCha20_ctr32 -.type ChaCha20_ctr32,%function .align 5 -ChaCha20_ctr32: -.LChaCha20_ctr32: +ENTRY(chacha20_arm) ldr r12,[sp,#0] @ pull pointer to counter and nonce stmdb sp!,{r0-r2,r4-r11,lr} -#if __ARM_ARCH__<7 && !defined(__thumb2__) - sub r14,pc,#16 @ ChaCha20_ctr32 -#else - adr r14,.LChaCha20_ctr32 -#endif cmp r2,#0 @ len==0? -#ifdef __thumb2__ +#ifdef __thumb2__ itt eq #endif addeq sp,sp,#4*3 - beq .Lno_data -#if __ARM_MAX_ARCH__>=7 - cmp r2,#192 @ test len - bls .Lshort - ldr r4,[r14,#-24] - ldr r4,[r14,r4] -# ifdef __APPLE__ - ldr r4,[r4] -# endif - tst r4,#ARMV7_NEON - bne .LChaCha20_neon -.Lshort: -#endif + beq .Lno_data_arm ldmia r12,{r4-r7} @ load counter and nonce sub sp,sp,#4*(16) @ off-load area - sub r14,r14,#64 @ .Lsigma +#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__) + sub r14,pc,#100 @ .Lsigma +#else + adr r14,.Lsigma @ .Lsigma +#endif stmdb sp!,{r4-r7} @ copy counter and nonce ldmia r3,{r4-r11} @ load key ldmia r14,{r0-r3} @ load sigma
@@ -191,7 +171,7 @@ ChaCha20_ctr32: @ rx and second half@sp+4*(16+8) cmp r11,#64 @ done yet? -#ifdef __thumb2__ +#ifdef __thumb2__ itete lo #endif addlo r12,sp,#4*(0) @ shortcut or ...
@@ -202,49 +182,49 @@ ChaCha20_ctr32: ldr r8,[sp,#4*(0)] @ load key material ldr r9,[sp,#4*(1)] -#if __ARM_ARCH__>=6 || !defined(__ARMEB__) -# if __ARM_ARCH__<7 +#if __LINUX_ARM_ARCH__ >= 6 || !defined(__ARMEB__) +#if __LINUX_ARM_ARCH__ < 7 orr r10,r12,r14 tst r10,#3 @ are input and output aligned? ldr r10,[sp,#4*(2)] bne .Lunaligned cmp r11,#64 @ restore flags -# else +#else ldr r10,[sp,#4*(2)] -# endif +#endif ldr r11,[sp,#4*(3)] add r0,r0,r8 @ accumulate key material add r1,r1,r9 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r8,[r12],#16 @ load input ldrhs r9,[r12,#-12] add r2,r2,r10 add r3,r3,r11 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r10,[r12,#-8] ldrhs r11,[r12,#-4] -# if __ARM_ARCH__>=6 && defined(__ARMEB__) +#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__) rev r0,r0 rev r1,r1 rev r2,r2 rev r3,r3 -# endif -# ifdef __thumb2__ +#endif +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r0,r0,r8 @ xor with input eorhs r1,r1,r9 add r8,sp,#4*(4) str r0,[r14],#16 @ store output -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r2,r2,r10 eorhs r3,r3,r11 ldmia r8,{r8-r11} @ load key material
@@ -254,34 +234,34 @@ ChaCha20_ctr32: add r4,r8,r4,ror#13 @ accumulate key material add r5,r9,r5,ror#13 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r8,[r12],#16 @ load input ldrhs r9,[r12,#-12] add r6,r10,r6,ror#13 add r7,r11,r7,ror#13 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r10,[r12,#-8] ldrhs r11,[r12,#-4] -# if __ARM_ARCH__>=6 && defined(__ARMEB__) +#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__) rev r4,r4 rev r5,r5 rev r6,r6 rev r7,r7 -# endif -# ifdef __thumb2__ +#endif +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r4,r4,r8 eorhs r5,r5,r9 add r8,sp,#4*(8) str r4,[r14],#16 @ store output -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r6,r6,r10 eorhs r7,r7,r11 str r5,[r14,#-12]
@@ -294,39 +274,39 @@ ChaCha20_ctr32: add r0,r0,r8 @ accumulate key material add r1,r1,r9 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r8,[r12],#16 @ load input ldrhs r9,[r12,#-12] -# ifdef __thumb2__ +#ifdef __thumb2__ itt hi -# endif +#endif strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it add r2,r2,r10 add r3,r3,r11 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r10,[r12,#-8] ldrhs r11,[r12,#-4] -# if __ARM_ARCH__>=6 && defined(__ARMEB__) +#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__) rev r0,r0 rev r1,r1 rev r2,r2 rev r3,r3 -# endif -# ifdef __thumb2__ +#endif +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r0,r0,r8 eorhs r1,r1,r9 add r8,sp,#4*(12) str r0,[r14],#16 @ store output -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r2,r2,r10 eorhs r3,r3,r11 str r1,[r14,#-12]
@@ -336,79 +316,79 @@ ChaCha20_ctr32: add r4,r8,r4,ror#24 @ accumulate key material add r5,r9,r5,ror#24 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hi -# endif +#endif addhi r8,r8,#1 @ next counter value strhi r8,[sp,#4*(12)] @ save next counter value -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r8,[r12],#16 @ load input ldrhs r9,[r12,#-12] add r6,r10,r6,ror#24 add r7,r11,r7,ror#24 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhs r10,[r12,#-8] ldrhs r11,[r12,#-4] -# if __ARM_ARCH__>=6 && defined(__ARMEB__) +#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__) rev r4,r4 rev r5,r5 rev r6,r6 rev r7,r7 -# endif -# ifdef __thumb2__ +#endif +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r4,r4,r8 eorhs r5,r5,r9 -# ifdef __thumb2__ +#ifdef __thumb2__ it ne -# endif +#endif ldrne r8,[sp,#4*(32+2)] @ re-load len -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif eorhs r6,r6,r10 eorhs r7,r7,r11 str r4,[r14],#16 @ store output str r5,[r14,#-12] -# ifdef __thumb2__ +#ifdef __thumb2__ it hs -# endif +#endif subhs r11,r8,#64 @ len-=64 str r6,[r14,#-8] str r7,[r14,#-4] bhi .Loop_outer beq .Ldone -# if __ARM_ARCH__<7 +#if __LINUX_ARM_ARCH__ < 7 b .Ltail .align 4 .Lunaligned: @ unaligned endian-neutral path cmp r11,#64 @ restore flags -# endif #endif -#if __ARM_ARCH__<7 +#endif +#if __LINUX_ARM_ARCH__ < 7 ldr r11,[sp,#4*(3)] add r0,r8,r0 @ accumulate key material add r1,r9,r1 add r2,r10,r2 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r8,r8,r8 @ zero or ... ldrhsb r8,[r12],#16 @ ... load input eorlo r9,r9,r9 ldrhsb r9,[r12,#-12] add r3,r11,r3 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r10,r10,r10 ldrhsb r10,[r12,#-8] eorlo r11,r11,r11
@@ -416,53 +396,53 @@ ChaCha20_ctr32: eor r0,r8,r0 @ xor with input (or zero) eor r1,r9,r1 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-15] @ load more input ldrhsb r9,[r12,#-11] eor r2,r10,r2 strb r0,[r14],#16 @ store output eor r3,r11,r3 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-7] ldrhsb r11,[r12,#-3] strb r1,[r14,#-12] eor r0,r8,r0,lsr#8 strb r2,[r14,#-8] eor r1,r9,r1,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-14] @ load more input ldrhsb r9,[r12,#-10] strb r3,[r14,#-4] eor r2,r10,r2,lsr#8 strb r0,[r14,#-15] eor r3,r11,r3,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-6] ldrhsb r11,[r12,#-2] strb r1,[r14,#-11] eor r0,r8,r0,lsr#8 strb r2,[r14,#-7] eor r1,r9,r1,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-13] @ load more input ldrhsb r9,[r12,#-9] strb r3,[r14,#-3] eor r2,r10,r2,lsr#8 strb r0,[r14,#-14] eor r3,r11,r3,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-5] ldrhsb r11,[r12,#-1] strb r1,[r14,#-10]
@@ -482,18 +462,18 @@ ChaCha20_ctr32: add r4,r8,r4,ror#13 @ accumulate key material add r5,r9,r5,ror#13 add r6,r10,r6,ror#13 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r8,r8,r8 @ zero or ... ldrhsb r8,[r12],#16 @ ... load input eorlo r9,r9,r9 ldrhsb r9,[r12,#-12] add r7,r11,r7,ror#13 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r10,r10,r10 ldrhsb r10,[r12,#-8] eorlo r11,r11,r11
@@ -501,53 +481,53 @@ ChaCha20_ctr32: eor r4,r8,r4 @ xor with input (or zero) eor r5,r9,r5 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-15] @ load more input ldrhsb r9,[r12,#-11] eor r6,r10,r6 strb r4,[r14],#16 @ store output eor r7,r11,r7 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-7] ldrhsb r11,[r12,#-3] strb r5,[r14,#-12] eor r4,r8,r4,lsr#8 strb r6,[r14,#-8] eor r5,r9,r5,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-14] @ load more input ldrhsb r9,[r12,#-10] strb r7,[r14,#-4] eor r6,r10,r6,lsr#8 strb r4,[r14,#-15] eor r7,r11,r7,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-6] ldrhsb r11,[r12,#-2] strb r5,[r14,#-11] eor r4,r8,r4,lsr#8 strb r6,[r14,#-7] eor r5,r9,r5,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-13] @ load more input ldrhsb r9,[r12,#-9] strb r7,[r14,#-3] eor r6,r10,r6,lsr#8 strb r4,[r14,#-14] eor r7,r11,r7,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-5] ldrhsb r11,[r12,#-1] strb r5,[r14,#-10]
@@ -564,26 +544,26 @@ ChaCha20_ctr32: add r8,sp,#4*(4+4) ldmia r8,{r8-r11} @ load key material ldmia r0,{r0-r7} @ load second half -# ifdef __thumb2__ +#ifdef __thumb2__ itt hi -# endif +#endif strhi r10,[sp,#4*(16+10)] @ copy "rx" strhi r11,[sp,#4*(16+11)] @ copy "rx" add r0,r8,r0 @ accumulate key material add r1,r9,r1 add r2,r10,r2 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r8,r8,r8 @ zero or ... ldrhsb r8,[r12],#16 @ ... load input eorlo r9,r9,r9 ldrhsb r9,[r12,#-12] add r3,r11,r3 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r10,r10,r10 ldrhsb r10,[r12,#-8] eorlo r11,r11,r11
@@ -591,53 +571,53 @@ ChaCha20_ctr32: eor r0,r8,r0 @ xor with input (or zero) eor r1,r9,r1 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-15] @ load more input ldrhsb r9,[r12,#-11] eor r2,r10,r2 strb r0,[r14],#16 @ store output eor r3,r11,r3 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-7] ldrhsb r11,[r12,#-3] strb r1,[r14,#-12] eor r0,r8,r0,lsr#8 strb r2,[r14,#-8] eor r1,r9,r1,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-14] @ load more input ldrhsb r9,[r12,#-10] strb r3,[r14,#-4] eor r2,r10,r2,lsr#8 strb r0,[r14,#-15] eor r3,r11,r3,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-6] ldrhsb r11,[r12,#-2] strb r1,[r14,#-11] eor r0,r8,r0,lsr#8 strb r2,[r14,#-7] eor r1,r9,r1,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-13] @ load more input ldrhsb r9,[r12,#-9] strb r3,[r14,#-3] eor r2,r10,r2,lsr#8 strb r0,[r14,#-14] eor r3,r11,r3,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-5] ldrhsb r11,[r12,#-1] strb r1,[r14,#-10]
@@ -654,25 +634,25 @@ ChaCha20_ctr32: add r8,sp,#4*(4+8) ldmia r8,{r8-r11} @ load key material add r4,r8,r4,ror#24 @ accumulate key material -# ifdef __thumb2__ +#ifdef __thumb2__ itt hi -# endif +#endif addhi r8,r8,#1 @ next counter value strhi r8,[sp,#4*(12)] @ save next counter value add r5,r9,r5,ror#24 add r6,r10,r6,ror#24 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r8,r8,r8 @ zero or ... ldrhsb r8,[r12],#16 @ ... load input eorlo r9,r9,r9 ldrhsb r9,[r12,#-12] add r7,r11,r7,ror#24 -# ifdef __thumb2__ +#ifdef __thumb2__ itete lo -# endif +#endif eorlo r10,r10,r10 ldrhsb r10,[r12,#-8] eorlo r11,r11,r11
@@ -680,53 +660,53 @@ ChaCha20_ctr32: eor r4,r8,r4 @ xor with input (or zero) eor r5,r9,r5 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-15] @ load more input ldrhsb r9,[r12,#-11] eor r6,r10,r6 strb r4,[r14],#16 @ store output eor r7,r11,r7 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-7] ldrhsb r11,[r12,#-3] strb r5,[r14,#-12] eor r4,r8,r4,lsr#8 strb r6,[r14,#-8] eor r5,r9,r5,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-14] @ load more input ldrhsb r9,[r12,#-10] strb r7,[r14,#-4] eor r6,r10,r6,lsr#8 strb r4,[r14,#-15] eor r7,r11,r7,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-6] ldrhsb r11,[r12,#-2] strb r5,[r14,#-11] eor r4,r8,r4,lsr#8 strb r6,[r14,#-7] eor r5,r9,r5,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r8,[r12,#-13] @ load more input ldrhsb r9,[r12,#-9] strb r7,[r14,#-3] eor r6,r10,r6,lsr#8 strb r4,[r14,#-14] eor r7,r11,r7,lsr#8 -# ifdef __thumb2__ +#ifdef __thumb2__ itt hs -# endif +#endif ldrhsb r10,[r12,#-5] ldrhsb r11,[r12,#-1] strb r5,[r14,#-10]
@@ -740,13 +720,13 @@ ChaCha20_ctr32: eor r7,r11,r7,lsr#8 strb r6,[r14,#-5] strb r7,[r14,#-1] -# ifdef __thumb2__ +#ifdef __thumb2__ it ne -# endif +#endif ldrne r8,[sp,#4*(32+2)] @ re-load len -# ifdef __thumb2__ +#ifdef __thumb2__ it hs -# endif +#endif subhs r11,r8,#64 @ len-=64 bhi .Loop_outer
@@ -768,20 +748,33 @@ ChaCha20_ctr32: .Ldone: add sp,sp,#4*(32+3) -.Lno_data: +.Lno_data_arm: ldmia sp!,{r4-r11,pc} -.size ChaCha20_ctr32,.-ChaCha20_ctr32 -#if __ARM_MAX_ARCH__>=7 +ENDPROC(chacha20_arm) + +#ifdef CONFIG_KERNEL_MODE_NEON +.align 5 +.Lsigma2: +.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral +.Lone2: +.long 1,0,0,0 +.word -1 + .arch armv7-a .fpu neon -.type ChaCha20_neon,%function .align 5 -ChaCha20_neon: +ENTRY(chacha20_neon) ldr r12,[sp,#0] @ pull pointer to counter and nonce stmdb sp!,{r0-r2,r4-r11,lr} -.LChaCha20_neon: - adr r14,.Lsigma + cmp r2,#0 @ len==0? +#ifdef __thumb2__ + itt eq +#endif + addeq sp,sp,#4*3 + beq .Lno_data_neon +.Lchacha20_neon_begin: + adr r14,.Lsigma2 vstmdb sp!,{d8-d15} @ ABI spec says so stmdb sp!,{r0-r3}
@@ -1121,12 +1114,12 @@ ChaCha20_neon: ldr r10,[r12,#-8] add r3,r3,r11 ldr r11,[r12,#-4] -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r0,r0 rev r1,r1 rev r2,r2 rev r3,r3 -# endif +#endif eor r0,r0,r8 @ xor with input add r8,sp,#4*(4) eor r1,r1,r9
@@ -1146,12 +1139,12 @@ ChaCha20_neon: ldr r10,[r12,#-8] add r7,r11,r7,ror#13 ldr r11,[r12,#-4] -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r4,r4 rev r5,r5 rev r6,r6 rev r7,r7 -# endif +#endif eor r4,r4,r8 add r8,sp,#4*(8) eor r5,r5,r9
@@ -1170,24 +1163,24 @@ ChaCha20_neon: ldr r8,[r12],#16 @ load input add r1,r1,r9 ldr r9,[r12,#-12] -# ifdef __thumb2__ +#ifdef __thumb2__ it hi -# endif +#endif strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it add r2,r2,r10 ldr r10,[r12,#-8] -# ifdef __thumb2__ +#ifdef __thumb2__ it hi -# endif +#endif strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it add r3,r3,r11 ldr r11,[r12,#-4] -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r0,r0 rev r1,r1 rev r2,r2 rev r3,r3 -# endif +#endif eor r0,r0,r8 add r8,sp,#4*(12) eor r1,r1,r9
@@ -1210,16 +1203,16 @@ ChaCha20_neon: add r7,r11,r7,ror#24 ldr r10,[r12,#-8] ldr r11,[r12,#-4] -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r4,r4 rev r5,r5 rev r6,r6 rev r7,r7 -# endif +#endif eor r4,r4,r8 -# ifdef __thumb2__ +#ifdef __thumb2__ it hi -# endif +#endif ldrhi r8,[sp,#4*(32+2)] @ re-load len eor r5,r5,r9 eor r6,r6,r10
@@ -1379,7 +1372,7 @@ ChaCha20_neon: add r6,r10,r6,ror#13 add r7,r11,r7,ror#13 ldmia r8,{r8-r11} @ load key material -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r0,r0 rev r1,r1 rev r2,r2
@@ -1388,7 +1381,7 @@ ChaCha20_neon: rev r5,r5 rev r6,r6 rev r7,r7 -# endif +#endif stmia sp,{r0-r7} add r0,sp,#4*(16+8)
@@ -1408,7 +1401,7 @@ ChaCha20_neon: add r6,r10,r6,ror#24 add r7,r11,r7,ror#24 ldr r11,[sp,#4*(32+2)] @ re-load len -# ifdef __ARMEB__ +#ifdef __ARMEB__ rev r0,r0 rev r1,r1 rev r2,r2
@@ -1417,7 +1410,7 @@ ChaCha20_neon: rev r5,r5 rev r6,r6 rev r7,r7 -# endif +#endif stmia r8,{r0-r7} add r10,sp,#4*(0) sub r11,r11,#64*3 @ len-=64*3
@@ -1434,7 +1427,7 @@ ChaCha20_neon: add sp,sp,#4*(32+4) vldmia sp,{d8-d15} add sp,sp,#4*(16+3) +.Lno_data_neon: ldmia sp!,{r4-r11,pc} -.size ChaCha20_neon,.-ChaCha20_neon -.comm OPENSSL_armcap_P,4,4 +ENDPROC(chacha20_neon) #endif
lib/zinc/chacha20/chacha20-arm64-cryptogams.S +13 -44
94 lines not on this page · raw message
Signature · 1 line
2.19.0
Jason A. Donenfeld
maintainerPATCH v6 7/23 zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldThese wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).
This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.
Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.
This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:
- ARM1176JZF-S [ARMv6]
- Cortex-A7 [ARMv7] - Cortex-A8 [ARMv7] - Cortex-A9 [ARMv7] - Cortex-A17 [ARMv7] - Cortex-A53 [ARMv8] - Cortex-A55 [ARMv8] - Cortex-A73 [ARMv8] - Cortex-A75 [ARMv8]
Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.
---
lib/zinc/Makefile | 2 +
lib/zinc/chacha20/chacha20-arm-glue.h | 88 +++
...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
...20-arm64-cryptogams.S => chacha20-arm64.S} | 0
lib/zinc/chacha20/chacha20.c | 2 +
5 files changed, 556 insertions(+), 38 deletions(-)
create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)lib/zinc/Makefile +2 -0
@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +88 -0
88 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464 -38
527 lines not on this page · raw message
lib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0 -0
lib/zinc/chacha20/chacha20.c +2 -0
@@ -16,6 +16,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "chacha20-x86_64-glue.h" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 ) +#include "chacha20-arm-glue.h" #else void __init chacha20_fpu_init (void ) {
Signature · 1 line
2.19.0
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 113 lines · Jason A. Donenfeld
These wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).
This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.
This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:
- ARM1176JZF-S [ARMv6]
- Cortex-A7 [ARMv7] - Cortex-A8 [ARMv7] - Cortex-A9 [ARMv7] - Cortex-A17 [ARMv7] - Cortex-A53 [ARMv8] - Cortex-A55 [ARMv8] - Cortex-A73 [ARMv8] - Cortex-A75 [ARMv8]Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.
Signed-off-by Jason A. Donenfeld <[email protected]>Co-authored-by Eric Biggers <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Russell King <[email protected]>Cc linux-arm-kernel at lists.infradead.org--- lib/zinc/Makefile | 2 + lib/zinc/chacha20/chacha20-arm-glue.h | 88 +++ ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++-- ...20-arm64-cryptogams.S => chacha20-arm64.S} | 0 lib/zinc/chacha20/chacha20.c | 2 + 5 files changed, 556 insertions(+), 38 deletions(-) create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%) rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)lib/zinc/Makefile +2
-0@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.olib/zinc/chacha20/chacha20-arm-glue.h new file +43
-0@@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <asm/hwcap.h> +#include <asm/neon.h> +#if defined(CONFIG_ARM ) +#include <asm/system_info.h> +#include <asm/cputype.h> +#endif + +asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#if defined(CONFIG_ARM ) +asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]); +#endif +#if defined(CONFIG_KERNEL_MODE_NEON ) +asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#endif + +static bool chacha20_use_neon__ro_after_init ; + +static void __init chacha20_fpu_init (void ) +{ +#if defined(CONFIG_ARM64 ) + chacha20_use_neon = elf_hwcap &HWCAP_ASIMD ; +#elif defined(CONFIG_ARM ) +switch (read_cpuid_part ()) { +case ARM_CPU_PART_CORTEX_A7 : +case ARM_CPU_PART_CORTEX_A5 : +/* The Cortex-A7 and Cortex-A5 do not perform well with the NEON +* implementation but do incredibly with the scalar one and use +* less power. +*/ +break ; +default : + chacha20_use_neon = elf_hwcap &HWCAP_NEON ; + } +#endif +} +
lib/zinc/chacha20/chacha20-arm-glue.h line 53
Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10
-0+static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{ +#if defined(CONFIG_KERNEL_MODE_NEON ) +if (chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && +simd_use (simd_context)) +chacha20_neon (dst, src, len, state->key, state->counter); + else +#endif
Better to use IS_ENABLED() here:
lib/zinc/chacha20/chacha20-arm-glue.h line 56
Quoted 3 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +3
-0+if (IS_ENABLED (CONFIG_KERNEL_MODE_NEON )) && + chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && + simd_use(simd_context))
Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.
Quoted 592 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +35
-0+chacha20_arm (dst, src, len, state->key, state->counter); + + state->counter[0 ] += (len +63 ) /64 ; +return true; +} + +static inline bool hchacha20_arch (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ], +simd_context_t *simd_context) +{ +#if defined(CONFIG_ARM ) +u32 x[] = {CHACHA20_CONSTANT_EXPA , +CHACHA20_CONSTANT_ND_3 , +CHACHA20_CONSTANT_2_BY , +CHACHA20_CONSTANT_TE_K , +get_unaligned_le32 (key +0 ), +get_unaligned_le32 (key +4 ), +get_unaligned_le32 (key +8 ), +get_unaligned_le32 (key +12 ), +get_unaligned_le32 (key +16 ), +get_unaligned_le32 (key +20 ), +get_unaligned_le32 (key +24 ), +get_unaligned_le32 (key +28 ), +get_unaligned_le32 (nonce +0 ), +get_unaligned_le32 (nonce +4 ), +get_unaligned_le32 (nonce +8 ), +get_unaligned_le32 (nonce +12 ) + }; +hchacha20_arm (x, derived_key); +return true; +#else +return false; +#endif +}lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464
-38@@ -1,13 +1,475 @@ /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ /* + * Copyright (C) 2018 Google, Inc. * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. - * - * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS. */ #include <linux/linkage.h> +/* + * The following scalar routine was written by Eric Biggers. + * + * Design notes: + * + * 16 registers would be needed to hold the state matrix, but only 14 are + * available because 'sp' and 'pc' cannot be used. So we spill the elements + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one + * 'ldrd' and one 'strd' instruction per round. + * + * All rotates are performed using the implicit rotate operand accepted by the + * 'add' and 'eor' instructions. This is faster than using explicit rotate + * instructions. To make this work, we allow the values in the second and last + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the + * wrong rotation amount. The rotation amount is then fixed up just in time + * when the values are used. 'brot' is the number of bits the values in row 'b' + * need to be rotated right to arrive at the correct values, and 'drot' + * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such + * that they end up as (25, 24) after every round. + */ + + // ChaCha state registers + X0 .req r0 + X1 .req r1 + X2 .req r2 + X3 .req r3 + X4 .req r4 + X5 .req r5 + X6 .req r6 + X7 .req r7 + X8_X10 .req r8 // shared by x8 and x10 + X9_X11 .req r9 // shared by x9 and x11 + X12 .req r10 + X13 .req r11 + X14 .req r12 + X15 .req r14 + +.Lexpand_32byte_k: + // "expand 32-byte k" + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574 + +#ifdef __thumb2__ +# define adrl adr +#endif + +.macro __rev out, in, t0, t1, t2 +.if __LINUX_ARM_ARCH__ >= 6 + rev \out, \in +.else + lsl \t0, \in, #24 + and \t1, \in, #0xff00 + and \t2, \in, #0xff0000 + orr \out, \t0, \in, lsr #24 + orr \out, \out, \t1, lsl #8 + orr \out, \out, \t2, lsr #8 +.endif +.endm + +.macro _le32_bswap x, t0, t1, t2 +#ifdef __ARMEB__ + __rev \x, \x, \t0, \t1, \t2 +#endif +.endm + +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2 + _le32_bswap \a, \t0, \t1, \t2 + _le32_bswap \b, \t0, \t1, \t2 + _le32_bswap \c, \t0, \t1, \t2 + _le32_bswap \d, \t0, \t1, \t2 +.endm + +.macro __ldrd a, b, src, offset +#if __LINUX_ARM_ARCH__ >= 6 + ldrd \a, \b, [\src, #\offset] +#else + ldr \a, [\src, #\offset] + ldr \b, [\src, #\offset + 4] +#endif +.endm + +.macro __strd a, b, dst, offset +#if __LINUX_ARM_ARCH__ >= 6 + strd \a, \b, [\dst, #\offset] +#else + str \a, [\dst, #\offset] + str \b, [\dst, #\offset + 4] +#endif +.endm + +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2 + + // a += b; d ^= a; d = rol(d, 16); + add \a1, \a1, \b1, ror #brot + add \a2, \a2, \b2, ror #brot + eor \d1, \a1, \d1, ror #drot + eor \d2, \a2, \d2, ror #drot + // drot == 32 - 16 == 16 + + // c += d; b ^= c; b = rol(b, 12); + add \c1, \c1, \d1, ror #16 + add \c2, \c2, \d2, ror #16 + eor \b1, \c1, \b1, ror #brot + eor \b2, \c2, \b2, ror #brot + // brot == 32 - 12 == 20 + + // a += b; d ^= a; d = rol(d, 8); + add \a1, \a1, \b1, ror #20 + add \a2, \a2, \b2, ror #20 + eor \d1, \a1, \d1, ror #16 + eor \d2, \a2, \d2, ror #16 + // drot == 32 - 8 == 24 + + // c += d; b ^= c; b = rol(b, 7); + add \c1, \c1, \d1, ror #24 + add \c2, \c2, \d2, ror #24 + eor \b1, \c1, \b1, ror #20 + eor \b2, \c2, \b2, ror #20 + // brot == 32 - 7 == 25 +.endm + +.macro _doubleround + + // column round + + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13) + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13 + + // save (x8, x9); restore (x10, x11) + __strd X8_X10, X9_X11, sp, 0 + __ldrd X8_X10, X9_X11, sp, 8 + + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15) + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15 + + .set brot, 25 + .set drot, 24 + + // diagonal round + + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12) + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12 + + // save (x10, x11); restore (x8, x9) + __strd X8_X10, X9_X11, sp, 8 + __ldrd X8_X10, X9_X11, sp, 0 + + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14) + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14 +.endm + +.macro _chacha_permute nrounds + .set brot, 0 + .set drot, 0 + .rept \nrounds / 2 + _doubleround + .endr +.endm + +.macro _chacha nrounds + +.Lnext_block\@: + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + + // Do the core ChaCha permutation to update x0-x15. + _chacha_permute \nrounds + + add sp, #8 + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15). + push {X8_X10, X9_X11, X12, X13, X14, X15} + + // Load (OUT, IN, LEN). + ldr r14, [sp, #96] + ldr r12, [sp, #100] + ldr r11, [sp, #104] + + orr r10, r14, r12 + + // Use slow path if fewer than 64 bytes remain. + cmp r11, #64 + blt .Lxor_slowpath\@ + + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on + // ARMv6+, since ldmia and stmia (used below) still require alignment. + tst r10, #3 + bne .Lxor_slowpath\@ + + // Fast path: XOR 64 bytes of aligned data. + + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // x0-x3 + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + ldmia r12!, {r8-r11} + eor X0, X0, r8 + eor X1, X1, r9 + eor X2, X2, r10 + eor X3, X3, r11 + stmia r14!, {X0-X3} + + // x4-x7 + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + ldmia r12!, {X0-X3} + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + eor X4, X4, X0 + eor X5, X5, X1 + eor X6, X6, X2 + eor X7, X7, X3 + stmia r14!, {X4-X7} + + // x8-x15 + pop {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + ldmia r12!, {r8-r11} + eor r0, r0, r8 // x8 + eor r1, r1, r9 // x9 + eor r6, r6, r10 // x10 + eor r7, r7, r11 // x11 + stmia r14!, {r0,r1,r6,r7} + ldmia r12!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + ldr r9, [sp, #72] // load LEN + eor r2, r2, r0 // x12 + eor r3, r3, r1 // x13 + eor r4, r4, r6 // x14 + eor r5, r5, r7 // x15 + subs r9, #64 // decrement and check LEN + stmia r14!, {r2-r5} + + beq .Ldone\@ + +.Lprepare_for_next_block\@: + + // Stack: x0-x15 OUT IN LEN + + // Increment block counter (x12) + add r8, #1 + + // Store updated (OUT, IN, LEN) + str r14, [sp, #64] + str r12, [sp, #68] + str r9, [sp, #72] + + mov r14, sp + + // Store updated block counter (x12) + str r8, [sp, #48] + + sub sp, #16 + + // Reload state and do next block + ldmia r14!, {r0-r11} // load x0-x11 + __strd r10, r11, sp, 8 // store x10-x11 before state + ldmia r14, {r10-r12,r14} // load x12-x15 + b .Lnext_block\@ + +.Lxor_slowpath\@: + // Slow path: < 64 bytes remaining, or unaligned input or output buffer. + // We handle it by storing the 64 bytes of keystream to the stack, then + // XOR-ing the needed portion with the data. + + // Allocate keystream buffer + sub sp, #64 + mov r14, sp + + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Save keystream for x0-x3 + __ldrd r8, r9, sp, 96 + __ldrd r10, r11, sp, 104 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + stmia r14!, {X0-X3} + + // Save keystream for x4-x7 + __ldrd r8, r9, sp, 112 + __ldrd r10, r11, sp, 120 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + add r8, sp, #64 + stmia r14!, {X4-X7} + + // Save keystream for x8-x15 + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 128 + __ldrd r10, r11, sp, 136 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + stmia r14!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 144 + __ldrd r10, r11, sp, 152 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + stmia r14, {r2-r5} + + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN + // Registers: r8 is block counter, r12 is IN. + + ldr r9, [sp, #168] // LEN + ldr r14, [sp, #160] // OUT + cmp r9, #64 + mov r0, sp + movle r1, r9 + movgt r1, #64 + // r1 is number of bytes to XOR, in range [1, 64] + +.if __LINUX_ARM_ARCH__ < 6 + orr r2, r12, r14 + tst r2, #3 // IN or OUT misaligned? + bne .Lxor_next_byte\@ +.endif + + // XOR a word at a time +.rept 16 + subs r1, #4 + blt .Lxor_words_done\@ + ldr r2, [r12], #4 + ldr r3, [r0], #4 + eor r2, r2, r3 + str r2, [r14], #4 +.endr + b .Lxor_slowpath_done\@ +.Lxor_words_done\@: + ands r1, r1, #3 + beq .Lxor_slowpath_done\@ + + // XOR a byte at a time +.Lxor_next_byte\@: + ldrb r2, [r12], #1 + ldrb r3, [r0], #1 + eor r2, r2, r3 + strb r2, [r14], #1 + subs r1, #1 + bne .Lxor_next_byte\@ + +.Lxor_slowpath_done\@: + subs r9, #64 + add sp, #96 + bgt .Lprepare_for_next_block\@ + +.Ldone\@: +.endm // _chacha + +/* + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8], + * const u32 iv[4]); + */ +ENTRY(chacha20_arm) + cmp r2, #0 // len == 0? + bxeq lr + + push {r0-r2,r4-r11,lr} + + // Push state x0-x15 onto stack. + // Also store an extra copy of x10-x11 just before the state. + + ldr r4, [sp, #48] // iv + mov r0, sp + sub sp, #80 + + // iv: x12-x15 + ldm r4, {X12,X13,X14,X15} + stmdb r0!, {X12,X13,X14,X15} + + // key: x4-x11 + __ldrd X8_X10, X9_X11, r3, 24 + __strd X8_X10, X9_X11, sp, 8 + stmdb r0!, {X8_X10, X9_X11} + ldm r3, {X4-X9_X11} + stmdb r0!, {X4-X9_X11} + + // constants: x0-x3 + adrl X3, .Lexpand_32byte_k + ldm X3, {X0-X3} + __strd X0, X1, sp, 16 + __strd X2, X3, sp, 24 + + _chacha 20 + + add sp, #76 + pop {r4-r11, pc} +ENDPROC(chacha20_arm) + +/* + * void hchacha20_arm(const u32 state[16], u32 out[8]); + */ +ENTRY(hchacha20_arm) + push {r1,r4-r11,lr} + + mov r14, r0 + ldmia r14!, {r0-r11} // load x0-x11 + push {r10-r11} // store x10-x11 to stack + ldm r14, {r10-r12,r14} // load x12-x15 + sub sp, #8 + + _chacha_permute 20 + + // Skip over (unused0-unused1, x10-x11) + add sp, #16 + + // Fix up rotations of x12-x15 + ror X12, X12, #drot + ror X13, X13, #drot + pop {r4} // load 'out' + ror X14, X14, #drot + ror X15, X15, #drot + + // Store (x0-x3,x12-x15) to 'out' + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15} + + pop {r4-r11,pc} +ENDPROC(hchacha20_arm) + +#ifdef CONFIG_KERNEL_MODE_NEON +/* + * This following NEON routine was ported from Andy Polyakov's implementation + * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine, + * since certain NEON code paths actually branch to it. + */ + .text #if defined(__thumb2__) || defined(__clang__) .syntax unified@@ -22,39 +484,6 @@ #define ldrhsb ldrbhs #endif -.align 5 -.Lsigma: -.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral -.Lone: -.long 1,0,0,0 -.word -1 - -.align 5 -ENTRY(chacha20_arm) - ldr r12,[sp,#0] @ pull pointer to counter and nonce - stmdb sp!,{r0-r2,r4-r11,lr} - cmp r2,#0 @ len==0? -#ifdef __thumb2__ - itt eq -#endif - addeq sp,sp,#4*3 - beq .Lno_data_arm - ldmia r12,{r4-r7} @ load counter and nonce - sub sp,sp,#4*(16) @ off-load area -#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__) - sub r14,pc,#100 @ .Lsigma -#else - adr r14,.Lsigma @ .Lsigma -#endif - stmdb sp!,{r4-r7} @ copy counter and nonce - ldmia r3,{r4-r11} @ load key - ldmia r14,{r0-r3} @ load sigma - stmdb sp!,{r4-r11} @ copy key - stmdb sp!,{r0-r3} @ copy sigma - str r10,[sp,#4*(16+10)] @ off-load "rx" - str r11,[sp,#4*(16+11)] @ off-load "rx" - b .Loop_outer_enter - .align 4 .Loop_outer: ldmia sp,{r0-r9} @ load key material@@ -748,11 +1177,8 @@ ENTRY(chacha20_arm) .Ldone: add sp,sp,#4*(32+3) -.Lno_data_arm: ldmia sp!,{r4-r11,pc} -ENDPROC(chacha20_arm) -#ifdef CONFIG_KERNEL_MODE_NEON .align 5 .Lsigma2: .long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutrallib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0
-0lib/zinc/chacha20/chacha20.c +2
-1@@ -16,6 +16,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "chacha20-x86_64-glue.h" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 ) +#include "chacha20-arm-glue.h" #else void __init chacha20_fpu_init (void ) { --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Hi Ard,
Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4
-0+static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{
Quoted 12 lines
lib/zinc/chacha20/chacha20-arm-glue.h line 53
Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6
-0+#if defined(CONFIG_KERNEL_MODE_NEON ) +if (chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && +simd_use (simd_context)) +chacha20_neon (dst, src, len, state->key, state->counter); + else +#endif Better to use
IS_ENABLED()here:Quoted 3 lines
(quoted diff) +3
-0+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) && + chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 && + simd_use(simd_context))
Good idea. I'll fix that up.
Quoted 4 lines
Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.
The vast majority of crypto routines in arch/*/crypto/ follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle. I made a change so that in chacha20poly1305.c, it calls simd_relax after handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now. And if it becomes one, this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.
Jason
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
(+ Herbert, Thomas)
Quoted 25 lines · Jason A. Donenfeld
Hi Ard,
Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4
-0+static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{Quoted 12 lines
lib/zinc/chacha20/chacha20-arm-glue.h line 53
Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6
-0+#if defined(CONFIG_KERNEL_MODE_NEON ) +if (chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && +simd_use (simd_context)) +chacha20_neon (dst, src, len, state->key, state->counter); + else +#endif Better to use
IS_ENABLED()here:Quoted 3 lines
(quoted diff) +3
-0+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) && + chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 && + simd_use(simd_context))Good idea. I'll fix that up.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.
The vast majority of crypto routines in
arch/*/crypto/follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle.
Anything that uses the scatterwalk API (AEADs and skciphers) will handle at most a page at a time. Hashes are different, which is why some of them have to handle it explicitly.
Quoted 7 lines
I made a change so that in
chacha20poly1305.c, it callssimd_relaxafter handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now.
The state of the tree does not capture all relevant context or history. The scheduling latency issue was brought up very recently by the -rt folks on the mailing lists.
Quoted 5 lines
And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.
You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.
Part of the [justified] criticism on the current state of the crypto API is on its complexity, and so I don't think it makes sense to keep it simple now and add the complexity later (and the same concern applies to async support btw).
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 3 lines · Ard Biesheuvel
I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).
Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 7 lines · Jason A. Donenfeld
Quoted 3 lines · Ard Biesheuvel
I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).
Quoted 10 lines
Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.
Framing it as /needless/ complexity does not help at all. The changes you are proposing are very useful, but nobody wants two crypto subsystems with two different maintainers in the kernel, so I would like to understand where this is going in the future. I am not saying it should block these patches though.
Also, I have spent a *lot* of time looking at your code, and trying to make it better, especially for use cases that weren't on your radar to begin with (e.g., 'pet projects' [your words] like the Cortex-A7 which will be in almost every new 32-bit Android phone). So characterizing my feedback as some kind of sabotage is not very productive either.
Contrary to what you seem to think, I am not deeply invested in the crypto API. What I do care about is that the ARM crypto pieces in the kernel are maintained, supported and improved by someone who understands the use cases Linaro's members care about, and is willing to make an effort to gain such understanding if he doesn't. I have no doubt that your involvement in the kernel's crypto subsystem will have a significant positive impact when it comes to code quality, robustness and usability. I'd just like to see a bit more consideration for other aspects of kernel programming, e.g., preemption under -rt, stack size constraints, coding style, importing code from other projects etc. - please try to be less dismissive of feedback first time around, but try to understand why people are raising these issues; I'm sure you will appreciate it when future contributors to zinc will do the same.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 5 lines · Ard Biesheuvel
Framing it as /needless/ complexity does not help at all. The changes you are proposing are very useful, but nobody wants two crypto subsystems with two different maintainers in the kernel, so I would like to understand where this is going in the future. I am not saying it should block these patches though.
Thanks for clarifying. I understood you to be intending to block the patches until they were converted to an async interface, which is not what Zinc's about. Seeing as you're just curious about future directions, that seems much more tenable.
Quoted 3 lines
Also, I have spent a *lot* of time looking at your code, and trying to make it better, especially for use cases that weren't on your radar to begin with
I am extremely grateful for a good portion of your reviews indeed. As I mentioned earlier, much is very useful. But in other places, I fear you're steering this in a direction I really am hesitant to go.
Quoted 1 line
(e.g., 'pet projects' [your words]
Taken out of context.
Quoted 3 lines
consideration for other aspects of kernel programming, e.g., preemption under -rt, stack size constraints, coding style, importing code from other projects etc.
And indeed all of these concerns I've been pretty amenable to, and continue to do so. What I'm commenting on are things outside of these.
Quoted 3 lines
- please try to be less dismissive of
feedback first time around, but try to understand why people are raising these issues
Apologies, and duly noted. I'll give you the benefit of the doubt.
Jason
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldQuoted 6 lines · Jason A. Donenfeld
Quoted 3 lines · Ard Biesheuvel
I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).
Quoted 10 lines
Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.
Can you please stop accusing Ard of "filibustering" your patchset? Spending too long in non-preemptible code is a real problem even on non-RT systems. syzkaller has been reporting bugs where the kernel spins too long without any preemption points, both in crypto-related code and elsewhere in the kernel. So we've had to add explicit preemption points to address those, as otherwise users can lock up all CPUs for tens of seconds. The issue being discussed here is basically the same except here preemption is being explicitly disabled via kernel_neon_begin(), so it becomes a problem even on non-CONFIG_PREEMPT kernels.
It's much better to address this problem (which is a regression from the current skcipher API which only maps a page at a time) up front rather than have to rush to patch it after the fact once the syzbot reports start coming in.
- Eric
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 8 lines · Eric Biggers
Can you please stop accusing Ard of "filibustering" your patchset? Spending too long in non-preemptible code is a real problem even on non-RT systems. syzkaller has been reporting bugs where the kernel spins too long without any preemption points, both in crypto-related code and elsewhere in the kernel. So we've had to add explicit preemption points to address those, as otherwise users can lock up all CPUs for tens of seconds. The issue being discussed here is basically the same except here preemption is being explicitly disabled via
kernel_neon_begin(), so it becomes a problem even on non-CONFIG_PREEMPT kernels.
The async distraction (re:filibustering) and the preempt concern are two totally different things. I've already posted some code elsewhere in this thread that addresses the preempt issue that looked good to Ard. This will be part of v7.
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Ard BiesheuvelQuoted 60 lines · Ard Biesheuvel
(+ Herbert, Thomas)
Quoted 25 lines · Jason A. Donenfeld
Hi Ard,
Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4
-0+static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{Quoted 12 lines
lib/zinc/chacha20/chacha20-arm-glue.h line 53
Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6
-0+#if defined(CONFIG_KERNEL_MODE_NEON ) +if (chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && +simd_use (simd_context)) +chacha20_neon (dst, src, len, state->key, state->counter); + else +#endif Better to use
IS_ENABLED()here:Quoted 3 lines
(quoted diff) +3
-0+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) && + chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 && + simd_use(simd_context))Good idea. I'll fix that up.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.
The vast majority of crypto routines in
arch/*/crypto/follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle.Anything that uses the scatterwalk API (AEADs and skciphers) will handle at most a page at a time. Hashes are different, which is why some of them have to handle it explicitly.
Quoted 7 lines
I made a change so that in
chacha20poly1305.c, it callssimd_relaxafter handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now.The state of the tree does not capture all relevant context or history. The scheduling latency issue was brought up very recently by the -rt folks on the mailing lists.
Quoted 5 lines
And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.
Quoted 4 lines
You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.
Actually, looking at the code again, the abstraction does appear to be fine, it is just the chacha20 code that does not make use of it.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 4 lines · Ard Biesheuvel
On Wed, 26 Sep 2018 at 16:02, Ard Biesheuvel <[email protected]> wrote:
Actually, looking at the code again, the abstraction does appear to be fine, it is just the chacha20 code that does not make use of it.
So what you have in mind is something like calling simd_relax() every 4096 bytes or so?
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 2 lines · Jason A. Donenfeld
So what you have in mind is something like calling
simd_relax()every 4096 bytes or so?
That was actually pretty easy, putting together both of your suggestions:
static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
u8 *src, size_t len,
simd_context_t *simd_context)
{
while (len > PAGE_SIZE) {
chacha20_arch(state, dst, src, PAGE_SIZE, simd_context);
len -= PAGE_SIZE;
src += PAGE_SIZE;
dst += PAGE_SIZE;
simd_relax(simd_context);
}
if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon &&
len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context))
chacha20_neon(dst, src, len, state->key, state->counter);
else
chacha20_arm(dst, src, len, state->key, state->counter);
state->counter[0] += (len + 63) / 64;
return true;
}Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 19 lines · Jason A. Donenfeld
Quoted 2 lines · Jason A. Donenfeld
So what you have in mind is something like calling
simd_relax()every 4096 bytes or so?That was actually pretty easy, putting together both of your suggestions:
static inline bool chacha20_arch(struct
chacha20_ctx*state, u8 *dst,
u8 *src,size_tlen,
simd_context_t*simd_context){ while (len > PAGE_SIZE) { chacha20_arch(state, dst, src, PAGE_SIZE, simd_context); len -= PAGE_SIZE; src += PAGE_SIZE; dst += PAGE_SIZE; simd_relax(simd_context); } if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon &&
Quoted 8 lines
len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context))
chacha20_neon(dst, src, len, state->key, state->counter);
else
chacha20_arm(dst, src, len, state->key, state->counter);
state->counter[0] += (len + 63) / 64;
return true;
}Nice one :-)
This works for me (but perhaps add a comment as well)
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 3 lines · Ard Biesheuvel
Nice one :-)
This works for me (but perhaps add a comment as well)
Sure. Just a prototype; it'll be clean for v7.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Ard BiesheuvelQuoted 26 lines · Ard Biesheuvel
Quoted 24 lines · Jason A. Donenfeld
Quoted 2 lines · Jason A. Donenfeld
So what you have in mind is something like calling
simd_relax()every 4096 bytes or so?That was actually pretty easy, putting together both of your suggestions:
static inline bool chacha20_arch(struct
chacha20_ctx*state, u8 *dst,
u8 *src,size_tlen,
simd_context_t*simd_context){ while (len > PAGE_SIZE) { chacha20_arch(state, dst, src, PAGE_SIZE, simd_context); len -= PAGE_SIZE; src += PAGE_SIZE; dst += PAGE_SIZE; simd_relax(simd_context); } if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context)) chacha20_neon(dst, src, len, state->key, state->counter); else chacha20_arm(dst, src, len, state->key, state->counter);
Quoted 7 lines
Quoted 3 lines
state->counter[0] += (len + 63) / 64; return true; }Nice one :-)
This works for me (but perhaps add a comment as well)
As elegant as my quick recursive solution was, gcc produced kind of bad code from it, as you might expect. So I've implemented this using a boring old loop that works the way it's supposed to. This is marked for v7.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Hi Thomas,
I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:
preempt_disable();
asm volatile(".fill N, 1, 0x90;");preempt_enable();
What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?
Thanks,
Jason
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Hey again Thomas,
Quoted 18 lines · Jason A. Donenfeld
Hi Thomas,
I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:
preempt_disable();
asmvolatile(".fill N, 1, 0x90;");preempt_enable();
Quoted 7 lines
What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?
Thanks,
JasonFrom talking to Peter (now CC'd) on IRC, it sounds like what you're
mostly interested in is clocktime latency on reasonable hardware, with a goal of around ~20?s as a maximum upper bound? I don't expect to get anywhere near this value at all, but if you can confirm that's a decent ballpark, it would make for some interesting calculations.
Regards,
Jason
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 28 lines · Jason A. Donenfeld
Hey again Thomas,
Quoted 22 lines · Jason A. Donenfeld
Hi Thomas,
I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:
preempt_disable();
asmvolatile(".fill N, 1, 0x90;");preempt_enable();What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?
Quoted 10 lines
Quoted 2 lines
Thanks,
JasonFrom talking to Peter (now CC'd) on IRC, it sounds like what you're mostly interested in is clocktime latency on reasonable hardware, with a goal of around ~20?s as a maximum upper bound? I don't expect to get anywhere near this value at all, but if you can confirm that's a decent ballpark, it would make for some interesting calculations.
I would add another consideration: if you can get better latency with negligible overhead (0.1%? 0.05%), then that might make sense too. For example, it seems plausible that checking need_resched() every few blocks adds basically no overhead, and the SIMD helpers could do this themselves or perhaps only ever do a block at a time.
need_resched() costs a cacheline access, but it?s usually a hot cacheline, and the actual check is just whether a certain bit in memory is set.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 3 lines · Andy Lutomirski
I would add another consideration: if you can get better latency with negligible overhead (0.1%? 0.05%), then that might make sense too. For example, it seems plausible that checking
need_resched()every few blocks adds basically no overhead, and the SIMD helpers could do this themselves or perhaps only ever do a block at a time.
need_resched()costs a cacheline access, but it?s usually a hot cacheline, and the actual check is just whether a certain bit in memory is set.
Yes you're right, I do plan to check quite often, rather than seldom, for this reason. I've been toying with the idea of instead processing 65k (maximum size of a UDP packet) at a time before checking need_resched(), but armed with the 20?s figure, this isn't remotely possible on most hardware. So I'll stick with the original conservative plan of checking very often, and not making things different from the aspects worked out by the present crypto API in this regard.
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Ard BiesheuvelQuoted 17 lines · Ard Biesheuvel
(+ Herbert, Thomas)
Quoted 10 lines · Jason A. Donenfeld
Hi Ard,
.And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.
Quoted 8 lines
You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.
Part of the [justified] criticism on the current state of the crypto API is on its complexity, and so I don't think it makes sense to keep it simple now and add the complexity later (and the same concern applies to async support btw).
Are, is what you?re saying that the Zinc chacha20 functions should call simd_relax() every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.
What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an __might_sleep or similar. Or simd_relax should learn to *not* schedule if the result of preempt_enable() leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.
As for async, ISTM a really good WireGuard accelerator would expose a different interface than crypto API supports, and it probably makes sense to wait for such hardware to show up before figuring out how to use it. And no matter what form it takes, I don?t think it should complicate the basic Zinc crypto entry points.
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 3 lines · Andy Lutomirski
Are, is what you?re saying that the Zinc chacha20 functions should call
simd_relax()every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an
__might_sleepor similar. Orsimd_relaxshould learn to *not* schedule if the result ofpreempt_enable()leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.
I'm not sure this is actually a problem. Namely:
preempt_disable();kernel_fpu_begin();kernel_fpu_end();schedule(); <--- bug!
Calling kernel_fpu_end() disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use of preempt_disable won't actually do anything until the outer preempt enable is called:
preempt_disable();kernel_fpu_begin();kernel_fpu_end();preempt_enable();schedule(); <--- works!
Or am I missing some more subtle point?
Jason
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 18 lines · Jason A. Donenfeld
Quoted 3 lines · Andy Lutomirski
Are, is what you?re saying that the Zinc chacha20 functions should call
simd_relax()every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an
__might_sleepor similar. Orsimd_relaxshould learn to *not* schedule if the result ofpreempt_enable()leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.I'm not sure this is actually a problem. Namely:
preempt_disable();kernel_fpu_begin();kernel_fpu_end();schedule(); <--- bug!Calling
kernel_fpu_end()disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use ofpreempt_disablewon't actually do anything until the outer preempt enable is called:
Quoted 8 lines
preempt_disable();kernel_fpu_begin();kernel_fpu_end();preempt_enable();schedule(); <--- works!Or am I missing some more subtle point?
No that seems accurate to me.
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldQuoted 21 lines · Jason A. Donenfeld
Quoted 5 lines · Andy Lutomirski
Are, is what you?re saying that the Zinc chacha20 functions should call
simd_relax()every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an
__might_sleepor similar. Orsimd_relaxshould learn to *not* schedule if the result ofpreempt_enable()leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.I'm not sure this is actually a problem. Namely:
preempt_disable();kernel_fpu_begin();kernel_fpu_end();schedule(); <--- bug!Calling
kernel_fpu_end()disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use ofpreempt_disablewon't actually do anything until the outer preempt enable is called:
Quoted 8 lines
preempt_disable();kernel_fpu_begin();kernel_fpu_end();preempt_enable();schedule(); <--- works!Or am I missing some more subtle point?
No, I think you?re right. I was mid-remembering precisely how simd_relax() worked.
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldQuoted 4 lines
Quoted 3 lines
Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.
Quoted 13 lines
The vast majority of crypto routines in
arch/*/crypto/follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle. I made a change so that inchacha20poly1305.c, it callssimd_relaxafter handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now. And if it becomes one, this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.
Hi Jason
This is not my area of expertise, so you should verify what i'm say here...
My guess is, IPSEC will mostly ask the crypto code to work on 1500 byte full MTU packets and 64 byte TCP ACK packets. Disk encryption i guess works on 4K blocks. So these requests are all quite small, keeping the latency reasonably bounded.
The wireguard interface claims it is GSO capable. This means the network stack will pass it big chunks of data and leave it to the network interface to perform the segmentation into 1500 byte MTU frames on the wire. I've not looked at how wireguard actually handles these big chunks. But to get maximum performance, it should try to keep them whole, just add a header and/or trailer. Will wireguard pass these big chunks of data to the crypto code? Do we now have 64K blocks being worked on? Does the latency jump from 4K to 64K? That might be new, so the existing state of the tree does not help you here.
Andrew
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 9 lines · Andrew Lunn
The wireguard interface claims it is GSO capable. This means the network stack will pass it big chunks of data and leave it to the network interface to perform the segmentation into 1500 byte MTU frames on the wire. I've not looked at how wireguard actually handles these big chunks. But to get maximum performance, it should try to keep them whole, just add a header and/or trailer. Will wireguard pass these big chunks of data to the crypto code? Do we now have 64K blocks being worked on? Does the latency jump from 4K to 64K? That might be new, so the existing state of the tree does not help you here.
No, it only requests GSO superpackets so that it can group the pieces and encrypt them on the same core. But they're each encrypted separately (broken up immediately after ndo_start_xmit), and so they wind up being ~1420 bytes each to encrypt. I spoke about this at netdev2.2 if you're interested in the architecture; there's a paper:
https://www.wireguard.com/papers/wireguard-netdev22.pdf
https://www.youtube.com/watch?v=54orFwtQ1XY
https://www.wireguard.com/talks/netdev2017-slides.pdf
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
In reply to Jason A. DonenfeldQuoted 11 lines · Jason A. Donenfeld
These wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).
This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.
Quoted 3 lines
Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.
Please put comments like this below the ---
Quoted 43 lines
This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:
- ARM1176JZF-S [ARMv6]
- Cortex-A7 [ARMv7] - Cortex-A8 [ARMv7] - Cortex-A9 [ARMv7] - Cortex-A17 [ARMv7] - Cortex-A53 [ARMv8] - Cortex-A55 [ARMv8] - Cortex-A73 [ARMv8] - Cortex-A75 [ARMv8]Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.
Signed-off-by Jason A. Donenfeld <[email protected]>Co-authored-by Eric Biggers <[email protected]>Cc Samuel Neves <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Greg KH <[email protected]>Cc Jean-Philippe Aumasson <[email protected]>Cc Russell King <[email protected]>Cc linux-arm-kernel at lists.infradead.org--- lib/zinc/Makefile | 2 + lib/zinc/chacha20/chacha20-arm-glue.h | 88 +++ ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++-- ...20-arm64-cryptogams.S => chacha20-arm64.S} | 0 lib/zinc/chacha20/chacha20.c | 2 + 5 files changed, 556 insertions(+), 38 deletions(-) create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%) rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)lib/zinc/Makefile +0
-0
lib/zinc/Makefile ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG · line 8
Quoted 5 lines
lib/zinc/Makefile +2
-0@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG zinc_chacha20-y := chacha20/chacha20.o zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
Are these CONFIG_ symbols defined anywhere at this point?
In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases, and the dependency on !CPU_32v3 you introduce (looking at the version of Kconfig at the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel but ARCH_RPC is built in v3 mode because of historical reasons while the actual core is a v4)
Quoted 19 lines
lib/zinc/Makefile +0
-0obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.olib/zinc/chacha20/chacha20-arm-glue.h new file +12
-0@@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* +* Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <asm/hwcap.h> +#include <asm/neon.h> +#if defined(CONFIG_ARM ) +#include <asm/system_info.h> +#include <asm/cputype.h> +#endif +
lib/zinc/chacha20/chacha20-arm-glue.h line 22
Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10
-0+asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#if defined(CONFIG_ARM ) +asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]); +#endif +#if defined(CONFIG_KERNEL_MODE_NEON ) +asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len, +const u32 key[8 ],const u32 counter[4 ]); +#endif +
No need to make asmlinkage declarations conditional
Quoted 16 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +16
-0+static bool chacha20_use_neon__ro_after_init ; + +static void __init chacha20_fpu_init (void ) +{ +#if defined(CONFIG_ARM64 ) + chacha20_use_neon = elf_hwcap &HWCAP_ASIMD ; +#elif defined(CONFIG_ARM ) +switch (read_cpuid_part ()) { +case ARM_CPU_PART_CORTEX_A7 : +case ARM_CPU_PART_CORTEX_A5 : +/* The Cortex-A7 and Cortex-A5 do not perform well with the NEON +* implementation but do incredibly with the scalar one and use +* less power. +*/ +break ; +default :
lib/zinc/chacha20/chacha20-arm-glue.h line 48
Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10
-0+ chacha20_use_neon = elf_hwcap & HWCAP_NEON; + } +#endif +} + +static inline bool chacha20_arch (struct chacha20_ctx *state,u8 *dst, +const u8 *src,size_t len, +simd_context_t *simd_context) +{ +#if defined(CONFIG_KERNEL_MODE_NEON )
if (IS_ENABLED())
Quoted 25 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +25
-0+if (chacha20_use_neon && len >=CHACHA20_BLOCK_SIZE *3 && +simd_use (simd_context)) +chacha20_neon (dst, src, len, state->key, state->counter); + else +#endif +chacha20_arm (dst, src, len, state->key, state->counter); + + state->counter[0 ] += (len +63 ) /64 ; +return true; +} + +static inline bool hchacha20_arch (u32 derived_key[CHACHA20_KEY_WORDS ], +const u8 nonce[HCHACHA20_NONCE_SIZE ], +const u8 key[HCHACHA20_KEY_SIZE ], +simd_context_t *simd_context) +{ +#if defined(CONFIG_ARM ) +u32 x[] = {CHACHA20_CONSTANT_EXPA , +CHACHA20_CONSTANT_ND_3 , +CHACHA20_CONSTANT_2_BY , +CHACHA20_CONSTANT_TE_K , +get_unaligned_le32 (key +0 ), +get_unaligned_le32 (key +4 ), +get_unaligned_le32 (key +8 ), + get_unaligned_le32(key + 12),
lib/zinc/chacha20/chacha20-arm-glue.h line 83
Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10
-0+get_unaligned_le32 (key +16 ), +get_unaligned_le32 (key +20 ), +get_unaligned_le32 (key +24 ), +get_unaligned_le32 (key +28 ), +get_unaligned_le32 (nonce +0 ), +get_unaligned_le32 (nonce +4 ), +get_unaligned_le32 (nonce +8 ), + get_unaligned_le32(nonce + 12) + }; +hchacha20_arm (x, derived_key);
"""
if (!IS_ENABLED(CONFIG_ARM))
return false;
hchacha20_arm(x, derived_key);
return true;
"""
and drop the #ifdefs
Quoted 551 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +5
-0+return true; +#else +return false; +#endif +}lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464
-38@@ -1,13 +1,475 @@ /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ /* + * Copyright (C) 2018 Google, Inc. * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved. * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved. - * - * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS. */ #include <linux/linkage.h> +/* + * The following scalar routine was written by Eric Biggers. + * + * Design notes: + * + * 16 registers would be needed to hold the state matrix, but only 14 are + * available because 'sp' and 'pc' cannot be used. So we spill the elements + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one + * 'ldrd' and one 'strd' instruction per round. + * + * All rotates are performed using the implicit rotate operand accepted by the + * 'add' and 'eor' instructions. This is faster than using explicit rotate + * instructions. To make this work, we allow the values in the second and last + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the + * wrong rotation amount. The rotation amount is then fixed up just in time + * when the values are used. 'brot' is the number of bits the values in row 'b' + * need to be rotated right to arrive at the correct values, and 'drot' + * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such + * that they end up as (25, 24) after every round. + */ + + // ChaCha state registers + X0 .req r0 + X1 .req r1 + X2 .req r2 + X3 .req r3 + X4 .req r4 + X5 .req r5 + X6 .req r6 + X7 .req r7 + X8_X10 .req r8 // shared by x8 and x10 + X9_X11 .req r9 // shared by x9 and x11 + X12 .req r10 + X13 .req r11 + X14 .req r12 + X15 .req r14 + +.Lexpand_32byte_k: + // "expand 32-byte k" + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574 + +#ifdef __thumb2__ +# define adrl adr +#endif + +.macro __rev out, in, t0, t1, t2 +.if __LINUX_ARM_ARCH__ >= 6 + rev \out, \in +.else + lsl \t0, \in, #24 + and \t1, \in, #0xff00 + and \t2, \in, #0xff0000 + orr \out, \t0, \in, lsr #24 + orr \out, \out, \t1, lsl #8 + orr \out, \out, \t2, lsr #8 +.endif +.endm + +.macro _le32_bswap x, t0, t1, t2 +#ifdef __ARMEB__ + __rev \x, \x, \t0, \t1, \t2 +#endif +.endm + +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2 + _le32_bswap \a, \t0, \t1, \t2 + _le32_bswap \b, \t0, \t1, \t2 + _le32_bswap \c, \t0, \t1, \t2 + _le32_bswap \d, \t0, \t1, \t2 +.endm + +.macro __ldrd a, b, src, offset +#if __LINUX_ARM_ARCH__ >= 6 + ldrd \a, \b, [\src, #\offset] +#else + ldr \a, [\src, #\offset] + ldr \b, [\src, #\offset + 4] +#endif +.endm + +.macro __strd a, b, dst, offset +#if __LINUX_ARM_ARCH__ >= 6 + strd \a, \b, [\dst, #\offset] +#else + str \a, [\dst, #\offset] + str \b, [\dst, #\offset + 4] +#endif +.endm + +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2 + + // a += b; d ^= a; d = rol(d, 16); + add \a1, \a1, \b1, ror #brot + add \a2, \a2, \b2, ror #brot + eor \d1, \a1, \d1, ror #drot + eor \d2, \a2, \d2, ror #drot + // drot == 32 - 16 == 16 + + // c += d; b ^= c; b = rol(b, 12); + add \c1, \c1, \d1, ror #16 + add \c2, \c2, \d2, ror #16 + eor \b1, \c1, \b1, ror #brot + eor \b2, \c2, \b2, ror #brot + // brot == 32 - 12 == 20 + + // a += b; d ^= a; d = rol(d, 8); + add \a1, \a1, \b1, ror #20 + add \a2, \a2, \b2, ror #20 + eor \d1, \a1, \d1, ror #16 + eor \d2, \a2, \d2, ror #16 + // drot == 32 - 8 == 24 + + // c += d; b ^= c; b = rol(b, 7); + add \c1, \c1, \d1, ror #24 + add \c2, \c2, \d2, ror #24 + eor \b1, \c1, \b1, ror #20 + eor \b2, \c2, \b2, ror #20 + // brot == 32 - 7 == 25 +.endm + +.macro _doubleround + + // column round + + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13) + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13 + + // save (x8, x9); restore (x10, x11) + __strd X8_X10, X9_X11, sp, 0 + __ldrd X8_X10, X9_X11, sp, 8 + + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15) + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15 + + .set brot, 25 + .set drot, 24 + + // diagonal round + + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12) + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12 + + // save (x10, x11); restore (x8, x9) + __strd X8_X10, X9_X11, sp, 8 + __ldrd X8_X10, X9_X11, sp, 0 + + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14) + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14 +.endm + +.macro _chacha_permute nrounds + .set brot, 0 + .set drot, 0 + .rept \nrounds / 2 + _doubleround + .endr +.endm + +.macro _chacha nrounds + +.Lnext_block\@: + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + + // Do the core ChaCha permutation to update x0-x15. + _chacha_permute \nrounds + + add sp, #8 + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers contain x0-x9,x12-x15. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15). + push {X8_X10, X9_X11, X12, X13, X14, X15} + + // Load (OUT, IN, LEN). + ldr r14, [sp, #96] + ldr r12, [sp, #100] + ldr r11, [sp, #104] + + orr r10, r14, r12 + + // Use slow path if fewer than 64 bytes remain. + cmp r11, #64 + blt .Lxor_slowpath\@ + + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on + // ARMv6+, since ldmia and stmia (used below) still require alignment. + tst r10, #3 + bne .Lxor_slowpath\@ + + // Fast path: XOR 64 bytes of aligned data. + + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // x0-x3 + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + ldmia r12!, {r8-r11} + eor X0, X0, r8 + eor X1, X1, r9 + eor X2, X2, r10 + eor X3, X3, r11 + stmia r14!, {X0-X3} + + // x4-x7 + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + ldmia r12!, {X0-X3} + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + eor X4, X4, X0 + eor X5, X5, X1 + eor X6, X6, X2 + eor X7, X7, X3 + stmia r14!, {X4-X7} + + // x8-x15 + pop {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 32 + __ldrd r10, r11, sp, 40 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + ldmia r12!, {r8-r11} + eor r0, r0, r8 // x8 + eor r1, r1, r9 // x9 + eor r6, r6, r10 // x10 + eor r7, r7, r11 // x11 + stmia r14!, {r0,r1,r6,r7} + ldmia r12!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 48 + __ldrd r10, r11, sp, 56 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + ldr r9, [sp, #72] // load LEN + eor r2, r2, r0 // x12 + eor r3, r3, r1 // x13 + eor r4, r4, r6 // x14 + eor r5, r5, r7 // x15 + subs r9, #64 // decrement and check LEN + stmia r14!, {r2-r5} + + beq .Ldone\@ + +.Lprepare_for_next_block\@: + + // Stack: x0-x15 OUT IN LEN + + // Increment block counter (x12) + add r8, #1 + + // Store updated (OUT, IN, LEN) + str r14, [sp, #64] + str r12, [sp, #68] + str r9, [sp, #72] + + mov r14, sp + + // Store updated block counter (x12) + str r8, [sp, #48] + + sub sp, #16 + + // Reload state and do next block + ldmia r14!, {r0-r11} // load x0-x11 + __strd r10, r11, sp, 8 // store x10-x11 before state + ldmia r14, {r10-r12,r14} // load x12-x15 + b .Lnext_block\@ + +.Lxor_slowpath\@: + // Slow path: < 64 bytes remaining, or unaligned input or output buffer. + // We handle it by storing the 64 bytes of keystream to the stack, then + // XOR-ing the needed portion with the data. + + // Allocate keystream buffer + sub sp, #64 + mov r14, sp + + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0. + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'. + + // Save keystream for x0-x3 + __ldrd r8, r9, sp, 96 + __ldrd r10, r11, sp, 104 + add X0, X0, r8 + add X1, X1, r9 + add X2, X2, r10 + add X3, X3, r11 + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10 + stmia r14!, {X0-X3} + + // Save keystream for x4-x7 + __ldrd r8, r9, sp, 112 + __ldrd r10, r11, sp, 120 + add X4, r8, X4, ror #brot + add X5, r9, X5, ror #brot + add X6, r10, X6, ror #brot + add X7, r11, X7, ror #brot + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10 + add r8, sp, #64 + stmia r14!, {X4-X7} + + // Save keystream for x8-x15 + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11) + __ldrd r8, r9, sp, 128 + __ldrd r10, r11, sp, 136 + add r0, r0, r8 // x8 + add r1, r1, r9 // x9 + add r6, r6, r10 // x10 + add r7, r7, r11 // x11 + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10 + stmia r14!, {r0,r1,r6,r7} + __ldrd r8, r9, sp, 144 + __ldrd r10, r11, sp, 152 + add r2, r8, r2, ror #drot // x12 + add r3, r9, r3, ror #drot // x13 + add r4, r10, r4, ror #drot // x14 + add r5, r11, r5, ror #drot // x15 + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11 + stmia r14, {r2-r5} + + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN + // Registers: r8 is block counter, r12 is IN. + + ldr r9, [sp, #168] // LEN + ldr r14, [sp, #160] // OUT + cmp r9, #64 + mov r0, sp + movle r1, r9 + movgt r1, #64 + // r1 is number of bytes to XOR, in range [1, 64] + +.if __LINUX_ARM_ARCH__ < 6 + orr r2, r12, r14 + tst r2, #3 // IN or OUT misaligned? + bne .Lxor_next_byte\@ +.endif + + // XOR a word at a time +.rept 16 + subs r1, #4 + blt .Lxor_words_done\@ + ldr r2, [r12], #4 + ldr r3, [r0], #4 + eor r2, r2, r3 + str r2, [r14], #4 +.endr + b .Lxor_slowpath_done\@ +.Lxor_words_done\@: + ands r1, r1, #3 + beq .Lxor_slowpath_done\@ + + // XOR a byte at a time +.Lxor_next_byte\@: + ldrb r2, [r12], #1 + ldrb r3, [r0], #1 + eor r2, r2, r3 + strb r2, [r14], #1 + subs r1, #1 + bne .Lxor_next_byte\@ + +.Lxor_slowpath_done\@: + subs r9, #64 + add sp, #96 + bgt .Lprepare_for_next_block\@ + +.Ldone\@: +.endm // _chacha + +/* + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8], + * const u32 iv[4]); + */ +ENTRY(chacha20_arm) + cmp r2, #0 // len == 0? + bxeq lr + + push {r0-r2,r4-r11,lr} + + // Push state x0-x15 onto stack. + // Also store an extra copy of x10-x11 just before the state. + + ldr r4, [sp, #48] // iv + mov r0, sp + sub sp, #80 + + // iv: x12-x15 + ldm r4, {X12,X13,X14,X15} + stmdb r0!, {X12,X13,X14,X15} + + // key: x4-x11 + __ldrd X8_X10, X9_X11, r3, 24 + __strd X8_X10, X9_X11, sp, 8 + stmdb r0!, {X8_X10, X9_X11} + ldm r3, {X4-X9_X11} + stmdb r0!, {X4-X9_X11} + + // constants: x0-x3 + adrl X3, .Lexpand_32byte_k + ldm X3, {X0-X3} + __strd X0, X1, sp, 16 + __strd X2, X3, sp, 24 + + _chacha 20 + + add sp, #76 + pop {r4-r11, pc} +ENDPROC(chacha20_arm) + +/* + * void hchacha20_arm(const u32 state[16], u32 out[8]); + */ +ENTRY(hchacha20_arm) + push {r1,r4-r11,lr} + + mov r14, r0 + ldmia r14!, {r0-r11} // load x0-x11 + push {r10-r11} // store x10-x11 to stack + ldm r14, {r10-r12,r14} // load x12-x15 + sub sp, #8 + + _chacha_permute 20 + + // Skip over (unused0-unused1, x10-x11) + add sp, #16 + + // Fix up rotations of x12-x15 + ror X12, X12, #drot + ror X13, X13, #drot + pop {r4} // load 'out' + ror X14, X14, #drot + ror X15, X15, #drot + + // Store (x0-x3,x12-x15) to 'out' + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15} + + pop {r4-r11,pc} +ENDPROC(hchacha20_arm) + +#ifdef CONFIG_KERNEL_MODE_NEON +/* + * This following NEON routine was ported from Andy Polyakov's implementation + * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine, + * since certain NEON code paths actually branch to it. + */ + .text #if defined(__thumb2__) || defined(__clang__) .syntax unified@@ -22,39 +484,6 @@ #define ldrhsb ldrbhs #endif -.align 5 -.Lsigma: -.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral -.Lone: -.long 1,0,0,0 -.word -1 - -.align 5 -ENTRY(chacha20_arm) - ldr r12,[sp,#0] @ pull pointer to counter and nonce - stmdb sp!,{r0-r2,r4-r11,lr} - cmp r2,#0 @ len==0? -#ifdef __thumb2__ - itt eq -#endif - addeq sp,sp,#4*3 - beq .Lno_data_arm - ldmia r12,{r4-r7} @ load counter and nonce - sub sp,sp,#4*(16) @ off-load area -#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__) - sub r14,pc,#100 @ .Lsigma -#else - adr r14,.Lsigma @ .Lsigma -#endif - stmdb sp!,{r4-r7} @ copy counter and nonce - ldmia r3,{r4-r11} @ load key - ldmia r14,{r0-r3} @ load sigma - stmdb sp!,{r4-r11} @ copy key - stmdb sp!,{r0-r3} @ copy sigma - str r10,[sp,#4*(16+10)] @ off-load "rx" - str r11,[sp,#4*(16+11)] @ off-load "rx" - b .Loop_outer_enter - .align 4 .Loop_outer: ldmia sp,{r0-r9} @ load key material@@ -748,11 +1177,8 @@ ENTRY(chacha20_arm) .Ldone: add sp,sp,#4*(32+3) -.Lno_data_arm: ldmia sp!,{r4-r11,pc} -ENDPROC(chacha20_arm) -#ifdef CONFIG_KERNEL_MODE_NEON .align 5 .Lsigma2: .long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutrallib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0
-0lib/zinc/chacha20/chacha20.c +0
-0
lib/zinc/chacha20/chacha20.c line 19
Quoted 4 lines
lib/zinc/chacha20/chacha20.c +1
-0@@ -16,6 +16,8 @@#if defined(CONFIG_ZINC_ARCH_X86_64 )#include "chacha20-x86_64-glue.h" +#elif defined(CONFIG_ZINC_ARCH_ARM ) || defined(CONFIG_ZINC_ARCH_ARM64 )
As above, just use CONFIG_ARM / CONFIG_ARM64 directly
Quoted 7 lines
lib/zinc/chacha20/chacha20.c +1
-1+#include "chacha20-arm-glue.h" #else void __init chacha20_fpu_init (void ) { --2.19.0
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Hi Ard,
Quoted 1 line · Ard Biesheuvel
Please put comments like this below the ---
git-notes is nice for this indeed.
Quoted 1 line
Are these
CONFIG_symbols defined anywhere at this point?
Yes, they're introduced in the first zinc commit. There's no git-blame on git.kernel.org, presumably because it's expensive to compute, but there is on my personal instance, so this might help:
https://git.zx2c4.com/linux-dev/blame/lib/zinc/Kconfig?h=jd/wireguard
Quoted 2 lines
In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases
You mean to say that since these nobs are def_bool y and are essentially "depends on ARM", then I should just straight up use CONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.
Quoted 6 lines
and the
dependency on !CPU_32v3 you introduce (looking at the version ofKconfigat the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel butARCH_RPCis built in v3 mode because of historical reasons while the actual core is a v4)
I added the !CPU_32v3 in my development tree after posting v6, so good to hear you're just looking straight at the updated tree. If you see things jump out in there prior to me posting v7, don't hesitate to let me know.
The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected, ARCH_RPC. Have a better suggestion than !CPU_32v3? It seems to me like so long as the kernel has CPU_32v3 as a thing in any form, I should mark Zinc as not supporting it, since we'll certainly be at least v4 and up. (Do you guys have any old Acorn ARM610 boxes sitting around for old time's sake at LinaroHQ? ;-)
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+#endif +No need to make asmlinkage declarations conditional
Yep, addressed in the IS_ENABLED cleanup.
Quoted 2 lines
if (
IS_ENABLED())
Sorted.
Quoted 10 lines
"""
if (!IS_ENABLED(CONFIG_ARM))
return false;
hchacha20_arm(x, derived_key);
return true;
"""and drop the #ifdefs
Also sorted.
Regards,
Jason
Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Quoted 18 lines · Jason A. Donenfeld
Hi Ard,
Quoted 1 line · Ard Biesheuvel
Please put comments like this below the ---
git-notes is nice for this indeed.
Quoted 1 line
Are these
CONFIG_symbols defined anywhere at this point?Yes, they're introduced in the first zinc commit. There's no git-blame on git.kernel.org, presumably because it's expensive to compute, but there is on my personal instance, so this might help:
https://git.zx2c4.com/linux-dev/blame/lib/zinc/Kconfig?h=jd/wireguardQuoted 2 lines
In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases
Quoted 9 lines
You mean to say that since these nobs are
def_booly and are essentially "depends on ARM", then I should just straight up useCONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.
Sure. But probably better to be consistent then, and stop using CONFIG_ARM directly in your code.
Quoted 12 lines
Quoted 6 lines
and the
dependency on !CPU_32v3 you introduce (looking at the version ofKconfigat the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel butARCH_RPCis built in v3 mode because of historical reasons while the actual core is a v4)I added the !CPU_32v3 in my development tree after posting v6, so good to hear you're just looking straight at the updated tree. If you see things jump out in there prior to me posting v7, don't hesitate to let me know.
Quoted 4 lines
The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected,ARCH_RPC. Have a better suggestion than !CPU_32v3?
Yes, you could just add
asflags-$(CONFIG_CPU_32v3) += -march=armv4
with a comment stating that we don't actually support ARMv3 but only use it as a code generation target for reasons unrelated to the ISA
Quoted 5 lines
It seems to me like so long as the kernel has
CPU_32v3as a thing in any form, I should mark Zinc as not supporting it, since we'll certainly be at least v4 and up. (Do you guys have any old Acorn ARM610 boxes sitting around for old time's sake at LinaroHQ? ;-)
AFAIK we only support the StrongARM flavor of RiscPC which is ARMv4. But yes, some people do care deeply about these antiquated platforms, including Russell, and there is no particular to leave this one behind.
Quoted 27 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+#endif +No need to make asmlinkage declarations conditional
Yep, addressed in the
IS_ENABLEDcleanup.Quoted 2 lines
if (
IS_ENABLED())Sorted.
Quoted 10 lines
"""
if (!IS_ENABLED(CONFIG_ARM))
return false;
hchacha20_arm(x, derived_key);
return true;
"""and drop the #ifdefs
Also sorted.
Regards,
Jason
Jason A. Donenfeld
maintainerRe: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
Hi Ard,
Quoted 11 lines · Ard Biesheuvel
Quoted 8 lines
You mean to say that since these nobs are
def_booly and are essentially "depends on ARM", then I should just straight up useCONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.Sure. But probably better to be consistent then, and stop using
CONFIG_ARMdirectly in your code.
Ack.
Quoted 11 lines
Quoted 4 lines
The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected,ARCH_RPC. Have a better suggestion than !CPU_32v3?Yes, you could just add
asflags-$(CONFIG_CPU_32v3) += -march=armv4
with a comment stating that we don't actually support ARMv3 but only use it as a code generation target for reasons unrelated to the ISA
Alright, I'll do exactly that. Though, if the rationale for this has to do only with codegen -- with what the C compiler does -- then shouldn't this be set globally for CONFIG_CPU_32v3? I couldn't find any macros that test against __LINUX_ARM_ARCH__ being 3 in the assembly, so this shouldn't be a problem I don't think. Maybe I'll send a patch.
Jason