Skip to contentSkip to conversation

WireGuard: Secure Network Tunnel

PATCH Superseded Jason A. Donenfeld 16 participants 146 messages Sep 25, 2018 – Nov 13, 2018 Page 1 of 2

Subsystems (30)

v6 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

v6 cover WireGuard: Secure Network Tunnel

Changes v5->v6, along with who suggested it.
--------------------------------------------
- [Eric Biggers] Cleaner and more efficient blake2s_final function.
- [Eric Biggers] Remove modulo trick that was working around a gcc 8.1 bug.
- [Eric Biggers] Use crypto_xor_cpy to avoid a memmove in chacha20.
- In the unlikely condition that we transition from SIMD back to scalar instructions for Poly1305, it's important that we also convert back from base 2^26 to 2^64. This is super rare and weird, and it's hard to imagine somebody actually doing this for whatever bad reason, but it is a possibility, so we shouldn't calculate things wrong in that instance.
- [Andrew Lunn] Change BUG_ON to WARN_ON in choice places.
- [Thomas Gleixner] Explicitly dual license files under X where X≠GPL2

    to be under X && GPL2.
  - [Andy Lutomirski] Use separate symbols for selftests that are of
    variable length, so that we neither waste space on disk nor waste space
    in memory (via __initconst).
  - [Thomas Gleixner] Make SPDX headers a standalone comment, and use the
    // commenting style in .h files, per the kernel style guide.
  - [Paul Burton] Allow assembler to fill branch delay slots on MIPS32r2,
    so that the implementation is more future-proof.
  - [Eric Biggers] Use Eric's scalar implementation for ChaCha20 ARM, which
    performs very well for Cortex-A7,5 and ARMv6, while using Andy
    Polyakov's NEON implementation for other ARMv7.
  - [Ard Biesheuvel] Reduce optimization from -O3 to -O2.
  - [Arn?d Bi?e(rgmann|shuvel)] Make sure stack frames stay inside 1024
    bytes on 32-bit and 1280 bytes on 64-bit.

- [Ard Biesheuvel] CRYPTOGAMS-related commit messages now have the corresponding OpenSSL commit written in them.
- [Eric Biggers] Keep HChaCha20 key in native endian, since in some cases it can trivially be passed into the ChaCha20 block later.
- Make constants follow a consistent naming scheme.
- Workaround gcc 8 stack mis-optimization on m68k.
- [Arnd Bergmann] Workaround gcc 8 stack mis-optimization with KASAN.

Many of the above ARM changes are a result of discussions between Ard, Eric, AndyL, AndyP, Samuel, and me. These discussions are ongoing, and so it's possible we'll do another revision with further ARM tweaks. But perhaps this one will be sufficient for merging now, and we can continue to refine later in the cycle.

-----------------------------------------------------------

This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:

* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

-----------------------------------------------------------

WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.

Ample information, including documentation, installation instructions, and project details, is available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:

* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/

The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:

* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf

The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:

* https://www.wireguard.com/formal-verification/

This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.

It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.

Enjoy,
Jason

author maintainer
#1 lore raw

PATCH v6 1/23 asm: simd context helper API

Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:

    simd_context_t simd_context;

    simd_get(&simd_context);
    while ((item = get_item_from_queue()) != NULL) {
        encrypt_item(item, simd_context);
        simd_relax(&simd_context);
    }
    simd_put(&simd_context);

The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.

On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:

   void encrypt_item(struct item *item, simd_context_t *simd_context)
   {
       if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
           wild_simd_code(item);
       else
           boring_scalar_code(item);
   }

The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Greg KH <[email protected]>
---
 arch/alpha/include/asm/Kbuild      |  5 ++-
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/simd.h        | 63 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/simd.h      | 51 +++++++++++++++++++++---
 arch/c6x/include/asm/Kbuild        |  3 +-
 arch/h8300/include/asm/Kbuild      |  3 +-
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/nds32/include/asm/Kbuild      |  7 ++--
 arch/nios2/include/asm/Kbuild      |  1 +
 arch/openrisc/include/asm/Kbuild   |  7 ++--
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  3 +-
 arch/riscv/include/asm/Kbuild      |  3 +-
 arch/s390/include/asm/Kbuild       |  3 +-
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/um/include/asm/Kbuild         |  3 +-
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/simd.h        | 44 ++++++++++++++++++++-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/simd.h         | 20 ++++++++++
 include/linux/simd.h               | 28 +++++++++++++
 26 files changed, 234 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm/include/asm/simd.h
 create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +3 -2
index 0580cb8c84b2..07b2c1025d34 100644
@@ -2,14 +2,15 @@
 
 
 generic-y += compat.h
+generic-y += current.h
 generic-y += exec.h
 generic-y += export.h
 generic-y += fb.h
 generic-y += irq_work.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h
arch/arc/include/asm/Kbuild +1 -0
index feed50ce89fa..a7f4255f1649 100644
@@ -22,6 +22,7 @@ generic-y += parport.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/arm/include/asm/simd.h new file +63 -0

63 lines not on this page · raw message

arch/arm64/include/asm/simd.h +45 -6

71 lines not on this page · raw message

arch/c6x/include/asm/Kbuild +2 -1
index 33a2c94fed0d..22f3d8333c74 100644
@@ -5,8 +5,8 @@ generic-y += compat.h
 generic-y += current.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h
 generic-y += preempt.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += tlbflush.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
index a5d0b2991f47..f5c2f12d593e 100644
@@ -8,8 +8,8 @@ generic-y += current.h
 generic-y += delay.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h
 generic-y += scatterlist.h
 generic-y += sections.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += spinlock.h
 generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
index dd2fd9c0d292..217d4695fd8a 100644
@@ -29,6 +29,7 @@ generic-y += rwsem.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
index 557bbc8ba9f5..41c5ebdf79e5 100644
@@ -4,6 +4,7 @@ generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += vtime.h
 generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
index a4b8d3331a9e..73898dd1a4d0 100644
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
index 569ba9e670c1..7a877eea99d3 100644
@@ -25,6 +25,7 @@ generic-y += parport.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += syscalls.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
index 58351e48421e..e8868e0fb2c3 100644
@@ -16,6 +16,7 @@ generic-y += qrwlock.h
 generic-y += qspinlock.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
index dbc4e5422550..603c1d020620 100644
@@ -7,14 +7,14 @@ generic-y += bug.h
 generic-y += bugs.h
 generic-y += checksum.h
 generic-y += clkdev.h
-generic-y += cmpxchg.h
 generic-y += cmpxchg-local.h
+generic-y += cmpxchg.h
 generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
 generic-y += shmbuf.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += stat.h
 generic-y += switch_to.h
 generic-y += timex.h
 generic-y += topology.h
 generic-y += trace_clock.h
-generic-y += xor.h
 generic-y += unaligned.h
 generic-y += user.h
 generic-y += vga.h
 generic-y += word-at-a-time.h
+generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
index 8fde4fa2c34f..571a9d9ad107 100644
@@ -33,6 +33,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
index eb87cd8327c8..5e9f2f4c4d39 100644
@@ -28,12 +28,13 @@ generic-y += module.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
-generic-y += qspinlock_types.h
-generic-y += qspinlock.h
-generic-y += qrwlock_types.h
 generic-y += qrwlock.h
+generic-y += qrwlock_types.h
+generic-y += qspinlock.h
+generic-y += qspinlock_types.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += string.h
 generic-y += switch_to.h
 generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
index 2013d639e735..97970b4d05ab 100644
@@ -17,6 +17,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += seccomp.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
index 3196d227e351..64290f48e733 100644
@@ -4,7 +4,8 @@ generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
+generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += vtime.h
-generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
index efdbe311e936..6669b7374c0a 100644
@@ -5,9 +5,9 @@ generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h
 generic-y += shmbuf.h
 generic-y += shmparam.h
 generic-y += signal.h
+generic-y += simd.h
 generic-y += socket.h
 generic-y += sockios.h
 generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
index e3239772887a..7a26dc6ce815 100644
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h
 generic-y += asm-offsets.h
 generic-y += cacheflush.h
 generic-y += device.h
+generic-y += div64.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
-generic-y += div64.h
 generic-y += emergency-restart.h
 generic-y += export.h
 generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
index 6a5609a55965..8e64ff35a933 100644
@@ -16,6 +16,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += trace_clock.h
 generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
index 410b263ef5c8..72b9e08fb350 100644
@@ -17,5 +17,6 @@ generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
index b10dde6cb793..d37288b08dd2 100644
@@ -16,15 +16,16 @@ generic-y += io.h
 generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += kdebug.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += param.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += switch_to.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
 generic-y += xor.h
-generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
index bfc7abe77905..98a908720bbd 100644
@@ -27,6 +27,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += syscalls.h
 generic-y += topology.h
arch/x86/include/asm/simd.h +43 -1

50 lines not on this page · raw message

arch/xtensa/include/asm/Kbuild +1 -0
index 82c756431b49..7950f359649d 100644
@@ -24,6 +24,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
include/asm-generic/simd.h +20 -0
index d0343d58a74a..b3dd61ac010e 100644
@@ -1,5 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
+
 #include <linux/hardirq.h>
 
 /*
@@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void)
 {
 	return !in_interrupt();
 }
+
+static inline void simd_get(simd_context_t *ctx)
+{
+	*ctx = HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+	return false;
+}
+
+#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +28 -0
new file mode 100644
index 000000000000..33bba21012ff
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _SIMD_H
+#define _SIMD_H
+
+typedef enum {
+	HAVE_NO_SIMD = 1 << 0,
+	HAVE_FULL_SIMD = 1 << 1,
+	HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+
+#include <linux/sched.h>
+#include <asm/simd.h>
+
+static inline void simd_relax(simd_context_t *ctx)
+{
+#ifdef CONFIG_PREEMPT
+	if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
+		simd_put(ctx);
+		simd_get(ctx);
+	}
+#endif
+}
+
+#endif /* _SIMD_H */
Signature · 1 line
2.19.0
#2 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 80 lines · Jason A. Donenfeld

Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:

    simd_context_t simd_context;

    simd_get(&simd_context);
    while ((item = get_item_from_queue()) != NULL) {
        encrypt_item(item, simd_context);
        simd_relax(&simd_context);
    }
    simd_put(&simd_context);

The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.

On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:

   void encrypt_item(struct item *item, simd_context_t *simd_context)
   {
       if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
           wild_simd_code(item);
       else
           boring_scalar_code(item);
   }

The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Greg KH <[email protected]>
---
 arch/alpha/include/asm/Kbuild      |  5 ++-
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/simd.h        | 63 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/simd.h      | 51 +++++++++++++++++++++---
 arch/c6x/include/asm/Kbuild        |  3 +-
 arch/h8300/include/asm/Kbuild      |  3 +-
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/nds32/include/asm/Kbuild      |  7 ++--
 arch/nios2/include/asm/Kbuild      |  1 +
 arch/openrisc/include/asm/Kbuild   |  7 ++--
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  3 +-
 arch/riscv/include/asm/Kbuild      |  3 +-
 arch/s390/include/asm/Kbuild       |  3 +-
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/um/include/asm/Kbuild         |  3 +-
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/simd.h        | 44 ++++++++++++++++++++-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/simd.h         | 20 ++++++++++
 include/linux/simd.h               | 28 +++++++++++++
 26 files changed, 234 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm/include/asm/simd.h
 create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +1 -0
index 0580cb8c84b2..07b2c1025d34 100644
@@ -2,14 +2,15 @@
 
 
 generic-y += compat.h
+generic-y += current.h
 generic-y += exec.h
 generic-y += export.h
 generic-y += fb.h

arch/alpha/include/asm/Kbuild line 15

Quoted 10 lines
arch/alpha/include/asm/Kbuild +2 -2
 generic-y += irq_work.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h

Given that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.

Quoted 25 lines
arch/arc/include/asm/Kbuild +1 -0
index feed50ce89fa..a7f4255f1649 100644
@@ -22,6 +22,7 @@ generic-y += parport.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/arm/include/asm/simd.h new file +7 -0
new file mode 100644
index 000000000000..263950dd69cb
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H

arch/arm/include/asm/simd.h line 17

Quoted 10 lines
arch/arm/include/asm/simd.h new file +10 -0
+#define _ASM_SIMD_H
+
+#ifdef CONFIG_KERNEL_MODE_NEON
+#include <asm/neon.h>
+
+static __must_check inline bool may_use_simd(void)
+{
+       return !in_interrupt();
+}
+

Remember this guy?

https://marc.info/?l=linux-arch&m=149631094625176&w=2

That was never merged, so let's get it right this time.

Quoted 551 lines
arch/arm/include/asm/simd.h new file +46 -0
+static inline void simd_get(simd_context_t *ctx)
+{
+       *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+       if (*ctx & HAVE_SIMD_IN_USE)
+               kernel_neon_end();
+       *ctx = HAVE_NO_SIMD;
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+       if (!(*ctx & HAVE_FULL_SIMD))
+               return false;
+       if (*ctx & HAVE_SIMD_IN_USE)
+               return true;
+       kernel_neon_begin();
+       *ctx |= HAVE_SIMD_IN_USE;
+       return true;
+}
+
+#else
+
+static __must_check inline bool may_use_simd(void)
+{
+       return false;
+}
+
+static inline void simd_get(simd_context_t *ctx)
+{
+       *ctx = HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+       return false;
+}
+#endif
+
+#endif /* _ASM_SIMD_H */
arch/arm64/include/asm/simd.h +45 -6
index 6495cc51246f..a45ff1600040 100644
@@ -1,11 +1,10 @@
-/*
- * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+/* SPDX-License-Identifier: GPL-2.0
  *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published
- * by the Free Software Foundation.
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  */
 
+#include <linux/simd.h>
 #ifndef __ASM_SIMD_H
 #define __ASM_SIMD_H
 
@@ -16,6 +15,8 @@
 #include <linux/types.h>
 
 #ifdef CONFIG_KERNEL_MODE_NEON
+#include <asm/neon.h>
+#include <asm/simd.h>
 
 DECLARE_PER_CPU(bool, kernel_neon_busy);
 
@@ -40,9 +41,47 @@ static __must_check inline bool may_use_simd(void)
                !this_cpu_read(kernel_neon_busy);
 }
 
+static inline void simd_get(simd_context_t *ctx)
+{
+       *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+       if (*ctx & HAVE_SIMD_IN_USE)
+               kernel_neon_end();
+       *ctx = HAVE_NO_SIMD;
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+       if (!(*ctx & HAVE_FULL_SIMD))
+               return false;
+       if (*ctx & HAVE_SIMD_IN_USE)
+               return true;
+       kernel_neon_begin();
+       *ctx |= HAVE_SIMD_IN_USE;
+       return true;
+}
+
 #else /* ! CONFIG_KERNEL_MODE_NEON */
 
-static __must_check inline bool may_use_simd(void) {
+static __must_check inline bool may_use_simd(void)
+{
+       return false;
+}
+
+static inline void simd_get(simd_context_t *ctx)
+{
+       *ctx = HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
        return false;
 }
 
arch/c6x/include/asm/Kbuild +2 -1
index 33a2c94fed0d..22f3d8333c74 100644
@@ -5,8 +5,8 @@ generic-y += compat.h
 generic-y += current.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h
 generic-y += preempt.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += tlbflush.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
index a5d0b2991f47..f5c2f12d593e 100644
@@ -8,8 +8,8 @@ generic-y += current.h
 generic-y += delay.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h
 generic-y += scatterlist.h
 generic-y += sections.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += spinlock.h
 generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
index dd2fd9c0d292..217d4695fd8a 100644
@@ -29,6 +29,7 @@ generic-y += rwsem.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
index 557bbc8ba9f5..41c5ebdf79e5 100644
@@ -4,6 +4,7 @@ generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += vtime.h
 generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
index a4b8d3331a9e..73898dd1a4d0 100644
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
index 569ba9e670c1..7a877eea99d3 100644
@@ -25,6 +25,7 @@ generic-y += parport.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += syscalls.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
index 58351e48421e..e8868e0fb2c3 100644
@@ -16,6 +16,7 @@ generic-y += qrwlock.h
 generic-y += qspinlock.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
index dbc4e5422550..603c1d020620 100644
@@ -7,14 +7,14 @@ generic-y += bug.h
 generic-y += bugs.h
 generic-y += checksum.h
 generic-y += clkdev.h
-generic-y += cmpxchg.h
 generic-y += cmpxchg-local.h
+generic-y += cmpxchg.h
 generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
 generic-y += shmbuf.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += stat.h
 generic-y += switch_to.h
 generic-y += timex.h
 generic-y += topology.h
 generic-y += trace_clock.h
-generic-y += xor.h
 generic-y += unaligned.h
 generic-y += user.h
 generic-y += vga.h
 generic-y += word-at-a-time.h
+generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
index 8fde4fa2c34f..571a9d9ad107 100644
@@ -33,6 +33,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
index eb87cd8327c8..5e9f2f4c4d39 100644
@@ -28,12 +28,13 @@ generic-y += module.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
-generic-y += qspinlock_types.h
-generic-y += qspinlock.h
-generic-y += qrwlock_types.h
 generic-y += qrwlock.h
+generic-y += qrwlock_types.h
+generic-y += qspinlock.h
+generic-y += qspinlock_types.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += string.h
 generic-y += switch_to.h
 generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
index 2013d639e735..97970b4d05ab 100644
@@ -17,6 +17,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += seccomp.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
index 3196d227e351..64290f48e733 100644
@@ -4,7 +4,8 @@ generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
+generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += vtime.h
-generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
index efdbe311e936..6669b7374c0a 100644
@@ -5,9 +5,9 @@ generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h
 generic-y += shmbuf.h
 generic-y += shmparam.h
 generic-y += signal.h
+generic-y += simd.h
 generic-y += socket.h
 generic-y += sockios.h
 generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
index e3239772887a..7a26dc6ce815 100644
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h
 generic-y += asm-offsets.h
 generic-y += cacheflush.h
 generic-y += device.h
+generic-y += div64.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
-generic-y += div64.h
 generic-y += emergency-restart.h
 generic-y += export.h
 generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
index 6a5609a55965..8e64ff35a933 100644
@@ -16,6 +16,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += trace_clock.h
 generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
index 410b263ef5c8..72b9e08fb350 100644
@@ -17,5 +17,6 @@ generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
index b10dde6cb793..d37288b08dd2 100644
@@ -16,15 +16,16 @@ generic-y += io.h
 generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += kdebug.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += param.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += switch_to.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
 generic-y += xor.h
-generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
index bfc7abe77905..98a908720bbd 100644
@@ -27,6 +27,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += syscalls.h
 generic-y += topology.h
arch/x86/include/asm/simd.h +43 -1
index a341c878e977..4aad7f158dcb 100644
@@ -1,4 +1,11 @@
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
 
 #include <asm/fpu/api.h>
 
@@ -10,3 +17,38 @@ static __must_check inline bool may_use_simd(void)
 {
        return irq_fpu_usable();
 }
+
+static inline void simd_get(simd_context_t *ctx)
+{
+#if !defined(CONFIG_UML)
+       *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
+#else
+       *ctx = HAVE_NO_SIMD;
+#endif
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+#if !defined(CONFIG_UML)
+       if (*ctx & HAVE_SIMD_IN_USE)
+               kernel_fpu_end();
+#endif
+       *ctx = HAVE_NO_SIMD;
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+#if !defined(CONFIG_UML)
+       if (!(*ctx & HAVE_FULL_SIMD))
+               return false;
+       if (*ctx & HAVE_SIMD_IN_USE)
+               return true;
+       kernel_fpu_begin();
+       *ctx |= HAVE_SIMD_IN_USE;
+       return true;
+#else
+       return false;
+#endif
+}
+
+#endif /* _ASM_SIMD_H */
arch/xtensa/include/asm/Kbuild +1 -0
index 82c756431b49..7950f359649d 100644
@@ -24,6 +24,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
include/asm-generic/simd.h +20 -0
index d0343d58a74a..b3dd61ac010e 100644
@@ -1,5 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
+
 #include <linux/hardirq.h>
 
 /*
@@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void)
 {
        return !in_interrupt();
 }
+
+static inline void simd_get(simd_context_t *ctx)
+{
+       *ctx = HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+       return false;
+}
+
+#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +15 -0
new file mode 100644
index 000000000000..33bba21012ff
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _SIMD_H
+#define _SIMD_H
+
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+
+#include <linux/sched.h>

include/linux/simd.h line 25

Quoted 10 lines
include/linux/simd.h new file +10 -0
+#include <asm/simd.h>
+
+static inline void simd_relax(simd_context_t *ctx)
+{
+#ifdef CONFIG_PREEMPT
+       if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
+               simd_put(ctx);
+               simd_get(ctx);
+       }
+#endif

Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.

Quoted 6 lines
include/linux/simd.h new file +3 -1
+}
+
+#endif /* _SIMD_H */
--

2.19.0

#3 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

(+ Joe)

Quoted 138 lines · Ard Biesheuvel
Quoted 80 lines · Jason A. Donenfeld

Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:

    simd_context_t simd_context;

    simd_get(&simd_context);
    while ((item = get_item_from_queue()) != NULL) {
        encrypt_item(item, simd_context);
        simd_relax(&simd_context);
    }
    simd_put(&simd_context);

The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.

On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:

   void encrypt_item(struct item *item, simd_context_t *simd_context)
   {
       if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
           wild_simd_code(item);
       else
           boring_scalar_code(item);
   }

The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Greg KH <[email protected]>
---
 arch/alpha/include/asm/Kbuild      |  5 ++-
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/simd.h        | 63 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/simd.h      | 51 +++++++++++++++++++++---
 arch/c6x/include/asm/Kbuild        |  3 +-
 arch/h8300/include/asm/Kbuild      |  3 +-
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/nds32/include/asm/Kbuild      |  7 ++--
 arch/nios2/include/asm/Kbuild      |  1 +
 arch/openrisc/include/asm/Kbuild   |  7 ++--
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  3 +-
 arch/riscv/include/asm/Kbuild      |  3 +-
 arch/s390/include/asm/Kbuild       |  3 +-
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/um/include/asm/Kbuild         |  3 +-
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/simd.h        | 44 ++++++++++++++++++++-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/simd.h         | 20 ++++++++++
 include/linux/simd.h               | 28 +++++++++++++
 26 files changed, 234 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm/include/asm/simd.h
 create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +1 -0
index 0580cb8c84b2..07b2c1025d34 100644
@@ -2,14 +2,15 @@
 
 
 generic-y += compat.h
+generic-y += current.h
 generic-y += exec.h
 generic-y += export.h
 generic-y += fb.h

arch/alpha/include/asm/Kbuild line 15

Quoted 10 lines
arch/alpha/include/asm/Kbuild +2 -2
 generic-y += irq_work.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h

Given that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.

Quoted 25 lines
arch/arc/include/asm/Kbuild +1 -0
index feed50ce89fa..a7f4255f1649 100644
@@ -22,6 +22,7 @@ generic-y += parport.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/arm/include/asm/simd.h new file +7 -0
new file mode 100644
index 000000000000..263950dd69cb
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H

arch/arm/include/asm/simd.h line 17

Quoted 10 lines
arch/arm/include/asm/simd.h new file +10 -0
+#define _ASM_SIMD_H
+
+#ifdef CONFIG_KERNEL_MODE_NEON
+#include <asm/neon.h>
+
+static __must_check inline bool may_use_simd(void)
+{
+       return !in_interrupt();
+}
+

Remember this guy?

https://marc.info/?l=linux-arch&m=149631094625176&w=2

That was never merged, so let's get it right this time.

...

Quoted 12 lines
Quoted 12 lines
include/linux/simd.h new file +6 -0
new file mode 100644
index 000000000000..33bba21012ff
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _SIMD_H
Quoted 8 lines

include/linux/simd.h line 14

Quoted 8 lines
include/linux/simd.h new file +8 -0
+#define _SIMD_H
+
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

Quoted 23 lines

include/linux/simd.h line 25

Quoted 11 lines
include/linux/simd.h new file +11 -0
+#include <linux/sched.h>
+#include <asm/simd.h>
+
+static inline void simd_relax(simd_context_t *ctx)
+{
+#ifdef CONFIG_PREEMPT
+       if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
+               simd_put(ctx);
+               simd_get(ctx);
+       }
+#endif

Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.

Quoted 6 lines
include/linux/simd.h new file +3 -1
+}
+
+#endif /* _SIMD_H */
--

2.19.0

author maintainer
#4 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

include/linux/simd.h line 14

Quoted 6 lines
include/linux/simd.h new file +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

#5 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

include/linux/simd.h line 14

Quoted 6 lines
include/linux/simd.h new file +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

Quoted 4 lines

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Yes that makes sense

author maintainer
#6 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 16 lines · Ard Biesheuvel
Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

include/linux/simd.h line 14

Quoted 6 lines
include/linux/simd.h new file +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

Quoted 6 lines
Quoted 4 lines

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

#7 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 18 lines · Jason A. Donenfeld
Quoted 16 lines · Ard Biesheuvel
Quoted 14 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

include/linux/simd.h line 14

Quoted 6 lines
include/linux/simd.h new file +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Yes just switch to a struct.

author maintainer
#8 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 25 lines · Ard Biesheuvel
Quoted 23 lines · Jason A. Donenfeld
Quoted 21 lines · Ard Biesheuvel
Quoted 18 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Yes just switch to a struct.

Okay. No problem with that, but will wait to hear from Joe first.

#9 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 27 lines · Jason A. Donenfeld
Quoted 25 lines · Ard Biesheuvel
Quoted 23 lines · Jason A. Donenfeld
Quoted 21 lines · Ard Biesheuvel
Quoted 18 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Yes just switch to a struct.

Okay. No problem with that, but will wait to hear from Joe first.

Why do you need to hear from me again?

As far as I know, the only info about typedef avoidance are in Documentation/process/coding-style.rst section 5.

#10 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 36 lines · Joe Perches
Quoted 33 lines · Jason A. Donenfeld
Quoted 29 lines · Ard Biesheuvel
Quoted 28 lines · Jason A. Donenfeld
Quoted 16 lines · Ard Biesheuvel
Quoted 16 lines · Jason A. Donenfeld
Quoted 11 lines · Ard Biesheuvel
Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+typedef enum {
+       HAVE_NO_SIMD = 1 << 0,
+       HAVE_FULL_SIMD = 1 << 1,
+       HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

Quoted 6 lines
Quoted 4 lines

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

Yes just switch to a struct.

Okay. No problem with that, but will wait to hear from Joe first.

Why do you need to hear from me again?

As far as I know, the only info about typedef avoidance are in Documentation/process/coding-style.rst section 5.

I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.

author maintainer
#11 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 18 lines · Andy Lutomirski
Quoted 18 lines
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Yes just switch to a struct.

Okay. No problem with that, but will wait to hear from Joe first.

Quoted 8 lines
Quoted 6 lines

Why do you need to hear from me again?

As far as I know, the only info about typedef avoidance are in Documentation/process/coding-style.rst section 5.

I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.

I'll stick with a typedef. Reading the style guide, this clearly falls into 5.a, 5.b, and maybe 5.c. For 5.a, at some point this will possibly contain architecture specific blobs. For 5.b, it is just an enum (integer) right now.

#12 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

Quoted 26 lines · Jason A. Donenfeld
Quoted 25 lines · Andy Lutomirski
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Oh, and another thing (and I'm surprised checkpatch.pl didn't complain about it): the use of typedef in new code is strongly discouraged. This policy predates my involvement, so perhaps Joe can elaborate on the rationale?

In case it matters, the motivation for making this a typedef is I could imagine this at some point turning into a more complicated struct on certain platforms and that would make refactoring easier. I could just make it struct simd_context now with 1 member though...

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Yes that makes sense

The rationale for it being a typedef or moving to a struct now?

Yes just switch to a struct.

Okay. No problem with that, but will wait to hear from Joe first.

Why do you need to hear from me again?

As far as I know, the only info about typedef avoidance are in Documentation/process/coding-style.rst section 5.

Quoted 6 lines
Quoted 1 line

I personally prefer it with the typedef. If this were my code, I’d say the coding style is silly for opaque tiny structs like this.

I'll stick with a typedef. Reading the style guide, this clearly falls into 5.a, 5.b, and maybe 5.c. For 5.a, at some point this will possibly contain architecture specific blobs. For 5.b, it is just an enum (integer) right now.

I can live with that, but other may still object.

In any case, since you are using the enum member as a bitfield, it would be better to typedef it to int instead, and retain the enum definition only for the symbolic constants.

author maintainer
#13 lore raw

Re: [PATCH net-next v6 01/23] asm: simd context helper API

In reply to Ard Biesheuvel
Quoted 3 lines · Ard Biesheuvel

Given that this patch applies to all architectures at once, it is probably better to drop the unrelated reordering hunks to avoid conflicts.

Ack. Will retain order for v7.

Quoted 11 lines

arch/arm/include/asm/simd.h line 17

Quoted 5 lines
arch/arm/include/asm/simd.h new file +5 -0
+static __must_check inline bool may_use_simd(void)
+{
+       return !in_interrupt();
+}
+

Remember this guy?

https://marc.info/?l=linux-arch&m=149631094625176&w=2

That was never merged, so let's get it right this time.

Wow, nice memory. I had forgotten all about that. Queued for v7.

Quoted 5 lines
Quoted 5 lines
include/linux/simd.h new file +5 -0
+static inline void simd_relax(simd_context_t *ctx)
+{
+#ifdef CONFIG_PREEMPT
+       if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
+               simd_put(ctx);
Quoted 8 lines

include/linux/simd.h line 25

Quoted 3 lines
include/linux/simd.h new file +3 -0
+               simd_get(ctx);
+       }
+#endif

Could we return a bool here indicating whether we rescheduled or not? In some cases, we could pass that into the asm code as a 'reload' param, allowing repeated loads of key schedules, round constant tables or S-boxes to be elided.

Sure, sounds easy enough.

author maintainer
#14 lore raw

PATCH v6 2/23 zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld

Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.

Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:

* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.

* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.

* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.

This last observation has seen itself play out several times over and over again within the kernel:

* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.

* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.

* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.

Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.

It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.

All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].

Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.

SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.

More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.

Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.

Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.

Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.

As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.

[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 MAINTAINERS       |  8 ++++++++
 lib/Kconfig       |  2 ++
 lib/Makefile      |  2 ++
 lib/zinc/Kconfig  | 32 ++++++++++++++++++++++++++++++++
 lib/zinc/Makefile |  3 +++
 5 files changed, 47 insertions(+)
 create mode 100644 lib/zinc/Kconfig
 create mode 100644 lib/zinc/Makefile
MAINTAINERS +8 -0
index 4327777dce57..5967c737f3ce 100644
@@ -16170,6 +16170,14 @@ Q:	https://patchwork.linuxtv.org/project/linux-media/list/
 S:	Maintained
 F:	drivers/media/dvb-frontends/zd1301_demod*
 
+ZINC CRYPTOGRAPHY LIBRARY
+M:	Jason A. Donenfeld <[email protected]>
+M:	Samuel Neves <[email protected]>
+S:	Maintained
+F:	lib/zinc/
+F:	include/zinc/
+L:	[email protected]
+
 ZPOOL COMPRESSED PAGE STORAGE API
 M:	Dan Streetman <[email protected]>
 L:	[email protected]
lib/Kconfig +2 -0
index a3928d4438b5..3e6848269c66 100644
@@ -485,6 +485,8 @@ config GLOB_SELFTEST
 	  module load) by a small amount, so you're welcome to play with
 	  it, but you probably don't need it.
 
+source "lib/zinc/Kconfig"
+
 #
 # Netlink attribute parsing support is select'ed if needed
 #
lib/Makefile +2 -0
index ca3f7ebb900d..571d28d3f143 100644
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o
 
 obj-$(CONFIG_ASN1) += asn1_decoder.o
 
+obj-y += zinc/
+
 obj-$(CONFIG_FONT_SUPPORT) += fonts/
 
 obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +32 -0
new file mode 100644
index 000000000000..4e2e59126a67
@@ -0,0 +1,32 @@
+config ZINC_DEBUG
+	bool "Zinc cryptography library debugging and self-tests"
+	help
+	  This builds a series of self-tests for the Zinc crypto library, which
+	  help diagnose any cryptographic algorithm implementation issues that
+	  might be at the root cause of potential bugs. It also adds various
+	  debugging traps.
+
+	  Unless you're developing and testing cryptographic routines, or are
+	  especially paranoid about correctness on your hardware, you may say
+	  N here.
+
+config ZINC_ARCH_ARM
+	def_bool y
+	depends on ARM && !64BIT
+
+config ZINC_ARCH_ARM64
+	def_bool y
+	depends on ARM64 && 64BIT
+
+config ZINC_ARCH_X86_64
+	def_bool y
+	depends on X86_64 && 64BIT
+	depends on !UML
+
+config ZINC_ARCH_MIPS
+	def_bool y
+	depends on MIPS && CPU_MIPS32_R2 && !64BIT
+
+config ZINC_ARCH_MIPS64
+	def_bool y
+	depends on MIPS && 64BIT
lib/zinc/Makefile new file +3 -0
new file mode 100644
index 000000000000..a61c80d676cb
@@ -0,0 +1,3 @@
+ccflags-y := -O2
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
Signature · 1 line
2.19.0
#15 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 6 lines · Jason A. Donenfeld

Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.

[]

lib/zinc/Makefile line 3

Quoted 9 lines
lib/zinc/Makefile new file +3 -0
new file mode 100644
index 000000000000..a61c80d676cb
@@ -0,0 +1,3 @@
+ccflags-y := -O2
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG

I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.

author maintainer
#16 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 3 lines · Joe Perches

I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.

There are about 2000 cases in the kernel of the same '#define pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that I pr_err from, why can't I just do this from the makefile, since I want that same pr_fmt to copy the whole directory?

#17 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 9 lines · Jason A. Donenfeld
Quoted 5 lines · Joe Perches

I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.

Quoted 5 lines

There are about 2000 cases in the kernel of the same '#define pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that I pr_err from, why can't I just do this from the makefile, since I want that same pr_fmt to copy the whole directory?

Because people like to be able to just read the C file to figure out what it does. Or we could adopt the Makefile approach kernel-wide, since this use of it seems reasonable.

author maintainer
#18 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 7 lines · Andy Lutomirski
Quoted 7 lines · Jason A. Donenfeld
Quoted 5 lines · Joe Perches

I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.

Quoted 7 lines
Quoted 5 lines

There are about 2000 cases in the kernel of the same '#define pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that I pr_err from, why can't I just do this from the makefile, since I want that same pr_fmt to copy the whole directory?

Because people like to be able to just read the C file to figure out what it does. Or we could adopt the Makefile approach kernel-wide, since this use of it seems reasonable.

It would indeed be nice to see this done tree-wide. I can recall various random bugs over the year where some dmesg messages are missing a prefix because the author forgot to copy and paste the thing to /yet another file/ in the same directory.

#19 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld
Quoted 9 lines · Jason A. Donenfeld
Quoted 3 lines · Joe Perches

I think the -Dpr_fmt is especially odd and not
really acceptable as it not used anywhere else
in the kernel.

There are about 2000 cases in the kernel of the same '#define pr_fmt...' being pasted into the top of the file, which seems a bit cumbersome. Rather than having to paste that into each and every file that I pr_err from,

As far as I can tell, zinc doesn't use pr_err, just pr_info in all the selftest blocks which are only used when DEBUG is #defined.

Perhaps all the pr_info uses should use pr_debug instead as well as dynamic debugging which already optionally adds KBUILD_MODINFO.

Quoted 2 lines

why can't I just do this from the makefile, since
I want that same pr_fmt to copy the whole directory?

Ideally all of those

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

uses will be removed one day soon and the default will change.

https://lore.kernel.org/patchwork/cover/904507/

This slightly odd use might complicate that.

author maintainer
#20 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Hi Joe,

Quoted 2 lines · Joe Perches

As far as I can tell, zinc doesn't use pr_err, just pr_info

Yes, pr_info, not pr_err. Apologies for my imprecision. But the distinction does not matter at all, since they both use pr_fmt in exactly the same way, which is sufficient for the purposes of this discussion.

Quoted 5 lines
Quoted 2 lines

why can't I just do this from the makefile, since
I want that same pr_fmt to copy the whole directory?

Ideally all of those

Quoted 8 lines

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

uses will be removed one day soon and the default
will change.

https://lore.kernel.org/patchwork/cover/904507/

This slightly odd use might complicate that.

Oh, that's good to see. In this case, I'm prefixing all of them with
zinc: instead of zinc_chacha20: (the modname), but I wouldn't object to that changing to KBUILD_MODNAME if your patch lands. IOW, once your patch lands, I'm happy to revisit this discussion and fall back to using the sane defaults you're in the process of setting up for everyone.

Jason

#21 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 23 lines · Jason A. Donenfeld

Hi Joe,

Quoted 2 lines · Joe Perches

As far as I can tell, zinc doesn't use pr_err, just pr_info

Yes, pr_info, not pr_err. Apologies for my imprecision. But the distinction does not matter at all, since they both use pr_fmt in exactly the same way, which is sufficient for the purposes of this discussion.

Quoted 12 lines
Quoted 2 lines

why can't I just do this from the makefile, since
I want that same pr_fmt to copy the whole directory?

Ideally all of those

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

uses will be removed one day soon and the default
will change.

https://lore.kernel.org/patchwork/cover/904507/

Quoted 8 lines
Quoted 1 line

This slightly odd use might complicate that.

Oh, that's good to see. In this case, I'm prefixing all of them with
zinc: instead of zinc_chacha20: (the modname), but I wouldn't object to that changing to KBUILD_MODNAME if your patch lands. IOW, once your patch lands, I'm happy to revisit this discussion and fall back to using the sane defaults you're in the process of setting up for everyone.

I'd still prefer as all these are effectively debugging output that you convert the pr_info uses pr_debug and so avoid using pr_fmt altogether.

author maintainer
#22 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Hi Joe,

Quoted 3 lines · Joe Perches

I'd still prefer as all these are effectively debugging output that you convert the pr_info uses pr_debug and so avoid using pr_fmt altogether.

I started to write back, "sure no problem, this will be in v7," but then I gave it a closer look, and I think actually I'll be changing these into pr_err/pr_info and not naming this "DEBUG" but rather "SELFTEST", since I think these are actually very good to have beyond mere debugging.

Jason

#23 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 7 lines · Jason A. Donenfeld

Hi Joe,

Quoted 3 lines · Joe Perches

I'd still prefer as all these are effectively debugging output that you convert the pr_info uses pr_debug and so avoid using pr_fmt altogether.

Quoted 5 lines

I started to write back, "sure no problem, this will be in v7," but then I gave it a closer look, and I think actually I'll be changing these into pr_err/pr_info and not naming this "DEBUG" but rather "SELFTEST", since I think these are actually very good to have beyond mere debugging.

Then you should change the CONFIG_ZINC_DEBUG option too.

Dunno what you should do with the

#ifdef DEBUG
	BUG_ON(...);

#endif

uses though.

author maintainer
#24 lore raw

Re: [PATCH net-next v6 02/23] zinc: introduce minimal cryptography library

Quoted 1 line · Joe Perches

Then you should change the CONFIG_ZINC_DEBUG option too.

Yes, indeed.

author maintainer
#25 lore raw

PATCH v6 3/23 zinc: ChaCha20 generic C implementation and selftest

In reply to Jason A. Donenfeld

This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.

Information: https://cr.yp.to/chacha.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20.h      |   65 +
 lib/zinc/Kconfig             |    4 +
 lib/zinc/Makefile            |    3 +
 lib/zinc/chacha20/chacha20.c |  179 +++
 lib/zinc/selftest/chacha20.h | 2676 ++++++++++++++++++++++++++++++++++
 5 files changed, 2927 insertions(+)
 create mode 100644 include/zinc/chacha20.h
 create mode 100644 lib/zinc/chacha20/chacha20.c
 create mode 100644 lib/zinc/selftest/chacha20.h
include/zinc/chacha20.h new file +65 -0
new file mode 100644
index 000000000000..14bbadd242c9
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20_H
+#define _ZINC_CHACHA20_H
+
+#include <asm/unaligned.h>
+#include <linux/simd.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+enum {
+	CHACHA20_NONCE_SIZE = 16,
+	CHACHA20_KEY_SIZE = 32,
+	CHACHA20_KEY_WORDS = CHACHA20_KEY_SIZE / sizeof(u32),
+	CHACHA20_BLOCK_SIZE = 64,
+	CHACHA20_BLOCK_WORDS = CHACHA20_BLOCK_SIZE / sizeof(u32),
+	HCHACHA20_NONCE_SIZE = CHACHA20_NONCE_SIZE,
+	HCHACHA20_KEY_SIZE = CHACHA20_KEY_SIZE
+};
+
+enum { /* expand 32-byte k */
+	CHACHA20_CONSTANT_EXPA = 0x61707865U,
+	CHACHA20_CONSTANT_ND_3 = 0x3320646eU,
+	CHACHA20_CONSTANT_2_BY = 0x79622d32U,
+	CHACHA20_CONSTANT_TE_K = 0x6b206574U
+};
+
+struct chacha20_ctx {
+	u32 constant[4];
+	u32 key[8];
+	u32 counter[4];
+} __aligned(32);
+
+static inline void chacha20_init(struct chacha20_ctx *state,
+				 const u8 key[CHACHA20_KEY_SIZE],
+				 const u64 nonce)
+{
+	state->constant[0] = CHACHA20_CONSTANT_EXPA;
+	state->constant[1] = CHACHA20_CONSTANT_ND_3;
+	state->constant[2] = CHACHA20_CONSTANT_2_BY;
+	state->constant[3] = CHACHA20_CONSTANT_TE_K;
+	state->key[0] = get_unaligned_le32(key + 0);
+	state->key[1] = get_unaligned_le32(key + 4);
+	state->key[2] = get_unaligned_le32(key + 8);
+	state->key[3] = get_unaligned_le32(key + 12);
+	state->key[4] = get_unaligned_le32(key + 16);
+	state->key[5] = get_unaligned_le32(key + 20);
+	state->key[6] = get_unaligned_le32(key + 24);
+	state->key[7] = get_unaligned_le32(key + 28);
+	state->counter[0] = 0;
+	state->counter[1] = 0;
+	state->counter[2] = nonce & U32_MAX;
+	state->counter[3] = nonce >> 32;
+}
+void chacha20(struct chacha20_ctx *state, u8 *dst, const u8 *src, u32 len,
+	      simd_context_t *simd_context);
+
+void hchacha20(u32 derived_key[CHACHA20_KEY_WORDS],
+	       const u8 nonce[HCHACHA20_NONCE_SIZE],
+	       const u8 key[HCHACHA20_KEY_SIZE], simd_context_t *simd_context);
+
+#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
index 4e2e59126a67..1ca1ae1e9ea9 100644
@@ -1,3 +1,7 @@
+config ZINC_CHACHA20
+	tristate
+	select CRYPTO_ALGAPI
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index a61c80d676cb..3d80144d55a6 100644
@@ -1,3 +1,6 @@
 ccflags-y := -O2
 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
+
+zinc_chacha20-y := chacha20/chacha20.o
+obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +179 -0

179 lines not on this page · raw message

lib/zinc/selftest/chacha20.h new file +2676 -0

2,676 lines not on this page · raw message

Signature · 1 line
2.19.0
#26 lore raw

Re: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest

Quoted 54 lines · Jason A. Donenfeld

This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.

Information: https://cr.yp.to/chacha.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20.h      |   65 +
 lib/zinc/Kconfig             |    4 +
 lib/zinc/Makefile            |    3 +
 lib/zinc/chacha20/chacha20.c |  179 +++
 lib/zinc/selftest/chacha20.h | 2676 ++++++++++++++++++++++++++++++++++
 5 files changed, 2927 insertions(+)
 create mode 100644 include/zinc/chacha20.h
 create mode 100644 lib/zinc/chacha20/chacha20.c
 create mode 100644 lib/zinc/selftest/chacha20.h
include/zinc/chacha20.h new file +26 -0
new file mode 100644
index 000000000000..14bbadd242c9
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20_H
+#define _ZINC_CHACHA20_H
+
+#include <asm/unaligned.h>
+#include <linux/simd.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+enum {
+       CHACHA20_NONCE_SIZE = 16,
+       CHACHA20_KEY_SIZE = 32,
+       CHACHA20_KEY_WORDS = CHACHA20_KEY_SIZE / sizeof(u32),
+       CHACHA20_BLOCK_SIZE = 64,
+       CHACHA20_BLOCK_WORDS = CHACHA20_BLOCK_SIZE / sizeof(u32),
+       HCHACHA20_NONCE_SIZE = CHACHA20_NONCE_SIZE,
+       HCHACHA20_KEY_SIZE = CHACHA20_KEY_SIZE
+};
+
+enum { /* expand 32-byte k */
+       CHACHA20_CONSTANT_EXPA = 0x61707865U,
+       CHACHA20_CONSTANT_ND_3 = 0x3320646eU,

include/zinc/chacha20.h line 36

Quoted 10 lines
include/zinc/chacha20.h new file +10 -0
+       CHACHA20_CONSTANT_2_BY = 0x79622d32U,
+       CHACHA20_CONSTANT_TE_K = 0x6b206574U
+};
+
+struct chacha20_ctx {
+       u32 constant[4];
+       u32 key[8];
+       u32 counter[4];
+} __aligned(32);
+

32 *byte* alignment? Is that right? If this is for performance and it actually helps, using __cacheline_aligned is more appropriate,

Quoted 64 lines
include/zinc/chacha20.h new file +29 -0
+static inline void chacha20_init(struct chacha20_ctx *state,
+                                const u8 key[CHACHA20_KEY_SIZE],
+                                const u64 nonce)
+{
+       state->constant[0] = CHACHA20_CONSTANT_EXPA;
+       state->constant[1] = CHACHA20_CONSTANT_ND_3;
+       state->constant[2] = CHACHA20_CONSTANT_2_BY;
+       state->constant[3] = CHACHA20_CONSTANT_TE_K;
+       state->key[0] = get_unaligned_le32(key + 0);
+       state->key[1] = get_unaligned_le32(key + 4);
+       state->key[2] = get_unaligned_le32(key + 8);
+       state->key[3] = get_unaligned_le32(key + 12);
+       state->key[4] = get_unaligned_le32(key + 16);
+       state->key[5] = get_unaligned_le32(key + 20);
+       state->key[6] = get_unaligned_le32(key + 24);
+       state->key[7] = get_unaligned_le32(key + 28);
+       state->counter[0] = 0;
+       state->counter[1] = 0;
+       state->counter[2] = nonce & U32_MAX;
+       state->counter[3] = nonce >> 32;
+}
+void chacha20(struct chacha20_ctx *state, u8 *dst, const u8 *src, u32 len,
+             simd_context_t *simd_context);
+
+void hchacha20(u32 derived_key[CHACHA20_KEY_WORDS],
+              const u8 nonce[HCHACHA20_NONCE_SIZE],
+              const u8 key[HCHACHA20_KEY_SIZE], simd_context_t *simd_context);
+
+#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
index 4e2e59126a67..1ca1ae1e9ea9 100644
@@ -1,3 +1,7 @@
+config ZINC_CHACHA20
+       tristate
+       select CRYPTO_ALGAPI
+
 config ZINC_DEBUG
        bool "Zinc cryptography library debugging and self-tests"
        help
lib/zinc/Makefile +3 -0
index a61c80d676cb..3d80144d55a6 100644
@@ -1,3 +1,6 @@
 ccflags-y := -O2
 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
+
+zinc_chacha20-y := chacha20/chacha20.o
+obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +6 -0
new file mode 100644
index 000000000000..c82d9fc71f21
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ *
+ * Implementation of the ChaCha20 stream cipher.
+ *

lib/zinc/chacha20/chacha20.c line 16

Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10 -0
+ * Information: https://cr.yp.to/chacha.html
+ */
+
+#include <zinc/chacha20.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <crypto/algapi.h>
+

I guess this include is for crypto_xor_cpy() ?

We may want to put a comment here, so we keep track of the interdependencies.

lib/zinc/chacha20/chacha20.c line 17

Quoted 1 line
lib/zinc/chacha20/chacha20.c new file +1 -0
+#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION

This #define is never set in subsequent patches, so just drop this #ifndef entirely (for this patch only)

Quoted 138 lines
lib/zinc/chacha20/chacha20.c new file +138 -0
+void __init chacha20_fpu_init(void)
+{
+}
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *out,
+                                const u8 *in, const size_t len,
+                                simd_context_t *simd_context)
+{
+       return false;
+}
+static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
+                                 const u8 nonce[HCHACHA20_NONCE_SIZE],
+                                 const u8 key[HCHACHA20_KEY_SIZE],
+                                 simd_context_t *simd_context)
+{
+       return false;
+}
+#endif
+
+#define QUARTER_ROUND(x, a, b, c, d) ( \
+       x[a] += x[b], \
+       x[d] = rol32((x[d] ^ x[a]), 16), \
+       x[c] += x[d], \
+       x[b] = rol32((x[b] ^ x[c]), 12), \
+       x[a] += x[b], \
+       x[d] = rol32((x[d] ^ x[a]), 8), \
+       x[c] += x[d], \
+       x[b] = rol32((x[b] ^ x[c]), 7) \
+)
+
+#define C(i, j) (i * 4 + j)
+
+#define DOUBLE_ROUND(x) ( \
+       /* Column Round */ \
+       QUARTER_ROUND(x, C(0, 0), C(1, 0), C(2, 0), C(3, 0)), \
+       QUARTER_ROUND(x, C(0, 1), C(1, 1), C(2, 1), C(3, 1)), \
+       QUARTER_ROUND(x, C(0, 2), C(1, 2), C(2, 2), C(3, 2)), \
+       QUARTER_ROUND(x, C(0, 3), C(1, 3), C(2, 3), C(3, 3)), \
+       /* Diagonal Round */ \
+       QUARTER_ROUND(x, C(0, 0), C(1, 1), C(2, 2), C(3, 3)), \
+       QUARTER_ROUND(x, C(0, 1), C(1, 2), C(2, 3), C(3, 0)), \
+       QUARTER_ROUND(x, C(0, 2), C(1, 3), C(2, 0), C(3, 1)), \
+       QUARTER_ROUND(x, C(0, 3), C(1, 0), C(2, 1), C(3, 2)) \
+)
+
+#define TWENTY_ROUNDS(x) ( \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x), \
+       DOUBLE_ROUND(x) \
+)
+
+static void chacha20_block_generic(__le32 *stream, u32 *state)
+{
+       u32 x[CHACHA20_BLOCK_WORDS];
+       int i;
+
+       for (i = 0; i < ARRAY_SIZE(x); ++i)
+               x[i] = state[i];
+
+       TWENTY_ROUNDS(x);
+
+       for (i = 0; i < ARRAY_SIZE(x); ++i)
+               stream[i] = cpu_to_le32(x[i] + state[i]);
+
+       ++state[12];
+}
+
+static void chacha20_generic(struct chacha20_ctx *state, u8 *out, const u8 *in,
+                            u32 len)
+{
+       __le32 buf[CHACHA20_BLOCK_WORDS];
+
+       while (len >= CHACHA20_BLOCK_SIZE) {
+               chacha20_block_generic(buf, (u32 *)state);
+               crypto_xor_cpy(out, in, (u8 *)buf, CHACHA20_BLOCK_SIZE);
+               len -= CHACHA20_BLOCK_SIZE;
+               out += CHACHA20_BLOCK_SIZE;
+               in += CHACHA20_BLOCK_SIZE;
+       }
+       if (len) {
+               chacha20_block_generic(buf, (u32 *)state);
+               crypto_xor_cpy(out, in, (u8 *)buf, len);
+       }
+}
+
+void chacha20(struct chacha20_ctx *state, u8 *dst, const u8 *src, u32 len,
+             simd_context_t *simd_context)
+{
+       if (!chacha20_arch(state, dst, src, len, simd_context))
+               chacha20_generic(state, dst, src, len);
+}
+EXPORT_SYMBOL(chacha20);
+
+static void hchacha20_generic(u32 derived_key[CHACHA20_KEY_WORDS],
+                             const u8 nonce[HCHACHA20_NONCE_SIZE],
+                             const u8 key[HCHACHA20_KEY_SIZE])
+{
+       u32 x[] = { CHACHA20_CONSTANT_EXPA,
+                   CHACHA20_CONSTANT_ND_3,
+                   CHACHA20_CONSTANT_2_BY,
+                   CHACHA20_CONSTANT_TE_K,
+                   get_unaligned_le32(key +  0),
+                   get_unaligned_le32(key +  4),
+                   get_unaligned_le32(key +  8),
+                   get_unaligned_le32(key + 12),
+                   get_unaligned_le32(key + 16),
+                   get_unaligned_le32(key + 20),
+                   get_unaligned_le32(key + 24),
+                   get_unaligned_le32(key + 28),
+                   get_unaligned_le32(nonce +  0),
+                   get_unaligned_le32(nonce +  4),
+                   get_unaligned_le32(nonce +  8),
+                   get_unaligned_le32(nonce + 12)
+       };
+
+       TWENTY_ROUNDS(x);
+
+       memcpy(derived_key + 0, x +  0, sizeof(u32) * 4);
+       memcpy(derived_key + 4, x + 12, sizeof(u32) * 4);
+}
+
+/* Derived key should be 32-bit aligned */
+void hchacha20(u32 derived_key[CHACHA20_KEY_WORDS],
+              const u8 nonce[HCHACHA20_NONCE_SIZE],
+              const u8 key[HCHACHA20_KEY_SIZE], simd_context_t *simd_context)
+{
+       if (!hchacha20_arch(derived_key, nonce, key, simd_context))
+               hchacha20_generic(derived_key, nonce, key);
+}
+EXPORT_SYMBOL(hchacha20);
+
+#include "../selftest/chacha20.h"

lib/zinc/chacha20/chacha20.c line 165

Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10 -0
+
+static bool nosimd __initdata = false;
+
+static int __init mod_init(void)
+{
+       if (!nosimd)
+               chacha20_fpu_init();
+#ifdef DEBUG
+       if (!chacha20_selftest())
+               return -ENOTRECOVERABLE;

Return values from initcalls are ignored, and given that chacha20 will be depended upon by random.c, it will never be a module in practice.

Given your previous statement that selftest should *not* be a DEBUG feature (which I wholeheartedly agree with), you could be a bit noisier here imo.
E.g.,

if (WARN_ON(!chach20_selftest())

    return ...
Quoted 2,699 lines
lib/zinc/chacha20/chacha20.c new file +14 -0
+#endif
+       return 0;
+}
+
+static void __exit mod_exit(void)
+{
+}
+
+module_param(nosimd, bool, 0);
+module_init(mod_init);
+module_exit(mod_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("ChaCha20 stream cipher");
+MODULE_AUTHOR("Jason A. Donenfeld <[email protected]>");
lib/zinc/selftest/chacha20.h new file +2676 -1
new file mode 100644
index 000000000000..397a4b930fd7
@@ -0,0 +1,2676 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifdef DEBUG
+
+struct chacha20_testvec {
+       const u8 *input, *output, *key;
+       u64 nonce;
+       size_t ilen;
+};
+
+struct hchacha20_testvec {
+       u8 key[HCHACHA20_KEY_SIZE];
+       u8 nonce[HCHACHA20_NONCE_SIZE];
+       u8 output[CHACHA20_KEY_SIZE];
+};
+
+/* These test vectors are generated by reference implementations and are
+ * designed to check chacha20 implementation block handling, as well as from
+ * the draft-arciszewski-xchacha-01 document.
+ */
+
+static const u8 input01[] __initconst = { };
+static const u8 output01[] __initconst = { };
+static const u8 key01[] __initconst = {
+       0x09, 0xf4, 0xe8, 0x57, 0x10, 0xf2, 0x12, 0xc3,
+       0xc6, 0x91, 0xc4, 0x09, 0x97, 0x46, 0xef, 0xfe,
+       0x02, 0x00, 0xe4, 0x5c, 0x82, 0xed, 0x16, 0xf3,
+       0x32, 0xbe, 0xec, 0x7a, 0xe6, 0x68, 0x12, 0x26
+};
+enum { nonce01 = 0x3834e2afca3c66d3ULL };
+
+static const u8 input02[] __initconst = {
+       0x9d
+};
+static const u8 output02[] __initconst = {
+       0x94
+};
+static const u8 key02[] __initconst = {
+       0x8c, 0x01, 0xac, 0xaf, 0x62, 0x63, 0x56, 0x7a,
+       0xad, 0x23, 0x4c, 0x58, 0x29, 0x29, 0xbe, 0xab,
+       0xe9, 0xf8, 0xdf, 0x6c, 0x8c, 0x74, 0x4d, 0x7d,
+       0x13, 0x94, 0x10, 0x02, 0x3d, 0x8e, 0x9f, 0x94
+};
+enum { nonce02 = 0x5d1b3bfdedd9f73aULL };
+
+static const u8 input03[] __initconst = {
+       0x04, 0x16
+};
+static const u8 output03[] __initconst = {
+       0x92, 0x07
+};
+static const u8 key03[] __initconst = {
+       0x22, 0x0c, 0x79, 0x2c, 0x38, 0x51, 0xbe, 0x99,
+       0xa9, 0x59, 0x24, 0x50, 0xef, 0x87, 0x38, 0xa6,
+       0xa0, 0x97, 0x20, 0xcb, 0xb4, 0x0c, 0x94, 0x67,
+       0x1f, 0x98, 0xdc, 0xc4, 0x83, 0xbc, 0x35, 0x4d
+};
+enum { nonce03 = 0x7a3353ad720a3e2eULL };
+
+static const u8 input04[] __initconst = {
+       0xc7, 0xcc, 0xd0
+};
+static const u8 output04[] __initconst = {
+       0xd8, 0x41, 0x80
+};
+static const u8 key04[] __initconst = {
+       0x81, 0x5e, 0x12, 0x01, 0xc4, 0x36, 0x15, 0x03,
+       0x11, 0xa0, 0xe9, 0x86, 0xbb, 0x5a, 0xdc, 0x45,
+       0x7d, 0x5e, 0x98, 0xf8, 0x06, 0x76, 0x1c, 0xec,
+       0xc0, 0xf7, 0xca, 0x4e, 0x99, 0xd9, 0x42, 0x38
+};
+enum { nonce04 = 0x6816e2fc66176da2ULL };
+
+static const u8 input05[] __initconst = {
+       0x48, 0xf1, 0x31, 0x5f
+};
+static const u8 output05[] __initconst = {
+       0x48, 0xf7, 0x13, 0x67
+};
+static const u8 key05[] __initconst = {
+       0x3f, 0xd6, 0xb6, 0x5e, 0x2f, 0xda, 0x82, 0x39,
+       0x97, 0x06, 0xd3, 0x62, 0x4f, 0xbd, 0xcb, 0x9b,
+       0x1d, 0xe6, 0x4a, 0x76, 0xab, 0xdd, 0x14, 0x50,
+       0x59, 0x21, 0xe3, 0xb2, 0xc7, 0x95, 0xbc, 0x45
+};
+enum { nonce05 = 0xc41a7490e228cc42ULL };
+
+static const u8 input06[] __initconst = {
+       0xae, 0xa2, 0x85, 0x1d, 0xc8
+};
+static const u8 output06[] __initconst = {
+       0xfa, 0xff, 0x45, 0x6b, 0x6f
+};
+static const u8 key06[] __initconst = {
+       0x04, 0x8d, 0xea, 0x67, 0x20, 0x78, 0xfb, 0x8f,
+       0x49, 0x80, 0x35, 0xb5, 0x7b, 0xe4, 0x31, 0x74,
+       0x57, 0x43, 0x3a, 0x64, 0x64, 0xb9, 0xe6, 0x23,
+       0x4d, 0xfe, 0xb8, 0x7b, 0x71, 0x4d, 0x9d, 0x21
+};
+enum { nonce06 = 0x251366db50b10903ULL };
+
+static const u8 input07[] __initconst = {
+       0x1a, 0x32, 0x85, 0xb6, 0xe8, 0x52
+};
+static const u8 output07[] __initconst = {
+       0xd3, 0x5f, 0xf0, 0x07, 0x69, 0xec
+};
+static const u8 key07[] __initconst = {
+       0xbf, 0x2d, 0x42, 0x99, 0x97, 0x76, 0x04, 0xad,
+       0xd3, 0x8f, 0x6e, 0x6a, 0x34, 0x85, 0xaf, 0x81,
+       0xef, 0x36, 0x33, 0xd5, 0x43, 0xa2, 0xaa, 0x08,
+       0x0f, 0x77, 0x42, 0x83, 0x58, 0xc5, 0x42, 0x2a
+};
+enum { nonce07 = 0xe0796da17dba9b58ULL };
+
+static const u8 input08[] __initconst = {
+       0x40, 0xae, 0xcd, 0xe4, 0x3d, 0x22, 0xe0
+};
+static const u8 output08[] __initconst = {
+       0xfd, 0x8a, 0x9f, 0x3d, 0x05, 0xc9, 0xd3
+};
+static const u8 key08[] __initconst = {
+       0xdc, 0x3f, 0x41, 0xe3, 0x23, 0x2a, 0x8d, 0xf6,
+       0x41, 0x2a, 0xa7, 0x66, 0x05, 0x68, 0xe4, 0x7b,
+       0xc4, 0x58, 0xd6, 0xcc, 0xdf, 0x0d, 0xc6, 0x25,
+       0x1b, 0x61, 0x32, 0x12, 0x4e, 0xf1, 0xe6, 0x29
+};
+enum { nonce08 = 0xb1d2536d9e159832ULL };
+
+static const u8 input09[] __initconst = {
+       0xba, 0x1d, 0x14, 0x16, 0x9f, 0x83, 0x67, 0x24
+};
+static const u8 output09[] __initconst = {
+       0x7c, 0xe3, 0x78, 0x1d, 0xa2, 0xe7, 0xe9, 0x39
+};
+static const u8 key09[] __initconst = {
+       0x17, 0x55, 0x90, 0x52, 0xa4, 0xce, 0x12, 0xae,
+       0xd4, 0xfd, 0xd4, 0xfb, 0xd5, 0x18, 0x59, 0x50,
+       0x4e, 0x51, 0x99, 0x32, 0x09, 0x31, 0xfc, 0xf7,
+       0x27, 0x10, 0x8e, 0xa2, 0x4b, 0xa5, 0xf5, 0x62
+};
+enum { nonce09 = 0x495fc269536d003ULL };
+
+static const u8 input10[] __initconst = {
+       0x09, 0xfd, 0x3c, 0x0b, 0x3d, 0x0e, 0xf3, 0x9d,
+       0x27
+};
+static const u8 output10[] __initconst = {
+       0xdc, 0xe4, 0x33, 0x60, 0x0c, 0x07, 0xcb, 0x51,
+       0x6b
+};
+static const u8 key10[] __initconst = {
+       0x4e, 0x00, 0x72, 0x37, 0x0f, 0x52, 0x4d, 0x6f,
+       0x37, 0x50, 0x3c, 0xb3, 0x51, 0x81, 0x49, 0x16,
+       0x7e, 0xfd, 0xb1, 0x51, 0x72, 0x2e, 0xe4, 0x16,
+       0x68, 0x5c, 0x5b, 0x8a, 0xc3, 0x90, 0x70, 0x04
+};
+enum { nonce10 = 0x1ad9d1114d88cbbdULL };
+
+static const u8 input11[] __initconst = {
+       0x70, 0x18, 0x52, 0x85, 0xba, 0x66, 0xff, 0x2c,
+       0x9a, 0x46
+};
+static const u8 output11[] __initconst = {
+       0xf5, 0x2a, 0x7a, 0xfd, 0x31, 0x7c, 0x91, 0x41,
+       0xb1, 0xcf
+};
+static const u8 key11[] __initconst = {
+       0x48, 0xb4, 0xd0, 0x7c, 0x88, 0xd1, 0x96, 0x0d,
+       0x80, 0x33, 0xb4, 0xd5, 0x31, 0x9a, 0x88, 0xca,
+       0x14, 0xdc, 0xf0, 0xa8, 0xf3, 0xac, 0xb8, 0x47,
+       0x75, 0x86, 0x7c, 0x88, 0x50, 0x11, 0x43, 0x40
+};
+enum { nonce11 = 0x47c35dd1f4f8aa4fULL };
+
+static const u8 input12[] __initconst = {
+       0x9e, 0x8e, 0x3d, 0x2a, 0x05, 0xfd, 0xe4, 0x90,
+       0x24, 0x1c, 0xd3
+};
+static const u8 output12[] __initconst = {
+       0x97, 0x72, 0x40, 0x9f, 0xc0, 0x6b, 0x05, 0x33,
+       0x42, 0x7e, 0x28
+};
+static const u8 key12[] __initconst = {
+       0xee, 0xff, 0x33, 0x33, 0xe0, 0x28, 0xdf, 0xa2,
+       0xb6, 0x5e, 0x25, 0x09, 0x52, 0xde, 0xa5, 0x9c,
+       0x8f, 0x95, 0xa9, 0x03, 0x77, 0x0f, 0xbe, 0xa1,
+       0xd0, 0x7d, 0x73, 0x2f, 0xf8, 0x7e, 0x51, 0x44
+};
+enum { nonce12 = 0xc22d044dc6ea4af3ULL };
+
+static const u8 input13[] __initconst = {
+       0x9c, 0x16, 0xa2, 0x22, 0x4d, 0xbe, 0x04, 0x9a,
+       0xb3, 0xb5, 0xc6, 0x58
+};
+static const u8 output13[] __initconst = {
+       0xf0, 0x81, 0xdb, 0x6d, 0xa3, 0xe9, 0xb2, 0xc6,
+       0x32, 0x50, 0x16, 0x9f
+};
+static const u8 key13[] __initconst = {
+       0x96, 0xb3, 0x01, 0xd2, 0x7a, 0x8c, 0x94, 0x09,
+       0x4f, 0x58, 0xbe, 0x80, 0xcc, 0xa9, 0x7e, 0x2d,
+       0xad, 0x58, 0x3b, 0x63, 0xb8, 0x5c, 0x17, 0xce,
+       0xbf, 0x43, 0x33, 0x7a, 0x7b, 0x82, 0x28, 0x2f
+};
+enum { nonce13 = 0x2a5d05d88cd7b0daULL };
+
+static const u8 input14[] __initconst = {
+       0x57, 0x4f, 0xaa, 0x30, 0xe6, 0x23, 0x50, 0x86,
+       0x91, 0xa5, 0x60, 0x96, 0x2b
+};
+static const u8 output14[] __initconst = {
+       0x6c, 0x1f, 0x3b, 0x42, 0xb6, 0x2f, 0xf0, 0xbd,
+       0x76, 0x60, 0xc7, 0x7e, 0x8d
+};
+static const u8 key14[] __initconst = {
+       0x22, 0x85, 0xaf, 0x8f, 0xa3, 0x53, 0xa0, 0xc4,
+       0xb5, 0x75, 0xc0, 0xba, 0x30, 0x92, 0xc3, 0x32,
+       0x20, 0x5a, 0x8f, 0x7e, 0x93, 0xda, 0x65, 0x18,
+       0xd1, 0xf6, 0x9a, 0x9b, 0x8f, 0x85, 0x30, 0xe6
+};
+enum { nonce14 = 0xf9946c166aa4475fULL };
+
+static const u8 input15[] __initconst = {
+       0x89, 0x81, 0xc7, 0xe2, 0x00, 0xac, 0x52, 0x70,
+       0xa4, 0x79, 0xab, 0xeb, 0x74, 0xf7
+};
+static const u8 output15[] __initconst = {
+       0xb4, 0xd0, 0xa9, 0x9d, 0x15, 0x5f, 0x48, 0xd6,
+       0x00, 0x7e, 0x4c, 0x77, 0x5a, 0x46
+};
+static const u8 key15[] __initconst = {
+       0x0a, 0x66, 0x36, 0xca, 0x5d, 0x82, 0x23, 0xb6,
+       0xe4, 0x9b, 0xad, 0x5e, 0xd0, 0x7f, 0xf6, 0x7a,
+       0x7b, 0x03, 0xa7, 0x4c, 0xfd, 0xec, 0xd5, 0xa1,
+       0xfc, 0x25, 0x54, 0xda, 0x5a, 0x5c, 0xf0, 0x2c
+};
+enum { nonce15 = 0x9ab2b87a35e772c8ULL };
+
+static const u8 input16[] __initconst = {
+       0x5f, 0x09, 0xc0, 0x8b, 0x1e, 0xde, 0xca, 0xd9,
+       0xb7, 0x5c, 0x23, 0xc9, 0x55, 0x1e, 0xcf
+};
+static const u8 output16[] __initconst = {
+       0x76, 0x9b, 0x53, 0xf3, 0x66, 0x88, 0x28, 0x60,
+       0x98, 0x80, 0x2c, 0xa8, 0x80, 0xa6, 0x48
+};
+static const u8 key16[] __initconst = {
+       0x80, 0xb5, 0x51, 0xdf, 0x17, 0x5b, 0xb0, 0xef,
+       0x8b, 0x5b, 0x2e, 0x3e, 0xc5, 0xe3, 0xa5, 0x86,
+       0xac, 0x0d, 0x8e, 0x32, 0x90, 0x9d, 0x82, 0x27,
+       0xf1, 0x23, 0x26, 0xc3, 0xea, 0x55, 0xb6, 0x63
+};
+enum { nonce16 = 0xa82e9d39e4d02ef5ULL };
+
+static const u8 input17[] __initconst = {
+       0x87, 0x0b, 0x36, 0x71, 0x7c, 0xb9, 0x0b, 0x80,
+       0x4d, 0x77, 0x5c, 0x4f, 0xf5, 0x51, 0x0e, 0x1a
+};
+static const u8 output17[] __initconst = {
+       0xf1, 0x12, 0x4a, 0x8a, 0xd9, 0xd0, 0x08, 0x67,
+       0x66, 0xd7, 0x34, 0xea, 0x32, 0x3b, 0x54, 0x0e
+};
+static const u8 key17[] __initconst = {
+       0xfb, 0x71, 0x5f, 0x3f, 0x7a, 0xc0, 0x9a, 0xc8,
+       0xc8, 0xcf, 0xe8, 0xbc, 0xfb, 0x09, 0xbf, 0x89,
+       0x6a, 0xef, 0xd5, 0xe5, 0x36, 0x87, 0x14, 0x76,
+       0x00, 0xb9, 0x32, 0x28, 0xb2, 0x00, 0x42, 0x53
+};
+enum { nonce17 = 0x229b87e73d557b96ULL };
+
+static const u8 input18[] __initconst = {
+       0x38, 0x42, 0xb5, 0x37, 0xb4, 0x3d, 0xfe, 0x59,
+       0x38, 0x68, 0x88, 0xfa, 0x89, 0x8a, 0x5f, 0x90,
+       0x3c
+};
+static const u8 output18[] __initconst = {
+       0xac, 0xad, 0x14, 0xe8, 0x7e, 0xd7, 0xce, 0x96,
+       0x3d, 0xb3, 0x78, 0x85, 0x22, 0x5a, 0xcb, 0x39,
+       0xd4
+};
+static const u8 key18[] __initconst = {
+       0xe1, 0xc1, 0xa8, 0xe0, 0x91, 0xe7, 0x38, 0x66,
+       0x80, 0x17, 0x12, 0x3c, 0x5e, 0x2d, 0xbb, 0xea,
+       0xeb, 0x6c, 0x8b, 0xc8, 0x1b, 0x6f, 0x7c, 0xea,
+       0x50, 0x57, 0x23, 0x1e, 0x65, 0x6f, 0x6d, 0x81
+};
+enum { nonce18 = 0xfaf5fcf8f30e57a9ULL };
+
+static const u8 input19[] __initconst = {
+       0x1c, 0x4a, 0x30, 0x26, 0xef, 0x9a, 0x32, 0xa7,
+       0x8f, 0xe5, 0xc0, 0x0f, 0x30, 0x3a, 0xbf, 0x38,
+       0x54, 0xba
+};
+static const u8 output19[] __initconst = {
+       0x57, 0x67, 0x54, 0x4f, 0x31, 0xd6, 0xef, 0x35,
+       0x0b, 0xd9, 0x52, 0xa7, 0x46, 0x7d, 0x12, 0x17,
+       0x1e, 0xe3
+};
+static const u8 key19[] __initconst = {
+       0x5a, 0x79, 0xc1, 0xea, 0x33, 0xb3, 0xc7, 0x21,
+       0xec, 0xf8, 0xcb, 0xd2, 0x58, 0x96, 0x23, 0xd6,
+       0x4d, 0xed, 0x2f, 0xdf, 0x8a, 0x79, 0xe6, 0x8b,
+       0x38, 0xa3, 0xc3, 0x7a, 0x33, 0xda, 0x02, 0xc7
+};
+enum { nonce19 = 0x2b23b61840429604ULL };
+
+static const u8 input20[] __initconst = {
+       0xab, 0xe9, 0x32, 0xbb, 0x35, 0x17, 0xe0, 0x60,
+       0x80, 0xb1, 0x27, 0xdc, 0xe6, 0x62, 0x9e, 0x0c,
+       0x77, 0xf4, 0x50
+};
+static const u8 output20[] __initconst = {
+       0x54, 0x6d, 0xaa, 0xfc, 0x08, 0xfb, 0x71, 0xa8,
+       0xd6, 0x1d, 0x7d, 0xf3, 0x45, 0x10, 0xb5, 0x4c,
+       0xcc, 0x4b, 0x45
+};
+static const u8 key20[] __initconst = {
+       0xa3, 0xfd, 0x3d, 0xa9, 0xeb, 0xea, 0x2c, 0x69,
+       0xcf, 0x59, 0x38, 0x13, 0x5b, 0xa7, 0x53, 0x8f,
+       0x5e, 0xa2, 0x33, 0x86, 0x4c, 0x75, 0x26, 0xaf,
+       0x35, 0x12, 0x09, 0x71, 0x81, 0xea, 0x88, 0x66
+};
+enum { nonce20 = 0x7459667a8fadff58ULL };
+
+static const u8 input21[] __initconst = {
+       0xa6, 0x82, 0x21, 0x23, 0xad, 0x27, 0x3f, 0xc6,
+       0xd7, 0x16, 0x0d, 0x6d, 0x24, 0x15, 0x54, 0xc5,
+       0x96, 0x72, 0x59, 0x8a
+};
+static const u8 output21[] __initconst = {
+       0x5f, 0x34, 0x32, 0xea, 0x06, 0xd4, 0x9e, 0x01,
+       0xdc, 0x32, 0x32, 0x40, 0x66, 0x73, 0x6d, 0x4a,
+       0x6b, 0x12, 0x20, 0xe8
+};
+static const u8 key21[] __initconst = {
+       0x96, 0xfd, 0x13, 0x23, 0xa9, 0x89, 0x04, 0xe6,
+       0x31, 0xa5, 0x2c, 0xc1, 0x40, 0xd5, 0x69, 0x5c,
+       0x32, 0x79, 0x56, 0xe0, 0x29, 0x93, 0x8f, 0xe8,
+       0x5f, 0x65, 0x53, 0x7f, 0xc1, 0xe9, 0xaf, 0xaf
+};
+enum { nonce21 = 0xba8defee9d8e13b5ULL };
+
+static const u8 input22[] __initconst = {
+       0xb8, 0x32, 0x1a, 0x81, 0xd8, 0x38, 0x89, 0x5a,
+       0xb0, 0x05, 0xbe, 0xf4, 0xd2, 0x08, 0xc6, 0xee,
+       0x79, 0x7b, 0x3a, 0x76, 0x59
+};
+static const u8 output22[] __initconst = {
+       0xb7, 0xba, 0xae, 0x80, 0xe4, 0x9f, 0x79, 0x84,
+       0x5a, 0x48, 0x50, 0x6d, 0xcb, 0xd0, 0x06, 0x0c,
+       0x15, 0x63, 0xa7, 0x5e, 0xbd
+};
+static const u8 key22[] __initconst = {
+       0x0f, 0x35, 0x3d, 0xeb, 0x5f, 0x0a, 0x82, 0x0d,
+       0x24, 0x59, 0x71, 0xd8, 0xe6, 0x2d, 0x5f, 0xe1,
+       0x7e, 0x0c, 0xae, 0xf6, 0xdc, 0x2c, 0xc5, 0x4a,
+       0x38, 0x88, 0xf2, 0xde, 0xd9, 0x5f, 0x76, 0x7c
+};
+enum { nonce22 = 0xe77f1760e9f5e192ULL };
+
+static const u8 input23[] __initconst = {
+       0x4b, 0x1e, 0x79, 0x99, 0xcf, 0xef, 0x64, 0x4b,
+       0xb0, 0x66, 0xae, 0x99, 0x2e, 0x68, 0x97, 0xf5,
+       0x5d, 0x9b, 0x3f, 0x7a, 0xa9, 0xd9
+};
+static const u8 output23[] __initconst = {
+       0x5f, 0xa4, 0x08, 0x39, 0xca, 0xfa, 0x2b, 0x83,
+       0x5d, 0x95, 0x70, 0x7c, 0x2e, 0xd4, 0xae, 0xfa,
+       0x45, 0x4a, 0x77, 0x7f, 0xa7, 0x65
+};
+static const u8 key23[] __initconst = {
+       0x4a, 0x06, 0x83, 0x64, 0xaa, 0xe3, 0x38, 0x32,
+       0x28, 0x5d, 0xa4, 0xb2, 0x5a, 0xee, 0xcf, 0x8e,
+       0x19, 0x67, 0xf1, 0x09, 0xe8, 0xc9, 0xf6, 0x40,
+       0x02, 0x6d, 0x0b, 0xde, 0xfa, 0x81, 0x03, 0xb1
+};
+enum { nonce23 = 0x9b3f349158709849ULL };
+
+static const u8 input24[] __initconst = {
+       0xc6, 0xfc, 0x47, 0x5e, 0xd8, 0xed, 0xa9, 0xe5,
+       0x4f, 0x82, 0x79, 0x35, 0xee, 0x3e, 0x7e, 0x3e,
+       0x35, 0x70, 0x6e, 0xfa, 0x6d, 0x08, 0xe8
+};
+static const u8 output24[] __initconst = {
+       0x3b, 0xc5, 0xf8, 0xc2, 0xbf, 0x2b, 0x90, 0x33,
+       0xa6, 0xae, 0xf5, 0x5a, 0x65, 0xb3, 0x3d, 0xe1,
+       0xcd, 0x5f, 0x55, 0xfa, 0xe7, 0xa5, 0x4a
+};
+static const u8 key24[] __initconst = {
+       0x00, 0x24, 0xc3, 0x65, 0x5f, 0xe6, 0x31, 0xbb,
+       0x6d, 0xfc, 0x20, 0x7b, 0x1b, 0xa8, 0x96, 0x26,
+       0x55, 0x21, 0x62, 0x25, 0x7e, 0xba, 0x23, 0x97,
+       0xc9, 0xb8, 0x53, 0xa8, 0xef, 0xab, 0xad, 0x61
+};
+enum { nonce24 = 0x13ee0b8f526177c3ULL };
+
+static const u8 input25[] __initconst = {
+       0x33, 0x07, 0x16, 0xb1, 0x34, 0x33, 0x67, 0x04,
+       0x9b, 0x0a, 0xce, 0x1b, 0xe9, 0xde, 0x1a, 0xec,
+       0xd0, 0x55, 0xfb, 0xc6, 0x33, 0xaf, 0x2d, 0xe3
+};
+static const u8 output25[] __initconst = {
+       0x05, 0x93, 0x10, 0xd1, 0x58, 0x6f, 0x68, 0x62,
+       0x45, 0xdb, 0x91, 0xae, 0x70, 0xcf, 0xd4, 0x5f,
+       0xee, 0xdf, 0xd5, 0xba, 0x9e, 0xde, 0x68, 0xe6
+};
+static const u8 key25[] __initconst = {
+       0x83, 0xa9, 0x4f, 0x5d, 0x74, 0xd5, 0x91, 0xb3,
+       0xc9, 0x97, 0x19, 0x15, 0xdb, 0x0d, 0x0b, 0x4a,
+       0x3d, 0x55, 0xcf, 0xab, 0xb2, 0x05, 0x21, 0x35,
+       0x45, 0x50, 0xeb, 0xf8, 0xf5, 0xbf, 0x36, 0x35
+};
+enum { nonce25 = 0x7c6f459e49ebfebcULL };
+
+static const u8 input26[] __initconst = {
+       0xc2, 0xd4, 0x7a, 0xa3, 0x92, 0xe1, 0xac, 0x46,
+       0x1a, 0x15, 0x38, 0xc9, 0xb5, 0xfd, 0xdf, 0x84,
+       0x38, 0xbc, 0x6b, 0x1d, 0xb0, 0x83, 0x43, 0x04,
+       0x39
+};
+static const u8 output26[] __initconst = {
+       0x7f, 0xde, 0xd6, 0x87, 0xcc, 0x34, 0xf4, 0x12,
+       0xae, 0x55, 0xa5, 0x89, 0x95, 0x29, 0xfc, 0x18,
+       0xd8, 0xc7, 0x7c, 0xd3, 0xcb, 0x85, 0x95, 0x21,
+       0xd2
+};
+static const u8 key26[] __initconst = {
+       0xe4, 0xd0, 0x54, 0x1d, 0x7d, 0x47, 0xa8, 0xc1,
+       0x08, 0xca, 0xe2, 0x42, 0x52, 0x95, 0x16, 0x43,
+       0xa3, 0x01, 0x23, 0x03, 0xcc, 0x3b, 0x81, 0x78,
+       0x23, 0xcc, 0xa7, 0x36, 0xd7, 0xa0, 0x97, 0x8d
+};
+enum { nonce26 = 0x524401012231683ULL };
+
+static const u8 input27[] __initconst = {
+       0x0d, 0xb0, 0xcf, 0xec, 0xfc, 0x38, 0x9d, 0x9d,
+       0x89, 0x00, 0x96, 0xf2, 0x79, 0x8a, 0xa1, 0x8d,
+       0x32, 0x5e, 0xc6, 0x12, 0x22, 0xec, 0xf6, 0x52,
+       0xc1, 0x0b
+};
+static const u8 output27[] __initconst = {
+       0xef, 0xe1, 0xf2, 0x67, 0x8e, 0x2c, 0x00, 0x9f,
+       0x1d, 0x4c, 0x66, 0x1f, 0x94, 0x58, 0xdc, 0xbb,
+       0xb9, 0x11, 0x8f, 0x74, 0xfd, 0x0e, 0x14, 0x01,
+       0xa8, 0x21
+};
+static const u8 key27[] __initconst = {
+       0x78, 0x71, 0xa4, 0xe6, 0xb2, 0x95, 0x44, 0x12,
+       0x81, 0xaa, 0x7e, 0x94, 0xa7, 0x8d, 0x44, 0xea,
+       0xc4, 0xbc, 0x01, 0xb7, 0x9e, 0xf7, 0x82, 0x9e,
+       0x3b, 0x23, 0x9f, 0x31, 0xdd, 0xb8, 0x0d, 0x18
+};
+enum { nonce27 = 0xd58fe0e58fb254d6ULL };
+
+static const u8 input28[] __initconst = {
+       0xaa, 0xb7, 0xaa, 0xd9, 0xa8, 0x91, 0xd7, 0x8a,
+       0x97, 0x9b, 0xdb, 0x7c, 0x47, 0x2b, 0xdb, 0xd2,
+       0xda, 0x77, 0xb1, 0xfa, 0x2d, 0x12, 0xe3, 0xe9,
+       0xc4, 0x7f, 0x54
+};
+static const u8 output28[] __initconst = {
+       0x87, 0x84, 0xa9, 0xa6, 0xad, 0x8f, 0xe6, 0x0f,
+       0x69, 0xf8, 0x21, 0xc3, 0x54, 0x95, 0x0f, 0xb0,
+       0x4e, 0xc7, 0x02, 0xe4, 0x04, 0xb0, 0x6c, 0x42,
+       0x8c, 0x63, 0xe3
+};
+static const u8 key28[] __initconst = {
+       0x12, 0x23, 0x37, 0x95, 0x04, 0xb4, 0x21, 0xe8,
+       0xbc, 0x65, 0x46, 0x7a, 0xf4, 0x01, 0x05, 0x3f,
+       0xb1, 0x34, 0x73, 0xd2, 0x49, 0xbf, 0x6f, 0x20,
+       0xbd, 0x23, 0x58, 0x5f, 0xd1, 0x73, 0x57, 0xa6
+};
+enum { nonce28 = 0x3a04d51491eb4e07ULL };
+
+static const u8 input29[] __initconst = {
+       0x55, 0xd0, 0xd4, 0x4b, 0x17, 0xc8, 0xc4, 0x2b,
+       0xc0, 0x28, 0xbd, 0x9d, 0x65, 0x4d, 0xaf, 0x77,
+       0x72, 0x7c, 0x36, 0x68, 0xa7, 0xb6, 0x87, 0x4d,
+       0xb9, 0x27, 0x25, 0x6c
+};
+static const u8 output29[] __initconst = {
+       0x0e, 0xac, 0x4c, 0xf5, 0x12, 0xb5, 0x56, 0xa5,
+       0x00, 0x9a, 0xd6, 0xe5, 0x1a, 0x59, 0x2c, 0xf6,
+       0x42, 0x22, 0xcf, 0x23, 0x98, 0x34, 0x29, 0xac,
+       0x6e, 0xe3, 0x37, 0x6d
+};
+static const u8 key29[] __initconst = {
+       0xda, 0x9d, 0x05, 0x0c, 0x0c, 0xba, 0x75, 0xb9,
+       0x9e, 0xb1, 0x8d, 0xd9, 0x73, 0x26, 0x2c, 0xa9,
+       0x3a, 0xb5, 0xcb, 0x19, 0x49, 0xa7, 0x4f, 0xf7,
+       0x64, 0x35, 0x23, 0x20, 0x2a, 0x45, 0x78, 0xc7
+};
+enum { nonce29 = 0xc25ac9982431cbfULL };
+
+static const u8 input30[] __initconst = {
+       0x4e, 0xd6, 0x85, 0xbb, 0xe7, 0x99, 0xfa, 0x04,
+       0x33, 0x24, 0xfd, 0x75, 0x18, 0xe3, 0xd3, 0x25,
+       0xcd, 0xca, 0xae, 0x00, 0xbe, 0x52, 0x56, 0x4a,
+       0x31, 0xe9, 0x4f, 0xae, 0x8a
+};
+static const u8 output30[] __initconst = {
+       0x30, 0x36, 0x32, 0xa2, 0x3c, 0xb6, 0xf9, 0xf9,
+       0x76, 0x70, 0xad, 0xa6, 0x10, 0x41, 0x00, 0x4a,
+       0xfa, 0xce, 0x1b, 0x86, 0x05, 0xdb, 0x77, 0x96,
+       0xb3, 0xb7, 0x8f, 0x61, 0x24
+};
+static const u8 key30[] __initconst = {
+       0x49, 0x35, 0x4c, 0x15, 0x98, 0xfb, 0xc6, 0x57,
+       0x62, 0x6d, 0x06, 0xc3, 0xd4, 0x79, 0x20, 0x96,
+       0x05, 0x2a, 0x31, 0x63, 0xc0, 0x44, 0x42, 0x09,
+       0x13, 0x13, 0xff, 0x1b, 0xc8, 0x63, 0x1f, 0x0b
+};
+enum { nonce30 = 0x4967f9c08e41568bULL };
+
+static const u8 input31[] __initconst = {
+       0x91, 0x04, 0x20, 0x47, 0x59, 0xee, 0xa6, 0x0f,
+       0x04, 0x75, 0xc8, 0x18, 0x95, 0x44, 0x01, 0x28,
+       0x20, 0x6f, 0x73, 0x68, 0x66, 0xb5, 0x03, 0xb3,
+       0x58, 0x27, 0x6e, 0x7a, 0x76, 0xb8
+};
+static const u8 output31[] __initconst = {
+       0xe8, 0x03, 0x78, 0x9d, 0x13, 0x15, 0x98, 0xef,
+       0x64, 0x68, 0x12, 0x41, 0xb0, 0x29, 0x94, 0x0c,
+       0x83, 0x35, 0x46, 0xa9, 0x74, 0xe1, 0x75, 0xf0,
+       0xb6, 0x96, 0xc3, 0x6f, 0xd7, 0x70
+};
+static const u8 key31[] __initconst = {
+       0xef, 0xcd, 0x5a, 0x4a, 0xf4, 0x7e, 0x6a, 0x3a,
+       0x11, 0x88, 0x72, 0x94, 0xb8, 0xae, 0x84, 0xc3,
+       0x66, 0xe0, 0xde, 0x4b, 0x00, 0xa5, 0xd6, 0x2d,
+       0x50, 0xb7, 0x28, 0xff, 0x76, 0x57, 0x18, 0x1f
+};
+enum { nonce31 = 0xcb6f428fa4192e19ULL };
+
+static const u8 input32[] __initconst = {
+       0x90, 0x06, 0x50, 0x4b, 0x98, 0x14, 0x30, 0xf1,
+       0xb8, 0xd7, 0xf0, 0xa4, 0x3e, 0x4e, 0xd8, 0x00,
+       0xea, 0xdb, 0x4f, 0x93, 0x05, 0xef, 0x02, 0x71,
+       0x1a, 0xcd, 0xa3, 0xb1, 0xae, 0xd3, 0x18
+};
+static const u8 output32[] __initconst = {
+       0xcb, 0x4a, 0x37, 0x3f, 0xea, 0x40, 0xab, 0x86,
+       0xfe, 0xcc, 0x07, 0xd5, 0xdc, 0xb2, 0x25, 0xb6,
+       0xfd, 0x2a, 0x72, 0xbc, 0x5e, 0xd4, 0x75, 0xff,
+       0x71, 0xfc, 0xce, 0x1e, 0x6f, 0x22, 0xc1
+};
+static const u8 key32[] __initconst = {
+       0xfc, 0x6d, 0xc3, 0x80, 0xce, 0xa4, 0x31, 0xa1,
+       0xcc, 0xfa, 0x9d, 0x10, 0x0b, 0xc9, 0x11, 0x77,
+       0x34, 0xdb, 0xad, 0x1b, 0xc4, 0xfc, 0xeb, 0x79,
+       0x91, 0xda, 0x59, 0x3b, 0x0d, 0xb1, 0x19, 0x3b
+};
+enum { nonce32 = 0x88551bf050059467ULL };
+
+static const u8 input33[] __initconst = {
+       0x88, 0x94, 0x71, 0x92, 0xe8, 0xd7, 0xf9, 0xbd,
+       0x55, 0xe3, 0x22, 0xdb, 0x99, 0x51, 0xfb, 0x50,
+       0xbf, 0x82, 0xb5, 0x70, 0x8b, 0x2b, 0x6a, 0x03,
+       0x37, 0xa0, 0xc6, 0x19, 0x5d, 0xc9, 0xbc, 0xcc
+};
+static const u8 output33[] __initconst = {
+       0xb6, 0x17, 0x51, 0xc8, 0xea, 0x8a, 0x14, 0xdc,
+       0x23, 0x1b, 0xd4, 0xed, 0xbf, 0x50, 0xb9, 0x38,
+       0x00, 0xc2, 0x3f, 0x78, 0x3d, 0xbf, 0xa0, 0x84,
+       0xef, 0x45, 0xb2, 0x7d, 0x48, 0x7b, 0x62, 0xa7
+};
+static const u8 key33[] __initconst = {
+       0xb9, 0x8f, 0x6a, 0xad, 0xb4, 0x6f, 0xb5, 0xdc,
+       0x48, 0xfa, 0x43, 0x57, 0x62, 0x97, 0xef, 0x89,
+       0x4c, 0x5a, 0x7b, 0x67, 0xb8, 0x9d, 0xf0, 0x42,
+       0x2b, 0x8f, 0xf3, 0x18, 0x05, 0x2e, 0x48, 0xd0
+};
+enum { nonce33 = 0x31f16488fe8447f5ULL };
+
+static const u8 input34[] __initconst = {
+       0xda, 0x2b, 0x3d, 0x63, 0x9e, 0x4f, 0xc2, 0xb8,
+       0x7f, 0xc2, 0x1a, 0x8b, 0x0d, 0x95, 0x65, 0x55,
+       0x52, 0xba, 0x51, 0x51, 0xc0, 0x61, 0x9f, 0x0a,
+       0x5d, 0xb0, 0x59, 0x8c, 0x64, 0x6a, 0xab, 0xf5,
+       0x57
+};
+static const u8 output34[] __initconst = {
+       0x5c, 0xf6, 0x62, 0x24, 0x8c, 0x45, 0xa3, 0x26,
+       0xd0, 0xe4, 0x88, 0x1c, 0xed, 0xc4, 0x26, 0x58,
+       0xb5, 0x5d, 0x92, 0xc4, 0x17, 0x44, 0x1c, 0xb8,
+       0x2c, 0xf3, 0x55, 0x7e, 0xd6, 0xe5, 0xb3, 0x65,
+       0xa8
+};
+static const u8 key34[] __initconst = {
+       0xde, 0xd1, 0x27, 0xb7, 0x7c, 0xfa, 0xa6, 0x78,
+       0x39, 0x80, 0xdf, 0xb7, 0x46, 0xac, 0x71, 0x26,
+       0xd0, 0x2a, 0x56, 0x79, 0x12, 0xeb, 0x26, 0x37,
+       0x01, 0x0d, 0x30, 0xe0, 0xe3, 0x66, 0xb2, 0xf4
+};
+enum { nonce34 = 0x92d0d9b252c24149ULL };
+
+static const u8 input35[] __initconst = {
+       0x3a, 0x15, 0x5b, 0x75, 0x6e, 0xd0, 0x52, 0x20,
+       0x6c, 0x82, 0xfa, 0xce, 0x5b, 0xea, 0xf5, 0x43,
+       0xc1, 0x81, 0x7c, 0xb2, 0xac, 0x16, 0x3f, 0xd3,
+       0x5a, 0xaf, 0x55, 0x98, 0xf4, 0xc6, 0xba, 0x71,
+       0x25, 0x8b
+};
+static const u8 output35[] __initconst = {
+       0xb3, 0xaf, 0xac, 0x6d, 0x4d, 0xc7, 0x68, 0x56,
+       0x50, 0x5b, 0x69, 0x2a, 0xe5, 0x90, 0xf9, 0x5f,
+       0x99, 0x88, 0xff, 0x0c, 0xa6, 0xb1, 0x83, 0xd6,
+       0x80, 0xa6, 0x1b, 0xde, 0x94, 0xa4, 0x2c, 0xc3,
+       0x74, 0xfa
+};
+static const u8 key35[] __initconst = {
+       0xd8, 0x24, 0xe2, 0x06, 0xd7, 0x7a, 0xce, 0x81,
+       0x52, 0x72, 0x02, 0x69, 0x89, 0xc4, 0xe9, 0x53,
+       0x3b, 0x08, 0x5f, 0x98, 0x1e, 0x1b, 0x99, 0x6e,
+       0x28, 0x17, 0x6d, 0xba, 0xc0, 0x96, 0xf9, 0x3c
+};
+enum { nonce35 = 0x7baf968c4c8e3a37ULL };
+
+static const u8 input36[] __initconst = {
+       0x31, 0x5d, 0x4f, 0xe3, 0xac, 0xad, 0x17, 0xa6,
+       0xb5, 0x01, 0xe2, 0xc6, 0xd4, 0x7e, 0xc4, 0x80,
+       0xc0, 0x59, 0x72, 0xbb, 0x4b, 0x74, 0x6a, 0x41,
+       0x0f, 0x9c, 0xf6, 0xca, 0x20, 0xb3, 0x73, 0x07,
+       0x6b, 0x02, 0x2a
+};
+static const u8 output36[] __initconst = {
+       0xf9, 0x09, 0x92, 0x94, 0x7e, 0x31, 0xf7, 0x53,
+       0xe8, 0x8a, 0x5b, 0x20, 0xef, 0x9b, 0x45, 0x81,
+       0xba, 0x5e, 0x45, 0x63, 0xc1, 0xc7, 0x9e, 0x06,
+       0x0e, 0xd9, 0x62, 0x8e, 0x96, 0xf9, 0xfa, 0x43,
+       0x4d, 0xd4, 0x28
+};
+static const u8 key36[] __initconst = {
+       0x13, 0x30, 0x4c, 0x06, 0xae, 0x18, 0xde, 0x03,
+       0x1d, 0x02, 0x40, 0xf5, 0xbb, 0x19, 0xe3, 0x88,
+       0x41, 0xb1, 0x29, 0x15, 0x97, 0xc2, 0x69, 0x3f,
+       0x32, 0x2a, 0x0c, 0x8b, 0xcf, 0x83, 0x8b, 0x6c
+};
+enum { nonce36 = 0x226d251d475075a0ULL };
+
+static const u8 input37[] __initconst = {
+       0x10, 0x18, 0xbe, 0xfd, 0x66, 0xc9, 0x77, 0xcc,
+       0x43, 0xe5, 0x46, 0x0b, 0x08, 0x8b, 0xae, 0x11,
+       0x86, 0x15, 0xc2, 0xf6, 0x45, 0xd4, 0x5f, 0xd6,
+       0xb6, 0x5f, 0x9f, 0x3e, 0x97, 0xb7, 0xd4, 0xad,
+       0x0b, 0xe8, 0x31, 0x94
+};
+static const u8 output37[] __initconst = {
+       0x03, 0x2c, 0x1c, 0xee, 0xc6, 0xdd, 0xed, 0x38,
+       0x80, 0x6d, 0x84, 0x16, 0xc3, 0xc2, 0x04, 0x63,
+       0xcd, 0xa7, 0x6e, 0x36, 0x8b, 0xed, 0x78, 0x63,
+       0x95, 0xfc, 0x69, 0x7a, 0x3f, 0x8d, 0x75, 0x6b,
+       0x6c, 0x26, 0x56, 0x4d
+};
+static const u8 key37[] __initconst = {
+       0xac, 0x84, 0x4d, 0xa9, 0x29, 0x49, 0x3c, 0x39,
+       0x7f, 0xd9, 0xa6, 0x01, 0xf3, 0x7e, 0xfa, 0x4a,
+       0x14, 0x80, 0x22, 0x74, 0xf0, 0x29, 0x30, 0x2d,
+       0x07, 0x21, 0xda, 0xc0, 0x4d, 0x70, 0x56, 0xa2
+};
+enum { nonce37 = 0x167823ce3b64925aULL };
+
+static const u8 input38[] __initconst = {
+       0x30, 0x8f, 0xfa, 0x24, 0x29, 0xb1, 0xfb, 0xce,
+       0x31, 0x62, 0xdc, 0xd0, 0x46, 0xab, 0xe1, 0x31,
+       0xd9, 0xae, 0x60, 0x0d, 0xca, 0x0a, 0x49, 0x12,
+       0x3d, 0x92, 0xe9, 0x91, 0x67, 0x12, 0x62, 0x18,
+       0x89, 0xe2, 0xf9, 0x1c, 0xcc
+};
+static const u8 output38[] __initconst = {
+       0x56, 0x9c, 0xc8, 0x7a, 0xc5, 0x98, 0xa3, 0x0f,
+       0xba, 0xd5, 0x3e, 0xe1, 0xc9, 0x33, 0x64, 0x33,
+       0xf0, 0xd5, 0xf7, 0x43, 0x66, 0x0e, 0x08, 0x9a,
+       0x6e, 0x09, 0xe4, 0x01, 0x0d, 0x1e, 0x2f, 0x4b,
+       0xed, 0x9c, 0x08, 0x8c, 0x03
+};
+static const u8 key38[] __initconst = {
+       0x77, 0x52, 0x2a, 0x23, 0xf1, 0xc5, 0x96, 0x2b,
+       0x89, 0x4f, 0x3e, 0xf3, 0xff, 0x0e, 0x94, 0xce,
+       0xf1, 0xbd, 0x53, 0xf5, 0x77, 0xd6, 0x9e, 0x47,
+       0x49, 0x3d, 0x16, 0x64, 0xff, 0x95, 0x42, 0x42
+};
+enum { nonce38 = 0xff629d7b82cef357ULL };
+
+static const u8 input39[] __initconst = {
+       0x38, 0x26, 0x27, 0xd0, 0xc2, 0xf5, 0x34, 0xba,
+       0xda, 0x0f, 0x1c, 0x1c, 0x9a, 0x70, 0xe5, 0x8a,
+       0x78, 0x2d, 0x8f, 0x9a, 0xbf, 0x89, 0x6a, 0xfd,
+       0xd4, 0x9c, 0x33, 0xf1, 0xb6, 0x89, 0x16, 0xe3,
+       0x6a, 0x00, 0xfa, 0x3a, 0x0f, 0x26
+};
+static const u8 output39[] __initconst = {
+       0x0f, 0xaf, 0x91, 0x6d, 0x9c, 0x99, 0xa4, 0xf7,
+       0x3b, 0x9d, 0x9a, 0x98, 0xca, 0xbb, 0x50, 0x48,
+       0xee, 0xcb, 0x5d, 0xa1, 0x37, 0x2d, 0x36, 0x09,
+       0x2a, 0xe2, 0x1c, 0x3d, 0x98, 0x40, 0x1c, 0x16,
+       0x56, 0xa7, 0x98, 0xe9, 0x7d, 0x2b
+};
+static const u8 key39[] __initconst = {
+       0x6e, 0x83, 0x15, 0x4d, 0xf8, 0x78, 0xa8, 0x0e,
+       0x71, 0x37, 0xd4, 0x6e, 0x28, 0x5c, 0x06, 0xa1,
+       0x2d, 0x6c, 0x72, 0x7a, 0xfd, 0xf8, 0x65, 0x1a,
+       0xb8, 0xe6, 0x29, 0x7b, 0xe5, 0xb3, 0x23, 0x79
+};
+enum { nonce39 = 0xa4d8c491cf093e9dULL };
+
+static const u8 input40[] __initconst = {
+       0x8f, 0x32, 0x7c, 0x40, 0x37, 0x95, 0x08, 0x00,
+       0x00, 0xfe, 0x2f, 0x95, 0x20, 0x12, 0x40, 0x18,
+       0x5e, 0x7e, 0x5e, 0x99, 0xee, 0x8d, 0x91, 0x7d,
+       0x50, 0x7d, 0x21, 0x45, 0x27, 0xe1, 0x7f, 0xd4,
+       0x73, 0x10, 0xe1, 0x33, 0xbc, 0xf8, 0xdd
+};
+static const u8 output40[] __initconst = {
+       0x78, 0x7c, 0xdc, 0x55, 0x2b, 0xd9, 0x2b, 0x3a,
+       0xdd, 0x56, 0x11, 0x52, 0xd3, 0x2e, 0xe0, 0x0d,
+       0x23, 0x20, 0x8a, 0xf1, 0x4f, 0xee, 0xf1, 0x68,
+       0xf6, 0xdc, 0x53, 0xcf, 0x17, 0xd4, 0xf0, 0x6c,
+       0xdc, 0x80, 0x5f, 0x1c, 0xa4, 0x91, 0x05
+};
+static const u8 key40[] __initconst = {
+       0x0d, 0x86, 0xbf, 0x8a, 0xba, 0x9e, 0x39, 0x91,
+       0xa8, 0xe7, 0x22, 0xf0, 0x0c, 0x43, 0x18, 0xe4,
+       0x1f, 0xb0, 0xaf, 0x8a, 0x34, 0x31, 0xf4, 0x41,
+       0xf0, 0x89, 0x85, 0xca, 0x5d, 0x05, 0x3b, 0x94
+};
+enum { nonce40 = 0xae7acc4f5986439eULL };
+
+static const u8 input41[] __initconst = {
+       0x20, 0x5f, 0xc1, 0x83, 0x36, 0x02, 0x76, 0x96,
+       0xf0, 0xbf, 0x8e, 0x0e, 0x1a, 0xd1, 0xc7, 0x88,
+       0x18, 0xc7, 0x09, 0xc4, 0x15, 0xd9, 0x4f, 0x5e,
+       0x1f, 0xb3, 0xb4, 0x6d, 0xcb, 0xa0, 0xd6, 0x8a,
+       0x3b, 0x40, 0x8e, 0x80, 0xf1, 0xe8, 0x8f, 0x5f
+};
+static const u8 output41[] __initconst = {
+       0x0b, 0xd1, 0x49, 0x9a, 0x9d, 0xe8, 0x97, 0xb8,
+       0xd1, 0xeb, 0x90, 0x62, 0x37, 0xd2, 0x99, 0x15,
+       0x67, 0x6d, 0x27, 0x93, 0xce, 0x37, 0x65, 0xa2,
+       0x94, 0x88, 0xd6, 0x17, 0xbc, 0x1c, 0x6e, 0xa2,
+       0xcc, 0xfb, 0x81, 0x0e, 0x30, 0x60, 0x5a, 0x6f
+};
+static const u8 key41[] __initconst = {
+       0x36, 0x27, 0x57, 0x01, 0x21, 0x68, 0x97, 0xc7,
+       0x00, 0x67, 0x7b, 0xe9, 0x0f, 0x55, 0x49, 0xbb,
+       0x92, 0x18, 0x98, 0xf5, 0x5e, 0xbc, 0xe7, 0x5a,
+       0x9d, 0x3d, 0xc7, 0xbd, 0x59, 0xec, 0x82, 0x8e
+};
+enum { nonce41 = 0x5da05e4c8dfab464ULL };
+
+static const u8 input42[] __initconst = {
+       0xca, 0x30, 0xcd, 0x63, 0xf0, 0x2d, 0xf1, 0x03,
+       0x4d, 0x0d, 0xf2, 0xf7, 0x6f, 0xae, 0xd6, 0x34,
+       0xea, 0xf6, 0x13, 0xcf, 0x1c, 0xa0, 0xd0, 0xe8,
+       0xa4, 0x78, 0x80, 0x3b, 0x1e, 0xa5, 0x32, 0x4c,
+       0x73, 0x12, 0xd4, 0x6a, 0x94, 0xbc, 0xba, 0x80,
+       0x5e
+};
+static const u8 output42[] __initconst = {
+       0xec, 0x3f, 0x18, 0x31, 0xc0, 0x7b, 0xb5, 0xe2,
+       0xad, 0xf3, 0xec, 0xa0, 0x16, 0x9d, 0xef, 0xce,
+       0x05, 0x65, 0x59, 0x9d, 0x5a, 0xca, 0x3e, 0x13,
+       0xb9, 0x5d, 0x5d, 0xb5, 0xeb, 0xae, 0xc0, 0x87,
+       0xbb, 0xfd, 0xe7, 0xe4, 0x89, 0x5b, 0xd2, 0x6c,
+       0x56
+};
+static const u8 key42[] __initconst = {
+       0x7c, 0x6b, 0x7e, 0x77, 0xcc, 0x8c, 0x1b, 0x03,
+       0x8b, 0x2a, 0xb3, 0x7c, 0x5a, 0x73, 0xcc, 0xac,
+       0xdd, 0x53, 0x54, 0x0c, 0x85, 0xed, 0xcd, 0x47,
+       0x24, 0xc1, 0xb8, 0x9b, 0x2e, 0x41, 0x92, 0x36
+};
+enum { nonce42 = 0xe4d7348b09682c9cULL };
+
+static const u8 input43[] __initconst = {
+       0x52, 0xf2, 0x4b, 0x7c, 0xe5, 0x58, 0xe8, 0xd2,
+       0xb7, 0xf3, 0xa1, 0x29, 0x68, 0xa2, 0x50, 0x50,
+       0xae, 0x9c, 0x1b, 0xe2, 0x67, 0x77, 0xe2, 0xdb,
+       0x85, 0x55, 0x7e, 0x84, 0x8a, 0x12, 0x3c, 0xb6,
+       0x2e, 0xed, 0xd3, 0xec, 0x47, 0x68, 0xfa, 0x52,
+       0x46, 0x9d
+};
+static const u8 output43[] __initconst = {
+       0x1b, 0xf0, 0x05, 0xe4, 0x1c, 0xd8, 0x74, 0x9a,
+       0xf0, 0xee, 0x00, 0x54, 0xce, 0x02, 0x83, 0x15,
+       0xfb, 0x23, 0x35, 0x78, 0xc3, 0xda, 0x98, 0xd8,
+       0x9d, 0x1b, 0xb2, 0x51, 0x82, 0xb0, 0xff, 0xbe,
+       0x05, 0xa9, 0xa4, 0x04, 0xba, 0xea, 0x4b, 0x73,
+       0x47, 0x6e
+};
+static const u8 key43[] __initconst = {
+       0xeb, 0xec, 0x0e, 0xa1, 0x65, 0xe2, 0x99, 0x46,
+       0xd8, 0x54, 0x8c, 0x4a, 0x93, 0xdf, 0x6d, 0xbf,
+       0x93, 0x34, 0x94, 0x57, 0xc9, 0x12, 0x9d, 0x68,
+       0x05, 0xc5, 0x05, 0xad, 0x5a, 0xc9, 0x2a, 0x3b
+};
+enum { nonce43 = 0xe14f6a902b7827fULL };
+
+static const u8 input44[] __initconst = {
+       0x3e, 0x22, 0x3e, 0x8e, 0xcd, 0x18, 0xe2, 0xa3,
+       0x8d, 0x8b, 0x38, 0xc3, 0x02, 0xa3, 0x31, 0x48,
+       0xc6, 0x0e, 0xec, 0x99, 0x51, 0x11, 0x6d, 0x8b,
+       0x32, 0x35, 0x3b, 0x08, 0x58, 0x76, 0x25, 0x30,
+       0xe2, 0xfc, 0xa2, 0x46, 0x7d, 0x6e, 0x34, 0x87,
+       0xac, 0x42, 0xbf
+};
+static const u8 output44[] __initconst = {
+       0x08, 0x92, 0x58, 0x02, 0x1a, 0xf4, 0x1f, 0x3d,
+       0x38, 0x7b, 0x6b, 0xf6, 0x84, 0x07, 0xa3, 0x19,
+       0x17, 0x2a, 0xed, 0x57, 0x1c, 0xf9, 0x55, 0x37,
+       0x4e, 0xf4, 0x68, 0x68, 0x82, 0x02, 0x4f, 0xca,
+       0x21, 0x00, 0xc6, 0x66, 0x79, 0x53, 0x19, 0xef,
+       0x7f, 0xdd, 0x74
+};
+static const u8 key44[] __initconst = {
+       0x73, 0xb6, 0x3e, 0xf4, 0x57, 0x52, 0xa6, 0x43,
+       0x51, 0xd8, 0x25, 0x00, 0xdb, 0xb4, 0x52, 0x69,
+       0xd6, 0x27, 0x49, 0xeb, 0x9b, 0xf1, 0x7b, 0xa0,
+       0xd6, 0x7c, 0x9c, 0xd8, 0x95, 0x03, 0x69, 0x26
+};
+enum { nonce44 = 0xf5e6dc4f35ce24e5ULL };
+
+static const u8 input45[] __initconst = {
+       0x55, 0x76, 0xc0, 0xf1, 0x74, 0x03, 0x7a, 0x6d,
+       0x14, 0xd8, 0x36, 0x2c, 0x9f, 0x9a, 0x59, 0x7a,
+       0x2a, 0xf5, 0x77, 0x84, 0x70, 0x7c, 0x1d, 0x04,
+       0x90, 0x45, 0xa4, 0xc1, 0x5e, 0xdd, 0x2e, 0x07,
+       0x18, 0x34, 0xa6, 0x85, 0x56, 0x4f, 0x09, 0xaf,
+       0x2f, 0x83, 0xe1, 0xc6
+};
+static const u8 output45[] __initconst = {
+       0x22, 0x46, 0xe4, 0x0b, 0x3a, 0x55, 0xcc, 0x9b,
+       0xf0, 0xc0, 0x53, 0xcd, 0x95, 0xc7, 0x57, 0x6c,
+       0x77, 0x46, 0x41, 0x72, 0x07, 0xbf, 0xa8, 0xe5,
+       0x68, 0x69, 0xd8, 0x1e, 0x45, 0xc1, 0xa2, 0x50,
+       0xa5, 0xd1, 0x62, 0xc9, 0x5a, 0x7d, 0x08, 0x14,
+       0xae, 0x44, 0x16, 0xb9
+};
+static const u8 key45[] __initconst = {
+       0x41, 0xf3, 0x88, 0xb2, 0x51, 0x25, 0x47, 0x02,
+       0x39, 0xe8, 0x15, 0x3a, 0x22, 0x78, 0x86, 0x0b,
+       0xf9, 0x1e, 0x8d, 0x98, 0xb2, 0x22, 0x82, 0xac,
+       0x42, 0x94, 0xde, 0x64, 0xf0, 0xfd, 0xb3, 0x6c
+};
+enum { nonce45 = 0xf51a582daf4aa01aULL };
+
+static const u8 input46[] __initconst = {
+       0xf6, 0xff, 0x20, 0xf9, 0x26, 0x7e, 0x0f, 0xa8,
+       0x6a, 0x45, 0x5a, 0x91, 0x73, 0xc4, 0x4c, 0x63,
+       0xe5, 0x61, 0x59, 0xca, 0xec, 0xc0, 0x20, 0x35,
+       0xbc, 0x9f, 0x58, 0x9c, 0x5e, 0xa1, 0x17, 0x46,
+       0xcc, 0xab, 0x6e, 0xd0, 0x4f, 0x24, 0xeb, 0x05,
+       0x4d, 0x40, 0x41, 0xe0, 0x9d
+};
+static const u8 output46[] __initconst = {
+       0x31, 0x6e, 0x63, 0x3f, 0x9c, 0xe6, 0xb1, 0xb7,
+       0xef, 0x47, 0x46, 0xd7, 0xb1, 0x53, 0x42, 0x2f,
+       0x2c, 0xc8, 0x01, 0xae, 0x8b, 0xec, 0x42, 0x2c,
+       0x6b, 0x2c, 0x9c, 0xb2, 0xf0, 0x29, 0x06, 0xa5,
+       0xcd, 0x7e, 0xc7, 0x3a, 0x38, 0x98, 0x8a, 0xde,
+       0x03, 0x29, 0x14, 0x8f, 0xf9
+};
+static const u8 key46[] __initconst = {
+       0xac, 0xa6, 0x44, 0x4a, 0x0d, 0x42, 0x10, 0xbc,
+       0xd3, 0xc9, 0x8e, 0x9e, 0x71, 0xa3, 0x1c, 0x14,
+       0x9d, 0x65, 0x0d, 0x49, 0x4d, 0x8c, 0xec, 0x46,
+       0xe1, 0x41, 0xcd, 0xf5, 0xfc, 0x82, 0x75, 0x34
+};
+enum { nonce46 = 0x25f85182df84dec5ULL };
+
+static const u8 input47[] __initconst = {
+       0xa1, 0xd2, 0xf2, 0x52, 0x2f, 0x79, 0x50, 0xb2,
+       0x42, 0x29, 0x5b, 0x44, 0x20, 0xf9, 0xbd, 0x85,
+       0xb7, 0x65, 0x77, 0x86, 0xce, 0x3e, 0x1c, 0xe4,
+       0x70, 0x80, 0xdd, 0x72, 0x07, 0x48, 0x0f, 0x84,
+       0x0d, 0xfd, 0x97, 0xc0, 0xb7, 0x48, 0x9b, 0xb4,
+       0xec, 0xff, 0x73, 0x14, 0x99, 0xe4
+};
+static const u8 output47[] __initconst = {
+       0xe5, 0x3c, 0x78, 0x66, 0x31, 0x1e, 0xd6, 0xc4,
+       0x9e, 0x71, 0xb3, 0xd7, 0xd5, 0xad, 0x84, 0xf2,
+       0x78, 0x61, 0x77, 0xf8, 0x31, 0xf0, 0x13, 0xad,
+       0x66, 0xf5, 0x31, 0x7d, 0xeb, 0xdf, 0xaf, 0xcb,
+       0xac, 0x28, 0x6c, 0xc2, 0x9e, 0xe7, 0x78, 0xa2,
+       0xa2, 0x58, 0xce, 0x84, 0x76, 0x70
+};
+static const u8 key47[] __initconst = {
+       0x05, 0x7f, 0xc0, 0x7f, 0x37, 0x20, 0x71, 0x02,
+       0x3a, 0xe7, 0x20, 0x5a, 0x0a, 0x8f, 0x79, 0x5a,
+       0xfe, 0xbb, 0x43, 0x4d, 0x2f, 0xcb, 0xf6, 0x9e,
+       0xa2, 0x97, 0x00, 0xad, 0x0d, 0x51, 0x7e, 0x17
+};
+enum { nonce47 = 0xae707c60f54de32bULL };
+
+static const u8 input48[] __initconst = {
+       0x80, 0x93, 0x77, 0x2e, 0x8d, 0xe8, 0xe6, 0xc1,
+       0x27, 0xe6, 0xf2, 0x89, 0x5b, 0x33, 0x62, 0x18,
+       0x80, 0x6e, 0x17, 0x22, 0x8e, 0x83, 0x31, 0x40,
+       0x8f, 0xc9, 0x5c, 0x52, 0x6c, 0x0e, 0xa5, 0xe9,
+       0x6c, 0x7f, 0xd4, 0x6a, 0x27, 0x56, 0x99, 0xce,
+       0x8d, 0x37, 0x59, 0xaf, 0xc0, 0x0e, 0xe1
+};
+static const u8 output48[] __initconst = {
+       0x02, 0xa4, 0x2e, 0x33, 0xb7, 0x7c, 0x2b, 0x9a,
+       0x18, 0x5a, 0xba, 0x53, 0x38, 0xaf, 0x00, 0xeb,
+       0xd8, 0x3d, 0x02, 0x77, 0x43, 0x45, 0x03, 0x91,
+       0xe2, 0x5e, 0x4e, 0xeb, 0x50, 0xd5, 0x5b, 0xe0,
+       0xf3, 0x33, 0xa7, 0xa2, 0xac, 0x07, 0x6f, 0xeb,
+       0x3f, 0x6c, 0xcd, 0xf2, 0x6c, 0x61, 0x64
+};
+static const u8 key48[] __initconst = {
+       0xf3, 0x79, 0xe7, 0xf8, 0x0e, 0x02, 0x05, 0x6b,
+       0x83, 0x1a, 0xe7, 0x86, 0x6b, 0xe6, 0x8f, 0x3f,
+       0xd3, 0xa3, 0xe4, 0x6e, 0x29, 0x06, 0xad, 0xbc,
+       0xe8, 0x33, 0x56, 0x39, 0xdf, 0xb0, 0xe2, 0xfe
+};
+enum { nonce48 = 0xd849b938c6569da0ULL };
+
+static const u8 input49[] __initconst = {
+       0x89, 0x3b, 0x88, 0x9e, 0x7b, 0x38, 0x16, 0x9f,
+       0xa1, 0x28, 0xf6, 0xf5, 0x23, 0x74, 0x28, 0xb0,
+       0xdf, 0x6c, 0x9e, 0x8a, 0x71, 0xaf, 0xed, 0x7a,
+       0x39, 0x21, 0x57, 0x7d, 0x31, 0x6c, 0xee, 0x0d,
+       0x11, 0x8d, 0x41, 0x9a, 0x5f, 0xb7, 0x27, 0x40,
+       0x08, 0xad, 0xc6, 0xe0, 0x00, 0x43, 0x9e, 0xae
+};
+static const u8 output49[] __initconst = {
+       0x4d, 0xfd, 0xdb, 0x4c, 0x77, 0xc1, 0x05, 0x07,
+       0x4d, 0x6d, 0x32, 0xcb, 0x2e, 0x0e, 0xff, 0x65,
+       0xc9, 0x27, 0xeb, 0xa9, 0x46, 0x5b, 0xab, 0x06,
+       0xe6, 0xb6, 0x5a, 0x1e, 0x00, 0xfb, 0xcf, 0xe4,
+       0xb9, 0x71, 0x40, 0x10, 0xef, 0x12, 0x39, 0xf0,
+       0xea, 0x40, 0xb8, 0x9a, 0xa2, 0x85, 0x38, 0x48
+};
+static const u8 key49[] __initconst = {
+       0xe7, 0x10, 0x40, 0xd9, 0x66, 0xc0, 0xa8, 0x6d,
+       0xa3, 0xcc, 0x8b, 0xdd, 0x93, 0xf2, 0x6e, 0xe0,
+       0x90, 0x7f, 0xd0, 0xf4, 0x37, 0x0c, 0x8b, 0x9b,
+       0x4c, 0x4d, 0xe6, 0xf2, 0x1f, 0xe9, 0x95, 0x24
+};
+enum { nonce49 = 0xf269817bdae01bc0ULL };
+
+static const u8 input50[] __initconst = {
+       0xda, 0x5b, 0x60, 0xcd, 0xed, 0x58, 0x8e, 0x7f,
+       0xae, 0xdd, 0xc8, 0x2e, 0x16, 0x90, 0xea, 0x4b,
+       0x0c, 0x74, 0x14, 0x35, 0xeb, 0xee, 0x2c, 0xff,
+       0x46, 0x99, 0x97, 0x6e, 0xae, 0xa7, 0x8e, 0x6e,
+       0x38, 0xfe, 0x63, 0xe7, 0x51, 0xd9, 0xaa, 0xce,
+       0x7b, 0x1e, 0x7e, 0x5d, 0xc0, 0xe8, 0x10, 0x06,
+       0x14
+};
+static const u8 output50[] __initconst = {
+       0xe4, 0xe5, 0x86, 0x1b, 0x66, 0x19, 0xac, 0x49,
+       0x1c, 0xbd, 0xee, 0x03, 0xaf, 0x11, 0xfc, 0x1f,
+       0x6a, 0xd2, 0x50, 0x5c, 0xea, 0x2c, 0xa5, 0x75,
+       0xfd, 0xb7, 0x0e, 0x80, 0x8f, 0xed, 0x3f, 0x31,
+       0x47, 0xac, 0x67, 0x43, 0xb8, 0x2e, 0xb4, 0x81,
+       0x6d, 0xe4, 0x1e, 0xb7, 0x8b, 0x0c, 0x53, 0xa9,
+       0x26
+};
+static const u8 key50[] __initconst = {
+       0xd7, 0xb2, 0x04, 0x76, 0x30, 0xcc, 0x38, 0x45,
+       0xef, 0xdb, 0xc5, 0x86, 0x08, 0x61, 0xf0, 0xee,
+       0x6d, 0xd8, 0x22, 0x04, 0x8c, 0xfb, 0xcb, 0x37,
+       0xa6, 0xfb, 0x95, 0x22, 0xe1, 0x87, 0xb7, 0x6f
+};
+enum { nonce50 = 0x3b44d09c45607d38ULL };
+
+static const u8 input51[] __initconst = {
+       0xa9, 0x41, 0x02, 0x4b, 0xd7, 0xd5, 0xd1, 0xf1,
+       0x21, 0x55, 0xb2, 0x75, 0x6d, 0x77, 0x1b, 0x86,
+       0xa9, 0xc8, 0x90, 0xfd, 0xed, 0x4a, 0x7b, 0x6c,
+       0xb2, 0x5f, 0x9b, 0x5f, 0x16, 0xa1, 0x54, 0xdb,
+       0xd6, 0x3f, 0x6a, 0x7f, 0x2e, 0x51, 0x9d, 0x49,
+       0x5b, 0xa5, 0x0e, 0xf9, 0xfb, 0x2a, 0x38, 0xff,
+       0x20, 0x8c
+};
+static const u8 output51[] __initconst = {
+       0x18, 0xf7, 0x88, 0xc1, 0x72, 0xfd, 0x90, 0x4b,
+       0xa9, 0x2d, 0xdb, 0x47, 0xb0, 0xa5, 0xc4, 0x37,
+       0x01, 0x95, 0xc4, 0xb1, 0xab, 0xc5, 0x5b, 0xcd,
+       0xe1, 0x97, 0x78, 0x13, 0xde, 0x6a, 0xff, 0x36,
+       0xce, 0xa4, 0x67, 0xc5, 0x4a, 0x45, 0x2b, 0xd9,
+       0xff, 0x8f, 0x06, 0x7c, 0x63, 0xbb, 0x83, 0x17,
+       0xb4, 0x6b
+};
+static const u8 key51[] __initconst = {
+       0x82, 0x1a, 0x79, 0xab, 0x9a, 0xb5, 0x49, 0x6a,
+       0x30, 0x6b, 0x99, 0x19, 0x11, 0xc7, 0xa2, 0xf4,
+       0xca, 0x55, 0xb9, 0xdd, 0xe7, 0x2f, 0xe7, 0xc1,
+       0xdd, 0x27, 0xad, 0x80, 0xf2, 0x56, 0xad, 0xf3
+};
+enum { nonce51 = 0xe93aff94ca71a4a6ULL };
+
+static const u8 input52[] __initconst = {
+       0x89, 0xdd, 0xf3, 0xfa, 0xb6, 0xc1, 0xaa, 0x9a,
+       0xc8, 0xad, 0x6b, 0x00, 0xa1, 0x65, 0xea, 0x14,
+       0x55, 0x54, 0x31, 0x8f, 0xf0, 0x03, 0x84, 0x51,
+       0x17, 0x1e, 0x0a, 0x93, 0x6e, 0x79, 0x96, 0xa3,
+       0x2a, 0x85, 0x9c, 0x89, 0xf8, 0xd1, 0xe2, 0x15,
+       0x95, 0x05, 0xf4, 0x43, 0x4d, 0x6b, 0xf0, 0x71,
+       0x3b, 0x3e, 0xba
+};
+static const u8 output52[] __initconst = {
+       0x0c, 0x42, 0x6a, 0xb3, 0x66, 0x63, 0x5d, 0x2c,
+       0x9f, 0x3d, 0xa6, 0x6e, 0xc7, 0x5f, 0x79, 0x2f,
+       0x50, 0xe3, 0xd6, 0x07, 0x56, 0xa4, 0x2b, 0x2d,
+       0x8d, 0x10, 0xc0, 0x6c, 0xa2, 0xfc, 0x97, 0xec,
+       0x3f, 0x5c, 0x8d, 0x59, 0xbe, 0x84, 0xf1, 0x3e,
+       0x38, 0x47, 0x4f, 0x75, 0x25, 0x66, 0x88, 0x14,
+       0x03, 0xdd, 0xde
+};
+static const u8 key52[] __initconst = {
+       0x4f, 0xb0, 0x27, 0xb6, 0xdd, 0x24, 0x0c, 0xdb,
+       0x6b, 0x71, 0x2e, 0xac, 0xfc, 0x3f, 0xa6, 0x48,
+       0x5d, 0xd5, 0xff, 0x53, 0xb5, 0x62, 0xf1, 0xe0,
+       0x93, 0xfe, 0x39, 0x4c, 0x9f, 0x03, 0x11, 0xa7
+};
+enum { nonce52 = 0xed8becec3bdf6f25ULL };
+
+static const u8 input53[] __initconst = {
+       0x68, 0xd1, 0xc7, 0x74, 0x44, 0x1c, 0x84, 0xde,
+       0x27, 0x27, 0x35, 0xf0, 0x18, 0x0b, 0x57, 0xaa,
+       0xd0, 0x1a, 0xd3, 0x3b, 0x5e, 0x5c, 0x62, 0x93,
+       0xd7, 0x6b, 0x84, 0x3b, 0x71, 0x83, 0x77, 0x01,
+       0x3e, 0x59, 0x45, 0xf4, 0x77, 0x6c, 0x6b, 0xcb,
+       0x88, 0x45, 0x09, 0x1d, 0xc6, 0x45, 0x6e, 0xdc,
+       0x6e, 0x51, 0xb8, 0x28
+};
+static const u8 output53[] __initconst = {
+       0xc5, 0x90, 0x96, 0x78, 0x02, 0xf5, 0xc4, 0x3c,
+       0xde, 0xd4, 0xd4, 0xc6, 0xa7, 0xad, 0x12, 0x47,
+       0x45, 0xce, 0xcd, 0x8c, 0x35, 0xcc, 0xa6, 0x9e,
+       0x5a, 0xc6, 0x60, 0xbb, 0xe3, 0xed, 0xec, 0x68,
+       0x3f, 0x64, 0xf7, 0x06, 0x63, 0x9c, 0x8c, 0xc8,
+       0x05, 0x3a, 0xad, 0x32, 0x79, 0x8b, 0x45, 0x96,
+       0x93, 0x73, 0x4c, 0xe0
+};
+static const u8 key53[] __initconst = {
+       0x42, 0x4b, 0x20, 0x81, 0x49, 0x50, 0xe9, 0xc2,
+       0x43, 0x69, 0x36, 0xe7, 0x68, 0xae, 0xd5, 0x7e,
+       0x42, 0x1a, 0x1b, 0xb4, 0x06, 0x4d, 0xa7, 0x17,
+       0xb5, 0x31, 0xd6, 0x0c, 0xb0, 0x5c, 0x41, 0x0b
+};
+enum { nonce53 = 0xf44ce1931fbda3d7ULL };
+
+static const u8 input54[] __initconst = {
+       0x7b, 0xf6, 0x8b, 0xae, 0xc0, 0xcb, 0x10, 0x8e,
+       0xe8, 0xd8, 0x2e, 0x3b, 0x14, 0xba, 0xb4, 0xd2,
+       0x58, 0x6b, 0x2c, 0xec, 0xc1, 0x81, 0x71, 0xb4,
+       0xc6, 0xea, 0x08, 0xc5, 0xc9, 0x78, 0xdb, 0xa2,
+       0xfa, 0x44, 0x50, 0x9b, 0xc8, 0x53, 0x8d, 0x45,
+       0x42, 0xe7, 0x09, 0xc4, 0x29, 0xd8, 0x75, 0x02,
+       0xbb, 0xb2, 0x78, 0xcf, 0xe7
+};
+static const u8 output54[] __initconst = {
+       0xaf, 0x2c, 0x83, 0x26, 0x6e, 0x7f, 0xa6, 0xe9,
+       0x03, 0x75, 0xfe, 0xfe, 0x87, 0x58, 0xcf, 0xb5,
+       0xbc, 0x3c, 0x9d, 0xa1, 0x6e, 0x13, 0xf1, 0x0f,
+       0x9e, 0xbc, 0xe0, 0x54, 0x24, 0x32, 0xce, 0x95,
+       0xe6, 0xa5, 0x59, 0x3d, 0x24, 0x1d, 0x8f, 0xb1,
+       0x74, 0x6c, 0x56, 0xe7, 0x96, 0xc1, 0x91, 0xc8,
+       0x2d, 0x0e, 0xb7, 0x51, 0x10
+};
+static const u8 key54[] __initconst = {
+       0x00, 0x68, 0x74, 0xdc, 0x30, 0x9e, 0xe3, 0x52,
+       0xa9, 0xae, 0xb6, 0x7c, 0xa1, 0xdc, 0x12, 0x2d,
+       0x98, 0x32, 0x7a, 0x77, 0xe1, 0xdd, 0xa3, 0x76,
+       0x72, 0x34, 0x83, 0xd8, 0xb7, 0x69, 0xba, 0x77
+};
+enum { nonce54 = 0xbea57d79b798b63aULL };
+
+static const u8 input55[] __initconst = {
+       0xb5, 0xf4, 0x2f, 0xc1, 0x5e, 0x10, 0xa7, 0x4e,
+       0x74, 0x3d, 0xa3, 0x96, 0xc0, 0x4d, 0x7b, 0x92,
+       0x8f, 0xdb, 0x2d, 0x15, 0x52, 0x6a, 0x95, 0x5e,
+       0x40, 0x81, 0x4f, 0x70, 0x73, 0xea, 0x84, 0x65,
+       0x3d, 0x9a, 0x4e, 0x03, 0x95, 0xf8, 0x5d, 0x2f,
+       0x07, 0x02, 0x13, 0x13, 0xdd, 0x82, 0xe6, 0x3b,
+       0xe1, 0x5f, 0xb3, 0x37, 0x9b, 0x88
+};
+static const u8 output55[] __initconst = {
+       0xc1, 0x88, 0xbd, 0x92, 0x77, 0xad, 0x7c, 0x5f,
+       0xaf, 0xa8, 0x57, 0x0e, 0x40, 0x0a, 0xdc, 0x70,
+       0xfb, 0xc6, 0x71, 0xfd, 0xc4, 0x74, 0x60, 0xcc,
+       0xa0, 0x89, 0x8e, 0x99, 0xf0, 0x06, 0xa6, 0x7c,
+       0x97, 0x42, 0x21, 0x81, 0x6a, 0x07, 0xe7, 0xb3,
+       0xf7, 0xa5, 0x03, 0x71, 0x50, 0x05, 0x63, 0x17,
+       0xa9, 0x46, 0x0b, 0xff, 0x30, 0x78
+};
+static const u8 key55[] __initconst = {
+       0x19, 0x8f, 0xe7, 0xd7, 0x6b, 0x7f, 0x6f, 0x69,
+       0x86, 0x91, 0x0f, 0xa7, 0x4a, 0x69, 0x8e, 0x34,
+       0xf3, 0xdb, 0xde, 0xaf, 0xf2, 0x66, 0x1d, 0x64,
+       0x97, 0x0c, 0xcf, 0xfa, 0x33, 0x84, 0xfd, 0x0c
+};
+enum { nonce55 = 0x80aa3d3e2c51ef06ULL };
+
+static const u8 input56[] __initconst = {
+       0x6b, 0xe9, 0x73, 0x42, 0x27, 0x5e, 0x12, 0xcd,
+       0xaa, 0x45, 0x12, 0x8b, 0xb3, 0xe6, 0x54, 0x33,
+       0x31, 0x7d, 0xe2, 0x25, 0xc6, 0x86, 0x47, 0x67,
+       0x86, 0x83, 0xe4, 0x46, 0xb5, 0x8f, 0x2c, 0xbb,
+       0xe4, 0xb8, 0x9f, 0xa2, 0xa4, 0xe8, 0x75, 0x96,
+       0x92, 0x51, 0x51, 0xac, 0x8e, 0x2e, 0x6f, 0xfc,
+       0xbd, 0x0d, 0xa3, 0x9f, 0x16, 0x55, 0x3e
+};
+static const u8 output56[] __initconst = {
+       0x42, 0x99, 0x73, 0x6c, 0xd9, 0x4b, 0x16, 0xe5,
+       0x18, 0x63, 0x1a, 0xd9, 0x0e, 0xf1, 0x15, 0x2e,
+       0x0f, 0x4b, 0xe4, 0x5f, 0xa0, 0x4d, 0xde, 0x9f,
+       0xa7, 0x18, 0xc1, 0x0c, 0x0b, 0xae, 0x55, 0xe4,
+       0x89, 0x18, 0xa4, 0x78, 0x9d, 0x25, 0x0d, 0xd5,
+       0x94, 0x0f, 0xf9, 0x78, 0xa3, 0xa6, 0xe9, 0x9e,
+       0x2c, 0x73, 0xf0, 0xf7, 0x35, 0xf3, 0x2b
+};
+static const u8 key56[] __initconst = {
+       0x7d, 0x12, 0xad, 0x51, 0xd5, 0x6f, 0x8f, 0x96,
+       0xc0, 0x5d, 0x9a, 0xd1, 0x7e, 0x20, 0x98, 0x0e,
+       0x3c, 0x0a, 0x67, 0x6b, 0x1b, 0x88, 0x69, 0xd4,
+       0x07, 0x8c, 0xaf, 0x0f, 0x3a, 0x28, 0xe4, 0x5d
+};
+enum { nonce56 = 0x70f4c372fb8b5984ULL };
+
+static const u8 input57[] __initconst = {
+       0x28, 0xa3, 0x06, 0xe8, 0xe7, 0x08, 0xb9, 0xef,
+       0x0d, 0x63, 0x15, 0x99, 0xb2, 0x78, 0x7e, 0xaf,
+       0x30, 0x50, 0xcf, 0xea, 0xc9, 0x91, 0x41, 0x2f,
+       0x3b, 0x38, 0x70, 0xc4, 0x87, 0xb0, 0x3a, 0xee,
+       0x4a, 0xea, 0xe3, 0x83, 0x68, 0x8b, 0xcf, 0xda,
+       0x04, 0xa5, 0xbd, 0xb2, 0xde, 0x3c, 0x55, 0x13,
+       0xfe, 0x96, 0xad, 0xc1, 0x61, 0x1b, 0x98, 0xde
+};
+static const u8 output57[] __initconst = {
+       0xf4, 0x44, 0xe9, 0xd2, 0x6d, 0xc2, 0x5a, 0xe9,
+       0xfd, 0x7e, 0x41, 0x54, 0x3f, 0xf4, 0x12, 0xd8,
+       0x55, 0x0d, 0x12, 0x9b, 0xd5, 0x2e, 0x95, 0xe5,
+       0x77, 0x42, 0x3f, 0x2c, 0xfb, 0x28, 0x9d, 0x72,
+       0x6d, 0x89, 0x82, 0x27, 0x64, 0x6f, 0x0d, 0x57,
+       0xa1, 0x25, 0xa3, 0x6b, 0x88, 0x9a, 0xac, 0x0c,
+       0x76, 0x19, 0x90, 0xe2, 0x50, 0x5a, 0xf8, 0x12
+};
+static const u8 key57[] __initconst = {
+       0x08, 0x26, 0xb8, 0xac, 0xf3, 0xa5, 0xc6, 0xa3,
+       0x7f, 0x09, 0x87, 0xf5, 0x6c, 0x5a, 0x85, 0x6c,
+       0x3d, 0xbd, 0xde, 0xd5, 0x87, 0xa3, 0x98, 0x7a,
+       0xaa, 0x40, 0x3e, 0xf7, 0xff, 0x44, 0x5d, 0xee
+};
+enum { nonce57 = 0xc03a6130bf06b089ULL };
+
+static const u8 input58[] __initconst = {
+       0x82, 0xa5, 0x38, 0x6f, 0xaa, 0xb4, 0xaf, 0xb2,
+       0x42, 0x01, 0xa8, 0x39, 0x3f, 0x15, 0x51, 0xa8,
+       0x11, 0x1b, 0x93, 0xca, 0x9c, 0xa0, 0x57, 0x68,
+       0x8f, 0xdb, 0x68, 0x53, 0x51, 0x6d, 0x13, 0x22,
+       0x12, 0x9b, 0xbd, 0x33, 0xa8, 0x52, 0x40, 0x57,
+       0x80, 0x9b, 0x98, 0xef, 0x56, 0x70, 0x11, 0xfa,
+       0x36, 0x69, 0x7d, 0x15, 0x48, 0xf9, 0x3b, 0xeb,
+       0x42
+};
+static const u8 output58[] __initconst = {
+       0xff, 0x3a, 0x74, 0xc3, 0x3e, 0x44, 0x64, 0x4d,
+       0x0e, 0x5f, 0x9d, 0xa8, 0xdb, 0xbe, 0x12, 0xef,
+       0xba, 0x56, 0x65, 0x50, 0x76, 0xaf, 0xa4, 0x4e,
+       0x01, 0xc1, 0xd3, 0x31, 0x14, 0xe2, 0xbe, 0x7b,
+       0xa5, 0x67, 0xb4, 0xe3, 0x68, 0x40, 0x9c, 0xb0,
+       0xb1, 0x78, 0xef, 0x49, 0x03, 0x0f, 0x2d, 0x56,
+       0xb4, 0x37, 0xdb, 0xbc, 0x2d, 0x68, 0x1c, 0x3c,
+       0xf1
+};
+static const u8 key58[] __initconst = {
+       0x7e, 0xf1, 0x7c, 0x20, 0x65, 0xed, 0xcd, 0xd7,
+       0x57, 0xe8, 0xdb, 0x90, 0x87, 0xdb, 0x5f, 0x63,
+       0x3d, 0xdd, 0xb8, 0x2b, 0x75, 0x8e, 0x04, 0xb5,
+       0xf4, 0x12, 0x79, 0xa9, 0x4d, 0x42, 0x16, 0x7f
+};
+enum { nonce58 = 0x92838183f80d2f7fULL };
+
+static const u8 input59[] __initconst = {
+       0x37, 0xf1, 0x9d, 0xdd, 0xd7, 0x08, 0x9f, 0x13,
+       0xc5, 0x21, 0x82, 0x75, 0x08, 0x9e, 0x25, 0x16,
+       0xb1, 0xd1, 0x71, 0x42, 0x28, 0x63, 0xac, 0x47,
+       0x71, 0x54, 0xb1, 0xfc, 0x39, 0xf0, 0x61, 0x4f,
+       0x7c, 0x6d, 0x4f, 0xc8, 0x33, 0xef, 0x7e, 0xc8,
+       0xc0, 0x97, 0xfc, 0x1a, 0x61, 0xb4, 0x87, 0x6f,
+       0xdd, 0x5a, 0x15, 0x7b, 0x1b, 0x95, 0x50, 0x94,
+       0x1d, 0xba
+};
+static const u8 output59[] __initconst = {
+       0x73, 0x67, 0xc5, 0x07, 0xbb, 0x57, 0x79, 0xd5,
+       0xc9, 0x04, 0xdd, 0x88, 0xf3, 0x86, 0xe5, 0x70,
+       0x49, 0x31, 0xe0, 0xcc, 0x3b, 0x1d, 0xdf, 0xb0,
+       0xaf, 0xf4, 0x2d, 0xe0, 0x06, 0x10, 0x91, 0x8d,
+       0x1c, 0xcf, 0x31, 0x0b, 0xf6, 0x73, 0xda, 0x1c,
+       0xf0, 0x17, 0x52, 0x9e, 0x20, 0x2e, 0x9f, 0x8c,
+       0xb3, 0x59, 0xce, 0xd4, 0xd3, 0xc1, 0x81, 0xe9,
+       0x11, 0x36
+};
+static const u8 key59[] __initconst = {
+       0xbd, 0x07, 0xd0, 0x53, 0x2c, 0xb3, 0xcc, 0x3f,
+       0xc4, 0x95, 0xfd, 0xe7, 0x81, 0xb3, 0x29, 0x99,
+       0x05, 0x45, 0xd6, 0x95, 0x25, 0x0b, 0x72, 0xd3,
+       0xcd, 0xbb, 0x73, 0xf8, 0xfa, 0xc0, 0x9b, 0x7a
+};
+enum { nonce59 = 0x4a0db819b0d519e2ULL };
+
+static const u8 input60[] __initconst = {
+       0x58, 0x4e, 0xdf, 0x94, 0x3c, 0x76, 0x0a, 0x79,
+       0x47, 0xf1, 0xbe, 0x88, 0xd3, 0xba, 0x94, 0xd8,
+       0xe2, 0x8f, 0xe3, 0x2f, 0x2f, 0x74, 0x82, 0x55,
+       0xc3, 0xda, 0xe2, 0x4e, 0x2c, 0x8c, 0x45, 0x1d,
+       0x72, 0x8f, 0x54, 0x41, 0xb5, 0xb7, 0x69, 0xe4,
+       0xdc, 0xd2, 0x36, 0x21, 0x5c, 0x28, 0x52, 0xf7,
+       0x98, 0x8e, 0x72, 0xa7, 0x6d, 0x57, 0xed, 0xdc,
+       0x3c, 0xe6, 0x6a
+};
+static const u8 output60[] __initconst = {
+       0xda, 0xaf, 0xb5, 0xe3, 0x30, 0x65, 0x5c, 0xb1,
+       0x48, 0x08, 0x43, 0x7b, 0x9e, 0xd2, 0x6a, 0x62,
+       0x56, 0x7c, 0xad, 0xd9, 0xe5, 0xf6, 0x09, 0x71,
+       0xcd, 0xe6, 0x05, 0x6b, 0x3f, 0x44, 0x3a, 0x5c,
+       0xf6, 0xf8, 0xd7, 0xce, 0x7d, 0xd1, 0xe0, 0x4f,
+       0x88, 0x15, 0x04, 0xd8, 0x20, 0xf0, 0x3e, 0xef,
+       0xae, 0xa6, 0x27, 0xa3, 0x0e, 0xfc, 0x18, 0x90,
+       0x33, 0xcd, 0xd3
+};
+static const u8 key60[] __initconst = {
+       0xbf, 0xfd, 0x25, 0xb5, 0xb2, 0xfc, 0x78, 0x0c,
+       0x8e, 0xb9, 0x57, 0x2f, 0x26, 0x4a, 0x7e, 0x71,
+       0xcc, 0xf2, 0xe0, 0xfd, 0x24, 0x11, 0x20, 0x23,
+       0x57, 0x00, 0xff, 0x80, 0x11, 0x0c, 0x1e, 0xff
+};
+enum { nonce60 = 0xf18df56fdb7954adULL };
+
+static const u8 input61[] __initconst = {
+       0xb0, 0xf3, 0x06, 0xbc, 0x22, 0xae, 0x49, 0x40,
+       0xae, 0xff, 0x1b, 0x31, 0xa7, 0x98, 0xab, 0x1d,
+       0xe7, 0x40, 0x23, 0x18, 0x4f, 0xab, 0x8e, 0x93,
+       0x82, 0xf4, 0x56, 0x61, 0xfd, 0x2b, 0xcf, 0xa7,
+       0xc4, 0xb4, 0x0a, 0xf4, 0xcb, 0xc7, 0x8c, 0x40,
+       0x57, 0xac, 0x0b, 0x3e, 0x2a, 0x0a, 0x67, 0x83,
+       0x50, 0xbf, 0xec, 0xb0, 0xc7, 0xf1, 0x32, 0x26,
+       0x98, 0x80, 0x33, 0xb4
+};
+static const u8 output61[] __initconst = {
+       0x9d, 0x23, 0x0e, 0xff, 0xcc, 0x7c, 0xd5, 0xcf,
+       0x1a, 0xb8, 0x59, 0x1e, 0x92, 0xfd, 0x7f, 0xca,
+       0xca, 0x3c, 0x18, 0x81, 0xde, 0xfa, 0x59, 0xc8,
+       0x6f, 0x9c, 0x24, 0x3f, 0x3a, 0xe6, 0x0b, 0xb4,
+       0x34, 0x48, 0x69, 0xfc, 0xb6, 0xea, 0xb2, 0xde,
+       0x9f, 0xfd, 0x92, 0x36, 0x18, 0x98, 0x99, 0xaa,
+       0x65, 0xe2, 0xea, 0xf4, 0xb1, 0x47, 0x8e, 0xb0,
+       0xe7, 0xd4, 0x7a, 0x2c
+};
+static const u8 key61[] __initconst = {
+       0xd7, 0xfd, 0x9b, 0xbd, 0x8f, 0x65, 0x0d, 0x00,
+       0xca, 0xa1, 0x6c, 0x85, 0x85, 0xa4, 0x6d, 0xf1,
+       0xb1, 0x68, 0x0c, 0x8b, 0x5d, 0x37, 0x72, 0xd0,
+       0xd8, 0xd2, 0x25, 0xab, 0x9f, 0x7b, 0x7d, 0x95
+};
+enum { nonce61 = 0xd82caf72a9c4864fULL };
+
+static const u8 input62[] __initconst = {
+       0x10, 0x77, 0xf3, 0x2f, 0xc2, 0x50, 0xd6, 0x0c,
+       0xba, 0xa8, 0x8d, 0xce, 0x0d, 0x58, 0x9e, 0x87,
+       0xb1, 0x59, 0x66, 0x0a, 0x4a, 0xb3, 0xd8, 0xca,
+       0x0a, 0x6b, 0xf8, 0xc6, 0x2b, 0x3f, 0x8e, 0x09,
+       0xe0, 0x0a, 0x15, 0x85, 0xfe, 0xaa, 0xc6, 0xbd,
+       0x30, 0xef, 0xe4, 0x10, 0x78, 0x03, 0xc1, 0xc7,
+       0x8a, 0xd9, 0xde, 0x0b, 0x51, 0x07, 0xc4, 0x7b,
+       0xe2, 0x2e, 0x36, 0x3a, 0xc2
+};
+static const u8 output62[] __initconst = {
+       0xa0, 0x0c, 0xfc, 0xc1, 0xf6, 0xaf, 0xc2, 0xb8,
+       0x5c, 0xef, 0x6e, 0xf3, 0xce, 0x15, 0x48, 0x05,
+       0xb5, 0x78, 0x49, 0x51, 0x1f, 0x9d, 0xf4, 0xbf,
+       0x2f, 0x53, 0xa2, 0xd1, 0x15, 0x20, 0x82, 0x6b,
+       0xd2, 0x22, 0x6c, 0x4e, 0x14, 0x87, 0xe3, 0xd7,
+       0x49, 0x45, 0x84, 0xdb, 0x5f, 0x68, 0x60, 0xc4,
+       0xb3, 0xe6, 0x3f, 0xd1, 0xfc, 0xa5, 0x73, 0xf3,
+       0xfc, 0xbb, 0xbe, 0xc8, 0x9d
+};
+static const u8 key62[] __initconst = {
+       0x6e, 0xc9, 0xaf, 0xce, 0x35, 0xb9, 0x86, 0xd1,
+       0xce, 0x5f, 0xd9, 0xbb, 0xd5, 0x1f, 0x7c, 0xcd,
+       0xfe, 0x19, 0xaa, 0x3d, 0xea, 0x64, 0xc1, 0x28,
+       0x40, 0xba, 0xa1, 0x28, 0xcd, 0x40, 0xb6, 0xf2
+};
+enum { nonce62 = 0xa1c0c265f900cde8ULL };
+
+static const u8 input63[] __initconst = {
+       0x7a, 0x70, 0x21, 0x2c, 0xef, 0xa6, 0x36, 0xd4,
+       0xe0, 0xab, 0x8c, 0x25, 0x73, 0x34, 0xc8, 0x94,
+       0x6c, 0x81, 0xcb, 0x19, 0x8d, 0x5a, 0x49, 0xaa,
+       0x6f, 0xba, 0x83, 0x72, 0x02, 0x5e, 0xf5, 0x89,
+       0xce, 0x79, 0x7e, 0x13, 0x3d, 0x5b, 0x98, 0x60,
+       0x5d, 0xd9, 0xfb, 0x15, 0x93, 0x4c, 0xf3, 0x51,
+       0x49, 0x55, 0xd1, 0x58, 0xdd, 0x7e, 0x6d, 0xfe,
+       0xdd, 0x84, 0x23, 0x05, 0xba, 0xe9
+};
+static const u8 output63[] __initconst = {
+       0x20, 0xb3, 0x5c, 0x03, 0x03, 0x78, 0x17, 0xfc,
+       0x3b, 0x35, 0x30, 0x9a, 0x00, 0x18, 0xf5, 0xc5,
+       0x06, 0x53, 0xf5, 0x04, 0x24, 0x9d, 0xd1, 0xb2,
+       0xac, 0x5a, 0xb6, 0x2a, 0xa5, 0xda, 0x50, 0x00,
+       0xec, 0xff, 0xa0, 0x7a, 0x14, 0x7b, 0xe4, 0x6b,
+       0x63, 0xe8, 0x66, 0x86, 0x34, 0xfd, 0x74, 0x44,
+       0xa2, 0x50, 0x97, 0x0d, 0xdc, 0xc3, 0x84, 0xf8,
+       0x71, 0x02, 0x31, 0x95, 0xed, 0x54
+};
+static const u8 key63[] __initconst = {
+       0x7d, 0x64, 0xb4, 0x12, 0x81, 0xe4, 0xe6, 0x8f,
+       0xcc, 0xe7, 0xd1, 0x1f, 0x70, 0x20, 0xfd, 0xb8,
+       0x3a, 0x7d, 0xa6, 0x53, 0x65, 0x30, 0x5d, 0xe3,
+       0x1a, 0x44, 0xbe, 0x62, 0xed, 0x90, 0xc4, 0xd1
+};
+enum { nonce63 = 0xe8e849596c942276ULL };
+
+static const u8 input64[] __initconst = {
+       0x84, 0xf8, 0xda, 0x87, 0x23, 0x39, 0x60, 0xcf,
+       0xc5, 0x50, 0x7e, 0xc5, 0x47, 0x29, 0x7c, 0x05,
+       0xc2, 0xb4, 0xf4, 0xb2, 0xec, 0x5d, 0x48, 0x36,
+       0xbf, 0xfc, 0x06, 0x8c, 0xf2, 0x0e, 0x88, 0xe7,
+       0xc9, 0xc5, 0xa4, 0xa2, 0x83, 0x20, 0xa1, 0x6f,
+       0x37, 0xe5, 0x2d, 0xa1, 0x72, 0xa1, 0x19, 0xef,
+       0x05, 0x42, 0x08, 0xf2, 0x57, 0x47, 0x31, 0x1e,
+       0x17, 0x76, 0x13, 0xd3, 0xcc, 0x75, 0x2c
+};
+static const u8 output64[] __initconst = {
+       0xcb, 0xec, 0x90, 0x88, 0xeb, 0x31, 0x69, 0x20,
+       0xa6, 0xdc, 0xff, 0x76, 0x98, 0xb0, 0x24, 0x49,
+       0x7b, 0x20, 0xd9, 0xd1, 0x1b, 0xe3, 0x61, 0xdc,
+       0xcf, 0x51, 0xf6, 0x70, 0x72, 0x33, 0x28, 0x94,
+       0xac, 0x73, 0x18, 0xcf, 0x93, 0xfd, 0xca, 0x08,
+       0x0d, 0xa2, 0xb9, 0x57, 0x1e, 0x51, 0xb6, 0x07,
+       0x5c, 0xc1, 0x13, 0x64, 0x1d, 0x18, 0x6f, 0xe6,
+       0x0b, 0xb7, 0x14, 0x03, 0x43, 0xb6, 0xaf
+};
+static const u8 key64[] __initconst = {
+       0xbf, 0x82, 0x65, 0xe4, 0x50, 0xf9, 0x5e, 0xea,
+       0x28, 0x91, 0xd1, 0xd2, 0x17, 0x7c, 0x13, 0x7e,
+       0xf5, 0xd5, 0x6b, 0x06, 0x1c, 0x20, 0xc2, 0x82,
+       0xa1, 0x7a, 0xa2, 0x14, 0xa1, 0xb0, 0x54, 0x58
+};
+enum { nonce64 = 0xe57c5095aa5723c9ULL };
+
+static const u8 input65[] __initconst = {
+       0x1c, 0xfb, 0xd3, 0x3f, 0x85, 0xd7, 0xba, 0x7b,
+       0xae, 0xb1, 0xa5, 0xd2, 0xe5, 0x40, 0xce, 0x4d,
+       0x3e, 0xab, 0x17, 0x9d, 0x7d, 0x9f, 0x03, 0x98,
+       0x3f, 0x9f, 0xc8, 0xdd, 0x36, 0x17, 0x43, 0x5c,
+       0x34, 0xd1, 0x23, 0xe0, 0x77, 0xbf, 0x35, 0x5d,
+       0x8f, 0xb1, 0xcb, 0x82, 0xbb, 0x39, 0x69, 0xd8,
+       0x90, 0x45, 0x37, 0xfd, 0x98, 0x25, 0xf7, 0x5b,
+       0xce, 0x06, 0x43, 0xba, 0x61, 0xa8, 0x47, 0xb9
+};
+static const u8 output65[] __initconst = {
+       0x73, 0xa5, 0x68, 0xab, 0x8b, 0xa5, 0xc3, 0x7e,
+       0x74, 0xf8, 0x9d, 0xf5, 0x93, 0x6e, 0xf2, 0x71,
+       0x6d, 0xde, 0x82, 0xc5, 0x40, 0xa0, 0x46, 0xb3,
+       0x9a, 0x78, 0xa8, 0xf7, 0xdf, 0xb1, 0xc3, 0xdd,
+       0x8d, 0x90, 0x00, 0x68, 0x21, 0x48, 0xe8, 0xba,
+       0x56, 0x9f, 0x8f, 0xe7, 0xa4, 0x4d, 0x36, 0x55,
+       0xd0, 0x34, 0x99, 0xa6, 0x1c, 0x4c, 0xc1, 0xe2,
+       0x65, 0x98, 0x14, 0x8e, 0x6a, 0x05, 0xb1, 0x2b
+};
+static const u8 key65[] __initconst = {
+       0xbd, 0x5c, 0x8a, 0xb0, 0x11, 0x29, 0xf3, 0x00,
+       0x7a, 0x78, 0x32, 0x63, 0x34, 0x00, 0xe6, 0x7d,
+       0x30, 0x54, 0xde, 0x37, 0xda, 0xc2, 0xc4, 0x3d,
+       0x92, 0x6b, 0x4c, 0xc2, 0x92, 0xe9, 0x9e, 0x2a
+};
+enum { nonce65 = 0xf654a3031de746f2ULL };
+
+static const u8 input66[] __initconst = {
+       0x4b, 0x27, 0x30, 0x8f, 0x28, 0xd8, 0x60, 0x46,
+       0x39, 0x06, 0x49, 0xea, 0x1b, 0x71, 0x26, 0xe0,
+       0x99, 0x2b, 0xd4, 0x8f, 0x64, 0x64, 0xcd, 0xac,
+       0x1d, 0x78, 0x88, 0x90, 0xe1, 0x5c, 0x24, 0x4b,
+       0xdc, 0x2d, 0xb7, 0xee, 0x3a, 0xe6, 0x86, 0x2c,
+       0x21, 0xe4, 0x2b, 0xfc, 0xe8, 0x19, 0xca, 0x65,
+       0xe7, 0xdd, 0x6f, 0x52, 0xb3, 0x11, 0xe1, 0xe2,
+       0xbf, 0xe8, 0x70, 0xe3, 0x0d, 0x45, 0xb8, 0xa5,
+       0x20, 0xb7, 0xb5, 0xaf, 0xff, 0x08, 0xcf, 0x23,
+       0x65, 0xdf, 0x8d, 0xc3, 0x31, 0xf3, 0x1e, 0x6a,
+       0x58, 0x8d, 0xcc, 0x45, 0x16, 0x86, 0x1f, 0x31,
+       0x5c, 0x27, 0xcd, 0xc8, 0x6b, 0x19, 0x1e, 0xec,
+       0x44, 0x75, 0x63, 0x97, 0xfd, 0x79, 0xf6, 0x62,
+       0xc5, 0xba, 0x17, 0xc7, 0xab, 0x8f, 0xbb, 0xed,
+       0x85, 0x2a, 0x98, 0x79, 0x21, 0xec, 0x6e, 0x4d,
+       0xdc, 0xfa, 0x72, 0x52, 0xba, 0xc8, 0x4c
+};
+static const u8 output66[] __initconst = {
+       0x76, 0x5b, 0x2c, 0xa7, 0x62, 0xb9, 0x08, 0x4a,
+       0xc6, 0x4a, 0x92, 0xc3, 0xbb, 0x10, 0xb3, 0xee,
+       0xff, 0xb9, 0x07, 0xc7, 0x27, 0xcb, 0x1e, 0xcf,
+       0x58, 0x6f, 0xa1, 0x64, 0xe8, 0xf1, 0x4e, 0xe1,
+       0xef, 0x18, 0x96, 0xab, 0x97, 0x28, 0xd1, 0x7c,
+       0x71, 0x6c, 0xd1, 0xe2, 0xfa, 0xd9, 0x75, 0xcb,
+       0xeb, 0xea, 0x0c, 0x86, 0x82, 0xd8, 0xf4, 0xcc,
+       0xea, 0xa3, 0x00, 0xfa, 0x82, 0xd2, 0xcd, 0xcb,
+       0xdb, 0x63, 0x28, 0xe2, 0x82, 0xe9, 0x01, 0xed,
+       0x31, 0xe6, 0x71, 0x45, 0x08, 0x89, 0x8a, 0x23,
+       0xa8, 0xb5, 0xc2, 0xe2, 0x9f, 0xe9, 0xb8, 0x9a,
+       0xc4, 0x79, 0x6d, 0x71, 0x52, 0x61, 0x74, 0x6c,
+       0x1b, 0xd7, 0x65, 0x6d, 0x03, 0xc4, 0x1a, 0xc0,
+       0x50, 0xba, 0xd6, 0xc9, 0x43, 0x50, 0xbe, 0x09,
+       0x09, 0x8a, 0xdb, 0xaa, 0x76, 0x4e, 0x3b, 0x61,
+       0x3c, 0x7c, 0x44, 0xe7, 0xdb, 0x10, 0xa7
+};
+static const u8 key66[] __initconst = {
+       0x88, 0xdf, 0xca, 0x68, 0xaf, 0x4f, 0xb3, 0xfd,
+       0x6e, 0xa7, 0x95, 0x35, 0x8a, 0xe8, 0x37, 0xe8,
+       0xc8, 0x55, 0xa2, 0x2a, 0x6d, 0x77, 0xf8, 0x93,
+       0x7a, 0x41, 0xf3, 0x7b, 0x95, 0xdf, 0x89, 0xf5
+};
+enum { nonce66 = 0x1024b4fdd415cf82ULL };
+
+static const u8 input67[] __initconst = {
+       0xd4, 0x2e, 0xfa, 0x92, 0xe9, 0x29, 0x68, 0xb7,
+       0x54, 0x2c, 0xf7, 0xa4, 0x2d, 0xb7, 0x50, 0xb5,
+       0xc5, 0xb2, 0x9d, 0x17, 0x5e, 0x0a, 0xca, 0x37,
+       0xbf, 0x60, 0xae, 0xd2, 0x98, 0xe9, 0xfa, 0x59,
+       0x67, 0x62, 0xe6, 0x43, 0x0c, 0x77, 0x80, 0x82,
+       0x33, 0x61, 0xa3, 0xff, 0xc1, 0xa0, 0x8f, 0x56,
+       0xbc, 0xec, 0x65, 0x43, 0x88, 0xa5, 0xff, 0x51,
+       0x64, 0x30, 0xee, 0x34, 0xb7, 0x5c, 0x28, 0x68,
+       0xc3, 0x52, 0xd2, 0xac, 0x78, 0x2a, 0xa6, 0x10,
+       0xb8, 0xb2, 0x4c, 0x80, 0x4f, 0x99, 0xb2, 0x36,
+       0x94, 0x8f, 0x66, 0xcb, 0xa1, 0x91, 0xed, 0x06,
+       0x42, 0x6d, 0xc1, 0xae, 0x55, 0x93, 0xdd, 0x93,
+       0x9e, 0x88, 0x34, 0x7f, 0x98, 0xeb, 0xbe, 0x61,
+       0xf9, 0xa9, 0x0f, 0xd9, 0xc4, 0x87, 0xd5, 0xef,
+       0xcc, 0x71, 0x8c, 0x0e, 0xce, 0xad, 0x02, 0xcf,
+       0xa2, 0x61, 0xdf, 0xb1, 0xfe, 0x3b, 0xdc, 0xc0,
+       0x58, 0xb5, 0x71, 0xa1, 0x83, 0xc9, 0xb4, 0xaf,
+       0x9d, 0x54, 0x12, 0xcd, 0xea, 0x06, 0xd6, 0x4e,
+       0xe5, 0x27, 0x0c, 0xc3, 0xbb, 0xa8, 0x0a, 0x81,
+       0x75, 0xc3, 0xc9, 0xd4, 0x35, 0x3e, 0x53, 0x9f,
+       0xaa, 0x20, 0xc0, 0x68, 0x39, 0x2c, 0x96, 0x39,
+       0x53, 0x81, 0xda, 0x07, 0x0f, 0x44, 0xa5, 0x47,
+       0x0e, 0xb3, 0x87, 0x0d, 0x1b, 0xc1, 0xe5, 0x41,
+       0x35, 0x12, 0x58, 0x96, 0x69, 0x8a, 0x1a, 0xa3,
+       0x9d, 0x3d, 0xd4, 0xb1, 0x8e, 0x1f, 0x96, 0x87,
+       0xda, 0xd3, 0x19, 0xe2, 0xb1, 0x3a, 0x19, 0x74,
+       0xa0, 0x00, 0x9f, 0x4d, 0xbc, 0xcb, 0x0c, 0xe9,
+       0xec, 0x10, 0xdf, 0x2a, 0x88, 0xdc, 0x30, 0x51,
+       0x46, 0x56, 0x53, 0x98, 0x6a, 0x26, 0x14, 0x05,
+       0x54, 0x81, 0x55, 0x0b, 0x3c, 0x85, 0xdd, 0x33,
+       0x81, 0x11, 0x29, 0x82, 0x46, 0x35, 0xe1, 0xdb,
+       0x59, 0x7b
+};
+static const u8 output67[] __initconst = {
+       0x64, 0x6c, 0xda, 0x7f, 0xd4, 0xa9, 0x2a, 0x5e,
+       0x22, 0xae, 0x8d, 0x67, 0xdb, 0xee, 0xfd, 0xd0,
+       0x44, 0x80, 0x17, 0xb2, 0xe3, 0x87, 0xad, 0x57,
+       0x15, 0xcb, 0x88, 0x64, 0xc0, 0xf1, 0x49, 0x3d,
+       0xfa, 0xbe, 0xa8, 0x9f, 0x12, 0xc3, 0x57, 0x56,
+       0x70, 0xa5, 0xc5, 0x6b, 0xf1, 0xab, 0xd5, 0xde,
+       0x77, 0x92, 0x6a, 0x56, 0x03, 0xf5, 0x21, 0x0d,
+       0xb6, 0xc4, 0xcc, 0x62, 0x44, 0x3f, 0xb1, 0xc1,
+       0x61, 0x41, 0x90, 0xb2, 0xd5, 0xb8, 0xf3, 0x57,
+       0xfb, 0xc2, 0x6b, 0x25, 0x58, 0xc8, 0x45, 0x20,
+       0x72, 0x29, 0x6f, 0x9d, 0xb5, 0x81, 0x4d, 0x2b,
+       0xb2, 0x89, 0x9e, 0x91, 0x53, 0x97, 0x1c, 0xd9,
+       0x3d, 0x79, 0xdc, 0x14, 0xae, 0x01, 0x73, 0x75,
+       0xf0, 0xca, 0xd5, 0xab, 0x62, 0x5c, 0x7a, 0x7d,
+       0x3f, 0xfe, 0x22, 0x7d, 0xee, 0xe2, 0xcb, 0x76,
+       0x55, 0xec, 0x06, 0xdd, 0x41, 0x47, 0x18, 0x62,
+       0x1d, 0x57, 0xd0, 0xd6, 0xb6, 0x0f, 0x4b, 0xfc,
+       0x79, 0x19, 0xf4, 0xd6, 0x37, 0x86, 0x18, 0x1f,
+       0x98, 0x0d, 0x9e, 0x15, 0x2d, 0xb6, 0x9a, 0x8a,
+       0x8c, 0x80, 0x22, 0x2f, 0x82, 0xc4, 0xc7, 0x36,
+       0xfa, 0xfa, 0x07, 0xbd, 0xc2, 0x2a, 0xe2, 0xea,
+       0x93, 0xc8, 0xb2, 0x90, 0x33, 0xf2, 0xee, 0x4b,
+       0x1b, 0xf4, 0x37, 0x92, 0x13, 0xbb, 0xe2, 0xce,
+       0xe3, 0x03, 0xcf, 0x07, 0x94, 0xab, 0x9a, 0xc9,
+       0xff, 0x83, 0x69, 0x3a, 0xda, 0x2c, 0xd0, 0x47,
+       0x3d, 0x6c, 0x1a, 0x60, 0x68, 0x47, 0xb9, 0x36,
+       0x52, 0xdd, 0x16, 0xef, 0x6c, 0xbf, 0x54, 0x11,
+       0x72, 0x62, 0xce, 0x8c, 0x9d, 0x90, 0xa0, 0x25,
+       0x06, 0x92, 0x3e, 0x12, 0x7e, 0x1a, 0x1d, 0xe5,
+       0xa2, 0x71, 0xce, 0x1c, 0x4c, 0x6a, 0x7c, 0xdc,
+       0x3d, 0xe3, 0x6e, 0x48, 0x9d, 0xb3, 0x64, 0x7d,
+       0x78, 0x40
+};
+static const u8 key67[] __initconst = {
+       0xa9, 0x20, 0x75, 0x89, 0x7e, 0x37, 0x85, 0x48,
+       0xa3, 0xfb, 0x7b, 0xe8, 0x30, 0xa7, 0xe3, 0x6e,
+       0xa6, 0xc1, 0x71, 0x17, 0xc1, 0x6c, 0x9b, 0xc2,
+       0xde, 0xf0, 0xa7, 0x19, 0xec, 0xce, 0xc6, 0x53
+};
+enum { nonce67 = 0x4adc4d1f968c8a10ULL };
+
+static const u8 input68[] __initconst = {
+       0x99, 0xae, 0x72, 0xfb, 0x16, 0xe1, 0xf1, 0x59,
+       0x43, 0x15, 0x4e, 0x33, 0xa0, 0x95, 0xe7, 0x6c,
+       0x74, 0x24, 0x31, 0xca, 0x3b, 0x2e, 0xeb, 0xd7,
+       0x11, 0xd8, 0xe0, 0x56, 0x92, 0x91, 0x61, 0x57,
+       0xe2, 0x82, 0x9f, 0x8f, 0x37, 0xf5, 0x3d, 0x24,
+       0x92, 0x9d, 0x87, 0x00, 0x8d, 0x89, 0xe0, 0x25,
+       0x8b, 0xe4, 0x20, 0x5b, 0x8a, 0x26, 0x2c, 0x61,
+       0x78, 0xb0, 0xa6, 0x3e, 0x82, 0x18, 0xcf, 0xdc,
+       0x2d, 0x24, 0xdd, 0x81, 0x42, 0xc4, 0x95, 0xf0,
+       0x48, 0x60, 0x71, 0xe3, 0xe3, 0xac, 0xec, 0xbe,
+       0x98, 0x6b, 0x0c, 0xb5, 0x6a, 0xa9, 0xc8, 0x79,
+       0x23, 0x2e, 0x38, 0x0b, 0x72, 0x88, 0x8c, 0xe7,
+       0x71, 0x8b, 0x36, 0xe3, 0x58, 0x3d, 0x9c, 0xa0,
+       0xa2, 0xea, 0xcf, 0x0c, 0x6a, 0x6c, 0x64, 0xdf,
+       0x97, 0x21, 0x8f, 0x93, 0xfb, 0xba, 0xf3, 0x5a,
+       0xd7, 0x8f, 0xa6, 0x37, 0x15, 0x50, 0x43, 0x02,
+       0x46, 0x7f, 0x93, 0x46, 0x86, 0x31, 0xe2, 0xaa,
+       0x24, 0xa8, 0x26, 0xae, 0xe6, 0xc0, 0x05, 0x73,
+       0x0b, 0x4f, 0x7e, 0xed, 0x65, 0xeb, 0x56, 0x1e,
+       0xb6, 0xb3, 0x0b, 0xc3, 0x0e, 0x31, 0x95, 0xa9,
+       0x18, 0x4d, 0xaf, 0x38, 0xd7, 0xec, 0xc6, 0x44,
+       0x72, 0x77, 0x4e, 0x25, 0x4b, 0x25, 0xdd, 0x1e,
+       0x8c, 0xa2, 0xdf, 0xf6, 0x2a, 0x97, 0x1a, 0x88,
+       0x2c, 0x8a, 0x5d, 0xfe, 0xe8, 0xfb, 0x35, 0xe8,
+       0x0f, 0x2b, 0x7a, 0x18, 0x69, 0x43, 0x31, 0x1d,
+       0x38, 0x6a, 0x62, 0x95, 0x0f, 0x20, 0x4b, 0xbb,
+       0x97, 0x3c, 0xe0, 0x64, 0x2f, 0x52, 0xc9, 0x2d,
+       0x4d, 0x9d, 0x54, 0x04, 0x3d, 0xc9, 0xea, 0xeb,
+       0xd0, 0x86, 0x52, 0xff, 0x42, 0xe1, 0x0d, 0x7a,
+       0xad, 0x88, 0xf9, 0x9b, 0x1e, 0x5e, 0x12, 0x27,
+       0x95, 0x3e, 0x0c, 0x2c, 0x13, 0x00, 0x6f, 0x8e,
+       0x93, 0x69, 0x0e, 0x01, 0x8c, 0xc1, 0xfd, 0xb3
+};
+static const u8 output68[] __initconst = {
+       0x26, 0x3e, 0xf2, 0xb1, 0xf5, 0xef, 0x81, 0xa4,
+       0xb7, 0x42, 0xd4, 0x26, 0x18, 0x4b, 0xdd, 0x6a,
+       0x47, 0x15, 0xcb, 0x0e, 0x57, 0xdb, 0xa7, 0x29,
+       0x7e, 0x7b, 0x3f, 0x47, 0x89, 0x57, 0xab, 0xea,
+       0x14, 0x7b, 0xcf, 0x37, 0xdb, 0x1c, 0xe1, 0x11,
+       0x77, 0xae, 0x2e, 0x4c, 0xd2, 0x08, 0x3f, 0xa6,
+       0x62, 0x86, 0xa6, 0xb2, 0x07, 0xd5, 0x3f, 0x9b,
+       0xdc, 0xc8, 0x50, 0x4b, 0x7b, 0xb9, 0x06, 0xe6,
+       0xeb, 0xac, 0x98, 0x8c, 0x36, 0x0c, 0x1e, 0xb2,
+       0xc8, 0xfb, 0x24, 0x60, 0x2c, 0x08, 0x17, 0x26,
+       0x5b, 0xc8, 0xc2, 0xdf, 0x9c, 0x73, 0x67, 0x4a,
+       0xdb, 0xcf, 0xd5, 0x2c, 0x2b, 0xca, 0x24, 0xcc,
+       0xdb, 0xc9, 0xa8, 0xf2, 0x5d, 0x67, 0xdf, 0x5c,
+       0x62, 0x0b, 0x58, 0xc0, 0x83, 0xde, 0x8b, 0xf6,
+       0x15, 0x0a, 0xd6, 0x32, 0xd8, 0xf5, 0xf2, 0x5f,
+       0x33, 0xce, 0x7e, 0xab, 0x76, 0xcd, 0x14, 0x91,
+       0xd8, 0x41, 0x90, 0x93, 0xa1, 0xaf, 0xf3, 0x45,
+       0x6c, 0x1b, 0x25, 0xbd, 0x48, 0x51, 0x6d, 0x15,
+       0x47, 0xe6, 0x23, 0x50, 0x32, 0x69, 0x1e, 0xb5,
+       0x94, 0xd3, 0x97, 0xba, 0xd7, 0x37, 0x4a, 0xba,
+       0xb9, 0xcd, 0xfb, 0x96, 0x9a, 0x90, 0xe0, 0x37,
+       0xf8, 0xdf, 0x91, 0x6c, 0x62, 0x13, 0x19, 0x21,
+       0x4b, 0xa9, 0xf1, 0x12, 0x66, 0xe2, 0x74, 0xd7,
+       0x81, 0xa0, 0x74, 0x8d, 0x7e, 0x7e, 0xc9, 0xb1,
+       0x69, 0x8f, 0xed, 0xb3, 0xf6, 0x97, 0xcd, 0x72,
+       0x78, 0x93, 0xd3, 0x54, 0x6b, 0x43, 0xac, 0x29,
+       0xb4, 0xbc, 0x7d, 0xa4, 0x26, 0x4b, 0x7b, 0xab,
+       0xd6, 0x67, 0x22, 0xff, 0x03, 0x92, 0xb6, 0xd4,
+       0x96, 0x94, 0x5a, 0xe5, 0x02, 0x35, 0x77, 0xfa,
+       0x3f, 0x54, 0x1d, 0xdd, 0x35, 0x39, 0xfe, 0x03,
+       0xdd, 0x8e, 0x3c, 0x8c, 0xc2, 0x69, 0x2a, 0xb1,
+       0xb7, 0xb3, 0xa1, 0x89, 0x84, 0xea, 0x16, 0xe2
+};
+static const u8 key68[] __initconst = {
+       0xd2, 0x49, 0x7f, 0xd7, 0x49, 0x66, 0x0d, 0xb3,
+       0x5a, 0x7e, 0x3c, 0xfc, 0x37, 0x83, 0x0e, 0xf7,
+       0x96, 0xd8, 0xd6, 0x33, 0x79, 0x2b, 0x84, 0x53,
+       0x06, 0xbc, 0x6c, 0x0a, 0x55, 0x84, 0xfe, 0xab
+};
+enum { nonce68 = 0x6a6df7ff0a20de06ULL };
+
+static const u8 input69[] __initconst = {
+       0xf9, 0x18, 0x4c, 0xd2, 0x3f, 0xf7, 0x22, 0xd9,
+       0x58, 0xb6, 0x3b, 0x38, 0x69, 0x79, 0xf4, 0x71,
+       0x5f, 0x38, 0x52, 0x1f, 0x17, 0x6f, 0x6f, 0xd9,
+       0x09, 0x2b, 0xfb, 0x67, 0xdc, 0xc9, 0xe8, 0x4a,
+       0x70, 0x9f, 0x2e, 0x3c, 0x06, 0xe5, 0x12, 0x20,
+       0x25, 0x29, 0xd0, 0xdc, 0x81, 0xc5, 0xc6, 0x0f,
+       0xd2, 0xa8, 0x81, 0x15, 0x98, 0xb2, 0x71, 0x5a,
+       0x9a, 0xe9, 0xfb, 0xaf, 0x0e, 0x5f, 0x8a, 0xf3,
+       0x16, 0x4a, 0x47, 0xf2, 0x5c, 0xbf, 0xda, 0x52,
+       0x9a, 0xa6, 0x36, 0xfd, 0xc6, 0xf7, 0x66, 0x00,
+       0xcc, 0x6c, 0xd4, 0xb3, 0x07, 0x6d, 0xeb, 0xfe,
+       0x92, 0x71, 0x25, 0xd0, 0xcf, 0x9c, 0xe8, 0x65,
+       0x45, 0x10, 0xcf, 0x62, 0x74, 0x7d, 0xf2, 0x1b,
+       0x57, 0xa0, 0xf1, 0x6b, 0xa4, 0xd5, 0xfa, 0x12,
+       0x27, 0x5a, 0xf7, 0x99, 0xfc, 0xca, 0xf3, 0xb8,
+       0x2c, 0x8b, 0xba, 0x28, 0x74, 0xde, 0x8f, 0x78,
+       0xa2, 0x8c, 0xaf, 0x89, 0x4b, 0x05, 0xe2, 0xf3,
+       0xf8, 0xd2, 0xef, 0xac, 0xa4, 0xc4, 0xe2, 0xe2,
+       0x36, 0xbb, 0x5e, 0xae, 0xe6, 0x87, 0x3d, 0x88,
+       0x9f, 0xb8, 0x11, 0xbb, 0xcf, 0x57, 0xce, 0xd0,
+       0xba, 0x62, 0xf4, 0xf8, 0x9b, 0x95, 0x04, 0xc9,
+       0xcf, 0x01, 0xe9, 0xf1, 0xc8, 0xc6, 0x22, 0xa4,
+       0xf2, 0x8b, 0x2f, 0x24, 0x0a, 0xf5, 0x6e, 0xb7,
+       0xd4, 0x2c, 0xb6, 0xf7, 0x5c, 0x97, 0x61, 0x0b,
+       0xd9, 0xb5, 0x06, 0xcd, 0xed, 0x3e, 0x1f, 0xc5,
+       0xb2, 0x6c, 0xa3, 0xea, 0xb8, 0xad, 0xa6, 0x42,
+       0x88, 0x7a, 0x52, 0xd5, 0x64, 0xba, 0xb5, 0x20,
+       0x10, 0xa0, 0x0f, 0x0d, 0xea, 0xef, 0x5a, 0x9b,
+       0x27, 0xb8, 0xca, 0x20, 0x19, 0x6d, 0xa8, 0xc4,
+       0x46, 0x04, 0xb3, 0xe8, 0xf8, 0x66, 0x1b, 0x0a,
+       0xce, 0x76, 0x5d, 0x59, 0x58, 0x05, 0xee, 0x3e,
+       0x3c, 0x86, 0x5b, 0x49, 0x1c, 0x72, 0x18, 0x01,
+       0x62, 0x92, 0x0f, 0x3e, 0xd1, 0x57, 0x5e, 0x20,
+       0x7b, 0xfb, 0x4d, 0x3c, 0xc5, 0x35, 0x43, 0x2f,
+       0xb0, 0xc5, 0x7c, 0xe4, 0xa2, 0x84, 0x13, 0x77
+};
+static const u8 output69[] __initconst = {
+       0xbb, 0x4a, 0x7f, 0x7c, 0xd5, 0x2f, 0x89, 0x06,
+       0xec, 0x20, 0xf1, 0x9a, 0x11, 0x09, 0x14, 0x2e,
+       0x17, 0x50, 0xf9, 0xd5, 0xf5, 0x48, 0x7c, 0x7a,
+       0x55, 0xc0, 0x57, 0x03, 0xe3, 0xc4, 0xb2, 0xb7,
+       0x18, 0x47, 0x95, 0xde, 0xaf, 0x80, 0x06, 0x3c,
+       0x5a, 0xf2, 0xc3, 0x53, 0xe3, 0x29, 0x92, 0xf8,
+       0xff, 0x64, 0x85, 0xb9, 0xf7, 0xd3, 0x80, 0xd2,
+       0x0c, 0x5d, 0x7b, 0x57, 0x0c, 0x51, 0x79, 0x86,
+       0xf3, 0x20, 0xd2, 0xb8, 0x6e, 0x0c, 0x5a, 0xce,
+       0xeb, 0x88, 0x02, 0x8b, 0x82, 0x1b, 0x7f, 0xf5,
+       0xde, 0x7f, 0x48, 0x48, 0xdf, 0xa0, 0x55, 0xc6,
+       0x0c, 0x22, 0xa1, 0x80, 0x8d, 0x3b, 0xcb, 0x40,
+       0x2d, 0x3d, 0x0b, 0xf2, 0xe0, 0x22, 0x13, 0x99,
+       0xe1, 0xa7, 0x27, 0x68, 0x31, 0xe1, 0x24, 0x5d,
+       0xd2, 0xee, 0x16, 0xc1, 0xd7, 0xa8, 0x14, 0x19,
+       0x23, 0x72, 0x67, 0x27, 0xdc, 0x5e, 0xb9, 0xc7,
+       0xd8, 0xe3, 0x55, 0x50, 0x40, 0x98, 0x7b, 0xe7,
+       0x34, 0x1c, 0x3b, 0x18, 0x14, 0xd8, 0x62, 0xc1,
+       0x93, 0x84, 0xf3, 0x5b, 0xdd, 0x9e, 0x1f, 0x3b,
+       0x0b, 0xbc, 0x4e, 0x5b, 0x79, 0xa3, 0xca, 0x74,
+       0x2a, 0x98, 0xe8, 0x04, 0x39, 0xef, 0xc6, 0x76,
+       0x6d, 0xee, 0x9f, 0x67, 0x5b, 0x59, 0x3a, 0xe5,
+       0xf2, 0x3b, 0xca, 0x89, 0xe8, 0x9b, 0x03, 0x3d,
+       0x11, 0xd2, 0x4a, 0x70, 0xaf, 0x88, 0xb0, 0x94,
+       0x96, 0x26, 0xab, 0x3c, 0xc1, 0xb8, 0xe4, 0xe7,
+       0x14, 0x61, 0x64, 0x3a, 0x61, 0x08, 0x0f, 0xa9,
+       0xce, 0x64, 0xb2, 0x40, 0xf8, 0x20, 0x3a, 0xa9,
+       0x31, 0xbd, 0x7e, 0x16, 0xca, 0xf5, 0x62, 0x0f,
+       0x91, 0x9f, 0x8e, 0x1d, 0xa4, 0x77, 0xf3, 0x87,
+       0x61, 0xe8, 0x14, 0xde, 0x18, 0x68, 0x4e, 0x9d,
+       0x73, 0xcd, 0x8a, 0xe4, 0x80, 0x84, 0x23, 0xaa,
+       0x9d, 0x64, 0x1c, 0x80, 0x41, 0xca, 0x82, 0x40,
+       0x94, 0x55, 0xe3, 0x28, 0xa1, 0x97, 0x71, 0xba,
+       0xf2, 0x2c, 0x39, 0x62, 0x29, 0x56, 0xd0, 0xff,
+       0xb2, 0x82, 0x20, 0x59, 0x1f, 0xc3, 0x64, 0x57
+};
+static const u8 key69[] __initconst = {
+       0x19, 0x09, 0xe9, 0x7c, 0xd9, 0x02, 0x4a, 0x0c,
+       0x52, 0x25, 0xad, 0x5c, 0x2e, 0x8d, 0x86, 0x10,
+       0x85, 0x2b, 0xba, 0xa4, 0x44, 0x5b, 0x39, 0x3e,
+       0x18, 0xaa, 0xce, 0x0e, 0xe2, 0x69, 0x3c, 0xcf
+};
+enum { nonce69 = 0xdb925a1948f0f060ULL };
+
+static const u8 input70[] __initconst = {
+       0x10, 0xe7, 0x83, 0xcf, 0x42, 0x9f, 0xf2, 0x41,
+       0xc7, 0xe4, 0xdb, 0xf9, 0xa3, 0x02, 0x1d, 0x8d,
+       0x50, 0x81, 0x2c, 0x6b, 0x92, 0xe0, 0x4e, 0xea,
+       0x26, 0x83, 0x2a, 0xd0, 0x31, 0xf1, 0x23, 0xf3,
+       0x0e, 0x88, 0x14, 0x31, 0xf9, 0x01, 0x63, 0x59,
+       0x21, 0xd1, 0x8b, 0xdd, 0x06, 0xd0, 0xc6, 0xab,
+       0x91, 0x71, 0x82, 0x4d, 0xd4, 0x62, 0x37, 0x17,
+       0xf9, 0x50, 0xf9, 0xb5, 0x74, 0xce, 0x39, 0x80,
+       0x80, 0x78, 0xf8, 0xdc, 0x1c, 0xdb, 0x7c, 0x3d,
+       0xd4, 0x86, 0x31, 0x00, 0x75, 0x7b, 0xd1, 0x42,
+       0x9f, 0x1b, 0x97, 0x88, 0x0e, 0x14, 0x0e, 0x1e,
+       0x7d, 0x7b, 0xc4, 0xd2, 0xf3, 0xc1, 0x6d, 0x17,
+       0x5d, 0xc4, 0x75, 0x54, 0x0f, 0x38, 0x65, 0x89,
+       0xd8, 0x7d, 0xab, 0xc9, 0xa7, 0x0a, 0x21, 0x0b,
+       0x37, 0x12, 0x05, 0x07, 0xb5, 0x68, 0x32, 0x32,
+       0xb9, 0xf8, 0x97, 0x17, 0x03, 0xed, 0x51, 0x8f,
+       0x3d, 0x5a, 0xd0, 0x12, 0x01, 0x6e, 0x2e, 0x91,
+       0x1c, 0xbe, 0x6b, 0xa3, 0xcc, 0x75, 0x62, 0x06,
+       0x8e, 0x65, 0xbb, 0xe2, 0x29, 0x71, 0x4b, 0x89,
+       0x6a, 0x9d, 0x85, 0x8c, 0x8c, 0xdf, 0x94, 0x95,
+       0x23, 0x66, 0xf8, 0x92, 0xee, 0x56, 0xeb, 0xb3,
+       0xeb, 0xd2, 0x4a, 0x3b, 0x77, 0x8a, 0x6e, 0xf6,
+       0xca, 0xd2, 0x34, 0x00, 0xde, 0xbe, 0x1d, 0x7a,
+       0x73, 0xef, 0x2b, 0x80, 0x56, 0x16, 0x29, 0xbf,
+       0x6e, 0x33, 0xed, 0x0d, 0xe2, 0x02, 0x60, 0x74,
+       0xe9, 0x0a, 0xbc, 0xd1, 0xc5, 0xe8, 0x53, 0x02,
+       0x79, 0x0f, 0x25, 0x0c, 0xef, 0xab, 0xd3, 0xbc,
+       0xb7, 0xfc, 0xf3, 0xb0, 0x34, 0xd1, 0x07, 0xd2,
+       0x5a, 0x31, 0x1f, 0xec, 0x1f, 0x87, 0xed, 0xdd,
+       0x6a, 0xc1, 0xe8, 0xb3, 0x25, 0x4c, 0xc6, 0x9b,
+       0x91, 0x73, 0xec, 0x06, 0x73, 0x9e, 0x57, 0x65,
+       0x32, 0x75, 0x11, 0x74, 0x6e, 0xa4, 0x7d, 0x0d,
+       0x74, 0x9f, 0x51, 0x10, 0x10, 0x47, 0xc9, 0x71,
+       0x6e, 0x97, 0xae, 0x44, 0x41, 0xef, 0x98, 0x78,
+       0xf4, 0xc5, 0xbd, 0x5e, 0x00, 0xe5, 0xfd, 0xe2,
+       0xbe, 0x8c, 0xc2, 0xae, 0xc2, 0xee, 0x59, 0xf6,
+       0xcb, 0x20, 0x54, 0x84, 0xc3, 0x31, 0x7e, 0x67,
+       0x71, 0xb6, 0x76, 0xbe, 0x81, 0x8f, 0x82, 0xad,
+       0x01, 0x8f, 0xc4, 0x00, 0x04, 0x3d, 0x8d, 0x34,
+       0xaa, 0xea, 0xc0, 0xea, 0x91, 0x42, 0xb6, 0xb8,
+       0x43, 0xf3, 0x17, 0xb2, 0x73, 0x64, 0x82, 0x97,
+       0xd5, 0xc9, 0x07, 0x77, 0xb1, 0x26, 0xe2, 0x00,
+       0x6a, 0xae, 0x70, 0x0b, 0xbe, 0xe6, 0xb8, 0x42,
+       0x81, 0x55, 0xf7, 0xb8, 0x96, 0x41, 0x9d, 0xd4,
+       0x2c, 0x27, 0x00, 0xcc, 0x91, 0x28, 0x22, 0xa4,
+       0x7b, 0x42, 0x51, 0x9e, 0xd6, 0xec, 0xf3, 0x6b,
+       0x00, 0xff, 0x5c, 0xa2, 0xac, 0x47, 0x33, 0x2d,
+       0xf8, 0x11, 0x65, 0x5f, 0x4d, 0x79, 0x8b, 0x4f,
+       0xad, 0xf0, 0x9d, 0xcd, 0xb9, 0x7b, 0x08, 0xf7,
+       0x32, 0x51, 0xfa, 0x39, 0xaa, 0x78, 0x05, 0xb1,
+       0xf3, 0x5d, 0xe8, 0x7c, 0x8e, 0x4f, 0xa2, 0xe0,
+       0x98, 0x0c, 0xb2, 0xa7, 0xf0, 0x35, 0x8e, 0x70,
+       0x7c, 0x82, 0xf3, 0x1b, 0x26, 0x28, 0x12, 0xe5,
+       0x23, 0x57, 0xe4, 0xb4, 0x9b, 0x00, 0x39, 0x97,
+       0xef, 0x7c, 0x46, 0x9b, 0x34, 0x6b, 0xe7, 0x0e,
+       0xa3, 0x2a, 0x18, 0x11, 0x64, 0xc6, 0x7c, 0x8b,
+       0x06, 0x02, 0xf5, 0x69, 0x76, 0xf9, 0xaa, 0x09,
+       0x5f, 0x68, 0xf8, 0x4a, 0x79, 0x58, 0xec, 0x37,
+       0xcf, 0x3a, 0xcc, 0x97, 0x70, 0x1d, 0x3e, 0x52,
+       0x18, 0x0a, 0xad, 0x28, 0x5b, 0x3b, 0xe9, 0x03,
+       0x84, 0xe9, 0x68, 0x50, 0xce, 0xc4, 0xbc, 0x3e,
+       0x21, 0xad, 0x63, 0xfe, 0xc6, 0xfd, 0x6e, 0x69,
+       0x84, 0xa9, 0x30, 0xb1, 0x7a, 0xc4, 0x31, 0x10,
+       0xc1, 0x1f, 0x6e, 0xeb, 0xa5, 0xa6, 0x01
+};
+static const u8 output70[] __initconst = {
+       0x0f, 0x93, 0x2a, 0x20, 0xb3, 0x87, 0x2d, 0xce,
+       0xd1, 0x3b, 0x30, 0xfd, 0x06, 0x6d, 0x0a, 0xaa,
+       0x3e, 0xc4, 0x29, 0x02, 0x8a, 0xde, 0xa6, 0x4b,
+       0x45, 0x1b, 0x4f, 0x25, 0x59, 0xd5, 0x56, 0x6a,
+       0x3b, 0x37, 0xbd, 0x3e, 0x47, 0x12, 0x2c, 0x4e,
+       0x60, 0x5f, 0x05, 0x75, 0x61, 0x23, 0x05, 0x74,
+       0xcb, 0xfc, 0x5a, 0xb3, 0xac, 0x5c, 0x3d, 0xab,
+       0x52, 0x5f, 0x05, 0xbc, 0x57, 0xc0, 0x7e, 0xcf,
+       0x34, 0x5d, 0x7f, 0x41, 0xa3, 0x17, 0x78, 0xd5,
+       0x9f, 0xec, 0x0f, 0x1e, 0xf9, 0xfe, 0xa3, 0xbd,
+       0x28, 0xb0, 0xba, 0x4d, 0x84, 0xdb, 0xae, 0x8f,
+       0x1d, 0x98, 0xb7, 0xdc, 0xf9, 0xad, 0x55, 0x9c,
+       0x89, 0xfe, 0x9b, 0x9c, 0xa9, 0x89, 0xf6, 0x97,
+       0x9c, 0x3f, 0x09, 0x3e, 0xc6, 0x02, 0xc2, 0x55,
+       0x58, 0x09, 0x54, 0x66, 0xe4, 0x36, 0x81, 0x35,
+       0xca, 0x88, 0x17, 0x89, 0x80, 0x24, 0x2b, 0x21,
+       0x89, 0xee, 0x45, 0x5a, 0xe7, 0x1f, 0xd5, 0xa5,
+       0x16, 0xa4, 0xda, 0x70, 0x7e, 0xe9, 0x4f, 0x24,
+       0x61, 0x97, 0xab, 0xa0, 0xe0, 0xe7, 0xb8, 0x5c,
+       0x0f, 0x25, 0x17, 0x37, 0x75, 0x12, 0xb5, 0x40,
+       0xde, 0x1c, 0x0d, 0x8a, 0x77, 0x62, 0x3c, 0x86,
+       0xd9, 0x70, 0x2e, 0x96, 0x30, 0xd2, 0x55, 0xb3,
+       0x6b, 0xc3, 0xf2, 0x9c, 0x47, 0xf3, 0x3a, 0x24,
+       0x52, 0xc6, 0x38, 0xd8, 0x22, 0xb3, 0x0c, 0xfd,
+       0x2f, 0xa3, 0x3c, 0xb5, 0xe8, 0x26, 0xe1, 0xa3,
+       0xad, 0xb0, 0x82, 0x17, 0xc1, 0x53, 0xb8, 0x34,
+       0x48, 0xee, 0x39, 0xae, 0x51, 0x43, 0xec, 0x82,
+       0xce, 0x87, 0xc6, 0x76, 0xb9, 0x76, 0xd3, 0x53,
+       0xfe, 0x49, 0x24, 0x7d, 0x02, 0x42, 0x2b, 0x72,
+       0xfb, 0xcb, 0xd8, 0x96, 0x02, 0xc6, 0x9a, 0x20,
+       0xf3, 0x5a, 0x67, 0xe8, 0x13, 0xf8, 0xb2, 0xcb,
+       0xa2, 0xec, 0x18, 0x20, 0x4a, 0xb0, 0x73, 0x53,
+       0x21, 0xb0, 0x77, 0x53, 0xd8, 0x76, 0xa1, 0x30,
+       0x17, 0x72, 0x2e, 0x33, 0x5f, 0x33, 0x6b, 0x28,
+       0xfb, 0xb0, 0xf4, 0xec, 0x8e, 0xed, 0x20, 0x7d,
+       0x57, 0x8c, 0x74, 0x28, 0x64, 0x8b, 0xeb, 0x59,
+       0x38, 0x3f, 0xe7, 0x83, 0x2e, 0xe5, 0x64, 0x4d,
+       0x5c, 0x1f, 0xe1, 0x3b, 0xd9, 0x84, 0xdb, 0xc9,
+       0xec, 0xd8, 0xc1, 0x7c, 0x1f, 0x1b, 0x68, 0x35,
+       0xc6, 0x34, 0x10, 0xef, 0x19, 0xc9, 0x0a, 0xd6,
+       0x43, 0x7f, 0xa6, 0xcb, 0x9d, 0xf4, 0xf0, 0x16,
+       0xb1, 0xb1, 0x96, 0x64, 0xec, 0x8d, 0x22, 0x4c,
+       0x4b, 0xe8, 0x1a, 0xba, 0x6f, 0xb7, 0xfc, 0xa5,
+       0x69, 0x3e, 0xad, 0x78, 0x79, 0x19, 0xb5, 0x04,
+       0x69, 0xe5, 0x3f, 0xff, 0x60, 0x8c, 0xda, 0x0b,
+       0x7b, 0xf7, 0xe7, 0xe6, 0x29, 0x3a, 0x85, 0xba,
+       0xb5, 0xb0, 0x35, 0xbd, 0x38, 0xce, 0x34, 0x5e,
+       0xf2, 0xdc, 0xd1, 0x8f, 0xc3, 0x03, 0x24, 0xa2,
+       0x03, 0xf7, 0x4e, 0x49, 0x5b, 0xcf, 0x6d, 0xb0,
+       0xeb, 0xe3, 0x30, 0x28, 0xd5, 0x5b, 0x82, 0x5f,
+       0xe4, 0x7c, 0x1e, 0xec, 0xd2, 0x39, 0xf9, 0x6f,
+       0x2e, 0xb3, 0xcd, 0x01, 0xb1, 0x67, 0xaa, 0xea,
+       0xaa, 0xb3, 0x63, 0xaf, 0xd9, 0xb2, 0x1f, 0xba,
+       0x05, 0x20, 0xeb, 0x19, 0x32, 0xf0, 0x6c, 0x3f,
+       0x40, 0xcc, 0x93, 0xb3, 0xd8, 0x25, 0xa6, 0xe4,
+       0xce, 0xd7, 0x7e, 0x48, 0x99, 0x65, 0x7f, 0x86,
+       0xc5, 0xd4, 0x79, 0x6b, 0xab, 0x43, 0xb8, 0x6b,
+       0xf1, 0x2f, 0xea, 0x4c, 0x5e, 0xf0, 0x3b, 0xb4,
+       0xb8, 0xb0, 0x94, 0x0c, 0x6b, 0xe7, 0x22, 0x93,
+       0xaa, 0x01, 0xcb, 0xf1, 0x11, 0x60, 0xf6, 0x69,
+       0xcf, 0x14, 0xde, 0xfb, 0x90, 0x05, 0x27, 0x0c,
+       0x1a, 0x9e, 0xf0, 0xb4, 0xc6, 0xa1, 0xe8, 0xdd,
+       0xd0, 0x4c, 0x25, 0x4f, 0x9c, 0xb7, 0xb1, 0xb0,
+       0x21, 0xdb, 0x87, 0x09, 0x03, 0xf2, 0xb3
+};
+static const u8 key70[] __initconst = {
+       0x3b, 0x5b, 0x59, 0x36, 0x44, 0xd1, 0xba, 0x71,
+       0x55, 0x87, 0x4d, 0x62, 0x3d, 0xc2, 0xfc, 0xaa,
+       0x3f, 0x4e, 0x1a, 0xe4, 0xca, 0x09, 0xfc, 0x6a,
+       0xb2, 0xd6, 0x5d, 0x79, 0xf9, 0x1a, 0x91, 0xa7
+};
+enum { nonce70 = 0x3fd6786dd147a85ULL };
+
+static const u8 input71[] __initconst = {
+       0x18, 0x78, 0xd6, 0x79, 0xe4, 0x9a, 0x6c, 0x73,
+       0x17, 0xd4, 0x05, 0x0f, 0x1e, 0x9f, 0xd9, 0x2b,
+       0x86, 0x48, 0x7d, 0xf4, 0xd9, 0x1c, 0x76, 0xfc,
+       0x8e, 0x22, 0x34, 0xe1, 0x48, 0x4a, 0x8d, 0x79,
+       0xb7, 0xbb, 0x88, 0xab, 0x90, 0xde, 0xc5, 0xb4,
+       0xb4, 0xe7, 0x85, 0x49, 0xda, 0x57, 0xeb, 0xc9,
+       0xcd, 0x21, 0xfc, 0x45, 0x6e, 0x32, 0x67, 0xf2,
+       0x4f, 0xa6, 0x54, 0xe5, 0x20, 0xed, 0xcf, 0xc6,
+       0x62, 0x25, 0x8e, 0x00, 0xf8, 0x6b, 0xa2, 0x80,
+       0xac, 0x88, 0xa6, 0x59, 0x27, 0x83, 0x95, 0x11,
+       0x3f, 0x70, 0x5e, 0x3f, 0x11, 0xfb, 0x26, 0xbf,
+       0xe1, 0x48, 0x75, 0xf9, 0x86, 0xbf, 0xa6, 0x5d,
+       0x15, 0x61, 0x66, 0xbf, 0x78, 0x8f, 0x6b, 0x9b,
+       0xda, 0x98, 0xb7, 0x19, 0xe2, 0xf2, 0xa3, 0x9c,
+       0x7c, 0x6a, 0x9a, 0xd8, 0x3d, 0x4c, 0x2c, 0xe1,
+       0x09, 0xb4, 0x28, 0x82, 0x4e, 0xab, 0x0c, 0x75,
+       0x63, 0xeb, 0xbc, 0xd0, 0x71, 0xa2, 0x73, 0x85,
+       0xed, 0x53, 0x7a, 0x3f, 0x68, 0x9f, 0xd0, 0xa9,
+       0x00, 0x5a, 0x9e, 0x80, 0x55, 0x00, 0xe6, 0xae,
+       0x0c, 0x03, 0x40, 0xed, 0xfc, 0x68, 0x4a, 0xb7,
+       0x1e, 0x09, 0x65, 0x30, 0x5a, 0x3d, 0x97, 0x4d,
+       0x5e, 0x51, 0x8e, 0xda, 0xc3, 0x55, 0x8c, 0xfb,
+       0xcf, 0x83, 0x05, 0x35, 0x0d, 0x08, 0x1b, 0xf3,
+       0x3a, 0x57, 0x96, 0xac, 0x58, 0x8b, 0xfa, 0x00,
+       0x49, 0x15, 0x78, 0xd2, 0x4b, 0xed, 0xb8, 0x59,
+       0x78, 0x9b, 0x7f, 0xaa, 0xfc, 0xe7, 0x46, 0xdc,
+       0x7b, 0x34, 0xd0, 0x34, 0xe5, 0x10, 0xff, 0x4d,
+       0x5a, 0x4d, 0x60, 0xa7, 0x16, 0x54, 0xc4, 0xfd,
+       0xca, 0x5d, 0x68, 0xc7, 0x4a, 0x01, 0x8d, 0x7f,
+       0x74, 0x5d, 0xff, 0xb8, 0x37, 0x15, 0x62, 0xfa,
+       0x44, 0x45, 0xcf, 0x77, 0x3b, 0x1d, 0xb2, 0xd2,
+       0x0d, 0x42, 0x00, 0x39, 0x68, 0x1f, 0xcc, 0x89,
+       0x73, 0x5d, 0xa9, 0x2e, 0xfd, 0x58, 0x62, 0xca,
+       0x35, 0x8e, 0x70, 0x70, 0xaa, 0x6e, 0x14, 0xe9,
+       0xa4, 0xe2, 0x10, 0x66, 0x71, 0xdc, 0x4c, 0xfc,
+       0xa9, 0xdc, 0x8f, 0x57, 0x4d, 0xc5, 0xac, 0xd7,
+       0xa9, 0xf3, 0xf3, 0xa1, 0xff, 0x62, 0xa0, 0x8f,
+       0xe4, 0x96, 0x3e, 0xcb, 0x9f, 0x76, 0x42, 0x39,
+       0x1f, 0x24, 0xfd, 0xfd, 0x79, 0xe8, 0x27, 0xdf,
+       0xa8, 0xf6, 0x33, 0x8b, 0x31, 0x59, 0x69, 0xcf,
+       0x6a, 0xef, 0x89, 0x4d, 0xa7, 0xf6, 0x7e, 0x97,
+       0x14, 0xbd, 0xda, 0xdd, 0xb4, 0x84, 0x04, 0x24,
+       0xe0, 0x17, 0xe1, 0x0f, 0x1f, 0x8a, 0x6a, 0x71,
+       0x74, 0x41, 0xdc, 0x59, 0x5c, 0x8f, 0x01, 0x25,
+       0x92, 0xf0, 0x2e, 0x15, 0x62, 0x71, 0x9a, 0x9f,
+       0x87, 0xdf, 0x62, 0x49, 0x7f, 0x86, 0x62, 0xfc,
+       0x20, 0x84, 0xd7, 0xe3, 0x3a, 0xd9, 0x37, 0x85,
+       0xb7, 0x84, 0x5a, 0xf9, 0xed, 0x21, 0x32, 0x94,
+       0x3e, 0x04, 0xe7, 0x8c, 0x46, 0x76, 0x21, 0x67,
+       0xf6, 0x95, 0x64, 0x92, 0xb7, 0x15, 0xf6, 0xe3,
+       0x41, 0x27, 0x9d, 0xd7, 0xe3, 0x79, 0x75, 0x92,
+       0xd0, 0xc1, 0xf3, 0x40, 0x92, 0x08, 0xde, 0x90,
+       0x22, 0x82, 0xb2, 0x69, 0xae, 0x1a, 0x35, 0x11,
+       0x89, 0xc8, 0x06, 0x82, 0x95, 0x23, 0x44, 0x08,
+       0x22, 0xf2, 0x71, 0x73, 0x1b, 0x88, 0x11, 0xcf,
+       0x1c, 0x7e, 0x8a, 0x2e, 0xdc, 0x79, 0x57, 0xce,
+       0x1f, 0xe7, 0x6c, 0x07, 0xd8, 0x06, 0xbe, 0xec,
+       0xa3, 0xcf, 0xf9, 0x68, 0xa5, 0xb8, 0xf0, 0xe3,
+       0x3f, 0x01, 0x92, 0xda, 0xf1, 0xa0, 0x2d, 0x7b,
+       0xab, 0x57, 0x58, 0x2a, 0xaf, 0xab, 0xbd, 0xf2,
+       0xe5, 0xaf, 0x7e, 0x1f, 0x46, 0x24, 0x9e, 0x20,
+       0x22, 0x0f, 0x84, 0x4c, 0xb7, 0xd8, 0x03, 0xe8,
+       0x09, 0x73, 0x6c, 0xc6, 0x9b, 0x90, 0xe0, 0xdb,
+       0xf2, 0x71, 0xba, 0xad, 0xb3, 0xec, 0xda, 0x7a
+};
+static const u8 output71[] __initconst = {
+       0x28, 0xc5, 0x9b, 0x92, 0xf9, 0x21, 0x4f, 0xbb,
+       0xef, 0x3b, 0xf0, 0xf5, 0x3a, 0x6d, 0x7f, 0xd6,
+       0x6a, 0x8d, 0xa1, 0x01, 0x5c, 0x62, 0x20, 0x8b,
+       0x5b, 0x39, 0xd5, 0xd3, 0xc2, 0xf6, 0x9d, 0x5e,
+       0xcc, 0xe1, 0xa2, 0x61, 0x16, 0xe2, 0xce, 0xe9,
+       0x86, 0xd0, 0xfc, 0xce, 0x9a, 0x28, 0x27, 0xc4,
+       0x0c, 0xb9, 0xaa, 0x8d, 0x48, 0xdb, 0xbf, 0x82,
+       0x7d, 0xd0, 0x35, 0xc4, 0x06, 0x34, 0xb4, 0x19,
+       0x51, 0x73, 0xf4, 0x7a, 0xf4, 0xfd, 0xe9, 0x1d,
+       0xdc, 0x0f, 0x7e, 0xf7, 0x96, 0x03, 0xe3, 0xb1,
+       0x2e, 0x22, 0x59, 0xb7, 0x6d, 0x1c, 0x97, 0x8c,
+       0xd7, 0x31, 0x08, 0x26, 0x4c, 0x6d, 0xc6, 0x14,
+       0xa5, 0xeb, 0x45, 0x6a, 0x88, 0xa3, 0xa2, 0x36,
+       0xc4, 0x35, 0xb1, 0x5a, 0xa0, 0xad, 0xf7, 0x06,
+       0x9b, 0x5d, 0xc1, 0x15, 0xc1, 0xce, 0x0a, 0xb0,
+       0x57, 0x2e, 0x3f, 0x6f, 0x0d, 0x10, 0xd9, 0x11,
+       0x2c, 0x9c, 0xad, 0x2d, 0xa5, 0x81, 0xfb, 0x4e,
+       0x8f, 0xd5, 0x32, 0x4e, 0xaf, 0x5c, 0xc1, 0x86,
+       0xde, 0x56, 0x5a, 0x33, 0x29, 0xf7, 0x67, 0xc6,
+       0x37, 0x6f, 0xb2, 0x37, 0x4e, 0xd4, 0x69, 0x79,
+       0xaf, 0xd5, 0x17, 0x79, 0xe0, 0xba, 0x62, 0xa3,
+       0x68, 0xa4, 0x87, 0x93, 0x8d, 0x7e, 0x8f, 0xa3,
+       0x9c, 0xef, 0xda, 0xe3, 0xa5, 0x1f, 0xcd, 0x30,
+       0xa6, 0x55, 0xac, 0x4c, 0x69, 0x74, 0x02, 0xc7,
+       0x5d, 0x95, 0x81, 0x4a, 0x68, 0x11, 0xd3, 0xa9,
+       0x98, 0xb1, 0x0b, 0x0d, 0xae, 0x40, 0x86, 0x65,
+       0xbf, 0xcc, 0x2d, 0xef, 0x57, 0xca, 0x1f, 0xe4,
+       0x34, 0x4e, 0xa6, 0x5e, 0x82, 0x6e, 0x61, 0xad,
+       0x0b, 0x3c, 0xf8, 0xeb, 0x01, 0x43, 0x7f, 0x87,
+       0xa2, 0xa7, 0x6a, 0xe9, 0x62, 0x23, 0x24, 0x61,
+       0xf1, 0xf7, 0x36, 0xdb, 0x10, 0xe5, 0x57, 0x72,
+       0x3a, 0xc2, 0xae, 0xcc, 0x75, 0xc7, 0x80, 0x05,
+       0x0a, 0x5c, 0x4c, 0x95, 0xda, 0x02, 0x01, 0x14,
+       0x06, 0x6b, 0x5c, 0x65, 0xc2, 0xb8, 0x4a, 0xd6,
+       0xd3, 0xb4, 0xd8, 0x12, 0x52, 0xb5, 0x60, 0xd3,
+       0x8e, 0x5f, 0x5c, 0x76, 0x33, 0x7a, 0x05, 0xe5,
+       0xcb, 0xef, 0x4f, 0x89, 0xf1, 0xba, 0x32, 0x6f,
+       0x33, 0xcd, 0x15, 0x8d, 0xa3, 0x0c, 0x3f, 0x63,
+       0x11, 0xe7, 0x0e, 0xe0, 0x00, 0x01, 0xe9, 0xe8,
+       0x8e, 0x36, 0x34, 0x8d, 0x96, 0xb5, 0x03, 0xcf,
+       0x55, 0x62, 0x49, 0x7a, 0x34, 0x44, 0xa5, 0xee,
+       0x8c, 0x46, 0x06, 0x22, 0xab, 0x1d, 0x53, 0x9c,
+       0xa1, 0xf9, 0x67, 0x18, 0x57, 0x89, 0xf9, 0xc2,
+       0xd1, 0x7e, 0xbe, 0x36, 0x40, 0xcb, 0xe9, 0x04,
+       0xde, 0xb1, 0x3b, 0x29, 0x52, 0xc5, 0x9a, 0xb5,
+       0xa2, 0x7c, 0x7b, 0xfe, 0xe5, 0x92, 0x73, 0xea,
+       0xea, 0x7b, 0xba, 0x0a, 0x8c, 0x88, 0x15, 0xe6,
+       0x53, 0xbf, 0x1c, 0x33, 0xf4, 0x9b, 0x9a, 0x5e,
+       0x8d, 0xae, 0x60, 0xdc, 0xcb, 0x5d, 0xfa, 0xbe,
+       0x06, 0xc3, 0x3f, 0x06, 0xe7, 0x00, 0x40, 0x7b,
+       0xaa, 0x94, 0xfa, 0x6d, 0x1f, 0xe4, 0xc5, 0xa9,
+       0x1b, 0x5f, 0x36, 0xea, 0x5a, 0xdd, 0xa5, 0x48,
+       0x6a, 0x55, 0xd2, 0x47, 0x28, 0xbf, 0x96, 0xf1,
+       0x9f, 0xb6, 0x11, 0x4b, 0xd3, 0x44, 0x7d, 0x48,
+       0x41, 0x61, 0xdb, 0x12, 0xd4, 0xc2, 0x59, 0x82,
+       0x4c, 0x47, 0x5c, 0x04, 0xf6, 0x7b, 0xd3, 0x92,
+       0x2e, 0xe8, 0x40, 0xef, 0x15, 0x32, 0x97, 0xdc,
+       0x35, 0x4c, 0x6e, 0xa4, 0x97, 0xe9, 0x24, 0xde,
+       0x63, 0x8b, 0xb1, 0x6b, 0x48, 0xbb, 0x46, 0x1f,
+       0x84, 0xd6, 0x17, 0xb0, 0x5a, 0x4a, 0x4e, 0xd5,
+       0x31, 0xd7, 0xcf, 0xa0, 0x39, 0xc6, 0x2e, 0xfc,
+       0xa6, 0xa3, 0xd3, 0x0f, 0xa4, 0x28, 0xac, 0xb2,
+       0xf4, 0x48, 0x8d, 0x50, 0xa5, 0x1c, 0x44, 0x5d,
+       0x6e, 0x38, 0xb7, 0x2b, 0x8a, 0x45, 0xa7, 0x3d
+};
+static const u8 key71[] __initconst = {
+       0x8b, 0x68, 0xc4, 0xb7, 0x0d, 0x81, 0xef, 0x52,
+       0x1e, 0x05, 0x96, 0x72, 0x62, 0x89, 0x27, 0x83,
+       0xd0, 0xc7, 0x33, 0x6d, 0xf2, 0xcc, 0x69, 0xf9,
+       0x23, 0xae, 0x99, 0xb1, 0xd1, 0x05, 0x4e, 0x54
+};
+enum { nonce71 = 0x983f03656d64b5f6ULL };
+
+static const u8 input72[] __initconst = {
+       0x6b, 0x09, 0xc9, 0x57, 0x3d, 0x79, 0x04, 0x8c,
+       0x65, 0xad, 0x4a, 0x0f, 0xa1, 0x31, 0x3a, 0xdd,
+       0x14, 0x8e, 0xe8, 0xfe, 0xbf, 0x42, 0x87, 0x98,
+       0x2e, 0x8d, 0x83, 0xa3, 0xf8, 0x55, 0x3d, 0x84,
+       0x1e, 0x0e, 0x05, 0x4a, 0x38, 0x9e, 0xe7, 0xfe,
+       0xd0, 0x4d, 0x79, 0x74, 0x3a, 0x0b, 0x9b, 0xe1,
+       0xfd, 0x51, 0x84, 0x4e, 0xb2, 0x25, 0xe4, 0x64,
+       0x4c, 0xda, 0xcf, 0x46, 0xec, 0xba, 0x12, 0xeb,
+       0x5a, 0x33, 0x09, 0x6e, 0x78, 0x77, 0x8f, 0x30,
+       0xb1, 0x7d, 0x3f, 0x60, 0x8c, 0xf2, 0x1d, 0x8e,
+       0xb4, 0x70, 0xa2, 0x90, 0x7c, 0x79, 0x1a, 0x2c,
+       0xf6, 0x28, 0x79, 0x7c, 0x53, 0xc5, 0xfa, 0xcc,
+       0x65, 0x9b, 0xe1, 0x51, 0xd1, 0x7f, 0x1d, 0xc4,
+       0xdb, 0xd4, 0xd9, 0x04, 0x61, 0x7d, 0xbe, 0x12,
+       0xfc, 0xcd, 0xaf, 0xe4, 0x0f, 0x9c, 0x20, 0xb5,
+       0x22, 0x40, 0x18, 0xda, 0xe4, 0xda, 0x8c, 0x2d,
+       0x84, 0xe3, 0x5f, 0x53, 0x17, 0xed, 0x78, 0xdc,
+       0x2f, 0xe8, 0x31, 0xc7, 0xe6, 0x39, 0x71, 0x40,
+       0xb4, 0x0f, 0xc9, 0xa9, 0x7e, 0x78, 0x87, 0xc1,
+       0x05, 0x78, 0xbb, 0x01, 0xf2, 0x8f, 0x33, 0xb0,
+       0x6e, 0x84, 0xcd, 0x36, 0x33, 0x5c, 0x5b, 0x8e,
+       0xf1, 0xac, 0x30, 0xfe, 0x33, 0xec, 0x08, 0xf3,
+       0x7e, 0xf2, 0xf0, 0x4c, 0xf2, 0xad, 0xd8, 0xc1,
+       0xd4, 0x4e, 0x87, 0x06, 0xd4, 0x75, 0xe7, 0xe3,
+       0x09, 0xd3, 0x4d, 0xe3, 0x21, 0x32, 0xba, 0xb4,
+       0x68, 0x68, 0xcb, 0x4c, 0xa3, 0x1e, 0xb3, 0x87,
+       0x7b, 0xd3, 0x0c, 0x63, 0x37, 0x71, 0x79, 0xfb,
+       0x58, 0x36, 0x57, 0x0f, 0x34, 0x1d, 0xc1, 0x42,
+       0x02, 0x17, 0xe7, 0xed, 0xe8, 0xe7, 0x76, 0xcb,
+       0x42, 0xc4, 0x4b, 0xe2, 0xb2, 0x5e, 0x42, 0xd5,
+       0xec, 0x9d, 0xc1, 0x32, 0x71, 0xe4, 0xeb, 0x10,
+       0x68, 0x1a, 0x6e, 0x99, 0x8e, 0x73, 0x12, 0x1f,
+       0x97, 0x0c, 0x9e, 0xcd, 0x02, 0x3e, 0x4c, 0xa0,
+       0xf2, 0x8d, 0xe5, 0x44, 0xca, 0x6d, 0xfe, 0x07,
+       0xe3, 0xe8, 0x9b, 0x76, 0xc1, 0x6d, 0xb7, 0x6e,
+       0x0d, 0x14, 0x00, 0x6f, 0x8a, 0xfd, 0x43, 0xc6,
+       0x43, 0xa5, 0x9c, 0x02, 0x47, 0x10, 0xd4, 0xb4,
+       0x9b, 0x55, 0x67, 0xc8, 0x7f, 0xc1, 0x8a, 0x1f,
+       0x1e, 0xd1, 0xbc, 0x99, 0x5d, 0x50, 0x4f, 0x89,
+       0xf1, 0xe6, 0x5d, 0x91, 0x40, 0xdc, 0x20, 0x67,
+       0x56, 0xc2, 0xef, 0xbd, 0x2c, 0xa2, 0x99, 0x38,
+       0xe0, 0x45, 0xec, 0x44, 0x05, 0x52, 0x65, 0x11,
+       0xfc, 0x3b, 0x19, 0xcb, 0x71, 0xc2, 0x8e, 0x0e,
+       0x03, 0x2a, 0x03, 0x3b, 0x63, 0x06, 0x31, 0x9a,
+       0xac, 0x53, 0x04, 0x14, 0xd4, 0x80, 0x9d, 0x6b,
+       0x42, 0x7e, 0x7e, 0x4e, 0xdc, 0xc7, 0x01, 0x49,
+       0x9f, 0xf5, 0x19, 0x86, 0x13, 0x28, 0x2b, 0xa6,
+       0xa6, 0xbe, 0xa1, 0x7e, 0x71, 0x05, 0x00, 0xff,
+       0x59, 0x2d, 0xb6, 0x63, 0xf0, 0x1e, 0x2e, 0x69,
+       0x9b, 0x85, 0xf1, 0x1e, 0x8a, 0x64, 0x39, 0xab,
+       0x00, 0x12, 0xe4, 0x33, 0x4b, 0xb5, 0xd8, 0xb3,
+       0x6b, 0x5b, 0x8b, 0x5c, 0xd7, 0x6f, 0x23, 0xcf,
+       0x3f, 0x2e, 0x5e, 0x47, 0xb9, 0xb8, 0x1f, 0xf0,
+       0x1d, 0xda, 0xe7, 0x4f, 0x6e, 0xab, 0xc3, 0x36,
+       0xb4, 0x74, 0x6b, 0xeb, 0xc7, 0x5d, 0x91, 0xe5,
+       0xda, 0xf2, 0xc2, 0x11, 0x17, 0x48, 0xf8, 0x9c,
+       0xc9, 0x8b, 0xc1, 0xa2, 0xf4, 0xcd, 0x16, 0xf8,
+       0x27, 0xd9, 0x6c, 0x6f, 0xb5, 0x8f, 0x77, 0xca,
+       0x1b, 0xd8, 0xef, 0x84, 0x68, 0x71, 0x53, 0xc1,
+       0x43, 0x0f, 0x9f, 0x98, 0xae, 0x7e, 0x31, 0xd2,
+       0x98, 0xfb, 0x20, 0xa2, 0xad, 0x00, 0x10, 0x83,
+       0x00, 0x8b, 0xeb, 0x56, 0xd2, 0xc4, 0xcc, 0x7f,
+       0x2f, 0x4e, 0xfa, 0x88, 0x13, 0xa4, 0x2c, 0xde,
+       0x6b, 0x77, 0x86, 0x10, 0x6a, 0xab, 0x43, 0x0a,
+       0x02
+};
+static const u8 output72[] __initconst = {
+       0x42, 0x89, 0xa4, 0x80, 0xd2, 0xcb, 0x5f, 0x7f,
+       0x2a, 0x1a, 0x23, 0x00, 0xa5, 0x6a, 0x95, 0xa3,
+       0x9a, 0x41, 0xa1, 0xd0, 0x2d, 0x1e, 0xd6, 0x13,
+       0x34, 0x40, 0x4e, 0x7f, 0x1a, 0xbe, 0xa0, 0x3d,
+       0x33, 0x9c, 0x56, 0x2e, 0x89, 0x25, 0x45, 0xf9,
+       0xf0, 0xba, 0x9c, 0x6d, 0xd1, 0xd1, 0xde, 0x51,
+       0x47, 0x63, 0xc9, 0xbd, 0xfa, 0xa2, 0x9e, 0xad,
+       0x6a, 0x7b, 0x21, 0x1a, 0x6c, 0x3e, 0xff, 0x46,
+       0xbe, 0xf3, 0x35, 0x7a, 0x6e, 0xb3, 0xb9, 0xf7,
+       0xda, 0x5e, 0xf0, 0x14, 0xb5, 0x70, 0xa4, 0x2b,
+       0xdb, 0xbb, 0xc7, 0x31, 0x4b, 0x69, 0x5a, 0x83,
+       0x70, 0xd9, 0x58, 0xd4, 0x33, 0x84, 0x23, 0xf0,
+       0xae, 0xbb, 0x6d, 0x26, 0x7c, 0xc8, 0x30, 0xf7,
+       0x24, 0xad, 0xbd, 0xe4, 0x2c, 0x38, 0x38, 0xac,
+       0xe1, 0x4a, 0x9b, 0xac, 0x33, 0x0e, 0x4a, 0xf4,
+       0x93, 0xed, 0x07, 0x82, 0x81, 0x4f, 0x8f, 0xb1,
+       0xdd, 0x73, 0xd5, 0x50, 0x6d, 0x44, 0x1e, 0xbe,
+       0xa7, 0xcd, 0x17, 0x57, 0xd5, 0x3b, 0x62, 0x36,
+       0xcf, 0x7d, 0xc8, 0xd8, 0xd1, 0x78, 0xd7, 0x85,
+       0x46, 0x76, 0x5d, 0xcc, 0xfe, 0xe8, 0x94, 0xc5,
+       0xad, 0xbc, 0x5e, 0xbc, 0x8d, 0x1d, 0xdf, 0x03,
+       0xc9, 0x6b, 0x1b, 0x81, 0xd1, 0xb6, 0x5a, 0x24,
+       0xe3, 0xdc, 0x3f, 0x20, 0xc9, 0x07, 0x73, 0x4c,
+       0x43, 0x13, 0x87, 0x58, 0x34, 0x0d, 0x14, 0x63,
+       0x0f, 0x6f, 0xad, 0x8d, 0xac, 0x7c, 0x67, 0x68,
+       0xa3, 0x9d, 0x7f, 0x00, 0xdf, 0x28, 0xee, 0x67,
+       0xf4, 0x5c, 0x26, 0xcb, 0xef, 0x56, 0x71, 0xc8,
+       0xc6, 0x67, 0x5f, 0x38, 0xbb, 0xa0, 0xb1, 0x5c,
+       0x1f, 0xb3, 0x08, 0xd9, 0x38, 0xcf, 0x74, 0x54,
+       0xc6, 0xa4, 0xc4, 0xc0, 0x9f, 0xb3, 0xd0, 0xda,
+       0x62, 0x67, 0x8b, 0x81, 0x33, 0xf0, 0xa9, 0x73,
+       0xa4, 0xd1, 0x46, 0x88, 0x8d, 0x85, 0x12, 0x40,
+       0xba, 0x1a, 0xcd, 0x82, 0xd8, 0x8d, 0xc4, 0x52,
+       0xe7, 0x01, 0x94, 0x2e, 0x0e, 0xd0, 0xaf, 0xe7,
+       0x2d, 0x3f, 0x3c, 0xaa, 0xf4, 0xf5, 0xa7, 0x01,
+       0x4c, 0x14, 0xe2, 0xc2, 0x96, 0x76, 0xbe, 0x05,
+       0xaa, 0x19, 0xb1, 0xbd, 0x95, 0xbb, 0x5a, 0xf9,
+       0xa5, 0xa7, 0xe6, 0x16, 0x38, 0x34, 0xf7, 0x9d,
+       0x19, 0x66, 0x16, 0x8e, 0x7f, 0x2b, 0x5a, 0xfb,
+       0xb5, 0x29, 0x79, 0xbf, 0x52, 0xae, 0x30, 0x95,
+       0x3f, 0x31, 0x33, 0x28, 0xde, 0xc5, 0x0d, 0x55,
+       0x89, 0xec, 0x21, 0x11, 0x0f, 0x8b, 0xfe, 0x63,
+       0x3a, 0xf1, 0x95, 0x5c, 0xcd, 0x50, 0xe4, 0x5d,
+       0x8f, 0xa7, 0xc8, 0xca, 0x93, 0xa0, 0x67, 0x82,
+       0x63, 0x5c, 0xd0, 0xed, 0xe7, 0x08, 0xc5, 0x60,
+       0xf8, 0xb4, 0x47, 0xf0, 0x1a, 0x65, 0x4e, 0xa3,
+       0x51, 0x68, 0xc7, 0x14, 0xa1, 0xd9, 0x39, 0x72,
+       0xa8, 0x6f, 0x7c, 0x7e, 0xf6, 0x03, 0x0b, 0x25,
+       0x9b, 0xf2, 0xca, 0x49, 0xae, 0x5b, 0xf8, 0x0f,
+       0x71, 0x51, 0x01, 0xa6, 0x23, 0xa9, 0xdf, 0xd0,
+       0x7a, 0x39, 0x19, 0xf5, 0xc5, 0x26, 0x44, 0x7b,
+       0x0a, 0x4a, 0x41, 0xbf, 0xf2, 0x8e, 0x83, 0x50,
+       0x91, 0x96, 0x72, 0x02, 0xf6, 0x80, 0xbf, 0x95,
+       0x41, 0xac, 0xda, 0xb0, 0xba, 0xe3, 0x76, 0xb1,
+       0x9d, 0xff, 0x1f, 0x33, 0x02, 0x85, 0xfc, 0x2a,
+       0x29, 0xe6, 0xe3, 0x9d, 0xd0, 0xef, 0xc2, 0xd6,
+       0x9c, 0x4a, 0x62, 0xac, 0xcb, 0xea, 0x8b, 0xc3,
+       0x08, 0x6e, 0x49, 0x09, 0x26, 0x19, 0xc1, 0x30,
+       0xcc, 0x27, 0xaa, 0xc6, 0x45, 0x88, 0xbd, 0xae,
+       0xd6, 0x79, 0xff, 0x4e, 0xfc, 0x66, 0x4d, 0x02,
+       0xa5, 0xee, 0x8e, 0xa5, 0xb6, 0x15, 0x72, 0x24,
+       0xb1, 0xbf, 0xbf, 0x64, 0xcf, 0xcc, 0x93, 0xe9,
+       0xb6, 0xfd, 0xb4, 0xb6, 0x21, 0xb5, 0x48, 0x08,
+       0x0f, 0x11, 0x65, 0xe1, 0x47, 0xee, 0x93, 0x29,
+       0xad
+};
+static const u8 key72[] __initconst = {
+       0xb9, 0xa2, 0xfc, 0x59, 0x06, 0x3f, 0x77, 0xa5,
+       0x66, 0xd0, 0x2b, 0x22, 0x74, 0x22, 0x4c, 0x1e,
+       0x6a, 0x39, 0xdf, 0xe1, 0x0d, 0x4c, 0x64, 0x99,
+       0x54, 0x8a, 0xba, 0x1d, 0x2c, 0x21, 0x5f, 0xc3
+};
+enum { nonce72 = 0x3d069308fa3db04bULL };
+
+static const u8 input73[] __initconst = {
+       0xe4, 0xdd, 0x36, 0xd4, 0xf5, 0x70, 0x51, 0x73,
+       0x97, 0x1d, 0x45, 0x05, 0x92, 0xe7, 0xeb, 0xb7,
+       0x09, 0x82, 0x6e, 0x25, 0x6c, 0x50, 0xf5, 0x40,
+       0x19, 0xba, 0xbc, 0xf4, 0x39, 0x14, 0xc5, 0x15,
+       0x83, 0x40, 0xbd, 0x26, 0xe0, 0xff, 0x3b, 0x22,
+       0x7c, 0x7c, 0xd7, 0x0b, 0xe9, 0x25, 0x0c, 0x3d,
+       0x92, 0x38, 0xbe, 0xe4, 0x22, 0x75, 0x65, 0xf1,
+       0x03, 0x85, 0x34, 0x09, 0xb8, 0x77, 0xfb, 0x48,
+       0xb1, 0x2e, 0x21, 0x67, 0x9b, 0x9d, 0xad, 0x18,
+       0x82, 0x0d, 0x6b, 0xc3, 0xcf, 0x00, 0x61, 0x6e,
+       0xda, 0xdc, 0xa7, 0x0b, 0x5c, 0x02, 0x1d, 0xa6,
+       0x4e, 0x0d, 0x7f, 0x37, 0x01, 0x5a, 0x37, 0xf3,
+       0x2b, 0xbf, 0xba, 0xe2, 0x1c, 0xb3, 0xa3, 0xbc,
+       0x1c, 0x93, 0x1a, 0xb1, 0x71, 0xaf, 0xe2, 0xdd,
+       0x17, 0xee, 0x53, 0xfa, 0xfb, 0x02, 0x40, 0x3e,
+       0x03, 0xca, 0xe7, 0xc3, 0x51, 0x81, 0xcc, 0x8c,
+       0xca, 0xcf, 0x4e, 0xc5, 0x78, 0x99, 0xfd, 0xbf,
+       0xea, 0xab, 0x38, 0x81, 0xfc, 0xd1, 0x9e, 0x41,
+       0x0b, 0x84, 0x25, 0xf1, 0x6b, 0x3c, 0xf5, 0x40,
+       0x0d, 0xc4, 0x3e, 0xb3, 0x6a, 0xec, 0x6e, 0x75,
+       0xdc, 0x9b, 0xdf, 0x08, 0x21, 0x16, 0xfb, 0x7a,
+       0x8e, 0x19, 0x13, 0x02, 0xa7, 0xfc, 0x58, 0x21,
+       0xc3, 0xb3, 0x59, 0x5a, 0x9c, 0xef, 0x38, 0xbd,
+       0x87, 0x55, 0xd7, 0x0d, 0x1f, 0x84, 0xdc, 0x98,
+       0x22, 0xca, 0x87, 0x96, 0x71, 0x6d, 0x68, 0x00,
+       0xcb, 0x4f, 0x2f, 0xc4, 0x64, 0x0c, 0xc1, 0x53,
+       0x0c, 0x90, 0xe7, 0x3c, 0x88, 0xca, 0xc5, 0x85,
+       0xa3, 0x2a, 0x96, 0x7c, 0x82, 0x6d, 0x45, 0xf5,
+       0xb7, 0x8d, 0x17, 0x69, 0xd6, 0xcd, 0x3c, 0xd3,
+       0xe7, 0x1c, 0xce, 0x93, 0x50, 0xd4, 0x59, 0xa2,
+       0xd8, 0x8b, 0x72, 0x60, 0x5b, 0x25, 0x14, 0xcd,
+       0x5a, 0xe8, 0x8c, 0xdb, 0x23, 0x8d, 0x2b, 0x59,
+       0x12, 0x13, 0x10, 0x47, 0xa4, 0xc8, 0x3c, 0xc1,
+       0x81, 0x89, 0x6c, 0x98, 0xec, 0x8f, 0x7b, 0x32,
+       0xf2, 0x87, 0xd9, 0xa2, 0x0d, 0xc2, 0x08, 0xf9,
+       0xd5, 0xf3, 0x91, 0xe7, 0xb3, 0x87, 0xa7, 0x0b,
+       0x64, 0x8f, 0xb9, 0x55, 0x1c, 0x81, 0x96, 0x6c,
+       0xa1, 0xc9, 0x6e, 0x3b, 0xcd, 0x17, 0x1b, 0xfc,
+       0xa6, 0x05, 0xba, 0x4a, 0x7d, 0x03, 0x3c, 0x59,
+       0xc8, 0xee, 0x50, 0xb2, 0x5b, 0xe1, 0x4d, 0x6a,
+       0x1f, 0x09, 0xdc, 0xa2, 0x51, 0xd1, 0x93, 0x3a,
+       0x5f, 0x72, 0x1d, 0x26, 0x14, 0x62, 0xa2, 0x41,
+       0x3d, 0x08, 0x70, 0x7b, 0x27, 0x3d, 0xbc, 0xdf,
+       0x15, 0xfa, 0xb9, 0x5f, 0xb5, 0x38, 0x84, 0x0b,
+       0x58, 0x3d, 0xee, 0x3f, 0x32, 0x65, 0x6d, 0xd7,
+       0xce, 0x97, 0x3c, 0x8d, 0xfb, 0x63, 0xb9, 0xb0,
+       0xa8, 0x4a, 0x72, 0x99, 0x97, 0x58, 0xc8, 0xa7,
+       0xf9, 0x4c, 0xae, 0xc1, 0x63, 0xb9, 0x57, 0x18,
+       0x8a, 0xfa, 0xab, 0xe9, 0xf3, 0x67, 0xe6, 0xfd,
+       0xd2, 0x9d, 0x5c, 0xa9, 0x8e, 0x11, 0x0a, 0xf4,
+       0x4b, 0xf1, 0xec, 0x1a, 0xaf, 0x50, 0x5d, 0x16,
+       0x13, 0x69, 0x2e, 0xbd, 0x0d, 0xe6, 0xf0, 0xb2,
+       0xed, 0xb4, 0x4c, 0x59, 0x77, 0x37, 0x00, 0x0b,
+       0xc7, 0xa7, 0x9e, 0x37, 0xf3, 0x60, 0x70, 0xef,
+       0xf3, 0xc1, 0x74, 0x52, 0x87, 0xc6, 0xa1, 0x81,
+       0xbd, 0x0a, 0x2c, 0x5d, 0x2c, 0x0c, 0x6a, 0x81,
+       0xa1, 0xfe, 0x26, 0x78, 0x6c, 0x03, 0x06, 0x07,
+       0x34, 0xaa, 0xd1, 0x1b, 0x40, 0x03, 0x39, 0x56,
+       0xcf, 0x2a, 0x92, 0xc1, 0x4e, 0xdf, 0x29, 0x24,
+       0x83, 0x22, 0x7a, 0xea, 0x67, 0x1e, 0xe7, 0x54,
+       0x64, 0xd3, 0xbd, 0x3a, 0x5d, 0xae, 0xca, 0xf0,
+       0x9c, 0xd6, 0x5a, 0x9a, 0x62, 0xc8, 0xc7, 0x83,
+       0xf9, 0x89, 0xde, 0x2d, 0x53, 0x64, 0x61, 0xf7,
+       0xa3, 0xa7, 0x31, 0x38, 0xc6, 0x22, 0x9c, 0xb4,
+       0x87, 0xe0
+};
+static const u8 output73[] __initconst = {
+       0x34, 0xed, 0x05, 0xb0, 0x14, 0xbc, 0x8c, 0xcc,
+       0x95, 0xbd, 0x99, 0x0f, 0xb1, 0x98, 0x17, 0x10,
+       0xae, 0xe0, 0x08, 0x53, 0xa3, 0x69, 0xd2, 0xed,
+       0x66, 0xdb, 0x2a, 0x34, 0x8d, 0x0c, 0x6e, 0xce,
+       0x63, 0x69, 0xc9, 0xe4, 0x57, 0xc3, 0x0c, 0x8b,
+       0xa6, 0x2c, 0xa7, 0xd2, 0x08, 0xff, 0x4f, 0xec,
+       0x61, 0x8c, 0xee, 0x0d, 0xfa, 0x6b, 0xe0, 0xe8,
+       0x71, 0xbc, 0x41, 0x46, 0xd7, 0x33, 0x1d, 0xc0,
+       0xfd, 0xad, 0xca, 0x8b, 0x34, 0x56, 0xa4, 0x86,
+       0x71, 0x62, 0xae, 0x5e, 0x3d, 0x2b, 0x66, 0x3e,
+       0xae, 0xd8, 0xc0, 0xe1, 0x21, 0x3b, 0xca, 0xd2,
+       0x6b, 0xa2, 0xb8, 0xc7, 0x98, 0x4a, 0xf3, 0xcf,
+       0xb8, 0x62, 0xd8, 0x33, 0xe6, 0x80, 0xdb, 0x2f,
+       0x0a, 0xaf, 0x90, 0x3c, 0xe1, 0xec, 0xe9, 0x21,
+       0x29, 0x42, 0x9e, 0xa5, 0x50, 0xe9, 0x93, 0xd3,
+       0x53, 0x1f, 0xac, 0x2a, 0x24, 0x07, 0xb8, 0xed,
+       0xed, 0x38, 0x2c, 0xc4, 0xa1, 0x2b, 0x31, 0x5d,
+       0x9c, 0x24, 0x7b, 0xbf, 0xd9, 0xbb, 0x4e, 0x87,
+       0x8f, 0x32, 0x30, 0xf1, 0x11, 0x29, 0x54, 0x94,
+       0x00, 0x95, 0x1d, 0x1d, 0x24, 0xc0, 0xd4, 0x34,
+       0x49, 0x1d, 0xd5, 0xe3, 0xa6, 0xde, 0x8b, 0xbf,
+       0x5a, 0x9f, 0x58, 0x5a, 0x9b, 0x70, 0xe5, 0x9b,
+       0xb3, 0xdb, 0xe8, 0xb8, 0xca, 0x1b, 0x43, 0xe3,
+       0xc6, 0x6f, 0x0a, 0xd6, 0x32, 0x11, 0xd4, 0x04,
+       0xef, 0xa3, 0xe4, 0x3f, 0x12, 0xd8, 0xc1, 0x73,
+       0x51, 0x87, 0x03, 0xbd, 0xba, 0x60, 0x79, 0xee,
+       0x08, 0xcc, 0xf7, 0xc0, 0xaa, 0x4c, 0x33, 0xc4,
+       0xc7, 0x09, 0xf5, 0x91, 0xcb, 0x74, 0x57, 0x08,
+       0x1b, 0x90, 0xa9, 0x1b, 0x60, 0x02, 0xd2, 0x3f,
+       0x7a, 0xbb, 0xfd, 0x78, 0xf0, 0x15, 0xf9, 0x29,
+       0x82, 0x8f, 0xc4, 0xb2, 0x88, 0x1f, 0xbc, 0xcc,
+       0x53, 0x27, 0x8b, 0x07, 0x5f, 0xfc, 0x91, 0x29,
+       0x82, 0x80, 0x59, 0x0a, 0x3c, 0xea, 0xc4, 0x7e,
+       0xad, 0xd2, 0x70, 0x46, 0xbd, 0x9e, 0x3b, 0x1c,
+       0x8a, 0x62, 0xea, 0x69, 0xbd, 0xf6, 0x96, 0x15,
+       0xb5, 0x57, 0xe8, 0x63, 0x5f, 0x65, 0x46, 0x84,
+       0x58, 0x50, 0x87, 0x4b, 0x0e, 0x5b, 0x52, 0x90,
+       0xb0, 0xae, 0x37, 0x0f, 0xdd, 0x7e, 0xa2, 0xa0,
+       0x8b, 0x78, 0xc8, 0x5a, 0x1f, 0x53, 0xdb, 0xc5,
+       0xbf, 0x73, 0x20, 0xa9, 0x44, 0xfb, 0x1e, 0xc7,
+       0x97, 0xb2, 0x3a, 0x5a, 0x17, 0xe6, 0x8b, 0x9b,
+       0xe8, 0xf8, 0x2a, 0x01, 0x27, 0xa3, 0x71, 0x28,
+       0xe3, 0x19, 0xc6, 0xaf, 0xf5, 0x3a, 0x26, 0xc0,
+       0x5c, 0x69, 0x30, 0x78, 0x75, 0x27, 0xf2, 0x0c,
+       0x22, 0x71, 0x65, 0xc6, 0x8e, 0x7b, 0x47, 0xe3,
+       0x31, 0xaf, 0x7b, 0xc6, 0xc2, 0x55, 0x68, 0x81,
+       0xaa, 0x1b, 0x21, 0x65, 0xfb, 0x18, 0x35, 0x45,
+       0x36, 0x9a, 0x44, 0xba, 0x5c, 0xff, 0x06, 0xde,
+       0x3a, 0xc8, 0x44, 0x0b, 0xaa, 0x8e, 0x34, 0xe2,
+       0x84, 0xac, 0x18, 0xfe, 0x9b, 0xe1, 0x4f, 0xaa,
+       0xb6, 0x90, 0x0b, 0x1c, 0x2c, 0xd9, 0x9a, 0x10,
+       0x18, 0xf9, 0x49, 0x41, 0x42, 0x1b, 0xb5, 0xe1,
+       0x26, 0xac, 0x2d, 0x38, 0x00, 0x00, 0xe4, 0xb4,
+       0x50, 0x6f, 0x14, 0x18, 0xd6, 0x3d, 0x00, 0x59,
+       0x3c, 0x45, 0xf3, 0x42, 0x13, 0x44, 0xb8, 0x57,
+       0xd4, 0x43, 0x5c, 0x8a, 0x2a, 0xb4, 0xfc, 0x0a,
+       0x25, 0x5a, 0xdc, 0x8f, 0x11, 0x0b, 0x11, 0x44,
+       0xc7, 0x0e, 0x54, 0x8b, 0x22, 0x01, 0x7e, 0x67,
+       0x2e, 0x15, 0x3a, 0xb9, 0xee, 0x84, 0x10, 0xd4,
+       0x80, 0x57, 0xd7, 0x75, 0xcf, 0x8b, 0xcb, 0x03,
+       0xc9, 0x92, 0x2b, 0x69, 0xd8, 0x5a, 0x9b, 0x06,
+       0x85, 0x47, 0xaa, 0x4c, 0x28, 0xde, 0x49, 0x58,
+       0xe6, 0x11, 0x1e, 0x5e, 0x64, 0x8e, 0x3b, 0xe0,
+       0x40, 0x2e, 0xac, 0x96, 0x97, 0x15, 0x37, 0x1e,
+       0x30, 0xdd
+};
+static const u8 key73[] __initconst = {
+       0x96, 0x06, 0x1e, 0xc1, 0x6d, 0xba, 0x49, 0x5b,
+       0x65, 0x80, 0x79, 0xdd, 0xf3, 0x67, 0xa8, 0x6e,
+       0x2d, 0x9c, 0x54, 0x46, 0xd8, 0x4a, 0xeb, 0x7e,
+       0x23, 0x86, 0x51, 0xd8, 0x49, 0x49, 0x56, 0xe0
+};
+enum { nonce73 = 0xbefb83cb67e11ffdULL };
+
+static const u8 input74[] __initconst = {
+       0x47, 0x22, 0x70, 0xe5, 0x2f, 0x41, 0x18, 0x45,
+       0x07, 0xd3, 0x6d, 0x32, 0x0d, 0x43, 0x92, 0x2b,
+       0x9b, 0x65, 0x73, 0x13, 0x1a, 0x4f, 0x49, 0x8f,
+       0xff, 0xf8, 0xcc, 0xae, 0x15, 0xab, 0x9d, 0x7d,
+       0xee, 0x22, 0x5d, 0x8b, 0xde, 0x81, 0x5b, 0x81,
+       0x83, 0x49, 0x35, 0x9b, 0xb4, 0xbc, 0x4e, 0x01,
+       0xc2, 0x29, 0xa7, 0xf1, 0xca, 0x3a, 0xce, 0x3f,
+       0xf5, 0x31, 0x93, 0xa8, 0xe2, 0xc9, 0x7d, 0x03,
+       0x26, 0xa4, 0xbc, 0xa8, 0x9c, 0xb9, 0x68, 0xf3,
+       0xb3, 0x91, 0xe8, 0xe6, 0xc7, 0x2b, 0x1a, 0xce,
+       0xd2, 0x41, 0x53, 0xbd, 0xa3, 0x2c, 0x54, 0x94,
+       0x21, 0xa1, 0x40, 0xae, 0xc9, 0x0c, 0x11, 0x92,
+       0xfd, 0x91, 0xa9, 0x40, 0xca, 0xde, 0x21, 0x4e,
+       0x1e, 0x3d, 0xcc, 0x2c, 0x87, 0x11, 0xef, 0x46,
+       0xed, 0x52, 0x03, 0x11, 0x19, 0x43, 0x25, 0xc7,
+       0x0d, 0xc3, 0x37, 0x5f, 0xd3, 0x6f, 0x0c, 0x6a,
+       0x45, 0x30, 0x88, 0xec, 0xf0, 0x21, 0xef, 0x1d,
+       0x7b, 0x38, 0x63, 0x4b, 0x49, 0x0c, 0x72, 0xf6,
+       0x4c, 0x40, 0xc3, 0xcc, 0x03, 0xa7, 0xae, 0xa8,
+       0x8c, 0x37, 0x03, 0x1c, 0x11, 0xae, 0x0d, 0x1b,
+       0x62, 0x97, 0x27, 0xfc, 0x56, 0x4b, 0xb7, 0xfd,
+       0xbc, 0xfb, 0x0e, 0xfc, 0x61, 0xad, 0xc6, 0xb5,
+       0x9c, 0x8c, 0xc6, 0x38, 0x27, 0x91, 0x29, 0x3d,
+       0x29, 0xc8, 0x37, 0xc9, 0x96, 0x69, 0xe3, 0xdc,
+       0x3e, 0x61, 0x35, 0x9b, 0x99, 0x4f, 0xb9, 0x4e,
+       0x5a, 0x29, 0x1c, 0x2e, 0xcf, 0x16, 0xcb, 0x69,
+       0x87, 0xe4, 0x1a, 0xc4, 0x6e, 0x78, 0x43, 0x00,
+       0x03, 0xb2, 0x8b, 0x03, 0xd0, 0xb4, 0xf1, 0xd2,
+       0x7d, 0x2d, 0x7e, 0xfc, 0x19, 0x66, 0x5b, 0xa3,
+       0x60, 0x3f, 0x9d, 0xbd, 0xfa, 0x3e, 0xca, 0x7b,
+       0x26, 0x08, 0x19, 0x16, 0x93, 0x5d, 0x83, 0xfd,
+       0xf9, 0x21, 0xc6, 0x31, 0x34, 0x6f, 0x0c, 0xaa,
+       0x28, 0xf9, 0x18, 0xa2, 0xc4, 0x78, 0x3b, 0x56,
+       0xc0, 0x88, 0x16, 0xba, 0x22, 0x2c, 0x07, 0x2f,
+       0x70, 0xd0, 0xb0, 0x46, 0x35, 0xc7, 0x14, 0xdc,
+       0xbb, 0x56, 0x23, 0x1e, 0x36, 0x36, 0x2d, 0x73,
+       0x78, 0xc7, 0xce, 0xf3, 0x58, 0xf7, 0x58, 0xb5,
+       0x51, 0xff, 0x33, 0x86, 0x0e, 0x3b, 0x39, 0xfb,
+       0x1a, 0xfd, 0xf8, 0x8b, 0x09, 0x33, 0x1b, 0x83,
+       0xf2, 0xe6, 0x38, 0x37, 0xef, 0x47, 0x84, 0xd9,
+       0x82, 0x77, 0x2b, 0x82, 0xcc, 0xf9, 0xee, 0x94,
+       0x71, 0x78, 0x81, 0xc8, 0x4d, 0x91, 0xd7, 0x35,
+       0x29, 0x31, 0x30, 0x5c, 0x4a, 0x23, 0x23, 0xb1,
+       0x38, 0x6b, 0xac, 0x22, 0x3f, 0x80, 0xc7, 0xe0,
+       0x7d, 0xfa, 0x76, 0x47, 0xd4, 0x6f, 0x93, 0xa0,
+       0xa0, 0x93, 0x5d, 0x68, 0xf7, 0x43, 0x25, 0x8f,
+       0x1b, 0xc7, 0x87, 0xea, 0x59, 0x0c, 0xa2, 0xfa,
+       0xdb, 0x2f, 0x72, 0x43, 0xcf, 0x90, 0xf1, 0xd6,
+       0x58, 0xf3, 0x17, 0x6a, 0xdf, 0xb3, 0x4e, 0x0e,
+       0x38, 0x24, 0x48, 0x1f, 0xb7, 0x01, 0xec, 0x81,
+       0xb1, 0x87, 0x5b, 0xec, 0x9c, 0x11, 0x1a, 0xff,
+       0xa5, 0xca, 0x5a, 0x63, 0x31, 0xb2, 0xe4, 0xc6,
+       0x3c, 0x1d, 0xaf, 0x27, 0xb2, 0xd4, 0x19, 0xa2,
+       0xcc, 0x04, 0x92, 0x42, 0xd2, 0xc1, 0x8c, 0x3b,
+       0xce, 0xf5, 0x74, 0xc1, 0x81, 0xf8, 0x20, 0x23,
+       0x6f, 0x20, 0x6d, 0x78, 0x36, 0x72, 0x2c, 0x52,
+       0xdf, 0x5e, 0xe8, 0x75, 0xce, 0x1c, 0x49, 0x9d,
+       0x93, 0x6f, 0x65, 0xeb, 0xb1, 0xbd, 0x8e, 0x5e,
+       0xe5, 0x89, 0xc4, 0x8a, 0x81, 0x3d, 0x9a, 0xa7,
+       0x11, 0x82, 0x8e, 0x38, 0x5b, 0x5b, 0xca, 0x7d,
+       0x4b, 0x72, 0xc2, 0x9c, 0x30, 0x5e, 0x7f, 0xc0,
+       0x6f, 0x91, 0xd5, 0x67, 0x8c, 0x3e, 0xae, 0xda,
+       0x2b, 0x3c, 0x53, 0xcc, 0x50, 0x97, 0x36, 0x0b,
+       0x79, 0xd6, 0x73, 0x6e, 0x7d, 0x42, 0x56, 0xe1,
+       0xaa, 0xfc, 0xb3, 0xa7, 0xc8, 0x01, 0xaa, 0xc1,
+       0xfc, 0x5c, 0x72, 0x8e, 0x63, 0xa8, 0x46, 0x18,
+       0xee, 0x11, 0xe7, 0x30, 0x09, 0x83, 0x6c, 0xd9,
+       0xf4, 0x7a, 0x7b, 0xb5, 0x1f, 0x6d, 0xc7, 0xbc,
+       0xcb, 0x55, 0xea, 0x40, 0x58, 0x7a, 0x00, 0x00,
+       0x90, 0x60, 0xc5, 0x64, 0x69, 0x05, 0x99, 0xd2,
+       0x49, 0x62, 0x4f, 0xcb, 0x97, 0xdf, 0xdd, 0x6b,
+       0x60, 0x75, 0xe2, 0xe0, 0x6f, 0x76, 0xd0, 0x37,
+       0x67, 0x0a, 0xcf, 0xff, 0xc8, 0x61, 0x84, 0x14,
+       0x80, 0x7c, 0x1d, 0x31, 0x8d, 0x90, 0xde, 0x0b,
+       0x1c, 0x74, 0x9f, 0x82, 0x96, 0x80, 0xda, 0xaf,
+       0x8d, 0x99, 0x86, 0x9f, 0x24, 0x99, 0x28, 0x3e,
+       0xe0, 0xa3, 0xc3, 0x90, 0x2d, 0x14, 0x65, 0x1e,
+       0x3b, 0xb9, 0xba, 0x13, 0xa5, 0x77, 0x73, 0x63,
+       0x9a, 0x06, 0x3d, 0xa9, 0x28, 0x9b, 0xba, 0x25,
+       0x61, 0xc9, 0xcd, 0xcf, 0x7a, 0x4d, 0x96, 0x09,
+       0xcb, 0xca, 0x03, 0x9c, 0x54, 0x34, 0x31, 0x85,
+       0xa0, 0x3d, 0xe5, 0xbc, 0xa5, 0x5f, 0x1b, 0xd3,
+       0x10, 0x63, 0x74, 0x9d, 0x01, 0x92, 0x88, 0xf0,
+       0x27, 0x9c, 0x28, 0xd9, 0xfd, 0xe2, 0x4e, 0x01,
+       0x8d, 0x61, 0x79, 0x60, 0x61, 0x5b, 0x76, 0xab,
+       0x06, 0xd3, 0x44, 0x87, 0x43, 0x52, 0xcd, 0x06,
+       0x68, 0x1e, 0x2d, 0xc5, 0xb0, 0x07, 0x25, 0xdf,
+       0x0a, 0x50, 0xd7, 0xd9, 0x08, 0x53, 0x65, 0xf1,
+       0x0c, 0x2c, 0xde, 0x3f, 0x9d, 0x03, 0x1f, 0xe1,
+       0x49, 0x43, 0x3c, 0x83, 0x81, 0x37, 0xf8, 0xa2,
+       0x0b, 0xf9, 0x61, 0x1c, 0xc1, 0xdb, 0x79, 0xbc,
+       0x64, 0xce, 0x06, 0x4e, 0x87, 0x89, 0x62, 0x73,
+       0x51, 0xbc, 0xa4, 0x32, 0xd4, 0x18, 0x62, 0xab,
+       0x65, 0x7e, 0xad, 0x1e, 0x91, 0xa3, 0xfa, 0x2d,
+       0x58, 0x9e, 0x2a, 0xe9, 0x74, 0x44, 0x64, 0x11,
+       0xe6, 0xb6, 0xb3, 0x00, 0x7e, 0xa3, 0x16, 0xef,
+       0x72
+};
+static const u8 output74[] __initconst = {
+       0xf5, 0xca, 0x45, 0x65, 0x50, 0x35, 0x47, 0x67,
+       0x6f, 0x4f, 0x67, 0xff, 0x34, 0xd9, 0xc3, 0x37,
+       0x2a, 0x26, 0xb0, 0x4f, 0x08, 0x1e, 0x45, 0x13,
+       0xc7, 0x2c, 0x14, 0x75, 0x33, 0xd8, 0x8e, 0x1e,
+       0x1b, 0x11, 0x0d, 0x97, 0x04, 0x33, 0x8a, 0xe4,
+       0xd8, 0x8d, 0x0e, 0x12, 0x8d, 0xdb, 0x6e, 0x02,
+       0xfa, 0xe5, 0xbd, 0x3a, 0xb5, 0x28, 0x07, 0x7d,
+       0x20, 0xf0, 0x12, 0x64, 0x83, 0x2f, 0x59, 0x79,
+       0x17, 0x88, 0x3c, 0x2d, 0x08, 0x2f, 0x55, 0xda,
+       0xcc, 0x02, 0x3a, 0x82, 0xcd, 0x03, 0x94, 0xdf,
+       0xdf, 0xab, 0x8a, 0x13, 0xf5, 0xe6, 0x74, 0xdf,
+       0x7b, 0xe2, 0xab, 0x34, 0xbc, 0x00, 0x85, 0xbf,
+       0x5a, 0x48, 0xc8, 0xff, 0x8d, 0x6c, 0x27, 0x48,
+       0x19, 0x2d, 0x08, 0xfa, 0x82, 0x62, 0x39, 0x55,
+       0x32, 0x11, 0xa8, 0xd7, 0xb9, 0x08, 0x2c, 0xd6,
+       0x7a, 0xd9, 0x83, 0x9f, 0x9b, 0xfb, 0xec, 0x3a,
+       0xd1, 0x08, 0xc7, 0xad, 0xdc, 0x98, 0x4c, 0xbc,
+       0x98, 0xeb, 0x36, 0xb0, 0x39, 0xf4, 0x3a, 0xd6,
+       0x53, 0x02, 0xa0, 0xa9, 0x73, 0xa1, 0xca, 0xef,
+       0xd8, 0xd2, 0xec, 0x0e, 0xf8, 0xf5, 0xac, 0x8d,
+       0x34, 0x41, 0x06, 0xa8, 0xc6, 0xc3, 0x31, 0xbc,
+       0xe5, 0xcc, 0x7e, 0x72, 0x63, 0x59, 0x3e, 0x63,
+       0xc2, 0x8d, 0x2b, 0xd5, 0xb9, 0xfd, 0x1e, 0x31,
+       0x69, 0x32, 0x05, 0xd6, 0xde, 0xc9, 0xe6, 0x4c,
+       0xac, 0x68, 0xf7, 0x1f, 0x9d, 0xcd, 0x0e, 0xa2,
+       0x15, 0x3d, 0xd6, 0x47, 0x99, 0xab, 0x08, 0x5f,
+       0x28, 0xc3, 0x4c, 0xc2, 0xd5, 0xdd, 0x10, 0xb7,
+       0xbd, 0xdb, 0x9b, 0xcf, 0x85, 0x27, 0x29, 0x76,
+       0x98, 0xeb, 0xad, 0x31, 0x64, 0xe7, 0xfb, 0x61,
+       0xe0, 0xd8, 0x1a, 0xa6, 0xe2, 0xe7, 0x43, 0x42,
+       0x77, 0xc9, 0x82, 0x00, 0xac, 0x85, 0xe0, 0xa2,
+       0xd4, 0x62, 0xe3, 0xb7, 0x17, 0x6e, 0xb2, 0x9e,
+       0x21, 0x58, 0x73, 0xa9, 0x53, 0x2d, 0x3c, 0xe1,
+       0xdd, 0xd6, 0x6e, 0x92, 0xf2, 0x1d, 0xc2, 0x22,
+       0x5f, 0x9a, 0x7e, 0xd0, 0x52, 0xbf, 0x54, 0x19,
+       0xd7, 0x80, 0x63, 0x3e, 0xd0, 0x08, 0x2d, 0x37,
+       0x0c, 0x15, 0xf7, 0xde, 0xab, 0x2b, 0xe3, 0x16,
+       0x21, 0x3a, 0xee, 0xa5, 0xdc, 0xdf, 0xde, 0xa3,
+       0x69, 0xcb, 0xfd, 0x92, 0x89, 0x75, 0xcf, 0xc9,
+       0x8a, 0xa4, 0xc8, 0xdd, 0xcc, 0x21, 0xe6, 0xfe,
+       0x9e, 0x43, 0x76, 0xb2, 0x45, 0x22, 0xb9, 0xb5,
+       0xac, 0x7e, 0x3d, 0x26, 0xb0, 0x53, 0xc8, 0xab,
+       0xfd, 0xea, 0x2c, 0xd1, 0x44, 0xc5, 0x60, 0x1b,
+       0x8a, 0x99, 0x0d, 0xa5, 0x0e, 0x67, 0x6e, 0x3a,
+       0x96, 0x55, 0xec, 0xe8, 0xcc, 0xbe, 0x49, 0xd9,
+       0xf2, 0x72, 0x9f, 0x30, 0x21, 0x97, 0x57, 0x19,
+       0xbe, 0x5e, 0x33, 0x0c, 0xee, 0xc0, 0x72, 0x0d,
+       0x2e, 0xd1, 0xe1, 0x52, 0xc2, 0xea, 0x41, 0xbb,
+       0xe1, 0x6d, 0xd4, 0x17, 0xa9, 0x8d, 0x89, 0xa9,
+       0xd6, 0x4b, 0xc6, 0x4c, 0xf2, 0x88, 0x97, 0x54,
+       0x3f, 0x4f, 0x57, 0xb7, 0x37, 0xf0, 0x2c, 0x11,
+       0x15, 0x56, 0xdb, 0x28, 0xb5, 0x16, 0x84, 0x66,
+       0xce, 0x45, 0x3f, 0x61, 0x75, 0xb6, 0xbe, 0x00,
+       0xd1, 0xe4, 0xf5, 0x27, 0x54, 0x7f, 0xc2, 0xf1,
+       0xb3, 0x32, 0x9a, 0xe8, 0x07, 0x02, 0xf3, 0xdb,
+       0xa9, 0xd1, 0xc2, 0xdf, 0xee, 0xad, 0xe5, 0x8a,
+       0x3c, 0xfa, 0x67, 0xec, 0x6b, 0xa4, 0x08, 0xfe,
+       0xba, 0x5a, 0x58, 0x0b, 0x78, 0x11, 0x91, 0x76,
+       0xe3, 0x1a, 0x28, 0x54, 0x5e, 0xbd, 0x71, 0x1b,
+       0x8b, 0xdc, 0x6c, 0xf4, 0x6f, 0xd7, 0xf4, 0xf3,
+       0xe1, 0x03, 0xa4, 0x3c, 0x8d, 0x91, 0x2e, 0xba,
+       0x5f, 0x7f, 0x8c, 0xaf, 0x69, 0x89, 0x29, 0x0a,
+       0x5b, 0x25, 0x13, 0xc4, 0x2e, 0x16, 0xc2, 0x15,
+       0x07, 0x5d, 0x58, 0x33, 0x7c, 0xe0, 0xf0, 0x55,
+       0x5f, 0xbf, 0x5e, 0xf0, 0x71, 0x48, 0x8f, 0xf7,
+       0x48, 0xb3, 0xf7, 0x0d, 0xa1, 0xd0, 0x63, 0xb1,
+       0xad, 0xae, 0xb5, 0xb0, 0x5f, 0x71, 0xaf, 0x24,
+       0x8b, 0xb9, 0x1c, 0x44, 0xd2, 0x1a, 0x53, 0xd1,
+       0xd5, 0xb4, 0xa9, 0xff, 0x88, 0x73, 0xb5, 0xaa,
+       0x15, 0x32, 0x5f, 0x59, 0x9d, 0x2e, 0xb5, 0xcb,
+       0xde, 0x21, 0x2e, 0xe9, 0x35, 0xed, 0xfd, 0x0f,
+       0xb6, 0xbb, 0xe6, 0x4b, 0x16, 0xf1, 0x45, 0x1e,
+       0xb4, 0x84, 0xe9, 0x58, 0x1c, 0x0c, 0x95, 0xc0,
+       0xcf, 0x49, 0x8b, 0x59, 0xa1, 0x78, 0xe6, 0x80,
+       0x12, 0x49, 0x7a, 0xd4, 0x66, 0x62, 0xdf, 0x9c,
+       0x18, 0xc8, 0x8c, 0xda, 0xc1, 0xa6, 0xbc, 0x65,
+       0x28, 0xd2, 0xa4, 0xe8, 0xf1, 0x35, 0xdb, 0x5a,
+       0x75, 0x1f, 0x73, 0x60, 0xec, 0xa8, 0xda, 0x5a,
+       0x43, 0x15, 0x83, 0x9b, 0xe7, 0xb1, 0xa6, 0x81,
+       0xbb, 0xef, 0xf3, 0x8f, 0x0f, 0xd3, 0x79, 0xa2,
+       0xe5, 0xaa, 0x42, 0xef, 0xa0, 0x13, 0x4e, 0x91,
+       0x2d, 0xcb, 0x61, 0x7a, 0x9a, 0x33, 0x14, 0x50,
+       0x77, 0x4a, 0xd0, 0x91, 0x48, 0xe0, 0x0c, 0xe0,
+       0x11, 0xcb, 0xdf, 0xb0, 0xce, 0x06, 0xd2, 0x79,
+       0x4d, 0x69, 0xb9, 0xc9, 0x36, 0x74, 0x8f, 0x81,
+       0x72, 0x73, 0xf3, 0x17, 0xb7, 0x13, 0xcb, 0x5b,
+       0xd2, 0x5c, 0x33, 0x61, 0xb7, 0x61, 0x79, 0xb0,
+       0xc0, 0x4d, 0xa1, 0xc7, 0x5d, 0x98, 0xc9, 0xe1,
+       0x98, 0xbd, 0x78, 0x5a, 0x2c, 0x64, 0x53, 0xaf,
+       0xaf, 0x66, 0x51, 0x47, 0xe4, 0x48, 0x66, 0x8b,
+       0x07, 0x52, 0xa3, 0x03, 0x93, 0x28, 0xad, 0xcc,
+       0xa3, 0x86, 0xad, 0x63, 0x04, 0x35, 0x6c, 0x49,
+       0xd5, 0x28, 0x0e, 0x00, 0x47, 0xf4, 0xd4, 0x32,
+       0x27, 0x19, 0xb3, 0x29, 0xe7, 0xbc, 0xbb, 0xce,
+       0x3e, 0x3e, 0xd5, 0x67, 0x20, 0xe4, 0x0b, 0x75,
+       0x95, 0x24, 0xe0, 0x6c, 0xb6, 0x29, 0x0c, 0x14,
+       0xfd
+};
+static const u8 key74[] __initconst = {
+       0xf0, 0x41, 0x5b, 0x00, 0x56, 0xc4, 0xac, 0xf6,
+       0xa2, 0x4c, 0x33, 0x41, 0x16, 0x09, 0x1b, 0x8e,
+       0x4d, 0xe8, 0x8c, 0xd9, 0x48, 0xab, 0x3e, 0x60,
+       0xcb, 0x49, 0x3e, 0xaf, 0x2b, 0x8b, 0xc8, 0xf0
+};
+enum { nonce74 = 0xcbdb0ffd0e923384ULL };
+
+static const struct chacha20_testvec chacha20_testvecs[] __initconst = {
+       { input01, output01, key01, nonce01, sizeof(input01) },
+       { input02, output02, key02, nonce02, sizeof(input02) },
+       { input03, output03, key03, nonce03, sizeof(input03) },
+       { input04, output04, key04, nonce04, sizeof(input04) },
+       { input05, output05, key05, nonce05, sizeof(input05) },
+       { input06, output06, key06, nonce06, sizeof(input06) },
+       { input07, output07, key07, nonce07, sizeof(input07) },
+       { input08, output08, key08, nonce08, sizeof(input08) },
+       { input09, output09, key09, nonce09, sizeof(input09) },
+       { input10, output10, key10, nonce10, sizeof(input10) },
+       { input11, output11, key11, nonce11, sizeof(input11) },
+       { input12, output12, key12, nonce12, sizeof(input12) },
+       { input13, output13, key13, nonce13, sizeof(input13) },
+       { input14, output14, key14, nonce14, sizeof(input14) },
+       { input15, output15, key15, nonce15, sizeof(input15) },
+       { input16, output16, key16, nonce16, sizeof(input16) },
+       { input17, output17, key17, nonce17, sizeof(input17) },
+       { input18, output18, key18, nonce18, sizeof(input18) },
+       { input19, output19, key19, nonce19, sizeof(input19) },
+       { input20, output20, key20, nonce20, sizeof(input20) },
+       { input21, output21, key21, nonce21, sizeof(input21) },
+       { input22, output22, key22, nonce22, sizeof(input22) },
+       { input23, output23, key23, nonce23, sizeof(input23) },
+       { input24, output24, key24, nonce24, sizeof(input24) },
+       { input25, output25, key25, nonce25, sizeof(input25) },
+       { input26, output26, key26, nonce26, sizeof(input26) },
+       { input27, output27, key27, nonce27, sizeof(input27) },
+       { input28, output28, key28, nonce28, sizeof(input28) },
+       { input29, output29, key29, nonce29, sizeof(input29) },
+       { input30, output30, key30, nonce30, sizeof(input30) },
+       { input31, output31, key31, nonce31, sizeof(input31) },
+       { input32, output32, key32, nonce32, sizeof(input32) },
+       { input33, output33, key33, nonce33, sizeof(input33) },
+       { input34, output34, key34, nonce34, sizeof(input34) },
+       { input35, output35, key35, nonce35, sizeof(input35) },
+       { input36, output36, key36, nonce36, sizeof(input36) },
+       { input37, output37, key37, nonce37, sizeof(input37) },
+       { input38, output38, key38, nonce38, sizeof(input38) },
+       { input39, output39, key39, nonce39, sizeof(input39) },
+       { input40, output40, key40, nonce40, sizeof(input40) },
+       { input41, output41, key41, nonce41, sizeof(input41) },
+       { input42, output42, key42, nonce42, sizeof(input42) },
+       { input43, output43, key43, nonce43, sizeof(input43) },
+       { input44, output44, key44, nonce44, sizeof(input44) },
+       { input45, output45, key45, nonce45, sizeof(input45) },
+       { input46, output46, key46, nonce46, sizeof(input46) },
+       { input47, output47, key47, nonce47, sizeof(input47) },
+       { input48, output48, key48, nonce48, sizeof(input48) },
+       { input49, output49, key49, nonce49, sizeof(input49) },
+       { input50, output50, key50, nonce50, sizeof(input50) },
+       { input51, output51, key51, nonce51, sizeof(input51) },
+       { input52, output52, key52, nonce52, sizeof(input52) },
+       { input53, output53, key53, nonce53, sizeof(input53) },
+       { input54, output54, key54, nonce54, sizeof(input54) },
+       { input55, output55, key55, nonce55, sizeof(input55) },
+       { input56, output56, key56, nonce56, sizeof(input56) },
+       { input57, output57, key57, nonce57, sizeof(input57) },
+       { input58, output58, key58, nonce58, sizeof(input58) },
+       { input59, output59, key59, nonce59, sizeof(input59) },
+       { input60, output60, key60, nonce60, sizeof(input60) },
+       { input61, output61, key61, nonce61, sizeof(input61) },
+       { input62, output62, key62, nonce62, sizeof(input62) },
+       { input63, output63, key63, nonce63, sizeof(input63) },
+       { input64, output64, key64, nonce64, sizeof(input64) },
+       { input65, output65, key65, nonce65, sizeof(input65) },
+       { input66, output66, key66, nonce66, sizeof(input66) },
+       { input67, output67, key67, nonce67, sizeof(input67) },
+       { input68, output68, key68, nonce68, sizeof(input68) },
+       { input69, output69, key69, nonce69, sizeof(input69) },
+       { input70, output70, key70, nonce70, sizeof(input70) },
+       { input71, output71, key71, nonce71, sizeof(input71) },
+       { input72, output72, key72, nonce72, sizeof(input72) },
+       { input73, output73, key73, nonce73, sizeof(input73) },
+       { input74, output74, key74, nonce74, sizeof(input74) }
+};
+
+static const struct hchacha20_testvec hchacha20_testvecs[] __initconst = {{
+       .key    = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
+                   0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
+                   0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
+                   0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f },
+       .nonce  = { 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, 0x4a,
+                   0x00, 0x00, 0x00, 0x00, 0x31, 0x41, 0x59, 0x27 },
+       .output = { 0x82, 0x41, 0x3b, 0x42, 0x27, 0xb2, 0x7b, 0xfe,
+                   0xd3, 0x0e, 0x42, 0x50, 0x8a, 0x87, 0x7d, 0x73,
+                   0xa0, 0xf9, 0xe4, 0xd5, 0x8a, 0x74, 0xa8, 0x53,
+                   0xc1, 0x2e, 0xc4, 0x13, 0x26, 0xd3, 0xec, 0xdc }
+}};
+
+static bool __init chacha20_selftest(void)
+{
+       enum { MAXIMUM_TEST_BUFFER_LEN = 1UL << 10 };
+       size_t i, j, k;
+       u32 derived_key[CHACHA20_KEY_WORDS];
+       u8 *offset_input = NULL, *computed_output = NULL;
+       u8 offset_key[CHACHA20_KEY_SIZE + 1]
+                       __aligned(__alignof__(unsigned long));
+       struct chacha20_ctx state;
+       bool success = true;
+       simd_context_t simd_context;
+
+       offset_input = kmalloc(MAXIMUM_TEST_BUFFER_LEN + 1, GFP_KERNEL);
+       computed_output = kmalloc(MAXIMUM_TEST_BUFFER_LEN + 1, GFP_KERNEL);
+       if (!computed_output || !offset_input) {
+               pr_info("chacha20 self-test malloc: FAIL\n");
+               success = false;
+               goto out;
+       }
+
+       simd_get(&simd_context);
+       for (i = 0; i < ARRAY_SIZE(chacha20_testvecs); ++i) {
+               /* Boring case */
+               memset(computed_output, 0, MAXIMUM_TEST_BUFFER_LEN + 1);
+               memset(&state, 0, sizeof(state));
+               chacha20_init(&state, chacha20_testvecs[i].key,
+                             chacha20_testvecs[i].nonce);
+               chacha20(&state, computed_output, chacha20_testvecs[i].input,
+                        chacha20_testvecs[i].ilen, &simd_context);
+               if (memcmp(computed_output, chacha20_testvecs[i].output,
+                          chacha20_testvecs[i].ilen)) {
+                       pr_info("chacha20 self-test %zu: FAIL\n", i + 1);
+                       success = false;
+               }
+               for (k = chacha20_testvecs[i].ilen;
+                    k < MAXIMUM_TEST_BUFFER_LEN + 1; ++k) {
+                       if (computed_output[k]) {
+                               pr_info("chacha20 self-test %zu (zero check): FAIL\n",
+                                       i + 1);
+                               success = false;
+                       }
+               }
+
+               /* Unaligned case */
+               memset(computed_output, 0, MAXIMUM_TEST_BUFFER_LEN + 1);
+               memset(&state, 0, sizeof(state));
+               memcpy(offset_input + 1, chacha20_testvecs[i].input,
+                      chacha20_testvecs[i].ilen);
+               memcpy(offset_key + 1, chacha20_testvecs[i].key,
+                      CHACHA20_KEY_SIZE);
+               chacha20_init(&state, offset_key + 1, chacha20_testvecs[i].nonce);
+               chacha20(&state, computed_output + 1, offset_input + 1,
+                        chacha20_testvecs[i].ilen, &simd_context);
+               if (memcmp(computed_output + 1, chacha20_testvecs[i].output,
+                          chacha20_testvecs[i].ilen)) {
+                       pr_info("chacha20 self-test %zu (unaligned): FAIL\n",
+                               i + 1);
+                       success = false;
+               }
+               if (computed_output[0]) {
+                       pr_info("chacha20 self-test %zu (unaligned, zero check): FAIL\n",
+                               i + 1);
+                       success = false;
+               }
+               for (k = chacha20_testvecs[i].ilen + 1;
+                    k < MAXIMUM_TEST_BUFFER_LEN + 1; ++k) {
+                       if (computed_output[k]) {
+                               pr_info("chacha20 self-test %zu (unaligned, zero check): FAIL\n",
+                                       i + 1);
+                               success = false;
+                       }
+               }
+
+               /* Chunked case */
+               if (chacha20_testvecs[i].ilen <= CHACHA20_BLOCK_SIZE)
+                       goto next_test;
+               memset(computed_output, 0, MAXIMUM_TEST_BUFFER_LEN + 1);
+               memset(&state, 0, sizeof(state));
+               chacha20_init(&state, chacha20_testvecs[i].key,
+                             chacha20_testvecs[i].nonce);
+               chacha20(&state, computed_output, chacha20_testvecs[i].input,
+                        CHACHA20_BLOCK_SIZE, &simd_context);
+               chacha20(&state, computed_output + CHACHA20_BLOCK_SIZE,
+                        chacha20_testvecs[i].input + CHACHA20_BLOCK_SIZE,
+                        chacha20_testvecs[i].ilen - CHACHA20_BLOCK_SIZE,
+                        &simd_context);
+               if (memcmp(computed_output, chacha20_testvecs[i].output,
+                          chacha20_testvecs[i].ilen)) {
+                       pr_info("chacha20 self-test %zu (chunked): FAIL\n",
+                               i + 1);
+                       success = false;
+               }
+               for (k = chacha20_testvecs[i].ilen;
+                    k < MAXIMUM_TEST_BUFFER_LEN + 1; ++k) {
+                       if (computed_output[k]) {
+                               pr_info("chacha20 self-test %zu (chunked, zero check): FAIL\n",
+                                       i + 1);
+                               success = false;
+                       }
+               }
+
+next_test:
+               /* Sliding unaligned case */
+               if (chacha20_testvecs[i].ilen > CHACHA20_BLOCK_SIZE + 1 ||
+                   !chacha20_testvecs[i].ilen)
+                       continue;
+               for (j = 1; j < CHACHA20_BLOCK_SIZE; ++j) {
+                       memset(computed_output, 0, MAXIMUM_TEST_BUFFER_LEN + 1);
+                       memset(&state, 0, sizeof(state));
+                       memcpy(offset_input + j, chacha20_testvecs[i].input,
+                              chacha20_testvecs[i].ilen);
+                       chacha20_init(&state, chacha20_testvecs[i].key,
+                                     chacha20_testvecs[i].nonce);
+                       chacha20(&state, computed_output + j, offset_input + j,
+                                chacha20_testvecs[i].ilen, &simd_context);
+                       if (memcmp(computed_output + j,
+                                  chacha20_testvecs[i].output,
+                                  chacha20_testvecs[i].ilen)) {
+                               pr_info("chacha20 self-test %zu (unaligned, slide %zu): FAIL\n",
+                                       i + 1, j);
+                               success = false;
+                       }
+                       for (k = j; k < j; ++k) {
+                               if (computed_output[k]) {
+                                       pr_info("chacha20 self-test %zu (unaligned, slide %zu, zero check): FAIL\n",
+                                               i + 1, j);
+                                       success = false;
+                               }
+                       }
+                       for (k = chacha20_testvecs[i].ilen + j;
+                            k < MAXIMUM_TEST_BUFFER_LEN + 1; ++k) {
+                               if (computed_output[k]) {
+                                       pr_info("chacha20 self-test %zu (unaligned, slide %zu, zero check): FAIL\n",
+                                               i + 1, j);
+                                       success = false;
+                               }
+                       }
+               }
+       }
+       for (i = 0; i < ARRAY_SIZE(hchacha20_testvecs); ++i) {
+               memset(&derived_key, 0, sizeof(derived_key));
+               hchacha20(derived_key, hchacha20_testvecs[i].nonce,
+                         hchacha20_testvecs[i].key, &simd_context);
+               cpu_to_le32_array(derived_key, ARRAY_SIZE(derived_key));
+               if (memcmp(derived_key, hchacha20_testvecs[i].output,
+                          CHACHA20_KEY_SIZE)) {
+                       pr_info("hchacha20 self-test %zu: FAIL\n", i + 1);
+                       success = false;
+               }
+       }
+       simd_put(&simd_context);
+       if (success)
+               pr_info("chacha20 self-tests: pass\n");
+
+out:
+       kfree(offset_input);
+       kfree(computed_output);
+       return success;
+}
+#endif
--

2.19.0

author maintainer
#27 lore raw

Re: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest

Hi Ard,

Quoted 9 lines · Ard Biesheuvel

include/zinc/chacha20.h line 36

Quoted 6 lines
include/zinc/chacha20.h new file +6 -0
+struct chacha20_ctx {
+       u32 constant[4];
+       u32 key[8];
+       u32 counter[4];
+} __aligned(32);
+

32 *byte* alignment? Is that right? If this is for performance and it actually helps, using __cacheline_aligned is more appropriate,

It was originally this way, I believe, for vmovdqa, which requires 32-byte alignment in vex.256 encoding, and I never removed it after changing some things. But I'll spend some time ensuring this is so and if it doesn't make sense anymore it'll be gone by v7. On the other hand, your suggestion of __cacheline_aligned may actually be something worth considering, especially on MIPS.

Quoted 3 lines

I guess this include is for crypto_xor_cpy() ?

We may want to put a comment here, so we keep track of the interdependencies.

Right, it's for crypto_xor_cpy. Good idea, I'll add the comment.

Quoted 4 lines
Quoted 1 line

+#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION

This #define is never set in subsequent patches, so just drop this #ifndef entirely (for this patch only)

Okay. It's also there for the other primitives too; I'll nix it for all of them.

Quoted 10 lines

Return values from initcalls are ignored, and given that chacha20 will be depended upon by random.c, it will never be a module in practice.

Given your previous statement that selftest should *not* be a DEBUG feature (which I wholeheartedly agree with), you could be a bit noisier here imo.
E.g.,

if (WARN_ON(!chach20_selftest())

    return ...

That's an excellent idea. We could bloat the whole thing with something like:

#ifdef MODULE
#define WARN_ON_IF_MODULE(x) (x)
#else
#define WARN_ON_IF_MODULE(x) WARN_ON(x)
#endif

But given that kind of thing would probably need to touch more files in the tree, and hence involve more drawn-out conversation, I'll keep it as the simple WARN_ON for now. Besides, being noisy no matter what might actually be the best strategy for receiving bug reports on what is potentially a pretty catastrophic error.

Thanks for the review.

Jason

maintainer
#28 lore raw

Re: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest

In reply to Jason A. Donenfeld
Quoted 11 lines · Jason A. Donenfeld

This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.

Information: https://cr.yp.to/chacha.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>

I don't have any problems with this interface.

However, I'm very unhappy about the fact that this adds another copy of the chacha20 implementation to the kernel. Either this is better than the one that we already have or it isn't.

So if it's better please replace the existing chacha20 C implementation with this or otherwise just adapt the existing implementation to this interface.

If we're replacing the existing one then you can either replace it directly within this patch or in a subsequent patch but please do it in the same series because otherwise we could wind up with two implementations for years to come.

Thanks,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author maintainer
#29 lore raw

Re: [PATCH net-next v6 03/23] zinc: ChaCha20 generic C implementation and selftest

Hi Herbert,

Quoted 6 lines · Herbert Xu

I don't have any problems with this interface.

However, I'm very unhappy about the fact that this adds another copy of the chacha20 implementation to the kernel. Either this is better than the one that we already have or it isn't.

Quoted 8 lines

So if it's better please replace the existing chacha20 C implementation with this or otherwise just adapt the existing implementation to this interface.

If we're replacing the existing one then you can either replace it directly within this patch or in a subsequent patch but please do it in the same series because otherwise we could wind up with two implementations for years to come.

Indeed this series takes care of removing the other implementation later on.

In case it helps to see things visualized, here's the layout of the whole series:
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

Note that comments on this are probably a bit more useful than on the version posted to the mailing list, since I've already started incorporating quite a bit of feedback from v6, and so v7 will be posted sometime later containing these.

Jason

author maintainer
#30 lore raw

PATCH v6 4/23 zinc: ChaCha20 x86_64 implementation

In reply to Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.

- When aligning the stack, the original code was saving %rsp to %r9.

    To keep objtool happy, we use instead the DRAP idiom to save %rsp
    to %r10:

      leaq    8(%rsp),%r10
      ... code here ...
      leaq    -8(%r10),%rsp

- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da

Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:

size old new
---- ---- ----
0 62 52
16 414 376
32 410 400
48 414 422
64 362 356
80 714 666
96 714 700
112 712 718
128 692 646
144 1042 674
160 1042 694
176 1042 726
192 1018 650
208 1366 686
224 1366 696
240 1366 722
256 640 656
272 988 1246
288 988 1276
304 992 1296
320 972 1222
336 1318 1256
352 1318 1276
368 1316 1294
384 1294 1218
400 1642 1258
416 1642 1282
432 1642 1302
448 1628 1224
464 1970 1258
480 1970 1280
496 1970 1300
512 656 676
528 1010 1290
544 1010 1306
560 1010 1332
576 986 1254
592 1340 1284
608 1334 1310
624 1340 1334
640 1314 1254
656 1664 1282
672 1674 1306
688 1662 1336
704 1638 1250
720 1992 1292
736 1994 1308
752 1988 1334
768 1252 1254
784 1596 1290
800 1596 1314
816 1596 1330
832 1576 1256
848 1922 1286
864 1922 1314
880 1926 1338
896 1898 1258
912 2248 1288
928 2248 1320
944 2248 1338
960 2226 1268
976 2574 1288
992 2576 1312
1008 2574 1340

Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:

size old new
---- ---- ----
0 64 54
16 386 372
32 388 396
48 388 420
64 366 350
80 708 666
96 708 692
112 706 736
128 692 648
144 1036 682
160 1036 708
176 1036 730
192 1016 658
208 1360 684
224 1362 708
240 1360 732
256 644 500
272 990 526
288 988 556
304 988 576
320 972 500
336 1314 532
352 1316 558
368 1318 578
384 1308 506
400 1644 532
416 1644 556
432 1644 594
448 1624 508
464 1970 534
480 1970 556
496 1968 582
512 660 624
528 1016 682
544 1016 702
560 1018 728
576 998 654
592 1344 680
608 1344 708
624 1344 730
640 1326 654
656 1670 686
672 1670 708
688 1670 732
704 1652 658
720 1998 682
736 1998 710
752 1996 734
768 1256 662
784 1606 688
800 1606 714
816 1606 736
832 1584 660
848 1948 688
864 1950 714
880 1948 736
896 1912 688
912 2258 718
928 2258 744
944 2256 768
960 2238 692
976 2584 718
992 2584 744
1008 2584 770

Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: [email protected]

---
 lib/zinc/Makefile                        |    1 +
 lib/zinc/chacha20/chacha20-x86_64-glue.h |  105 +
 lib/zinc/chacha20/chacha20-x86_64.S      | 2632 ++++++++++++++++++++++
 lib/zinc/chacha20/chacha20.c             |    4 +-
 4 files changed, 2741 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +1 -0
index 3d80144d55a6..223a0816c918 100644
@@ -3,4 +3,5 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +105 -0

105 lines not on this page · raw message

lib/zinc/chacha20/chacha20-x86_64.S new file +2632 -0

2,632 lines not on this page · raw message

lib/zinc/chacha20/chacha20.c +3 -1
index c82d9fc71f21..4354b874a6a5 100644
@@ -14,7 +14,9 @@
 #include <linux/init.h>
 #include <crypto/algapi.h>
 
-#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "chacha20-x86_64-glue.h"
+#else
 void __init chacha20_fpu_init(void)
 {
 }
Signature · 1 line
2.19.0
#31 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Quoted 30 lines · Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.

- When aligning the stack, the original code was saving %rsp to %r9.

    To keep objtool happy, we use instead the DRAP idiom to save %rsp
    to %r10:

      leaq    8(%rsp),%r10
      ... code here ...
      leaq    -8(%r10),%rsp

- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.

Quoted 9 lines

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da

I'd still prefer the kernel side changes to be presented as a separate followup patch (preferably based on the .pl but I understand that is more difficult with x86 than with ARM code)

Quoted 138 lines

Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:

size    old     new
----    ----    ----
0       62      52
16      414     376
32      410     400
48      414     422
64      362     356
80      714     666
96      714     700
112     712     718
128     692     646
144     1042    674
160     1042    694
176     1042    726
192     1018    650
208     1366    686
224     1366    696
240     1366    722
256     640     656
272     988     1246
288     988     1276
304     992     1296
320     972     1222
336     1318    1256
352     1318    1276
368     1316    1294
384     1294    1218
400     1642    1258
416     1642    1282
432     1642    1302
448     1628    1224
464     1970    1258
480     1970    1280
496     1970    1300
512     656     676
528     1010    1290
544     1010    1306
560     1010    1332
576     986     1254
592     1340    1284
608     1334    1310
624     1340    1334
640     1314    1254
656     1664    1282
672     1674    1306
688     1662    1336
704     1638    1250
720     1992    1292
736     1994    1308
752     1988    1334
768     1252    1254
784     1596    1290
800     1596    1314
816     1596    1330
832     1576    1256
848     1922    1286
864     1922    1314
880     1926    1338
896     1898    1258
912     2248    1288
928     2248    1320
944     2248    1338
960     2226    1268
976     2574    1288
992     2576    1312
1008    2574    1340

Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:

size    old     new
----    ----    ----
0       64      54
16      386     372
32      388     396
48      388     420
64      366     350
80      708     666
96      708     692
112     706     736
128     692     648
144     1036    682
160     1036    708
176     1036    730
192     1016    658
208     1360    684
224     1362    708
240     1360    732
256     644     500
272     990     526
288     988     556
304     988     576
320     972     500
336     1314    532
352     1316    558
368     1318    578
384     1308    506
400     1644    532
416     1644    556
432     1644    594
448     1624    508
464     1970    534
480     1970    556
496     1968    582
512     660     624
528     1016    682
544     1016    702
560     1018    728
576     998     654
592     1344    680
608     1344    708
624     1344    730
640     1326    654
656     1670    686
672     1670    708
688     1670    732
704     1652    658
720     1998    682
736     1998    710
752     1996    734
768     1256    662
784     1606    688
800     1606    714
816     1606    736
832     1584    660
848     1948    688
864     1950    714
880     1948    736
896     1912    688
912     2258    718
928     2258    744
944     2256    768
960     2238    692
976     2584    718
992     2584    744
1008    2584    770
Quoted 2 lines

Please drop this SOB line: SOB is not about (co-)authorship but about who handled the patch on its way into mainline. You are sending the patch so your SOB should come last.

Quoted 51 lines

Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: [email protected]

---
 lib/zinc/Makefile                        |    1 +
 lib/zinc/chacha20/chacha20-x86_64-glue.h |  105 +
 lib/zinc/chacha20/chacha20-x86_64.S      | 2632 ++++++++++++++++++++++
 lib/zinc/chacha20/chacha20.c             |    4 +-
 4 files changed, 2741 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +1 -0
index 3d80144d55a6..223a0816c918 100644
@@ -3,4 +3,5 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +17 -0
new file mode 100644
index 000000000000..9b47001661a6
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/fpu/api.h>
+#include <asm/cpufeature.h>
+#include <asm/processor.h>
+#include <asm/intel-family.h>
+
+#ifdef CONFIG_AS_SSSE3
+asmlinkage void hchacha20_ssse3(u32 *derived_key, const u8 *nonce,
+                               const u8 *key);
+asmlinkage void chacha20_ssse3(u8 *out, const u8 *in, const size_t len,
+                              const u32 key[8], const u32 counter[4]);
+#endif
+#ifdef CONFIG_AS_AVX2

lib/zinc/chacha20/chacha20-x86_64-glue.h line 27

Quoted 10 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +10 -0
+asmlinkage void chacha20_avx2(u8 *out, const u8 *in, const size_t len,
+                             const u32 key[8], const u32 counter[4]);
+#endif
+#ifdef CONFIG_AS_AVX512
+asmlinkage void chacha20_avx512(u8 *out, const u8 *in, const size_t len,
+                               const u32 key[8], const u32 counter[4]);
+asmlinkage void chacha20_avx512vl(u8 *out, const u8 *in, const size_t len,
+                                 const u32 key[8], const u32 counter[4]);
+#endif
+

You can drop the #ifdefs above ...

Quoted 28 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +28 -0
+static bool chacha20_use_ssse3 __ro_after_init;
+static bool chacha20_use_avx2 __ro_after_init;
+static bool chacha20_use_avx512 __ro_after_init;
+static bool chacha20_use_avx512vl __ro_after_init;
+
+static void __init chacha20_fpu_init(void)
+{
+       chacha20_use_ssse3 = boot_cpu_has(X86_FEATURE_SSSE3);
+       chacha20_use_avx2 =
+               boot_cpu_has(X86_FEATURE_AVX) &&
+               boot_cpu_has(X86_FEATURE_AVX2) &&
+               cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL);
+       chacha20_use_avx512 =
+               boot_cpu_has(X86_FEATURE_AVX) &&
+               boot_cpu_has(X86_FEATURE_AVX2) &&
+               boot_cpu_has(X86_FEATURE_AVX512F) &&
+               cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
+                                 XFEATURE_MASK_AVX512, NULL) &&
+               /* Skylake downclocks unacceptably much when using zmm. */
+               boot_cpu_data.x86_model != INTEL_FAM6_SKYLAKE_X;
+       chacha20_use_avx512vl =
+               boot_cpu_has(X86_FEATURE_AVX) &&
+               boot_cpu_has(X86_FEATURE_AVX2) &&
+               boot_cpu_has(X86_FEATURE_AVX512F) &&
+               boot_cpu_has(X86_FEATURE_AVX512VL) &&
+               cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
+                                 XFEATURE_MASK_AVX512, NULL);
+}

lib/zinc/chacha20/chacha20-x86_64-glue.h line 65

Quoted 10 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +10 -0
+
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, const size_t len,
+                                simd_context_t *simd_context)
+{
+       if (!chacha20_use_ssse3 || len <= CHACHA20_BLOCK_SIZE ||
+           !simd_use(simd_context))
+               return false;
+
+#ifdef CONFIG_AS_AVX512

... and use IS_ENABLED(CONFIG_AS_AVX512) here inside the if().

Quoted 40 lines
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +40 -0
+       if (chacha20_use_avx512 && len >= CHACHA20_BLOCK_SIZE * 8) {
+               chacha20_avx512(dst, src, len, state->key, state->counter);
+               goto success;
+       }
+       if (chacha20_use_avx512vl && len >= CHACHA20_BLOCK_SIZE * 4) {
+               chacha20_avx512vl(dst, src, len, state->key, state->counter);
+               goto success;
+       }
+#endif
+#ifdef CONFIG_AS_AVX2
+       if (chacha20_use_avx2 && len >= CHACHA20_BLOCK_SIZE * 4) {
+               chacha20_avx2(dst, src, len, state->key, state->counter);
+               goto success;
+       }
+#endif
+#ifdef CONFIG_AS_SSSE3
+       if (chacha20_use_ssse3) {
+               chacha20_ssse3(dst, src, len, state->key, state->counter);
+               goto success;
+       }
+#endif
+       return false;
+success:
+       state->counter[0] += (len + 63) / 64;
+       return true;
+}
+
+static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
+                                 const u8 nonce[HCHACHA20_NONCE_SIZE],
+                                 const u8 key[HCHACHA20_KEY_SIZE],
+                                 simd_context_t *simd_context)
+{
+#if defined(CONFIG_AS_SSSE3)
+       if (chacha20_use_ssse3 && simd_use(simd_context)) {
+               hchacha20_ssse3(derived_key, nonce, key);
+               return true;
+       }
+#endif
+       return false;
+}

...

lib/zinc/chacha20/chacha20.c line 17

Quoted 9 lines
lib/zinc/chacha20/chacha20.c +0 -1
index c82d9fc71f21..4354b874a6a5 100644
@@ -14,7 +14,9 @@
 #include <linux/init.h>
 #include <crypto/algapi.h>
 
-#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION

As I mentioned in reply to the previous patch, please get rid of this CPP symbol ^^^

Quoted 9 lines
lib/zinc/chacha20/chacha20.c +3 -1
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "chacha20-x86_64-glue.h"
+#else
 void __init chacha20_fpu_init(void)
 {
 }
--

2.19.0

author maintainer
#32 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Hi Ard,

Quoted 2 lines · Ard Biesheuvel

Please drop this SOB line: SOB is not about (co-)authorship but about who handled the patch on its way into mainline.

The s-o-b line is about the DCoO, which is given here by both of us, with the code starting in a shared tree that only had that s-o-b. I can remove it though if you think it's strange to have two s-o-bs.

Quoted 2 lines

You are sending the
patch so your SOB should come last.

I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.

Quoted 1 line

You can drop the #ifdefs above ...

Yep. All of those ifdefs in this and in the other primitives are already gone, and everything is fully IS_ENABLEDenabled now.

Quoted 3 lines
Quoted 1 line

+#ifdef CONFIG_AS_AVX512

... and use IS_ENABLED(CONFIG_AS_AVX512) here inside the if().

Already done, per your suggestion a few days ago.

Quoted 4 lines
Quoted 1 line

-#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION

As I mentioned in reply to the previous patch, please get rid of this CPP symbol ^^^

Ack.

Jason

maintainer
#33 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Quoted 3 lines · Jason A. Donenfeld

I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.

Documentation/process/submitting-patches.rst

section 12, first sentence.

The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#34 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Quoted 5 lines · Borislav Petkov
Quoted 3 lines · Jason A. Donenfeld

I was wondering about the ordering of these, actually. I've seen s-o-b on top and s-o-b on bottom of the cc list in lots of commits and haven't yet divined the One True Position.

Quoted 7 lines

Documentation/process/submitting-patches.rst

section 12, first sentence.

The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.

Note that this is the author of the *patch* not necessarily the author of the code.

Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.

maintainer
#35 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Quoted 8 lines · Ard Biesheuvel

Note that this is the author of the *patch* not necessarily the author of the code.

Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.

... and the actual author could be named with Originally-by or Co-Developed-by:, and even in free text in the commit message.

But the correct SOB chain denoting who handled the patch on its way up, is important.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#36 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Quoted 4 lines · Borislav Petkov
Quoted 3 lines · Ard Biesheuvel

Note that this is the author of the *patch* not necessarily the author of the code.

Quoted 8 lines
Quoted 5 lines

Anyone is free to submit patches adding code authored by others as long as the author has made it available under a suitable license, and this is actually the whole point of the S-o-B: you are stating to the next guy that the code included in your patch was made available to you under a compatible license.

... and the actual author could be named with Originally-by or Co-Developed-by:, and even in free text in the commit message.

Or you could just put the original author as the first S-o-b and then another (second) S-o-b line where you put yourself as a submitter.

AFAIK, that's how people usually do it.

Quoted 9 lines

But the correct SOB chain denoting who handled the patch on its way up, is important.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
author maintainer
#37 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

In reply to Borislav Petkov

Hi Borislav,

Quoted 6 lines · Borislav Petkov

Documentation/process/submitting-patches.rst

section 12, first sentence.

The SOB chain needs to clearly express the path of the patch from author(s) to the upstream kernel.

Thanks for the clarification.

I've received the SOB directly from the individual in question as it traveled through an intermediate tree on its way to the tree being submitted here. Therefore I will retain that SOB so that it shows the full path.

Jason

author maintainer
#38 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

In reply to Ard Biesheuvel

Hi Ard,

Quoted 6 lines · Ard Biesheuvel
Quoted 3 lines

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da

I'd still prefer the kernel side changes to be presented as a separate followup patch

I've restructured things for the MIPS64 and x86_64 and it will be as you've asked for v7.

Jason

maintainer
#39 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

In reply to Jason A. Donenfeld
Quoted 36 lines · Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.

- When aligning the stack, the original code was saving %rsp to %r9. To keep objtool happy, we use instead the DRAP idiom to save %rsp to %r10:

     leaq    8(%rsp),%r10
     ... code here ...
     leaq    -8(%r10),%rsp

- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

Quoted 8 lines

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit cded951378069a478391843f5f8653c1eb5128da

Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:

size    old     new
----    ----    ----
0       62      52

What is the old column? Is it the existing x86-64 implementation in the kernel or something else? This needs to be made clear in the patch description.

The same comment from the previous patch also stands here. Please ensure that we only have one copy of chacha x86-64 code in the kernel. Either replace the existing one right here or in a follow-up patch in the same series.

Thanks,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author maintainer
#40 lore raw

Re: [PATCH net-next v6 04/23] zinc: ChaCha20 x86_64 implementation

Hi Herbert,

Quoted 7 lines · Herbert Xu
Quoted 3 lines
size    old     new
----    ----    ----
0       62      52

What is the old column? Is it the existing x86-64 implementation in the kernel or something else? This needs to be made clear in the patch description.

Good idea. I'll make that more clear.

Quoted 4 lines

The same comment from the previous patch also stands here. Please ensure that we only have one copy of chacha x86-64 code in the kernel. Either replace the existing one right here or in a follow-up patch in the same series.

Yes indeed this happens within this same series.

Jason

author maintainer
#41 lore raw

PATCH v6 5/23 zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2

Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org

---
 lib/zinc/chacha20/chacha20-arm-cryptogams.S   | 1440 ++++++++++++
 lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++
 2 files changed, 3413 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S
 create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.S
lib/zinc/chacha20/chacha20-arm-cryptogams.S new file +1440 -0
new file mode 100644
index 000000000000..05a3a9e6e93f
@@ -0,0 +1,1440 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ */
+
+#include "arm_arch.h"
+
+.text
+#if defined(__thumb2__) || defined(__clang__)
+.syntax	unified
+#endif
+#if defined(__thumb2__)
+.thumb
+#else
+.code	32
+#endif
+
+#if defined(__thumb2__) || defined(__clang__)
+#define ldrhsb	ldrbhs
+#endif
+
+.align	5
+.Lsigma:
+.long	0x61707865,0x3320646e,0x79622d32,0x6b206574	@ endian-neutral
+.Lone:
+.long	1,0,0,0
+.Lrot8:
+.long	0x02010003,0x06050407
+#if __ARM_MAX_ARCH__>=7
+.LOPENSSL_armcap:
+.word   OPENSSL_armcap_P-.LChaCha20_ctr32
+#else
+.word	-1
+#endif
+
+.globl	ChaCha20_ctr32
+.type	ChaCha20_ctr32,%function
+.align	5
+ChaCha20_ctr32:
+.LChaCha20_ctr32:
+	ldr	r12,[sp,#0]		@ pull pointer to counter and nonce
+	stmdb	sp!,{r0-r2,r4-r11,lr}
+#if __ARM_ARCH__<7 && !defined(__thumb2__)
+	sub	r14,pc,#16		@ ChaCha20_ctr32
+#else
+	adr	r14,.LChaCha20_ctr32
+#endif
+	cmp	r2,#0			@ len==0?
+#ifdef	__thumb2__
+	itt	eq
+#endif
+	addeq	sp,sp,#4*3
+	beq	.Lno_data
+#if __ARM_MAX_ARCH__>=7
+	cmp	r2,#192			@ test len
+	bls	.Lshort
+	ldr	r4,[r14,#-24]
+	ldr	r4,[r14,r4]
+# ifdef	__APPLE__
+	ldr	r4,[r4]
+# endif
+	tst	r4,#ARMV7_NEON
+	bne	.LChaCha20_neon
+.Lshort:
+#endif
+	ldmia	r12,{r4-r7}		@ load counter and nonce
+	sub	sp,sp,#4*(16)		@ off-load area
+	sub	r14,r14,#64		@ .Lsigma
+	stmdb	sp!,{r4-r7}		@ copy counter and nonce
+	ldmia	r3,{r4-r11}		@ load key
+	ldmia	r14,{r0-r3}		@ load sigma
+	stmdb	sp!,{r4-r11}		@ copy key
+	stmdb	sp!,{r0-r3}		@ copy sigma
+	str	r10,[sp,#4*(16+10)]	@ off-load "rx"
+	str	r11,[sp,#4*(16+11)]	@ off-load "rx"
+	b	.Loop_outer_enter
+
+.align	4
+.Loop_outer:
+	ldmia	sp,{r0-r9}		@ load key material
+	str	r11,[sp,#4*(32+2)]	@ save len
+	str	r12,  [sp,#4*(32+1)]	@ save inp
+	str	r14,  [sp,#4*(32+0)]	@ save out
+.Loop_outer_enter:
+	ldr	r11, [sp,#4*(15)]
+	 mov	r4,r4,ror#19	@ twist b[0..3]
+	ldr	r12,[sp,#4*(12)]	@ modulo-scheduled load
+	 mov	r5,r5,ror#19
+	ldr	r10, [sp,#4*(13)]
+	 mov	r6,r6,ror#19
+	ldr	r14,[sp,#4*(14)]
+	 mov	r7,r7,ror#19
+	mov	r11,r11,ror#8	@ twist d[0..3]
+	mov	r12,r12,ror#8
+	mov	r10,r10,ror#8
+	mov	r14,r14,ror#8
+	str	r11, [sp,#4*(16+15)]
+	mov	r11,#10
+	b	.Loop
+
+.align	4
+.Loop:
+	subs	r11,r11,#1
+	add	r0,r0,r4,ror#13
+	add	r1,r1,r5,ror#13
+	eor	r12,r0,r12,ror#24
+	eor	r10,r1,r10,ror#24
+	add	r8,r8,r12,ror#16
+	add	r9,r9,r10,ror#16
+	eor	r4,r8,r4,ror#13
+	eor	r5,r9,r5,ror#13
+	add	r0,r0,r4,ror#20
+	add	r1,r1,r5,ror#20
+	eor	r12,r0,r12,ror#16
+	eor	r10,r1,r10,ror#16
+	add	r8,r8,r12,ror#24
+	str	r10,[sp,#4*(16+13)]
+	add	r9,r9,r10,ror#24
+	ldr	r10,[sp,#4*(16+15)]
+	str	r8,[sp,#4*(16+8)]
+	eor	r4,r4,r8,ror#12
+	str	r9,[sp,#4*(16+9)]
+	eor	r5,r5,r9,ror#12
+	ldr	r8,[sp,#4*(16+10)]
+	add	r2,r2,r6,ror#13
+	ldr	r9,[sp,#4*(16+11)]
+	add	r3,r3,r7,ror#13
+	eor	r14,r2,r14,ror#24
+	eor	r10,r3,r10,ror#24
+	add	r8,r8,r14,ror#16
+	add	r9,r9,r10,ror#16
+	eor	r6,r8,r6,ror#13
+	eor	r7,r9,r7,ror#13
+	add	r2,r2,r6,ror#20
+	add	r3,r3,r7,ror#20
+	eor	r14,r2,r14,ror#16
+	eor	r10,r3,r10,ror#16
+	add	r8,r8,r14,ror#24
+	add	r9,r9,r10,ror#24
+	eor	r6,r6,r8,ror#12
+	eor	r7,r7,r9,ror#12
+	add	r0,r0,r5,ror#13
+	add	r1,r1,r6,ror#13
+	eor	r10,r0,r10,ror#24
+	eor	r12,r1,r12,ror#24
+	add	r8,r8,r10,ror#16
+	add	r9,r9,r12,ror#16
+	eor	r5,r8,r5,ror#13
+	eor	r6,r9,r6,ror#13
+	add	r0,r0,r5,ror#20
+	add	r1,r1,r6,ror#20
+	eor	r10,r0,r10,ror#16
+	eor	r12,r1,r12,ror#16
+	str	r10,[sp,#4*(16+15)]
+	add	r8,r8,r10,ror#24
+	ldr	r10,[sp,#4*(16+13)]
+	add	r9,r9,r12,ror#24
+	str	r8,[sp,#4*(16+10)]
+	eor	r5,r5,r8,ror#12
+	str	r9,[sp,#4*(16+11)]
+	eor	r6,r6,r9,ror#12
+	ldr	r8,[sp,#4*(16+8)]
+	add	r2,r2,r7,ror#13
+	ldr	r9,[sp,#4*(16+9)]
+	add	r3,r3,r4,ror#13
+	eor	r10,r2,r10,ror#24
+	eor	r14,r3,r14,ror#24
+	add	r8,r8,r10,ror#16
+	add	r9,r9,r14,ror#16
+	eor	r7,r8,r7,ror#13
+	eor	r4,r9,r4,ror#13
+	add	r2,r2,r7,ror#20
+	add	r3,r3,r4,ror#20
+	eor	r10,r2,r10,ror#16
+	eor	r14,r3,r14,ror#16
+	add	r8,r8,r10,ror#24
+	add	r9,r9,r14,ror#24
+	eor	r7,r7,r8,ror#12
+	eor	r4,r4,r9,ror#12
+	bne	.Loop
+
+	ldr	r11,[sp,#4*(32+2)]	@ load len
+
+	str	r8, [sp,#4*(16+8)]	@ modulo-scheduled store
+	str	r9, [sp,#4*(16+9)]
+	str	r12,[sp,#4*(16+12)]
+	str	r10, [sp,#4*(16+13)]
+	str	r14,[sp,#4*(16+14)]
+
+	@ at this point we have first half of 512-bit result in
+	@ rx and second half at sp+4*(16+8)
+
+	cmp	r11,#64		@ done yet?
+#ifdef	__thumb2__
+	itete	lo
+#endif
+	addlo	r12,sp,#4*(0)		@ shortcut or ...
+	ldrhs	r12,[sp,#4*(32+1)]	@ ... load inp
+	addlo	r14,sp,#4*(0)		@ shortcut or ...
+	ldrhs	r14,[sp,#4*(32+0)]	@ ... load out
+
+	ldr	r8,[sp,#4*(0)]	@ load key material
+	ldr	r9,[sp,#4*(1)]
+
+#if __ARM_ARCH__>=6 || !defined(__ARMEB__)
+# if __ARM_ARCH__<7
+	orr	r10,r12,r14
+	tst	r10,#3		@ are input and output aligned?
+	ldr	r10,[sp,#4*(2)]
+	bne	.Lunaligned
+	cmp	r11,#64		@ restore flags
+# else
+	ldr	r10,[sp,#4*(2)]
+# endif
+	ldr	r11,[sp,#4*(3)]
+
+	add	r0,r0,r8	@ accumulate key material
+	add	r1,r1,r9
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r8,[r12],#16		@ load input
+	ldrhs	r9,[r12,#-12]
+
+	add	r2,r2,r10
+	add	r3,r3,r11
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r10,[r12,#-8]
+	ldrhs	r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+	rev	r0,r0
+	rev	r1,r1
+	rev	r2,r2
+	rev	r3,r3
+# endif
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r0,r0,r8	@ xor with input
+	eorhs	r1,r1,r9
+	 add	r8,sp,#4*(4)
+	str	r0,[r14],#16		@ store output
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r2,r2,r10
+	eorhs	r3,r3,r11
+	 ldmia	r8,{r8-r11}	@ load key material
+	str	r1,[r14,#-12]
+	str	r2,[r14,#-8]
+	str	r3,[r14,#-4]
+
+	add	r4,r8,r4,ror#13 @ accumulate key material
+	add	r5,r9,r5,ror#13
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r8,[r12],#16		@ load input
+	ldrhs	r9,[r12,#-12]
+	add	r6,r10,r6,ror#13
+	add	r7,r11,r7,ror#13
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r10,[r12,#-8]
+	ldrhs	r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+	rev	r4,r4
+	rev	r5,r5
+	rev	r6,r6
+	rev	r7,r7
+# endif
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r4,r4,r8
+	eorhs	r5,r5,r9
+	 add	r8,sp,#4*(8)
+	str	r4,[r14],#16		@ store output
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r6,r6,r10
+	eorhs	r7,r7,r11
+	str	r5,[r14,#-12]
+	 ldmia	r8,{r8-r11}	@ load key material
+	str	r6,[r14,#-8]
+	 add	r0,sp,#4*(16+8)
+	str	r7,[r14,#-4]
+
+	ldmia	r0,{r0-r7}	@ load second half
+
+	add	r0,r0,r8	@ accumulate key material
+	add	r1,r1,r9
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r8,[r12],#16		@ load input
+	ldrhs	r9,[r12,#-12]
+# ifdef	__thumb2__
+	itt	hi
+# endif
+	 strhi	r10,[sp,#4*(16+10)]	@ copy "rx" while at it
+	 strhi	r11,[sp,#4*(16+11)]	@ copy "rx" while at it
+	add	r2,r2,r10
+	add	r3,r3,r11
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r10,[r12,#-8]
+	ldrhs	r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+	rev	r0,r0
+	rev	r1,r1
+	rev	r2,r2
+	rev	r3,r3
+# endif
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r0,r0,r8
+	eorhs	r1,r1,r9
+	 add	r8,sp,#4*(12)
+	str	r0,[r14],#16		@ store output
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r2,r2,r10
+	eorhs	r3,r3,r11
+	str	r1,[r14,#-12]
+	 ldmia	r8,{r8-r11}	@ load key material
+	str	r2,[r14,#-8]
+	str	r3,[r14,#-4]
+
+	add	r4,r8,r4,ror#24 @ accumulate key material
+	add	r5,r9,r5,ror#24
+# ifdef	__thumb2__
+	itt	hi
+# endif
+	 addhi	r8,r8,#1		@ next counter value
+	 strhi	r8,[sp,#4*(12)]	@ save next counter value
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r8,[r12],#16		@ load input
+	ldrhs	r9,[r12,#-12]
+	add	r6,r10,r6,ror#24
+	add	r7,r11,r7,ror#24
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhs	r10,[r12,#-8]
+	ldrhs	r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+	rev	r4,r4
+	rev	r5,r5
+	rev	r6,r6
+	rev	r7,r7
+# endif
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r4,r4,r8
+	eorhs	r5,r5,r9
+# ifdef	__thumb2__
+	 it	ne
+# endif
+	 ldrne	r8,[sp,#4*(32+2)]	@ re-load len
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	eorhs	r6,r6,r10
+	eorhs	r7,r7,r11
+	str	r4,[r14],#16		@ store output
+	str	r5,[r14,#-12]
+# ifdef	__thumb2__
+	it	hs
+# endif
+	 subhs	r11,r8,#64		@ len-=64
+	str	r6,[r14,#-8]
+	str	r7,[r14,#-4]
+	bhi	.Loop_outer
+
+	beq	.Ldone
+# if __ARM_ARCH__<7
+	b	.Ltail
+
+.align	4
+.Lunaligned:				@ unaligned endian-neutral path
+	cmp	r11,#64		@ restore flags
+# endif
+#endif
+#if __ARM_ARCH__<7
+	ldr	r11,[sp,#4*(3)]
+	add	r0,r8,r0	@ accumulate key material
+	add	r1,r9,r1
+	add	r2,r10,r2
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r8,r8,r8		@ zero or ...
+	ldrhsb	r8,[r12],#16			@ ... load input
+	eorlo	r9,r9,r9
+	ldrhsb	r9,[r12,#-12]
+
+	add	r3,r11,r3
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r10,r10,r10
+	ldrhsb	r10,[r12,#-8]
+	eorlo	r11,r11,r11
+	ldrhsb	r11,[r12,#-4]
+
+	eor	r0,r8,r0		@ xor with input (or zero)
+	eor	r1,r9,r1
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-15]		@ load more input
+	ldrhsb	r9,[r12,#-11]
+	eor	r2,r10,r2
+	 strb	r0,[r14],#16		@ store output
+	eor	r3,r11,r3
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-7]
+	ldrhsb	r11,[r12,#-3]
+	 strb	r1,[r14,#-12]
+	eor	r0,r8,r0,lsr#8
+	 strb	r2,[r14,#-8]
+	eor	r1,r9,r1,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-14]		@ load more input
+	ldrhsb	r9,[r12,#-10]
+	 strb	r3,[r14,#-4]
+	eor	r2,r10,r2,lsr#8
+	 strb	r0,[r14,#-15]
+	eor	r3,r11,r3,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-6]
+	ldrhsb	r11,[r12,#-2]
+	 strb	r1,[r14,#-11]
+	eor	r0,r8,r0,lsr#8
+	 strb	r2,[r14,#-7]
+	eor	r1,r9,r1,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-13]		@ load more input
+	ldrhsb	r9,[r12,#-9]
+	 strb	r3,[r14,#-3]
+	eor	r2,r10,r2,lsr#8
+	 strb	r0,[r14,#-14]
+	eor	r3,r11,r3,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-5]
+	ldrhsb	r11,[r12,#-1]
+	 strb	r1,[r14,#-10]
+	 strb	r2,[r14,#-6]
+	eor	r0,r8,r0,lsr#8
+	 strb	r3,[r14,#-2]
+	eor	r1,r9,r1,lsr#8
+	 strb	r0,[r14,#-13]
+	eor	r2,r10,r2,lsr#8
+	 strb	r1,[r14,#-9]
+	eor	r3,r11,r3,lsr#8
+	 strb	r2,[r14,#-5]
+	 strb	r3,[r14,#-1]
+	add	r8,sp,#4*(4+0)
+	ldmia	r8,{r8-r11}		@ load key material
+	add	r0,sp,#4*(16+8)
+	add	r4,r8,r4,ror#13	@ accumulate key material
+	add	r5,r9,r5,ror#13
+	add	r6,r10,r6,ror#13
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r8,r8,r8		@ zero or ...
+	ldrhsb	r8,[r12],#16			@ ... load input
+	eorlo	r9,r9,r9
+	ldrhsb	r9,[r12,#-12]
+
+	add	r7,r11,r7,ror#13
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r10,r10,r10
+	ldrhsb	r10,[r12,#-8]
+	eorlo	r11,r11,r11
+	ldrhsb	r11,[r12,#-4]
+
+	eor	r4,r8,r4		@ xor with input (or zero)
+	eor	r5,r9,r5
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-15]		@ load more input
+	ldrhsb	r9,[r12,#-11]
+	eor	r6,r10,r6
+	 strb	r4,[r14],#16		@ store output
+	eor	r7,r11,r7
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-7]
+	ldrhsb	r11,[r12,#-3]
+	 strb	r5,[r14,#-12]
+	eor	r4,r8,r4,lsr#8
+	 strb	r6,[r14,#-8]
+	eor	r5,r9,r5,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-14]		@ load more input
+	ldrhsb	r9,[r12,#-10]
+	 strb	r7,[r14,#-4]
+	eor	r6,r10,r6,lsr#8
+	 strb	r4,[r14,#-15]
+	eor	r7,r11,r7,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-6]
+	ldrhsb	r11,[r12,#-2]
+	 strb	r5,[r14,#-11]
+	eor	r4,r8,r4,lsr#8
+	 strb	r6,[r14,#-7]
+	eor	r5,r9,r5,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-13]		@ load more input
+	ldrhsb	r9,[r12,#-9]
+	 strb	r7,[r14,#-3]
+	eor	r6,r10,r6,lsr#8
+	 strb	r4,[r14,#-14]
+	eor	r7,r11,r7,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-5]
+	ldrhsb	r11,[r12,#-1]
+	 strb	r5,[r14,#-10]
+	 strb	r6,[r14,#-6]
+	eor	r4,r8,r4,lsr#8
+	 strb	r7,[r14,#-2]
+	eor	r5,r9,r5,lsr#8
+	 strb	r4,[r14,#-13]
+	eor	r6,r10,r6,lsr#8
+	 strb	r5,[r14,#-9]
+	eor	r7,r11,r7,lsr#8
+	 strb	r6,[r14,#-5]
+	 strb	r7,[r14,#-1]
+	add	r8,sp,#4*(4+4)
+	ldmia	r8,{r8-r11}		@ load key material
+	ldmia	r0,{r0-r7}		@ load second half
+# ifdef	__thumb2__
+	itt	hi
+# endif
+	strhi	r10,[sp,#4*(16+10)]		@ copy "rx"
+	strhi	r11,[sp,#4*(16+11)]		@ copy "rx"
+	add	r0,r8,r0	@ accumulate key material
+	add	r1,r9,r1
+	add	r2,r10,r2
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r8,r8,r8		@ zero or ...
+	ldrhsb	r8,[r12],#16			@ ... load input
+	eorlo	r9,r9,r9
+	ldrhsb	r9,[r12,#-12]
+
+	add	r3,r11,r3
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r10,r10,r10
+	ldrhsb	r10,[r12,#-8]
+	eorlo	r11,r11,r11
+	ldrhsb	r11,[r12,#-4]
+
+	eor	r0,r8,r0		@ xor with input (or zero)
+	eor	r1,r9,r1
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-15]		@ load more input
+	ldrhsb	r9,[r12,#-11]
+	eor	r2,r10,r2
+	 strb	r0,[r14],#16		@ store output
+	eor	r3,r11,r3
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-7]
+	ldrhsb	r11,[r12,#-3]
+	 strb	r1,[r14,#-12]
+	eor	r0,r8,r0,lsr#8
+	 strb	r2,[r14,#-8]
+	eor	r1,r9,r1,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-14]		@ load more input
+	ldrhsb	r9,[r12,#-10]
+	 strb	r3,[r14,#-4]
+	eor	r2,r10,r2,lsr#8
+	 strb	r0,[r14,#-15]
+	eor	r3,r11,r3,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-6]
+	ldrhsb	r11,[r12,#-2]
+	 strb	r1,[r14,#-11]
+	eor	r0,r8,r0,lsr#8
+	 strb	r2,[r14,#-7]
+	eor	r1,r9,r1,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-13]		@ load more input
+	ldrhsb	r9,[r12,#-9]
+	 strb	r3,[r14,#-3]
+	eor	r2,r10,r2,lsr#8
+	 strb	r0,[r14,#-14]
+	eor	r3,r11,r3,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-5]
+	ldrhsb	r11,[r12,#-1]
+	 strb	r1,[r14,#-10]
+	 strb	r2,[r14,#-6]
+	eor	r0,r8,r0,lsr#8
+	 strb	r3,[r14,#-2]
+	eor	r1,r9,r1,lsr#8
+	 strb	r0,[r14,#-13]
+	eor	r2,r10,r2,lsr#8
+	 strb	r1,[r14,#-9]
+	eor	r3,r11,r3,lsr#8
+	 strb	r2,[r14,#-5]
+	 strb	r3,[r14,#-1]
+	add	r8,sp,#4*(4+8)
+	ldmia	r8,{r8-r11}		@ load key material
+	add	r4,r8,r4,ror#24	@ accumulate key material
+# ifdef	__thumb2__
+	itt	hi
+# endif
+	addhi	r8,r8,#1			@ next counter value
+	strhi	r8,[sp,#4*(12)]		@ save next counter value
+	add	r5,r9,r5,ror#24
+	add	r6,r10,r6,ror#24
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r8,r8,r8		@ zero or ...
+	ldrhsb	r8,[r12],#16			@ ... load input
+	eorlo	r9,r9,r9
+	ldrhsb	r9,[r12,#-12]
+
+	add	r7,r11,r7,ror#24
+# ifdef	__thumb2__
+	itete	lo
+# endif
+	eorlo	r10,r10,r10
+	ldrhsb	r10,[r12,#-8]
+	eorlo	r11,r11,r11
+	ldrhsb	r11,[r12,#-4]
+
+	eor	r4,r8,r4		@ xor with input (or zero)
+	eor	r5,r9,r5
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-15]		@ load more input
+	ldrhsb	r9,[r12,#-11]
+	eor	r6,r10,r6
+	 strb	r4,[r14],#16		@ store output
+	eor	r7,r11,r7
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-7]
+	ldrhsb	r11,[r12,#-3]
+	 strb	r5,[r14,#-12]
+	eor	r4,r8,r4,lsr#8
+	 strb	r6,[r14,#-8]
+	eor	r5,r9,r5,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-14]		@ load more input
+	ldrhsb	r9,[r12,#-10]
+	 strb	r7,[r14,#-4]
+	eor	r6,r10,r6,lsr#8
+	 strb	r4,[r14,#-15]
+	eor	r7,r11,r7,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-6]
+	ldrhsb	r11,[r12,#-2]
+	 strb	r5,[r14,#-11]
+	eor	r4,r8,r4,lsr#8
+	 strb	r6,[r14,#-7]
+	eor	r5,r9,r5,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r8,[r12,#-13]		@ load more input
+	ldrhsb	r9,[r12,#-9]
+	 strb	r7,[r14,#-3]
+	eor	r6,r10,r6,lsr#8
+	 strb	r4,[r14,#-14]
+	eor	r7,r11,r7,lsr#8
+# ifdef	__thumb2__
+	itt	hs
+# endif
+	ldrhsb	r10,[r12,#-5]
+	ldrhsb	r11,[r12,#-1]
+	 strb	r5,[r14,#-10]
+	 strb	r6,[r14,#-6]
+	eor	r4,r8,r4,lsr#8
+	 strb	r7,[r14,#-2]
+	eor	r5,r9,r5,lsr#8
+	 strb	r4,[r14,#-13]
+	eor	r6,r10,r6,lsr#8
+	 strb	r5,[r14,#-9]
+	eor	r7,r11,r7,lsr#8
+	 strb	r6,[r14,#-5]
+	 strb	r7,[r14,#-1]
+# ifdef	__thumb2__
+	it	ne
+# endif
+	ldrne	r8,[sp,#4*(32+2)]		@ re-load len
+# ifdef	__thumb2__
+	it	hs
+# endif
+	subhs	r11,r8,#64			@ len-=64
+	bhi	.Loop_outer
+
+	beq	.Ldone
+#endif
+
+.Ltail:
+	ldr	r12,[sp,#4*(32+1)]	@ load inp
+	add	r9,sp,#4*(0)
+	ldr	r14,[sp,#4*(32+0)]	@ load out
+
+.Loop_tail:
+	ldrb	r10,[r9],#1	@ read buffer on stack
+	ldrb	r11,[r12],#1		@ read input
+	subs	r8,r8,#1
+	eor	r11,r11,r10
+	strb	r11,[r14],#1		@ store output
+	bne	.Loop_tail
+
+.Ldone:
+	add	sp,sp,#4*(32+3)
+.Lno_data:
+	ldmia	sp!,{r4-r11,pc}
+.size	ChaCha20_ctr32,.-ChaCha20_ctr32
+#if __ARM_MAX_ARCH__>=7
+.arch	armv7-a
+.fpu	neon
+
+.type	ChaCha20_neon,%function
+.align	5
+ChaCha20_neon:
+	ldr		r12,[sp,#0]		@ pull pointer to counter and nonce
+	stmdb		sp!,{r0-r2,r4-r11,lr}
+.LChaCha20_neon:
+	adr		r14,.Lsigma
+	vstmdb		sp!,{d8-d15}		@ ABI spec says so
+	stmdb		sp!,{r0-r3}
+
+	vld1.32		{q1-q2},[r3]		@ load key
+	ldmia		r3,{r4-r11}		@ load key
+
+	sub		sp,sp,#4*(16+16)
+	vld1.32		{q3},[r12]		@ load counter and nonce
+	add		r12,sp,#4*8
+	ldmia		r14,{r0-r3}		@ load sigma
+	vld1.32		{q0},[r14]!		@ load sigma
+	vld1.32		{q12},[r14]!		@ one
+	@ vld1.32	{d30},[r14]		@ rot8
+	vst1.32		{q2-q3},[r12]		@ copy 1/2key|counter|nonce
+	vst1.32		{q0-q1},[sp]		@ copy sigma|1/2key
+
+	str		r10,[sp,#4*(16+10)]	@ off-load "rx"
+	str		r11,[sp,#4*(16+11)]	@ off-load "rx"
+	vshl.i32	d26,d24,#1	@ two
+	vstr		d24,[sp,#4*(16+0)]
+	vshl.i32	d28,d24,#2	@ four
+	vstr		d26,[sp,#4*(16+2)]
+	vmov		q4,q0
+	vstr		d28,[sp,#4*(16+4)]
+	vmov		q8,q0
+	@ vstr		d30,[sp,#4*(16+6)]
+	vmov		q5,q1
+	vmov		q9,q1
+	b		.Loop_neon_enter
+
+.align	4
+.Loop_neon_outer:
+	ldmia		sp,{r0-r9}		@ load key material
+	cmp		r11,#64*2		@ if len<=64*2
+	bls		.Lbreak_neon		@ switch to integer-only
+	@ vldr		d30,[sp,#4*(16+6)]	@ rot8
+	vmov		q4,q0
+	str		r11,[sp,#4*(32+2)]	@ save len
+	vmov		q8,q0
+	str		r12,  [sp,#4*(32+1)]	@ save inp
+	vmov		q5,q1
+	str		r14,  [sp,#4*(32+0)]	@ save out
+	vmov		q9,q1
+.Loop_neon_enter:
+	ldr		r11, [sp,#4*(15)]
+	 mov		r4,r4,ror#19	@ twist b[0..3]
+	vadd.i32	q7,q3,q12		@ counter+1
+	ldr		r12,[sp,#4*(12)]	@ modulo-scheduled load
+	 mov		r5,r5,ror#19
+	vmov		q6,q2
+	ldr		r10, [sp,#4*(13)]
+	 mov		r6,r6,ror#19
+	vmov		q10,q2
+	ldr		r14,[sp,#4*(14)]
+	 mov		r7,r7,ror#19
+	vadd.i32	q11,q7,q12		@ counter+2
+	add		r12,r12,#3	@ counter+3
+	mov		r11,r11,ror#8	@ twist d[0..3]
+	mov		r12,r12,ror#8
+	mov		r10,r10,ror#8
+	mov		r14,r14,ror#8
+	str		r11, [sp,#4*(16+15)]
+	mov		r11,#10
+	b		.Loop_neon
+
+.align	4
+.Loop_neon:
+	subs		r11,r11,#1
+	vadd.i32	q0,q0,q1
+	add	r0,r0,r4,ror#13
+	vadd.i32	q4,q4,q5
+	add	r1,r1,r5,ror#13
+	vadd.i32	q8,q8,q9
+	eor	r12,r0,r12,ror#24
+	veor	q3,q3,q0
+	eor	r10,r1,r10,ror#24
+	veor	q7,q7,q4
+	add	r8,r8,r12,ror#16
+	veor	q11,q11,q8
+	add	r9,r9,r10,ror#16
+	vrev32.16	q3,q3
+	eor	r4,r8,r4,ror#13
+	vrev32.16	q7,q7
+	eor	r5,r9,r5,ror#13
+	vrev32.16	q11,q11
+	add	r0,r0,r4,ror#20
+	vadd.i32	q2,q2,q3
+	add	r1,r1,r5,ror#20
+	vadd.i32	q6,q6,q7
+	eor	r12,r0,r12,ror#16
+	vadd.i32	q10,q10,q11
+	eor	r10,r1,r10,ror#16
+	veor	q12,q1,q2
+	add	r8,r8,r12,ror#24
+	veor	q13,q5,q6
+	str	r10,[sp,#4*(16+13)]
+	veor	q14,q9,q10
+	add	r9,r9,r10,ror#24
+	vshr.u32	q1,q12,#20
+	ldr	r10,[sp,#4*(16+15)]
+	vshr.u32	q5,q13,#20
+	str	r8,[sp,#4*(16+8)]
+	vshr.u32	q9,q14,#20
+	eor	r4,r4,r8,ror#12
+	vsli.32	q1,q12,#12
+	str	r9,[sp,#4*(16+9)]
+	vsli.32	q5,q13,#12
+	eor	r5,r5,r9,ror#12
+	vsli.32	q9,q14,#12
+	ldr	r8,[sp,#4*(16+10)]
+	vadd.i32	q0,q0,q1
+	add	r2,r2,r6,ror#13
+	vadd.i32	q4,q4,q5
+	ldr	r9,[sp,#4*(16+11)]
+	vadd.i32	q8,q8,q9
+	add	r3,r3,r7,ror#13
+	veor	q12,q3,q0
+	eor	r14,r2,r14,ror#24
+	veor	q13,q7,q4
+	eor	r10,r3,r10,ror#24
+	veor	q14,q11,q8
+	add	r8,r8,r14,ror#16
+	vshr.u32	q3,q12,#24
+	add	r9,r9,r10,ror#16
+	vshr.u32	q7,q13,#24
+	eor	r6,r8,r6,ror#13
+	vshr.u32	q11,q14,#24
+	eor	r7,r9,r7,ror#13
+	vsli.32	q3,q12,#8
+	add	r2,r2,r6,ror#20
+	vsli.32	q7,q13,#8
+	add	r3,r3,r7,ror#20
+	vsli.32	q11,q14,#8
+	eor	r14,r2,r14,ror#16
+	vadd.i32	q2,q2,q3
+	eor	r10,r3,r10,ror#16
+	vadd.i32	q6,q6,q7
+	add	r8,r8,r14,ror#24
+	vadd.i32	q10,q10,q11
+	add	r9,r9,r10,ror#24
+	veor	q12,q1,q2
+	eor	r6,r6,r8,ror#12
+	veor	q13,q5,q6
+	eor	r7,r7,r9,ror#12
+	veor	q14,q9,q10
+	vshr.u32	q1,q12,#25
+	vshr.u32	q5,q13,#25
+	vshr.u32	q9,q14,#25
+	vsli.32	q1,q12,#7
+	vsli.32	q5,q13,#7
+	vsli.32	q9,q14,#7
+	vext.8	q2,q2,q2,#8
+	vext.8	q6,q6,q6,#8
+	vext.8	q10,q10,q10,#8
+	vext.8	q1,q1,q1,#4
+	vext.8	q5,q5,q5,#4
+	vext.8	q9,q9,q9,#4
+	vext.8	q3,q3,q3,#12
+	vext.8	q7,q7,q7,#12
+	vext.8	q11,q11,q11,#12
+	vadd.i32	q0,q0,q1
+	add	r0,r0,r5,ror#13
+	vadd.i32	q4,q4,q5
+	add	r1,r1,r6,ror#13
+	vadd.i32	q8,q8,q9
+	eor	r10,r0,r10,ror#24
+	veor	q3,q3,q0
+	eor	r12,r1,r12,ror#24
+	veor	q7,q7,q4
+	add	r8,r8,r10,ror#16
+	veor	q11,q11,q8
+	add	r9,r9,r12,ror#16
+	vrev32.16	q3,q3
+	eor	r5,r8,r5,ror#13
+	vrev32.16	q7,q7
+	eor	r6,r9,r6,ror#13
+	vrev32.16	q11,q11
+	add	r0,r0,r5,ror#20
+	vadd.i32	q2,q2,q3
+	add	r1,r1,r6,ror#20
+	vadd.i32	q6,q6,q7
+	eor	r10,r0,r10,ror#16
+	vadd.i32	q10,q10,q11
+	eor	r12,r1,r12,ror#16
+	veor	q12,q1,q2
+	str	r10,[sp,#4*(16+15)]
+	veor	q13,q5,q6
+	add	r8,r8,r10,ror#24
+	veor	q14,q9,q10
+	ldr	r10,[sp,#4*(16+13)]
+	vshr.u32	q1,q12,#20
+	add	r9,r9,r12,ror#24
+	vshr.u32	q5,q13,#20
+	str	r8,[sp,#4*(16+10)]
+	vshr.u32	q9,q14,#20
+	eor	r5,r5,r8,ror#12
+	vsli.32	q1,q12,#12
+	str	r9,[sp,#4*(16+11)]
+	vsli.32	q5,q13,#12
+	eor	r6,r6,r9,ror#12
+	vsli.32	q9,q14,#12
+	ldr	r8,[sp,#4*(16+8)]
+	vadd.i32	q0,q0,q1
+	add	r2,r2,r7,ror#13
+	vadd.i32	q4,q4,q5
+	ldr	r9,[sp,#4*(16+9)]
+	vadd.i32	q8,q8,q9
+	add	r3,r3,r4,ror#13
+	veor	q12,q3,q0
+	eor	r10,r2,r10,ror#24
+	veor	q13,q7,q4
+	eor	r14,r3,r14,ror#24
+	veor	q14,q11,q8
+	add	r8,r8,r10,ror#16
+	vshr.u32	q3,q12,#24
+	add	r9,r9,r14,ror#16
+	vshr.u32	q7,q13,#24
+	eor	r7,r8,r7,ror#13
+	vshr.u32	q11,q14,#24
+	eor	r4,r9,r4,ror#13
+	vsli.32	q3,q12,#8
+	add	r2,r2,r7,ror#20
+	vsli.32	q7,q13,#8
+	add	r3,r3,r4,ror#20
+	vsli.32	q11,q14,#8
+	eor	r10,r2,r10,ror#16
+	vadd.i32	q2,q2,q3
+	eor	r14,r3,r14,ror#16
+	vadd.i32	q6,q6,q7
+	add	r8,r8,r10,ror#24
+	vadd.i32	q10,q10,q11
+	add	r9,r9,r14,ror#24
+	veor	q12,q1,q2
+	eor	r7,r7,r8,ror#12
+	veor	q13,q5,q6
+	eor	r4,r4,r9,ror#12
+	veor	q14,q9,q10
+	vshr.u32	q1,q12,#25
+	vshr.u32	q5,q13,#25
+	vshr.u32	q9,q14,#25
+	vsli.32	q1,q12,#7
+	vsli.32	q5,q13,#7
+	vsli.32	q9,q14,#7
+	vext.8	q2,q2,q2,#8
+	vext.8	q6,q6,q6,#8
+	vext.8	q10,q10,q10,#8
+	vext.8	q1,q1,q1,#12
+	vext.8	q5,q5,q5,#12
+	vext.8	q9,q9,q9,#12
+	vext.8	q3,q3,q3,#4
+	vext.8	q7,q7,q7,#4
+	vext.8	q11,q11,q11,#4
+	bne		.Loop_neon
+
+	add		r11,sp,#32
+	vld1.32		{q12-q13},[sp]		@ load key material
+	vld1.32		{q14-q15},[r11]
+
+	ldr		r11,[sp,#4*(32+2)]	@ load len
+
+	str		r8, [sp,#4*(16+8)]	@ modulo-scheduled store
+	str		r9, [sp,#4*(16+9)]
+	str		r12,[sp,#4*(16+12)]
+	str		r10, [sp,#4*(16+13)]
+	str		r14,[sp,#4*(16+14)]
+
+	@ at this point we have first half of 512-bit result in
+	@ rx and second half at sp+4*(16+8)
+
+	ldr		r12,[sp,#4*(32+1)]	@ load inp
+	ldr		r14,[sp,#4*(32+0)]	@ load out
+
+	vadd.i32	q0,q0,q12		@ accumulate key material
+	vadd.i32	q4,q4,q12
+	vadd.i32	q8,q8,q12
+	vldr		d24,[sp,#4*(16+0)]	@ one
+
+	vadd.i32	q1,q1,q13
+	vadd.i32	q5,q5,q13
+	vadd.i32	q9,q9,q13
+	vldr		d26,[sp,#4*(16+2)]	@ two
+
+	vadd.i32	q2,q2,q14
+	vadd.i32	q6,q6,q14
+	vadd.i32	q10,q10,q14
+	vadd.i32	d14,d14,d24	@ counter+1
+	vadd.i32	d22,d22,d26	@ counter+2
+
+	vadd.i32	q3,q3,q15
+	vadd.i32	q7,q7,q15
+	vadd.i32	q11,q11,q15
+
+	cmp		r11,#64*4
+	blo		.Ltail_neon
+
+	vld1.8		{q12-q13},[r12]!	@ load input
+	 mov		r11,sp
+	vld1.8		{q14-q15},[r12]!
+	veor		q0,q0,q12		@ xor with input
+	veor		q1,q1,q13
+	vld1.8		{q12-q13},[r12]!
+	veor		q2,q2,q14
+	veor		q3,q3,q15
+	vld1.8		{q14-q15},[r12]!
+
+	veor		q4,q4,q12
+	 vst1.8		{q0-q1},[r14]!	@ store output
+	veor		q5,q5,q13
+	vld1.8		{q12-q13},[r12]!
+	veor		q6,q6,q14
+	 vst1.8		{q2-q3},[r14]!
+	veor		q7,q7,q15
+	vld1.8		{q14-q15},[r12]!
+
+	veor		q8,q8,q12
+	 vld1.32	{q0-q1},[r11]!	@ load for next iteration
+	 veor		d25,d25,d25
+	 vldr		d24,[sp,#4*(16+4)]	@ four
+	veor		q9,q9,q13
+	 vld1.32	{q2-q3},[r11]
+	veor		q10,q10,q14
+	 vst1.8		{q4-q5},[r14]!
+	veor		q11,q11,q15
+	 vst1.8		{q6-q7},[r14]!
+
+	vadd.i32	d6,d6,d24	@ next counter value
+	vldr		d24,[sp,#4*(16+0)]	@ one
+
+	ldmia		sp,{r8-r11}	@ load key material
+	add		r0,r0,r8	@ accumulate key material
+	ldr		r8,[r12],#16		@ load input
+	 vst1.8		{q8-q9},[r14]!
+	add		r1,r1,r9
+	ldr		r9,[r12,#-12]
+	 vst1.8		{q10-q11},[r14]!
+	add		r2,r2,r10
+	ldr		r10,[r12,#-8]
+	add		r3,r3,r11
+	ldr		r11,[r12,#-4]
+# ifdef	__ARMEB__
+	rev		r0,r0
+	rev		r1,r1
+	rev		r2,r2
+	rev		r3,r3
+# endif
+	eor		r0,r0,r8	@ xor with input
+	 add		r8,sp,#4*(4)
+	eor		r1,r1,r9
+	str		r0,[r14],#16		@ store output
+	eor		r2,r2,r10
+	str		r1,[r14,#-12]
+	eor		r3,r3,r11
+	 ldmia		r8,{r8-r11}	@ load key material
+	str		r2,[r14,#-8]
+	str		r3,[r14,#-4]
+
+	add		r4,r8,r4,ror#13 @ accumulate key material
+	ldr		r8,[r12],#16		@ load input
+	add		r5,r9,r5,ror#13
+	ldr		r9,[r12,#-12]
+	add		r6,r10,r6,ror#13
+	ldr		r10,[r12,#-8]
+	add		r7,r11,r7,ror#13
+	ldr		r11,[r12,#-4]
+# ifdef	__ARMEB__
+	rev		r4,r4
+	rev		r5,r5
+	rev		r6,r6
+	rev		r7,r7
+# endif
+	eor		r4,r4,r8
+	 add		r8,sp,#4*(8)
+	eor		r5,r5,r9
+	str		r4,[r14],#16		@ store output
+	eor		r6,r6,r10
+	str		r5,[r14,#-12]
+	eor		r7,r7,r11
+	 ldmia		r8,{r8-r11}	@ load key material
+	str		r6,[r14,#-8]
+	 add		r0,sp,#4*(16+8)
+	str		r7,[r14,#-4]
+
+	ldmia		r0,{r0-r7}	@ load second half
+
+	add		r0,r0,r8	@ accumulate key material
+	ldr		r8,[r12],#16		@ load input
+	add		r1,r1,r9
+	ldr		r9,[r12,#-12]
+# ifdef	__thumb2__
+	it	hi
+# endif
+	 strhi		r10,[sp,#4*(16+10)]	@ copy "rx" while at it
+	add		r2,r2,r10
+	ldr		r10,[r12,#-8]
+# ifdef	__thumb2__
+	it	hi
+# endif
+	 strhi		r11,[sp,#4*(16+11)]	@ copy "rx" while at it
+	add		r3,r3,r11
+	ldr		r11,[r12,#-4]
+# ifdef	__ARMEB__
+	rev		r0,r0
+	rev		r1,r1
+	rev		r2,r2
+	rev		r3,r3
+# endif
+	eor		r0,r0,r8
+	 add		r8,sp,#4*(12)
+	eor		r1,r1,r9
+	str		r0,[r14],#16		@ store output
+	eor		r2,r2,r10
+	str		r1,[r14,#-12]
+	eor		r3,r3,r11
+	 ldmia		r8,{r8-r11}	@ load key material
+	str		r2,[r14,#-8]
+	str		r3,[r14,#-4]
+
+	add		r4,r8,r4,ror#24 @ accumulate key material
+	 add		r8,r8,#4		@ next counter value
+	add		r5,r9,r5,ror#24
+	 str		r8,[sp,#4*(12)]	@ save next counter value
+	ldr		r8,[r12],#16		@ load input
+	add		r6,r10,r6,ror#24
+	 add		r4,r4,#3		@ counter+3
+	ldr		r9,[r12,#-12]
+	add		r7,r11,r7,ror#24
+	ldr		r10,[r12,#-8]
+	ldr		r11,[r12,#-4]
+# ifdef	__ARMEB__
+	rev		r4,r4
+	rev		r5,r5
+	rev		r6,r6
+	rev		r7,r7
+# endif
+	eor		r4,r4,r8
+# ifdef	__thumb2__
+	it	hi
+# endif
+	 ldrhi		r8,[sp,#4*(32+2)]	@ re-load len
+	eor		r5,r5,r9
+	eor		r6,r6,r10
+	str		r4,[r14],#16		@ store output
+	eor		r7,r7,r11
+	str		r5,[r14,#-12]
+	 sub		r11,r8,#64*4	@ len-=64*4
+	str		r6,[r14,#-8]
+	str		r7,[r14,#-4]
+	bhi		.Loop_neon_outer
+
+	b		.Ldone_neon
+
+.align	4
+.Lbreak_neon:
+	@ harmonize NEON and integer-only stack frames: load data
+	@ from NEON frame, but save to integer-only one; distance
+	@ between the two is 4*(32+4+16-32)=4*(20).
+
+	str		r11, [sp,#4*(20+32+2)]	@ save len
+	 add		r11,sp,#4*(32+4)
+	str		r12,   [sp,#4*(20+32+1)]	@ save inp
+	str		r14,   [sp,#4*(20+32+0)]	@ save out
+
+	ldr		r12,[sp,#4*(16+10)]
+	ldr		r14,[sp,#4*(16+11)]
+	 vldmia		r11,{d8-d15}			@ fulfill ABI requirement
+	str		r12,[sp,#4*(20+16+10)]	@ copy "rx"
+	str		r14,[sp,#4*(20+16+11)]	@ copy "rx"
+
+	ldr		r11, [sp,#4*(15)]
+	 mov		r4,r4,ror#19		@ twist b[0..3]
+	ldr		r12,[sp,#4*(12)]		@ modulo-scheduled load
+	 mov		r5,r5,ror#19
+	ldr		r10, [sp,#4*(13)]
+	 mov		r6,r6,ror#19
+	ldr		r14,[sp,#4*(14)]
+	 mov		r7,r7,ror#19
+	mov		r11,r11,ror#8		@ twist d[0..3]
+	mov		r12,r12,ror#8
+	mov		r10,r10,ror#8
+	mov		r14,r14,ror#8
+	str		r11, [sp,#4*(20+16+15)]
+	add		r11,sp,#4*(20)
+	vst1.32		{q0-q1},[r11]!		@ copy key
+	add		sp,sp,#4*(20)			@ switch frame
+	vst1.32		{q2-q3},[r11]
+	mov		r11,#10
+	b		.Loop				@ go integer-only
+
+.align	4
+.Ltail_neon:
+	cmp		r11,#64*3
+	bhs		.L192_or_more_neon
+	cmp		r11,#64*2
+	bhs		.L128_or_more_neon
+	cmp		r11,#64*1
+	bhs		.L64_or_more_neon
+
+	add		r8,sp,#4*(8)
+	vst1.8		{q0-q1},[sp]
+	add		r10,sp,#4*(0)
+	vst1.8		{q2-q3},[r8]
+	b		.Loop_tail_neon
+
+.align	4
+.L64_or_more_neon:
+	vld1.8		{q12-q13},[r12]!
+	vld1.8		{q14-q15},[r12]!
+	veor		q0,q0,q12
+	veor		q1,q1,q13
+	veor		q2,q2,q14
+	veor		q3,q3,q15
+	vst1.8		{q0-q1},[r14]!
+	vst1.8		{q2-q3},[r14]!
+
+	beq		.Ldone_neon
+
+	add		r8,sp,#4*(8)
+	vst1.8		{q4-q5},[sp]
+	add		r10,sp,#4*(0)
+	vst1.8		{q6-q7},[r8]
+	sub		r11,r11,#64*1	@ len-=64*1
+	b		.Loop_tail_neon
+
+.align	4
+.L128_or_more_neon:
+	vld1.8		{q12-q13},[r12]!
+	vld1.8		{q14-q15},[r12]!
+	veor		q0,q0,q12
+	veor		q1,q1,q13
+	vld1.8		{q12-q13},[r12]!
+	veor		q2,q2,q14
+	veor		q3,q3,q15
+	vld1.8		{q14-q15},[r12]!
+
+	veor		q4,q4,q12
+	veor		q5,q5,q13
+	 vst1.8		{q0-q1},[r14]!
+	veor		q6,q6,q14
+	 vst1.8		{q2-q3},[r14]!
+	veor		q7,q7,q15
+	vst1.8		{q4-q5},[r14]!
+	vst1.8		{q6-q7},[r14]!
+
+	beq		.Ldone_neon
+
+	add		r8,sp,#4*(8)
+	vst1.8		{q8-q9},[sp]
+	add		r10,sp,#4*(0)
+	vst1.8		{q10-q11},[r8]
+	sub		r11,r11,#64*2	@ len-=64*2
+	b		.Loop_tail_neon
+
+.align	4
+.L192_or_more_neon:
+	vld1.8		{q12-q13},[r12]!
+	vld1.8		{q14-q15},[r12]!
+	veor		q0,q0,q12
+	veor		q1,q1,q13
+	vld1.8		{q12-q13},[r12]!
+	veor		q2,q2,q14
+	veor		q3,q3,q15
+	vld1.8		{q14-q15},[r12]!
+
+	veor		q4,q4,q12
+	veor		q5,q5,q13
+	vld1.8		{q12-q13},[r12]!
+	veor		q6,q6,q14
+	 vst1.8		{q0-q1},[r14]!
+	veor		q7,q7,q15
+	vld1.8		{q14-q15},[r12]!
+
+	veor		q8,q8,q12
+	 vst1.8		{q2-q3},[r14]!
+	veor		q9,q9,q13
+	 vst1.8		{q4-q5},[r14]!
+	veor		q10,q10,q14
+	 vst1.8		{q6-q7},[r14]!
+	veor		q11,q11,q15
+	vst1.8		{q8-q9},[r14]!
+	vst1.8		{q10-q11},[r14]!
+
+	beq		.Ldone_neon
+
+	ldmia		sp,{r8-r11}	@ load key material
+	add		r0,r0,r8	@ accumulate key material
+	 add		r8,sp,#4*(4)
+	add		r1,r1,r9
+	add		r2,r2,r10
+	add		r3,r3,r11
+	 ldmia		r8,{r8-r11}	@ load key material
+
+	add		r4,r8,r4,ror#13 @ accumulate key material
+	 add		r8,sp,#4*(8)
+	add		r5,r9,r5,ror#13
+	add		r6,r10,r6,ror#13
+	add		r7,r11,r7,ror#13
+	 ldmia		r8,{r8-r11}	@ load key material
+# ifdef	__ARMEB__
+	rev		r0,r0
+	rev		r1,r1
+	rev		r2,r2
+	rev		r3,r3
+	rev		r4,r4
+	rev		r5,r5
+	rev		r6,r6
+	rev		r7,r7
+# endif
+	stmia		sp,{r0-r7}
+	 add		r0,sp,#4*(16+8)
+
+	ldmia		r0,{r0-r7}	@ load second half
+
+	add		r0,r0,r8	@ accumulate key material
+	 add		r8,sp,#4*(12)
+	add		r1,r1,r9
+	add		r2,r2,r10
+	add		r3,r3,r11
+	 ldmia		r8,{r8-r11}	@ load key material
+
+	add		r4,r8,r4,ror#24 @ accumulate key material
+	 add		r8,sp,#4*(8)
+	add		r5,r9,r5,ror#24
+	 add		r4,r4,#3		@ counter+3
+	add		r6,r10,r6,ror#24
+	add		r7,r11,r7,ror#24
+	 ldr		r11,[sp,#4*(32+2)]	@ re-load len
+# ifdef	__ARMEB__
+	rev		r0,r0
+	rev		r1,r1
+	rev		r2,r2
+	rev		r3,r3
+	rev		r4,r4
+	rev		r5,r5
+	rev		r6,r6
+	rev		r7,r7
+# endif
+	stmia		r8,{r0-r7}
+	 add		r10,sp,#4*(0)
+	 sub		r11,r11,#64*3	@ len-=64*3
+
+.Loop_tail_neon:
+	ldrb		r8,[r10],#1	@ read buffer on stack
+	ldrb		r9,[r12],#1		@ read input
+	subs		r11,r11,#1
+	eor		r8,r8,r9
+	strb		r8,[r14],#1		@ store output
+	bne		.Loop_tail_neon
+
+.Ldone_neon:
+	add		sp,sp,#4*(32+4)
+	vldmia		sp,{d8-d15}
+	add		sp,sp,#4*(16+3)
+	ldmia		sp!,{r4-r11,pc}
+.size	ChaCha20_neon,.-ChaCha20_neon
+.comm	OPENSSL_armcap_P,4,4
+#endif
lib/zinc/chacha20/chacha20-arm64-cryptogams.S new file +1973 -0

1,560 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

#42 lore raw

Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations

Quoted 10 lines · Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2

Quoted 8 lines

Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: [email protected]

As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.

Quoted 3,435 lines
---
 lib/zinc/chacha20/chacha20-arm-cryptogams.S   | 1440 ++++++++++++
 lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++
 2 files changed, 3413 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S
 create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.S
lib/zinc/chacha20/chacha20-arm-cryptogams.S new file +1440 -0
new file mode 100644
index 000000000000..05a3a9e6e93f
@@ -0,0 +1,1440 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ */
+
+#include "arm_arch.h"
+
+.text
+#if defined(__thumb2__) || defined(__clang__)
+.syntax        unified
+#endif
+#if defined(__thumb2__)
+.thumb
+#else
+.code  32
+#endif
+
+#if defined(__thumb2__) || defined(__clang__)
+#define ldrhsb ldrbhs
+#endif
+
+.align 5
+.Lsigma:
+.long  0x61707865,0x3320646e,0x79622d32,0x6b206574     @ endian-neutral
+.Lone:
+.long  1,0,0,0
+.Lrot8:
+.long  0x02010003,0x06050407
+#if __ARM_MAX_ARCH__>=7
+.LOPENSSL_armcap:
+.word   OPENSSL_armcap_P-.LChaCha20_ctr32
+#else
+.word  -1
+#endif
+
+.globl ChaCha20_ctr32
+.type  ChaCha20_ctr32,%function
+.align 5
+ChaCha20_ctr32:
+.LChaCha20_ctr32:
+       ldr     r12,[sp,#0]             @ pull pointer to counter and nonce
+       stmdb   sp!,{r0-r2,r4-r11,lr}
+#if __ARM_ARCH__<7 && !defined(__thumb2__)
+       sub     r14,pc,#16              @ ChaCha20_ctr32
+#else
+       adr     r14,.LChaCha20_ctr32
+#endif
+       cmp     r2,#0                   @ len==0?
+#ifdef __thumb2__
+       itt     eq
+#endif
+       addeq   sp,sp,#4*3
+       beq     .Lno_data
+#if __ARM_MAX_ARCH__>=7
+       cmp     r2,#192                 @ test len
+       bls     .Lshort
+       ldr     r4,[r14,#-24]
+       ldr     r4,[r14,r4]
+# ifdef        __APPLE__
+       ldr     r4,[r4]
+# endif
+       tst     r4,#ARMV7_NEON
+       bne     .LChaCha20_neon
+.Lshort:
+#endif
+       ldmia   r12,{r4-r7}             @ load counter and nonce
+       sub     sp,sp,#4*(16)           @ off-load area
+       sub     r14,r14,#64             @ .Lsigma
+       stmdb   sp!,{r4-r7}             @ copy counter and nonce
+       ldmia   r3,{r4-r11}             @ load key
+       ldmia   r14,{r0-r3}             @ load sigma
+       stmdb   sp!,{r4-r11}            @ copy key
+       stmdb   sp!,{r0-r3}             @ copy sigma
+       str     r10,[sp,#4*(16+10)]     @ off-load "rx"
+       str     r11,[sp,#4*(16+11)]     @ off-load "rx"
+       b       .Loop_outer_enter
+
+.align 4
+.Loop_outer:
+       ldmia   sp,{r0-r9}              @ load key material
+       str     r11,[sp,#4*(32+2)]      @ save len
+       str     r12,  [sp,#4*(32+1)]    @ save inp
+       str     r14,  [sp,#4*(32+0)]    @ save out
+.Loop_outer_enter:
+       ldr     r11, [sp,#4*(15)]
+        mov    r4,r4,ror#19    @ twist b[0..3]
+       ldr     r12,[sp,#4*(12)]        @ modulo-scheduled load
+        mov    r5,r5,ror#19
+       ldr     r10, [sp,#4*(13)]
+        mov    r6,r6,ror#19
+       ldr     r14,[sp,#4*(14)]
+        mov    r7,r7,ror#19
+       mov     r11,r11,ror#8   @ twist d[0..3]
+       mov     r12,r12,ror#8
+       mov     r10,r10,ror#8
+       mov     r14,r14,ror#8
+       str     r11, [sp,#4*(16+15)]
+       mov     r11,#10
+       b       .Loop
+
+.align 4
+.Loop:
+       subs    r11,r11,#1
+       add     r0,r0,r4,ror#13
+       add     r1,r1,r5,ror#13
+       eor     r12,r0,r12,ror#24
+       eor     r10,r1,r10,ror#24
+       add     r8,r8,r12,ror#16
+       add     r9,r9,r10,ror#16
+       eor     r4,r8,r4,ror#13
+       eor     r5,r9,r5,ror#13
+       add     r0,r0,r4,ror#20
+       add     r1,r1,r5,ror#20
+       eor     r12,r0,r12,ror#16
+       eor     r10,r1,r10,ror#16
+       add     r8,r8,r12,ror#24
+       str     r10,[sp,#4*(16+13)]
+       add     r9,r9,r10,ror#24
+       ldr     r10,[sp,#4*(16+15)]
+       str     r8,[sp,#4*(16+8)]
+       eor     r4,r4,r8,ror#12
+       str     r9,[sp,#4*(16+9)]
+       eor     r5,r5,r9,ror#12
+       ldr     r8,[sp,#4*(16+10)]
+       add     r2,r2,r6,ror#13
+       ldr     r9,[sp,#4*(16+11)]
+       add     r3,r3,r7,ror#13
+       eor     r14,r2,r14,ror#24
+       eor     r10,r3,r10,ror#24
+       add     r8,r8,r14,ror#16
+       add     r9,r9,r10,ror#16
+       eor     r6,r8,r6,ror#13
+       eor     r7,r9,r7,ror#13
+       add     r2,r2,r6,ror#20
+       add     r3,r3,r7,ror#20
+       eor     r14,r2,r14,ror#16
+       eor     r10,r3,r10,ror#16
+       add     r8,r8,r14,ror#24
+       add     r9,r9,r10,ror#24
+       eor     r6,r6,r8,ror#12
+       eor     r7,r7,r9,ror#12
+       add     r0,r0,r5,ror#13
+       add     r1,r1,r6,ror#13
+       eor     r10,r0,r10,ror#24
+       eor     r12,r1,r12,ror#24
+       add     r8,r8,r10,ror#16
+       add     r9,r9,r12,ror#16
+       eor     r5,r8,r5,ror#13
+       eor     r6,r9,r6,ror#13
+       add     r0,r0,r5,ror#20
+       add     r1,r1,r6,ror#20
+       eor     r10,r0,r10,ror#16
+       eor     r12,r1,r12,ror#16
+       str     r10,[sp,#4*(16+15)]
+       add     r8,r8,r10,ror#24
+       ldr     r10,[sp,#4*(16+13)]
+       add     r9,r9,r12,ror#24
+       str     r8,[sp,#4*(16+10)]
+       eor     r5,r5,r8,ror#12
+       str     r9,[sp,#4*(16+11)]
+       eor     r6,r6,r9,ror#12
+       ldr     r8,[sp,#4*(16+8)]
+       add     r2,r2,r7,ror#13
+       ldr     r9,[sp,#4*(16+9)]
+       add     r3,r3,r4,ror#13
+       eor     r10,r2,r10,ror#24
+       eor     r14,r3,r14,ror#24
+       add     r8,r8,r10,ror#16
+       add     r9,r9,r14,ror#16
+       eor     r7,r8,r7,ror#13
+       eor     r4,r9,r4,ror#13
+       add     r2,r2,r7,ror#20
+       add     r3,r3,r4,ror#20
+       eor     r10,r2,r10,ror#16
+       eor     r14,r3,r14,ror#16
+       add     r8,r8,r10,ror#24
+       add     r9,r9,r14,ror#24
+       eor     r7,r7,r8,ror#12
+       eor     r4,r4,r9,ror#12
+       bne     .Loop
+
+       ldr     r11,[sp,#4*(32+2)]      @ load len
+
+       str     r8, [sp,#4*(16+8)]      @ modulo-scheduled store
+       str     r9, [sp,#4*(16+9)]
+       str     r12,[sp,#4*(16+12)]
+       str     r10, [sp,#4*(16+13)]
+       str     r14,[sp,#4*(16+14)]
+
+       @ at this point we have first half of 512-bit result in
+       @ rx and second half at sp+4*(16+8)
+
+       cmp     r11,#64         @ done yet?
+#ifdef __thumb2__
+       itete   lo
+#endif
+       addlo   r12,sp,#4*(0)           @ shortcut or ...
+       ldrhs   r12,[sp,#4*(32+1)]      @ ... load inp
+       addlo   r14,sp,#4*(0)           @ shortcut or ...
+       ldrhs   r14,[sp,#4*(32+0)]      @ ... load out
+
+       ldr     r8,[sp,#4*(0)]  @ load key material
+       ldr     r9,[sp,#4*(1)]
+
+#if __ARM_ARCH__>=6 || !defined(__ARMEB__)
+# if __ARM_ARCH__<7
+       orr     r10,r12,r14
+       tst     r10,#3          @ are input and output aligned?
+       ldr     r10,[sp,#4*(2)]
+       bne     .Lunaligned
+       cmp     r11,#64         @ restore flags
+# else
+       ldr     r10,[sp,#4*(2)]
+# endif
+       ldr     r11,[sp,#4*(3)]
+
+       add     r0,r0,r8        @ accumulate key material
+       add     r1,r1,r9
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r8,[r12],#16            @ load input
+       ldrhs   r9,[r12,#-12]
+
+       add     r2,r2,r10
+       add     r3,r3,r11
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r10,[r12,#-8]
+       ldrhs   r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+       rev     r0,r0
+       rev     r1,r1
+       rev     r2,r2
+       rev     r3,r3
+# endif
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r0,r0,r8        @ xor with input
+       eorhs   r1,r1,r9
+        add    r8,sp,#4*(4)
+       str     r0,[r14],#16            @ store output
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r2,r2,r10
+       eorhs   r3,r3,r11
+        ldmia  r8,{r8-r11}     @ load key material
+       str     r1,[r14,#-12]
+       str     r2,[r14,#-8]
+       str     r3,[r14,#-4]
+
+       add     r4,r8,r4,ror#13 @ accumulate key material
+       add     r5,r9,r5,ror#13
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r8,[r12],#16            @ load input
+       ldrhs   r9,[r12,#-12]
+       add     r6,r10,r6,ror#13
+       add     r7,r11,r7,ror#13
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r10,[r12,#-8]
+       ldrhs   r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+       rev     r4,r4
+       rev     r5,r5
+       rev     r6,r6
+       rev     r7,r7
+# endif
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r4,r4,r8
+       eorhs   r5,r5,r9
+        add    r8,sp,#4*(8)
+       str     r4,[r14],#16            @ store output
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r6,r6,r10
+       eorhs   r7,r7,r11
+       str     r5,[r14,#-12]
+        ldmia  r8,{r8-r11}     @ load key material
+       str     r6,[r14,#-8]
+        add    r0,sp,#4*(16+8)
+       str     r7,[r14,#-4]
+
+       ldmia   r0,{r0-r7}      @ load second half
+
+       add     r0,r0,r8        @ accumulate key material
+       add     r1,r1,r9
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r8,[r12],#16            @ load input
+       ldrhs   r9,[r12,#-12]
+# ifdef        __thumb2__
+       itt     hi
+# endif
+        strhi  r10,[sp,#4*(16+10)]     @ copy "rx" while at it
+        strhi  r11,[sp,#4*(16+11)]     @ copy "rx" while at it
+       add     r2,r2,r10
+       add     r3,r3,r11
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r10,[r12,#-8]
+       ldrhs   r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+       rev     r0,r0
+       rev     r1,r1
+       rev     r2,r2
+       rev     r3,r3
+# endif
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r0,r0,r8
+       eorhs   r1,r1,r9
+        add    r8,sp,#4*(12)
+       str     r0,[r14],#16            @ store output
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r2,r2,r10
+       eorhs   r3,r3,r11
+       str     r1,[r14,#-12]
+        ldmia  r8,{r8-r11}     @ load key material
+       str     r2,[r14,#-8]
+       str     r3,[r14,#-4]
+
+       add     r4,r8,r4,ror#24 @ accumulate key material
+       add     r5,r9,r5,ror#24
+# ifdef        __thumb2__
+       itt     hi
+# endif
+        addhi  r8,r8,#1                @ next counter value
+        strhi  r8,[sp,#4*(12)] @ save next counter value
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r8,[r12],#16            @ load input
+       ldrhs   r9,[r12,#-12]
+       add     r6,r10,r6,ror#24
+       add     r7,r11,r7,ror#24
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhs   r10,[r12,#-8]
+       ldrhs   r11,[r12,#-4]
+# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+       rev     r4,r4
+       rev     r5,r5
+       rev     r6,r6
+       rev     r7,r7
+# endif
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r4,r4,r8
+       eorhs   r5,r5,r9
+# ifdef        __thumb2__
+        it     ne
+# endif
+        ldrne  r8,[sp,#4*(32+2)]       @ re-load len
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       eorhs   r6,r6,r10
+       eorhs   r7,r7,r11
+       str     r4,[r14],#16            @ store output
+       str     r5,[r14,#-12]
+# ifdef        __thumb2__
+       it      hs
+# endif
+        subhs  r11,r8,#64              @ len-=64
+       str     r6,[r14,#-8]
+       str     r7,[r14,#-4]
+       bhi     .Loop_outer
+
+       beq     .Ldone
+# if __ARM_ARCH__<7
+       b       .Ltail
+
+.align 4
+.Lunaligned:                           @ unaligned endian-neutral path
+       cmp     r11,#64         @ restore flags
+# endif
+#endif
+#if __ARM_ARCH__<7
+       ldr     r11,[sp,#4*(3)]
+       add     r0,r8,r0        @ accumulate key material
+       add     r1,r9,r1
+       add     r2,r10,r2
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r8,r8,r8                @ zero or ...
+       ldrhsb  r8,[r12],#16                    @ ... load input
+       eorlo   r9,r9,r9
+       ldrhsb  r9,[r12,#-12]
+
+       add     r3,r11,r3
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r10,r10,r10
+       ldrhsb  r10,[r12,#-8]
+       eorlo   r11,r11,r11
+       ldrhsb  r11,[r12,#-4]
+
+       eor     r0,r8,r0                @ xor with input (or zero)
+       eor     r1,r9,r1
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-15]           @ load more input
+       ldrhsb  r9,[r12,#-11]
+       eor     r2,r10,r2
+        strb   r0,[r14],#16            @ store output
+       eor     r3,r11,r3
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-7]
+       ldrhsb  r11,[r12,#-3]
+        strb   r1,[r14,#-12]
+       eor     r0,r8,r0,lsr#8
+        strb   r2,[r14,#-8]
+       eor     r1,r9,r1,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-14]           @ load more input
+       ldrhsb  r9,[r12,#-10]
+        strb   r3,[r14,#-4]
+       eor     r2,r10,r2,lsr#8
+        strb   r0,[r14,#-15]
+       eor     r3,r11,r3,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-6]
+       ldrhsb  r11,[r12,#-2]
+        strb   r1,[r14,#-11]
+       eor     r0,r8,r0,lsr#8
+        strb   r2,[r14,#-7]
+       eor     r1,r9,r1,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-13]           @ load more input
+       ldrhsb  r9,[r12,#-9]
+        strb   r3,[r14,#-3]
+       eor     r2,r10,r2,lsr#8
+        strb   r0,[r14,#-14]
+       eor     r3,r11,r3,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-5]
+       ldrhsb  r11,[r12,#-1]
+        strb   r1,[r14,#-10]
+        strb   r2,[r14,#-6]
+       eor     r0,r8,r0,lsr#8
+        strb   r3,[r14,#-2]
+       eor     r1,r9,r1,lsr#8
+        strb   r0,[r14,#-13]
+       eor     r2,r10,r2,lsr#8
+        strb   r1,[r14,#-9]
+       eor     r3,r11,r3,lsr#8
+        strb   r2,[r14,#-5]
+        strb   r3,[r14,#-1]
+       add     r8,sp,#4*(4+0)
+       ldmia   r8,{r8-r11}             @ load key material
+       add     r0,sp,#4*(16+8)
+       add     r4,r8,r4,ror#13 @ accumulate key material
+       add     r5,r9,r5,ror#13
+       add     r6,r10,r6,ror#13
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r8,r8,r8                @ zero or ...
+       ldrhsb  r8,[r12],#16                    @ ... load input
+       eorlo   r9,r9,r9
+       ldrhsb  r9,[r12,#-12]
+
+       add     r7,r11,r7,ror#13
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r10,r10,r10
+       ldrhsb  r10,[r12,#-8]
+       eorlo   r11,r11,r11
+       ldrhsb  r11,[r12,#-4]
+
+       eor     r4,r8,r4                @ xor with input (or zero)
+       eor     r5,r9,r5
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-15]           @ load more input
+       ldrhsb  r9,[r12,#-11]
+       eor     r6,r10,r6
+        strb   r4,[r14],#16            @ store output
+       eor     r7,r11,r7
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-7]
+       ldrhsb  r11,[r12,#-3]
+        strb   r5,[r14,#-12]
+       eor     r4,r8,r4,lsr#8
+        strb   r6,[r14,#-8]
+       eor     r5,r9,r5,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-14]           @ load more input
+       ldrhsb  r9,[r12,#-10]
+        strb   r7,[r14,#-4]
+       eor     r6,r10,r6,lsr#8
+        strb   r4,[r14,#-15]
+       eor     r7,r11,r7,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-6]
+       ldrhsb  r11,[r12,#-2]
+        strb   r5,[r14,#-11]
+       eor     r4,r8,r4,lsr#8
+        strb   r6,[r14,#-7]
+       eor     r5,r9,r5,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-13]           @ load more input
+       ldrhsb  r9,[r12,#-9]
+        strb   r7,[r14,#-3]
+       eor     r6,r10,r6,lsr#8
+        strb   r4,[r14,#-14]
+       eor     r7,r11,r7,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-5]
+       ldrhsb  r11,[r12,#-1]
+        strb   r5,[r14,#-10]
+        strb   r6,[r14,#-6]
+       eor     r4,r8,r4,lsr#8
+        strb   r7,[r14,#-2]
+       eor     r5,r9,r5,lsr#8
+        strb   r4,[r14,#-13]
+       eor     r6,r10,r6,lsr#8
+        strb   r5,[r14,#-9]
+       eor     r7,r11,r7,lsr#8
+        strb   r6,[r14,#-5]
+        strb   r7,[r14,#-1]
+       add     r8,sp,#4*(4+4)
+       ldmia   r8,{r8-r11}             @ load key material
+       ldmia   r0,{r0-r7}              @ load second half
+# ifdef        __thumb2__
+       itt     hi
+# endif
+       strhi   r10,[sp,#4*(16+10)]             @ copy "rx"
+       strhi   r11,[sp,#4*(16+11)]             @ copy "rx"
+       add     r0,r8,r0        @ accumulate key material
+       add     r1,r9,r1
+       add     r2,r10,r2
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r8,r8,r8                @ zero or ...
+       ldrhsb  r8,[r12],#16                    @ ... load input
+       eorlo   r9,r9,r9
+       ldrhsb  r9,[r12,#-12]
+
+       add     r3,r11,r3
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r10,r10,r10
+       ldrhsb  r10,[r12,#-8]
+       eorlo   r11,r11,r11
+       ldrhsb  r11,[r12,#-4]
+
+       eor     r0,r8,r0                @ xor with input (or zero)
+       eor     r1,r9,r1
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-15]           @ load more input
+       ldrhsb  r9,[r12,#-11]
+       eor     r2,r10,r2
+        strb   r0,[r14],#16            @ store output
+       eor     r3,r11,r3
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-7]
+       ldrhsb  r11,[r12,#-3]
+        strb   r1,[r14,#-12]
+       eor     r0,r8,r0,lsr#8
+        strb   r2,[r14,#-8]
+       eor     r1,r9,r1,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-14]           @ load more input
+       ldrhsb  r9,[r12,#-10]
+        strb   r3,[r14,#-4]
+       eor     r2,r10,r2,lsr#8
+        strb   r0,[r14,#-15]
+       eor     r3,r11,r3,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-6]
+       ldrhsb  r11,[r12,#-2]
+        strb   r1,[r14,#-11]
+       eor     r0,r8,r0,lsr#8
+        strb   r2,[r14,#-7]
+       eor     r1,r9,r1,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-13]           @ load more input
+       ldrhsb  r9,[r12,#-9]
+        strb   r3,[r14,#-3]
+       eor     r2,r10,r2,lsr#8
+        strb   r0,[r14,#-14]
+       eor     r3,r11,r3,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-5]
+       ldrhsb  r11,[r12,#-1]
+        strb   r1,[r14,#-10]
+        strb   r2,[r14,#-6]
+       eor     r0,r8,r0,lsr#8
+        strb   r3,[r14,#-2]
+       eor     r1,r9,r1,lsr#8
+        strb   r0,[r14,#-13]
+       eor     r2,r10,r2,lsr#8
+        strb   r1,[r14,#-9]
+       eor     r3,r11,r3,lsr#8
+        strb   r2,[r14,#-5]
+        strb   r3,[r14,#-1]
+       add     r8,sp,#4*(4+8)
+       ldmia   r8,{r8-r11}             @ load key material
+       add     r4,r8,r4,ror#24 @ accumulate key material
+# ifdef        __thumb2__
+       itt     hi
+# endif
+       addhi   r8,r8,#1                        @ next counter value
+       strhi   r8,[sp,#4*(12)]         @ save next counter value
+       add     r5,r9,r5,ror#24
+       add     r6,r10,r6,ror#24
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r8,r8,r8                @ zero or ...
+       ldrhsb  r8,[r12],#16                    @ ... load input
+       eorlo   r9,r9,r9
+       ldrhsb  r9,[r12,#-12]
+
+       add     r7,r11,r7,ror#24
+# ifdef        __thumb2__
+       itete   lo
+# endif
+       eorlo   r10,r10,r10
+       ldrhsb  r10,[r12,#-8]
+       eorlo   r11,r11,r11
+       ldrhsb  r11,[r12,#-4]
+
+       eor     r4,r8,r4                @ xor with input (or zero)
+       eor     r5,r9,r5
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-15]           @ load more input
+       ldrhsb  r9,[r12,#-11]
+       eor     r6,r10,r6
+        strb   r4,[r14],#16            @ store output
+       eor     r7,r11,r7
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-7]
+       ldrhsb  r11,[r12,#-3]
+        strb   r5,[r14,#-12]
+       eor     r4,r8,r4,lsr#8
+        strb   r6,[r14,#-8]
+       eor     r5,r9,r5,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-14]           @ load more input
+       ldrhsb  r9,[r12,#-10]
+        strb   r7,[r14,#-4]
+       eor     r6,r10,r6,lsr#8
+        strb   r4,[r14,#-15]
+       eor     r7,r11,r7,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-6]
+       ldrhsb  r11,[r12,#-2]
+        strb   r5,[r14,#-11]
+       eor     r4,r8,r4,lsr#8
+        strb   r6,[r14,#-7]
+       eor     r5,r9,r5,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r8,[r12,#-13]           @ load more input
+       ldrhsb  r9,[r12,#-9]
+        strb   r7,[r14,#-3]
+       eor     r6,r10,r6,lsr#8
+        strb   r4,[r14,#-14]
+       eor     r7,r11,r7,lsr#8
+# ifdef        __thumb2__
+       itt     hs
+# endif
+       ldrhsb  r10,[r12,#-5]
+       ldrhsb  r11,[r12,#-1]
+        strb   r5,[r14,#-10]
+        strb   r6,[r14,#-6]
+       eor     r4,r8,r4,lsr#8
+        strb   r7,[r14,#-2]
+       eor     r5,r9,r5,lsr#8
+        strb   r4,[r14,#-13]
+       eor     r6,r10,r6,lsr#8
+        strb   r5,[r14,#-9]
+       eor     r7,r11,r7,lsr#8
+        strb   r6,[r14,#-5]
+        strb   r7,[r14,#-1]
+# ifdef        __thumb2__
+       it      ne
+# endif
+       ldrne   r8,[sp,#4*(32+2)]               @ re-load len
+# ifdef        __thumb2__
+       it      hs
+# endif
+       subhs   r11,r8,#64                      @ len-=64
+       bhi     .Loop_outer
+
+       beq     .Ldone
+#endif
+
+.Ltail:
+       ldr     r12,[sp,#4*(32+1)]      @ load inp
+       add     r9,sp,#4*(0)
+       ldr     r14,[sp,#4*(32+0)]      @ load out
+
+.Loop_tail:
+       ldrb    r10,[r9],#1     @ read buffer on stack
+       ldrb    r11,[r12],#1            @ read input
+       subs    r8,r8,#1
+       eor     r11,r11,r10
+       strb    r11,[r14],#1            @ store output
+       bne     .Loop_tail
+
+.Ldone:
+       add     sp,sp,#4*(32+3)
+.Lno_data:
+       ldmia   sp!,{r4-r11,pc}
+.size  ChaCha20_ctr32,.-ChaCha20_ctr32
+#if __ARM_MAX_ARCH__>=7
+.arch  armv7-a
+.fpu   neon
+
+.type  ChaCha20_neon,%function
+.align 5
+ChaCha20_neon:
+       ldr             r12,[sp,#0]             @ pull pointer to counter and nonce
+       stmdb           sp!,{r0-r2,r4-r11,lr}
+.LChaCha20_neon:
+       adr             r14,.Lsigma
+       vstmdb          sp!,{d8-d15}            @ ABI spec says so
+       stmdb           sp!,{r0-r3}
+
+       vld1.32         {q1-q2},[r3]            @ load key
+       ldmia           r3,{r4-r11}             @ load key
+
+       sub             sp,sp,#4*(16+16)
+       vld1.32         {q3},[r12]              @ load counter and nonce
+       add             r12,sp,#4*8
+       ldmia           r14,{r0-r3}             @ load sigma
+       vld1.32         {q0},[r14]!             @ load sigma
+       vld1.32         {q12},[r14]!            @ one
+       @ vld1.32       {d30},[r14]             @ rot8
+       vst1.32         {q2-q3},[r12]           @ copy 1/2key|counter|nonce
+       vst1.32         {q0-q1},[sp]            @ copy sigma|1/2key
+
+       str             r10,[sp,#4*(16+10)]     @ off-load "rx"
+       str             r11,[sp,#4*(16+11)]     @ off-load "rx"
+       vshl.i32        d26,d24,#1      @ two
+       vstr            d24,[sp,#4*(16+0)]
+       vshl.i32        d28,d24,#2      @ four
+       vstr            d26,[sp,#4*(16+2)]
+       vmov            q4,q0
+       vstr            d28,[sp,#4*(16+4)]
+       vmov            q8,q0
+       @ vstr          d30,[sp,#4*(16+6)]
+       vmov            q5,q1
+       vmov            q9,q1
+       b               .Loop_neon_enter
+
+.align 4
+.Loop_neon_outer:
+       ldmia           sp,{r0-r9}              @ load key material
+       cmp             r11,#64*2               @ if len<=64*2
+       bls             .Lbreak_neon            @ switch to integer-only
+       @ vldr          d30,[sp,#4*(16+6)]      @ rot8
+       vmov            q4,q0
+       str             r11,[sp,#4*(32+2)]      @ save len
+       vmov            q8,q0
+       str             r12,  [sp,#4*(32+1)]    @ save inp
+       vmov            q5,q1
+       str             r14,  [sp,#4*(32+0)]    @ save out
+       vmov            q9,q1
+.Loop_neon_enter:
+       ldr             r11, [sp,#4*(15)]
+        mov            r4,r4,ror#19    @ twist b[0..3]
+       vadd.i32        q7,q3,q12               @ counter+1
+       ldr             r12,[sp,#4*(12)]        @ modulo-scheduled load
+        mov            r5,r5,ror#19
+       vmov            q6,q2
+       ldr             r10, [sp,#4*(13)]
+        mov            r6,r6,ror#19
+       vmov            q10,q2
+       ldr             r14,[sp,#4*(14)]
+        mov            r7,r7,ror#19
+       vadd.i32        q11,q7,q12              @ counter+2
+       add             r12,r12,#3      @ counter+3
+       mov             r11,r11,ror#8   @ twist d[0..3]
+       mov             r12,r12,ror#8
+       mov             r10,r10,ror#8
+       mov             r14,r14,ror#8
+       str             r11, [sp,#4*(16+15)]
+       mov             r11,#10
+       b               .Loop_neon
+
+.align 4
+.Loop_neon:
+       subs            r11,r11,#1
+       vadd.i32        q0,q0,q1
+       add     r0,r0,r4,ror#13
+       vadd.i32        q4,q4,q5
+       add     r1,r1,r5,ror#13
+       vadd.i32        q8,q8,q9
+       eor     r12,r0,r12,ror#24
+       veor    q3,q3,q0
+       eor     r10,r1,r10,ror#24
+       veor    q7,q7,q4
+       add     r8,r8,r12,ror#16
+       veor    q11,q11,q8
+       add     r9,r9,r10,ror#16
+       vrev32.16       q3,q3
+       eor     r4,r8,r4,ror#13
+       vrev32.16       q7,q7
+       eor     r5,r9,r5,ror#13
+       vrev32.16       q11,q11
+       add     r0,r0,r4,ror#20
+       vadd.i32        q2,q2,q3
+       add     r1,r1,r5,ror#20
+       vadd.i32        q6,q6,q7
+       eor     r12,r0,r12,ror#16
+       vadd.i32        q10,q10,q11
+       eor     r10,r1,r10,ror#16
+       veor    q12,q1,q2
+       add     r8,r8,r12,ror#24
+       veor    q13,q5,q6
+       str     r10,[sp,#4*(16+13)]
+       veor    q14,q9,q10
+       add     r9,r9,r10,ror#24
+       vshr.u32        q1,q12,#20
+       ldr     r10,[sp,#4*(16+15)]
+       vshr.u32        q5,q13,#20
+       str     r8,[sp,#4*(16+8)]
+       vshr.u32        q9,q14,#20
+       eor     r4,r4,r8,ror#12
+       vsli.32 q1,q12,#12
+       str     r9,[sp,#4*(16+9)]
+       vsli.32 q5,q13,#12
+       eor     r5,r5,r9,ror#12
+       vsli.32 q9,q14,#12
+       ldr     r8,[sp,#4*(16+10)]
+       vadd.i32        q0,q0,q1
+       add     r2,r2,r6,ror#13
+       vadd.i32        q4,q4,q5
+       ldr     r9,[sp,#4*(16+11)]
+       vadd.i32        q8,q8,q9
+       add     r3,r3,r7,ror#13
+       veor    q12,q3,q0
+       eor     r14,r2,r14,ror#24
+       veor    q13,q7,q4
+       eor     r10,r3,r10,ror#24
+       veor    q14,q11,q8
+       add     r8,r8,r14,ror#16
+       vshr.u32        q3,q12,#24
+       add     r9,r9,r10,ror#16
+       vshr.u32        q7,q13,#24
+       eor     r6,r8,r6,ror#13
+       vshr.u32        q11,q14,#24
+       eor     r7,r9,r7,ror#13
+       vsli.32 q3,q12,#8
+       add     r2,r2,r6,ror#20
+       vsli.32 q7,q13,#8
+       add     r3,r3,r7,ror#20
+       vsli.32 q11,q14,#8
+       eor     r14,r2,r14,ror#16
+       vadd.i32        q2,q2,q3
+       eor     r10,r3,r10,ror#16
+       vadd.i32        q6,q6,q7
+       add     r8,r8,r14,ror#24
+       vadd.i32        q10,q10,q11
+       add     r9,r9,r10,ror#24
+       veor    q12,q1,q2
+       eor     r6,r6,r8,ror#12
+       veor    q13,q5,q6
+       eor     r7,r7,r9,ror#12
+       veor    q14,q9,q10
+       vshr.u32        q1,q12,#25
+       vshr.u32        q5,q13,#25
+       vshr.u32        q9,q14,#25
+       vsli.32 q1,q12,#7
+       vsli.32 q5,q13,#7
+       vsli.32 q9,q14,#7
+       vext.8  q2,q2,q2,#8
+       vext.8  q6,q6,q6,#8
+       vext.8  q10,q10,q10,#8
+       vext.8  q1,q1,q1,#4
+       vext.8  q5,q5,q5,#4
+       vext.8  q9,q9,q9,#4
+       vext.8  q3,q3,q3,#12
+       vext.8  q7,q7,q7,#12
+       vext.8  q11,q11,q11,#12
+       vadd.i32        q0,q0,q1
+       add     r0,r0,r5,ror#13
+       vadd.i32        q4,q4,q5
+       add     r1,r1,r6,ror#13
+       vadd.i32        q8,q8,q9
+       eor     r10,r0,r10,ror#24
+       veor    q3,q3,q0
+       eor     r12,r1,r12,ror#24
+       veor    q7,q7,q4
+       add     r8,r8,r10,ror#16
+       veor    q11,q11,q8
+       add     r9,r9,r12,ror#16
+       vrev32.16       q3,q3
+       eor     r5,r8,r5,ror#13
+       vrev32.16       q7,q7
+       eor     r6,r9,r6,ror#13
+       vrev32.16       q11,q11
+       add     r0,r0,r5,ror#20
+       vadd.i32        q2,q2,q3
+       add     r1,r1,r6,ror#20
+       vadd.i32        q6,q6,q7
+       eor     r10,r0,r10,ror#16
+       vadd.i32        q10,q10,q11
+       eor     r12,r1,r12,ror#16
+       veor    q12,q1,q2
+       str     r10,[sp,#4*(16+15)]
+       veor    q13,q5,q6
+       add     r8,r8,r10,ror#24
+       veor    q14,q9,q10
+       ldr     r10,[sp,#4*(16+13)]
+       vshr.u32        q1,q12,#20
+       add     r9,r9,r12,ror#24
+       vshr.u32        q5,q13,#20
+       str     r8,[sp,#4*(16+10)]
+       vshr.u32        q9,q14,#20
+       eor     r5,r5,r8,ror#12
+       vsli.32 q1,q12,#12
+       str     r9,[sp,#4*(16+11)]
+       vsli.32 q5,q13,#12
+       eor     r6,r6,r9,ror#12
+       vsli.32 q9,q14,#12
+       ldr     r8,[sp,#4*(16+8)]
+       vadd.i32        q0,q0,q1
+       add     r2,r2,r7,ror#13
+       vadd.i32        q4,q4,q5
+       ldr     r9,[sp,#4*(16+9)]
+       vadd.i32        q8,q8,q9
+       add     r3,r3,r4,ror#13
+       veor    q12,q3,q0
+       eor     r10,r2,r10,ror#24
+       veor    q13,q7,q4
+       eor     r14,r3,r14,ror#24
+       veor    q14,q11,q8
+       add     r8,r8,r10,ror#16
+       vshr.u32        q3,q12,#24
+       add     r9,r9,r14,ror#16
+       vshr.u32        q7,q13,#24
+       eor     r7,r8,r7,ror#13
+       vshr.u32        q11,q14,#24
+       eor     r4,r9,r4,ror#13
+       vsli.32 q3,q12,#8
+       add     r2,r2,r7,ror#20
+       vsli.32 q7,q13,#8
+       add     r3,r3,r4,ror#20
+       vsli.32 q11,q14,#8
+       eor     r10,r2,r10,ror#16
+       vadd.i32        q2,q2,q3
+       eor     r14,r3,r14,ror#16
+       vadd.i32        q6,q6,q7
+       add     r8,r8,r10,ror#24
+       vadd.i32        q10,q10,q11
+       add     r9,r9,r14,ror#24
+       veor    q12,q1,q2
+       eor     r7,r7,r8,ror#12
+       veor    q13,q5,q6
+       eor     r4,r4,r9,ror#12
+       veor    q14,q9,q10
+       vshr.u32        q1,q12,#25
+       vshr.u32        q5,q13,#25
+       vshr.u32        q9,q14,#25
+       vsli.32 q1,q12,#7
+       vsli.32 q5,q13,#7
+       vsli.32 q9,q14,#7
+       vext.8  q2,q2,q2,#8
+       vext.8  q6,q6,q6,#8
+       vext.8  q10,q10,q10,#8
+       vext.8  q1,q1,q1,#12
+       vext.8  q5,q5,q5,#12
+       vext.8  q9,q9,q9,#12
+       vext.8  q3,q3,q3,#4
+       vext.8  q7,q7,q7,#4
+       vext.8  q11,q11,q11,#4
+       bne             .Loop_neon
+
+       add             r11,sp,#32
+       vld1.32         {q12-q13},[sp]          @ load key material
+       vld1.32         {q14-q15},[r11]
+
+       ldr             r11,[sp,#4*(32+2)]      @ load len
+
+       str             r8, [sp,#4*(16+8)]      @ modulo-scheduled store
+       str             r9, [sp,#4*(16+9)]
+       str             r12,[sp,#4*(16+12)]
+       str             r10, [sp,#4*(16+13)]
+       str             r14,[sp,#4*(16+14)]
+
+       @ at this point we have first half of 512-bit result in
+       @ rx and second half at sp+4*(16+8)
+
+       ldr             r12,[sp,#4*(32+1)]      @ load inp
+       ldr             r14,[sp,#4*(32+0)]      @ load out
+
+       vadd.i32        q0,q0,q12               @ accumulate key material
+       vadd.i32        q4,q4,q12
+       vadd.i32        q8,q8,q12
+       vldr            d24,[sp,#4*(16+0)]      @ one
+
+       vadd.i32        q1,q1,q13
+       vadd.i32        q5,q5,q13
+       vadd.i32        q9,q9,q13
+       vldr            d26,[sp,#4*(16+2)]      @ two
+
+       vadd.i32        q2,q2,q14
+       vadd.i32        q6,q6,q14
+       vadd.i32        q10,q10,q14
+       vadd.i32        d14,d14,d24     @ counter+1
+       vadd.i32        d22,d22,d26     @ counter+2
+
+       vadd.i32        q3,q3,q15
+       vadd.i32        q7,q7,q15
+       vadd.i32        q11,q11,q15
+
+       cmp             r11,#64*4
+       blo             .Ltail_neon
+
+       vld1.8          {q12-q13},[r12]!        @ load input
+        mov            r11,sp
+       vld1.8          {q14-q15},[r12]!
+       veor            q0,q0,q12               @ xor with input
+       veor            q1,q1,q13
+       vld1.8          {q12-q13},[r12]!
+       veor            q2,q2,q14
+       veor            q3,q3,q15
+       vld1.8          {q14-q15},[r12]!
+
+       veor            q4,q4,q12
+        vst1.8         {q0-q1},[r14]!  @ store output
+       veor            q5,q5,q13
+       vld1.8          {q12-q13},[r12]!
+       veor            q6,q6,q14
+        vst1.8         {q2-q3},[r14]!
+       veor            q7,q7,q15
+       vld1.8          {q14-q15},[r12]!
+
+       veor            q8,q8,q12
+        vld1.32        {q0-q1},[r11]!  @ load for next iteration
+        veor           d25,d25,d25
+        vldr           d24,[sp,#4*(16+4)]      @ four
+       veor            q9,q9,q13
+        vld1.32        {q2-q3},[r11]
+       veor            q10,q10,q14
+        vst1.8         {q4-q5},[r14]!
+       veor            q11,q11,q15
+        vst1.8         {q6-q7},[r14]!
+
+       vadd.i32        d6,d6,d24       @ next counter value
+       vldr            d24,[sp,#4*(16+0)]      @ one
+
+       ldmia           sp,{r8-r11}     @ load key material
+       add             r0,r0,r8        @ accumulate key material
+       ldr             r8,[r12],#16            @ load input
+        vst1.8         {q8-q9},[r14]!
+       add             r1,r1,r9
+       ldr             r9,[r12,#-12]
+        vst1.8         {q10-q11},[r14]!
+       add             r2,r2,r10
+       ldr             r10,[r12,#-8]
+       add             r3,r3,r11
+       ldr             r11,[r12,#-4]
+# ifdef        __ARMEB__
+       rev             r0,r0
+       rev             r1,r1
+       rev             r2,r2
+       rev             r3,r3
+# endif
+       eor             r0,r0,r8        @ xor with input
+        add            r8,sp,#4*(4)
+       eor             r1,r1,r9
+       str             r0,[r14],#16            @ store output
+       eor             r2,r2,r10
+       str             r1,[r14,#-12]
+       eor             r3,r3,r11
+        ldmia          r8,{r8-r11}     @ load key material
+       str             r2,[r14,#-8]
+       str             r3,[r14,#-4]
+
+       add             r4,r8,r4,ror#13 @ accumulate key material
+       ldr             r8,[r12],#16            @ load input
+       add             r5,r9,r5,ror#13
+       ldr             r9,[r12,#-12]
+       add             r6,r10,r6,ror#13
+       ldr             r10,[r12,#-8]
+       add             r7,r11,r7,ror#13
+       ldr             r11,[r12,#-4]
+# ifdef        __ARMEB__
+       rev             r4,r4
+       rev             r5,r5
+       rev             r6,r6
+       rev             r7,r7
+# endif
+       eor             r4,r4,r8
+        add            r8,sp,#4*(8)
+       eor             r5,r5,r9
+       str             r4,[r14],#16            @ store output
+       eor             r6,r6,r10
+       str             r5,[r14,#-12]
+       eor             r7,r7,r11
+        ldmia          r8,{r8-r11}     @ load key material
+       str             r6,[r14,#-8]
+        add            r0,sp,#4*(16+8)
+       str             r7,[r14,#-4]
+
+       ldmia           r0,{r0-r7}      @ load second half
+
+       add             r0,r0,r8        @ accumulate key material
+       ldr             r8,[r12],#16            @ load input
+       add             r1,r1,r9
+       ldr             r9,[r12,#-12]
+# ifdef        __thumb2__
+       it      hi
+# endif
+        strhi          r10,[sp,#4*(16+10)]     @ copy "rx" while at it
+       add             r2,r2,r10
+       ldr             r10,[r12,#-8]
+# ifdef        __thumb2__
+       it      hi
+# endif
+        strhi          r11,[sp,#4*(16+11)]     @ copy "rx" while at it
+       add             r3,r3,r11
+       ldr             r11,[r12,#-4]
+# ifdef        __ARMEB__
+       rev             r0,r0
+       rev             r1,r1
+       rev             r2,r2
+       rev             r3,r3
+# endif
+       eor             r0,r0,r8
+        add            r8,sp,#4*(12)
+       eor             r1,r1,r9
+       str             r0,[r14],#16            @ store output
+       eor             r2,r2,r10
+       str             r1,[r14,#-12]
+       eor             r3,r3,r11
+        ldmia          r8,{r8-r11}     @ load key material
+       str             r2,[r14,#-8]
+       str             r3,[r14,#-4]
+
+       add             r4,r8,r4,ror#24 @ accumulate key material
+        add            r8,r8,#4                @ next counter value
+       add             r5,r9,r5,ror#24
+        str            r8,[sp,#4*(12)] @ save next counter value
+       ldr             r8,[r12],#16            @ load input
+       add             r6,r10,r6,ror#24
+        add            r4,r4,#3                @ counter+3
+       ldr             r9,[r12,#-12]
+       add             r7,r11,r7,ror#24
+       ldr             r10,[r12,#-8]
+       ldr             r11,[r12,#-4]
+# ifdef        __ARMEB__
+       rev             r4,r4
+       rev             r5,r5
+       rev             r6,r6
+       rev             r7,r7
+# endif
+       eor             r4,r4,r8
+# ifdef        __thumb2__
+       it      hi
+# endif
+        ldrhi          r8,[sp,#4*(32+2)]       @ re-load len
+       eor             r5,r5,r9
+       eor             r6,r6,r10
+       str             r4,[r14],#16            @ store output
+       eor             r7,r7,r11
+       str             r5,[r14,#-12]
+        sub            r11,r8,#64*4    @ len-=64*4
+       str             r6,[r14,#-8]
+       str             r7,[r14,#-4]
+       bhi             .Loop_neon_outer
+
+       b               .Ldone_neon
+
+.align 4
+.Lbreak_neon:
+       @ harmonize NEON and integer-only stack frames: load data
+       @ from NEON frame, but save to integer-only one; distance
+       @ between the two is 4*(32+4+16-32)=4*(20).
+
+       str             r11, [sp,#4*(20+32+2)]  @ save len
+        add            r11,sp,#4*(32+4)
+       str             r12,   [sp,#4*(20+32+1)]        @ save inp
+       str             r14,   [sp,#4*(20+32+0)]        @ save out
+
+       ldr             r12,[sp,#4*(16+10)]
+       ldr             r14,[sp,#4*(16+11)]
+        vldmia         r11,{d8-d15}                    @ fulfill ABI requirement
+       str             r12,[sp,#4*(20+16+10)]  @ copy "rx"
+       str             r14,[sp,#4*(20+16+11)]  @ copy "rx"
+
+       ldr             r11, [sp,#4*(15)]
+        mov            r4,r4,ror#19            @ twist b[0..3]
+       ldr             r12,[sp,#4*(12)]                @ modulo-scheduled load
+        mov            r5,r5,ror#19
+       ldr             r10, [sp,#4*(13)]
+        mov            r6,r6,ror#19
+       ldr             r14,[sp,#4*(14)]
+        mov            r7,r7,ror#19
+       mov             r11,r11,ror#8           @ twist d[0..3]
+       mov             r12,r12,ror#8
+       mov             r10,r10,ror#8
+       mov             r14,r14,ror#8
+       str             r11, [sp,#4*(20+16+15)]
+       add             r11,sp,#4*(20)
+       vst1.32         {q0-q1},[r11]!          @ copy key
+       add             sp,sp,#4*(20)                   @ switch frame
+       vst1.32         {q2-q3},[r11]
+       mov             r11,#10
+       b               .Loop                           @ go integer-only
+
+.align 4
+.Ltail_neon:
+       cmp             r11,#64*3
+       bhs             .L192_or_more_neon
+       cmp             r11,#64*2
+       bhs             .L128_or_more_neon
+       cmp             r11,#64*1
+       bhs             .L64_or_more_neon
+
+       add             r8,sp,#4*(8)
+       vst1.8          {q0-q1},[sp]
+       add             r10,sp,#4*(0)
+       vst1.8          {q2-q3},[r8]
+       b               .Loop_tail_neon
+
+.align 4
+.L64_or_more_neon:
+       vld1.8          {q12-q13},[r12]!
+       vld1.8          {q14-q15},[r12]!
+       veor            q0,q0,q12
+       veor            q1,q1,q13
+       veor            q2,q2,q14
+       veor            q3,q3,q15
+       vst1.8          {q0-q1},[r14]!
+       vst1.8          {q2-q3},[r14]!
+
+       beq             .Ldone_neon
+
+       add             r8,sp,#4*(8)
+       vst1.8          {q4-q5},[sp]
+       add             r10,sp,#4*(0)
+       vst1.8          {q6-q7},[r8]
+       sub             r11,r11,#64*1   @ len-=64*1
+       b               .Loop_tail_neon
+
+.align 4
+.L128_or_more_neon:
+       vld1.8          {q12-q13},[r12]!
+       vld1.8          {q14-q15},[r12]!
+       veor            q0,q0,q12
+       veor            q1,q1,q13
+       vld1.8          {q12-q13},[r12]!
+       veor            q2,q2,q14
+       veor            q3,q3,q15
+       vld1.8          {q14-q15},[r12]!
+
+       veor            q4,q4,q12
+       veor            q5,q5,q13
+        vst1.8         {q0-q1},[r14]!
+       veor            q6,q6,q14
+        vst1.8         {q2-q3},[r14]!
+       veor            q7,q7,q15
+       vst1.8          {q4-q5},[r14]!
+       vst1.8          {q6-q7},[r14]!
+
+       beq             .Ldone_neon
+
+       add             r8,sp,#4*(8)
+       vst1.8          {q8-q9},[sp]
+       add             r10,sp,#4*(0)
+       vst1.8          {q10-q11},[r8]
+       sub             r11,r11,#64*2   @ len-=64*2
+       b               .Loop_tail_neon
+
+.align 4
+.L192_or_more_neon:
+       vld1.8          {q12-q13},[r12]!
+       vld1.8          {q14-q15},[r12]!
+       veor            q0,q0,q12
+       veor            q1,q1,q13
+       vld1.8          {q12-q13},[r12]!
+       veor            q2,q2,q14
+       veor            q3,q3,q15
+       vld1.8          {q14-q15},[r12]!
+
+       veor            q4,q4,q12
+       veor            q5,q5,q13
+       vld1.8          {q12-q13},[r12]!
+       veor            q6,q6,q14
+        vst1.8         {q0-q1},[r14]!
+       veor            q7,q7,q15
+       vld1.8          {q14-q15},[r12]!
+
+       veor            q8,q8,q12
+        vst1.8         {q2-q3},[r14]!
+       veor            q9,q9,q13
+        vst1.8         {q4-q5},[r14]!
+       veor            q10,q10,q14
+        vst1.8         {q6-q7},[r14]!
+       veor            q11,q11,q15
+       vst1.8          {q8-q9},[r14]!
+       vst1.8          {q10-q11},[r14]!
+
+       beq             .Ldone_neon
+
+       ldmia           sp,{r8-r11}     @ load key material
+       add             r0,r0,r8        @ accumulate key material
+        add            r8,sp,#4*(4)
+       add             r1,r1,r9
+       add             r2,r2,r10
+       add             r3,r3,r11
+        ldmia          r8,{r8-r11}     @ load key material
+
+       add             r4,r8,r4,ror#13 @ accumulate key material
+        add            r8,sp,#4*(8)
+       add             r5,r9,r5,ror#13
+       add             r6,r10,r6,ror#13
+       add             r7,r11,r7,ror#13
+        ldmia          r8,{r8-r11}     @ load key material
+# ifdef        __ARMEB__
+       rev             r0,r0
+       rev             r1,r1
+       rev             r2,r2
+       rev             r3,r3
+       rev             r4,r4
+       rev             r5,r5
+       rev             r6,r6
+       rev             r7,r7
+# endif
+       stmia           sp,{r0-r7}
+        add            r0,sp,#4*(16+8)
+
+       ldmia           r0,{r0-r7}      @ load second half
+
+       add             r0,r0,r8        @ accumulate key material
+        add            r8,sp,#4*(12)
+       add             r1,r1,r9
+       add             r2,r2,r10
+       add             r3,r3,r11
+        ldmia          r8,{r8-r11}     @ load key material
+
+       add             r4,r8,r4,ror#24 @ accumulate key material
+        add            r8,sp,#4*(8)
+       add             r5,r9,r5,ror#24
+        add            r4,r4,#3                @ counter+3
+       add             r6,r10,r6,ror#24
+       add             r7,r11,r7,ror#24
+        ldr            r11,[sp,#4*(32+2)]      @ re-load len
+# ifdef        __ARMEB__
+       rev             r0,r0
+       rev             r1,r1
+       rev             r2,r2
+       rev             r3,r3
+       rev             r4,r4
+       rev             r5,r5
+       rev             r6,r6
+       rev             r7,r7
+# endif
+       stmia           r8,{r0-r7}
+        add            r10,sp,#4*(0)
+        sub            r11,r11,#64*3   @ len-=64*3
+
+.Loop_tail_neon:
+       ldrb            r8,[r10],#1     @ read buffer on stack
+       ldrb            r9,[r12],#1             @ read input
+       subs            r11,r11,#1
+       eor             r8,r8,r9
+       strb            r8,[r14],#1             @ store output
+       bne             .Loop_tail_neon
+
+.Ldone_neon:
+       add             sp,sp,#4*(32+4)
+       vldmia          sp,{d8-d15}
+       add             sp,sp,#4*(16+3)
+       ldmia           sp!,{r4-r11,pc}
+.size  ChaCha20_neon,.-ChaCha20_neon
+.comm  OPENSSL_armcap_P,4,4
+#endif
lib/zinc/chacha20/chacha20-arm64-cryptogams.S new file +1973 -1
new file mode 100644
index 000000000000..4d029bfdad3a
@@ -0,0 +1,1973 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ */
+
+#include "arm_arch.h"
+
+.text
+
+
+
+.align 5
+.Lsigma:
+.quad  0x3320646e61707865,0x6b20657479622d32           // endian-neutral
+.Lone:
+.long  1,0,0,0
+.LOPENSSL_armcap_P:
+#ifdef __ILP32__
+.long  OPENSSL_armcap_P-.
+#else
+.quad  OPENSSL_armcap_P-.
+#endif
+.byte  67,104,97,67,104,97,50,48,32,102,111,114,32,65,82,77,118,56,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0
+.align 2
+
+.globl ChaCha20_ctr32
+.type  ChaCha20_ctr32,%function
+.align 5
+ChaCha20_ctr32:
+       cbz     x2,.Labort
+       adr     x5,.LOPENSSL_armcap_P
+       cmp     x2,#192
+       b.lo    .Lshort
+#ifdef __ILP32__
+       ldrsw   x6,[x5]
+#else
+       ldr     x6,[x5]
+#endif
+       ldr     w17,[x6,x5]
+       tst     w17,#ARMV7_NEON
+       b.ne    ChaCha20_neon
+
+.Lshort:
+       stp     x29,x30,[sp,#-96]!
+       add     x29,sp,#0
+
+       adr     x5,.Lsigma
+       stp     x19,x20,[sp,#16]
+       stp     x21,x22,[sp,#32]
+       stp     x23,x24,[sp,#48]
+       stp     x25,x26,[sp,#64]
+       stp     x27,x28,[sp,#80]
+       sub     sp,sp,#64
+
+       ldp     x22,x23,[x5]            // load sigma
+       ldp     x24,x25,[x3]            // load key
+       ldp     x26,x27,[x3,#16]
+       ldp     x28,x30,[x4]            // load counter
+#ifdef __ARMEB__
+       ror     x24,x24,#32
+       ror     x25,x25,#32
+       ror     x26,x26,#32
+       ror     x27,x27,#32
+       ror     x28,x28,#32
+       ror     x30,x30,#32
+#endif
+
+.Loop_outer:
+       mov     w5,w22                  // unpack key block
+       lsr     x6,x22,#32
+       mov     w7,w23
+       lsr     x8,x23,#32
+       mov     w9,w24
+       lsr     x10,x24,#32
+       mov     w11,w25
+       lsr     x12,x25,#32
+       mov     w13,w26
+       lsr     x14,x26,#32
+       mov     w15,w27
+       lsr     x16,x27,#32
+       mov     w17,w28
+       lsr     x19,x28,#32
+       mov     w20,w30
+       lsr     x21,x30,#32
+
+       mov     x4,#10
+       subs    x2,x2,#64
+.Loop:
+       sub     x4,x4,#1
+       add     w5,w5,w9
+       add     w6,w6,w10
+       add     w7,w7,w11
+       add     w8,w8,w12
+       eor     w17,w17,w5
+       eor     w19,w19,w6
+       eor     w20,w20,w7
+       eor     w21,w21,w8
+       ror     w17,w17,#16
+       ror     w19,w19,#16
+       ror     w20,w20,#16
+       ror     w21,w21,#16
+       add     w13,w13,w17
+       add     w14,w14,w19
+       add     w15,w15,w20
+       add     w16,w16,w21
+       eor     w9,w9,w13
+       eor     w10,w10,w14
+       eor     w11,w11,w15
+       eor     w12,w12,w16
+       ror     w9,w9,#20
+       ror     w10,w10,#20
+       ror     w11,w11,#20
+       ror     w12,w12,#20
+       add     w5,w5,w9
+       add     w6,w6,w10
+       add     w7,w7,w11
+       add     w8,w8,w12
+       eor     w17,w17,w5
+       eor     w19,w19,w6
+       eor     w20,w20,w7
+       eor     w21,w21,w8
+       ror     w17,w17,#24
+       ror     w19,w19,#24
+       ror     w20,w20,#24
+       ror     w21,w21,#24
+       add     w13,w13,w17
+       add     w14,w14,w19
+       add     w15,w15,w20
+       add     w16,w16,w21
+       eor     w9,w9,w13
+       eor     w10,w10,w14
+       eor     w11,w11,w15
+       eor     w12,w12,w16
+       ror     w9,w9,#25
+       ror     w10,w10,#25
+       ror     w11,w11,#25
+       ror     w12,w12,#25
+       add     w5,w5,w10
+       add     w6,w6,w11
+       add     w7,w7,w12
+       add     w8,w8,w9
+       eor     w21,w21,w5
+       eor     w17,w17,w6
+       eor     w19,w19,w7
+       eor     w20,w20,w8
+       ror     w21,w21,#16
+       ror     w17,w17,#16
+       ror     w19,w19,#16
+       ror     w20,w20,#16
+       add     w15,w15,w21
+       add     w16,w16,w17
+       add     w13,w13,w19
+       add     w14,w14,w20
+       eor     w10,w10,w15
+       eor     w11,w11,w16
+       eor     w12,w12,w13
+       eor     w9,w9,w14
+       ror     w10,w10,#20
+       ror     w11,w11,#20
+       ror     w12,w12,#20
+       ror     w9,w9,#20
+       add     w5,w5,w10
+       add     w6,w6,w11
+       add     w7,w7,w12
+       add     w8,w8,w9
+       eor     w21,w21,w5
+       eor     w17,w17,w6
+       eor     w19,w19,w7
+       eor     w20,w20,w8
+       ror     w21,w21,#24
+       ror     w17,w17,#24
+       ror     w19,w19,#24
+       ror     w20,w20,#24
+       add     w15,w15,w21
+       add     w16,w16,w17
+       add     w13,w13,w19
+       add     w14,w14,w20
+       eor     w10,w10,w15
+       eor     w11,w11,w16
+       eor     w12,w12,w13
+       eor     w9,w9,w14
+       ror     w10,w10,#25
+       ror     w11,w11,#25
+       ror     w12,w12,#25
+       ror     w9,w9,#25
+       cbnz    x4,.Loop
+
+       add     w5,w5,w22               // accumulate key block
+       add     x6,x6,x22,lsr#32
+       add     w7,w7,w23
+       add     x8,x8,x23,lsr#32
+       add     w9,w9,w24
+       add     x10,x10,x24,lsr#32
+       add     w11,w11,w25
+       add     x12,x12,x25,lsr#32
+       add     w13,w13,w26
+       add     x14,x14,x26,lsr#32
+       add     w15,w15,w27
+       add     x16,x16,x27,lsr#32
+       add     w17,w17,w28
+       add     x19,x19,x28,lsr#32
+       add     w20,w20,w30
+       add     x21,x21,x30,lsr#32
+
+       b.lo    .Ltail
+
+       add     x5,x5,x6,lsl#32 // pack
+       add     x7,x7,x8,lsl#32
+       ldp     x6,x8,[x1,#0]           // load input
+       add     x9,x9,x10,lsl#32
+       add     x11,x11,x12,lsl#32
+       ldp     x10,x12,[x1,#16]
+       add     x13,x13,x14,lsl#32
+       add     x15,x15,x16,lsl#32
+       ldp     x14,x16,[x1,#32]
+       add     x17,x17,x19,lsl#32
+       add     x20,x20,x21,lsl#32
+       ldp     x19,x21,[x1,#48]
+       add     x1,x1,#64
+#ifdef __ARMEB__
+       rev     x5,x5
+       rev     x7,x7
+       rev     x9,x9
+       rev     x11,x11
+       rev     x13,x13
+       rev     x15,x15
+       rev     x17,x17
+       rev     x20,x20
+#endif
+       eor     x5,x5,x6
+       eor     x7,x7,x8
+       eor     x9,x9,x10
+       eor     x11,x11,x12
+       eor     x13,x13,x14
+       eor     x15,x15,x16
+       eor     x17,x17,x19
+       eor     x20,x20,x21
+
+       stp     x5,x7,[x0,#0]           // store output
+       add     x28,x28,#1                      // increment counter
+       stp     x9,x11,[x0,#16]
+       stp     x13,x15,[x0,#32]
+       stp     x17,x20,[x0,#48]
+       add     x0,x0,#64
+
+       b.hi    .Loop_outer
+
+       ldp     x19,x20,[x29,#16]
+       add     sp,sp,#64
+       ldp     x21,x22,[x29,#32]
+       ldp     x23,x24,[x29,#48]
+       ldp     x25,x26,[x29,#64]
+       ldp     x27,x28,[x29,#80]
+       ldp     x29,x30,[sp],#96
+.Labort:
+       ret
+
+.align 4
+.Ltail:
+       add     x2,x2,#64
+.Less_than_64:
+       sub     x0,x0,#1
+       add     x1,x1,x2
+       add     x0,x0,x2
+       add     x4,sp,x2
+       neg     x2,x2
+
+       add     x5,x5,x6,lsl#32 // pack
+       add     x7,x7,x8,lsl#32
+       add     x9,x9,x10,lsl#32
+       add     x11,x11,x12,lsl#32
+       add     x13,x13,x14,lsl#32
+       add     x15,x15,x16,lsl#32
+       add     x17,x17,x19,lsl#32
+       add     x20,x20,x21,lsl#32
+#ifdef __ARMEB__
+       rev     x5,x5
+       rev     x7,x7
+       rev     x9,x9
+       rev     x11,x11
+       rev     x13,x13
+       rev     x15,x15
+       rev     x17,x17
+       rev     x20,x20
+#endif
+       stp     x5,x7,[sp,#0]
+       stp     x9,x11,[sp,#16]
+       stp     x13,x15,[sp,#32]
+       stp     x17,x20,[sp,#48]
+
+.Loop_tail:
+       ldrb    w10,[x1,x2]
+       ldrb    w11,[x4,x2]
+       add     x2,x2,#1
+       eor     w10,w10,w11
+       strb    w10,[x0,x2]
+       cbnz    x2,.Loop_tail
+
+       stp     xzr,xzr,[sp,#0]
+       stp     xzr,xzr,[sp,#16]
+       stp     xzr,xzr,[sp,#32]
+       stp     xzr,xzr,[sp,#48]
+
+       ldp     x19,x20,[x29,#16]
+       add     sp,sp,#64
+       ldp     x21,x22,[x29,#32]
+       ldp     x23,x24,[x29,#48]
+       ldp     x25,x26,[x29,#64]
+       ldp     x27,x28,[x29,#80]
+       ldp     x29,x30,[sp],#96
+       ret
+.size  ChaCha20_ctr32,.-ChaCha20_ctr32
+
+.type  ChaCha20_neon,%function
+.align 5
+ChaCha20_neon:
+       stp     x29,x30,[sp,#-96]!
+       add     x29,sp,#0
+
+       adr     x5,.Lsigma
+       stp     x19,x20,[sp,#16]
+       stp     x21,x22,[sp,#32]
+       stp     x23,x24,[sp,#48]
+       stp     x25,x26,[sp,#64]
+       stp     x27,x28,[sp,#80]
+       cmp     x2,#512
+       b.hs    .L512_or_more_neon
+
+       sub     sp,sp,#64
+
+       ldp     x22,x23,[x5]            // load sigma
+       ld1     {v24.4s},[x5],#16
+       ldp     x24,x25,[x3]            // load key
+       ldp     x26,x27,[x3,#16]
+       ld1     {v25.4s,v26.4s},[x3]
+       ldp     x28,x30,[x4]            // load counter
+       ld1     {v27.4s},[x4]
+       ld1     {v31.4s},[x5]
+#ifdef __ARMEB__
+       rev64   v24.4s,v24.4s
+       ror     x24,x24,#32
+       ror     x25,x25,#32
+       ror     x26,x26,#32
+       ror     x27,x27,#32
+       ror     x28,x28,#32
+       ror     x30,x30,#32
+#endif
+       add     v27.4s,v27.4s,v31.4s            // += 1
+       add     v28.4s,v27.4s,v31.4s
+       add     v29.4s,v28.4s,v31.4s
+       shl     v31.4s,v31.4s,#2                        // 1 -> 4
+
+.Loop_outer_neon:
+       mov     w5,w22                  // unpack key block
+       lsr     x6,x22,#32
+       mov     v0.16b,v24.16b
+       mov     w7,w23
+       lsr     x8,x23,#32
+       mov     v4.16b,v24.16b
+       mov     w9,w24
+       lsr     x10,x24,#32
+       mov     v16.16b,v24.16b
+       mov     w11,w25
+       mov     v1.16b,v25.16b
+       lsr     x12,x25,#32
+       mov     v5.16b,v25.16b
+       mov     w13,w26
+       mov     v17.16b,v25.16b
+       lsr     x14,x26,#32
+       mov     v3.16b,v27.16b
+       mov     w15,w27
+       mov     v7.16b,v28.16b
+       lsr     x16,x27,#32
+       mov     v19.16b,v29.16b
+       mov     w17,w28
+       mov     v2.16b,v26.16b
+       lsr     x19,x28,#32
+       mov     v6.16b,v26.16b
+       mov     w20,w30
+       mov     v18.16b,v26.16b
+       lsr     x21,x30,#32
+
+       mov     x4,#10
+       subs    x2,x2,#256
+.Loop_neon:
+       sub     x4,x4,#1
+       add     v0.4s,v0.4s,v1.4s
+       add     w5,w5,w9
+       add     v4.4s,v4.4s,v5.4s
+       add     w6,w6,w10
+       add     v16.4s,v16.4s,v17.4s
+       add     w7,w7,w11
+       eor     v3.16b,v3.16b,v0.16b
+       add     w8,w8,w12
+       eor     v7.16b,v7.16b,v4.16b
+       eor     w17,w17,w5
+       eor     v19.16b,v19.16b,v16.16b
+       eor     w19,w19,w6
+       rev32   v3.8h,v3.8h
+       eor     w20,w20,w7
+       rev32   v7.8h,v7.8h
+       eor     w21,w21,w8
+       rev32   v19.8h,v19.8h
+       ror     w17,w17,#16
+       add     v2.4s,v2.4s,v3.4s
+       ror     w19,w19,#16
+       add     v6.4s,v6.4s,v7.4s
+       ror     w20,w20,#16
+       add     v18.4s,v18.4s,v19.4s
+       ror     w21,w21,#16
+       eor     v20.16b,v1.16b,v2.16b
+       add     w13,w13,w17
+       eor     v21.16b,v5.16b,v6.16b
+       add     w14,w14,w19
+       eor     v22.16b,v17.16b,v18.16b
+       add     w15,w15,w20
+       ushr    v1.4s,v20.4s,#20
+       add     w16,w16,w21
+       ushr    v5.4s,v21.4s,#20
+       eor     w9,w9,w13
+       ushr    v17.4s,v22.4s,#20
+       eor     w10,w10,w14
+       sli     v1.4s,v20.4s,#12
+       eor     w11,w11,w15
+       sli     v5.4s,v21.4s,#12
+       eor     w12,w12,w16
+       sli     v17.4s,v22.4s,#12
+       ror     w9,w9,#20
+       add     v0.4s,v0.4s,v1.4s
+       ror     w10,w10,#20
+       add     v4.4s,v4.4s,v5.4s
+       ror     w11,w11,#20
+       add     v16.4s,v16.4s,v17.4s
+       ror     w12,w12,#20
+       eor     v20.16b,v3.16b,v0.16b
+       add     w5,w5,w9
+       eor     v21.16b,v7.16b,v4.16b
+       add     w6,w6,w10
+       eor     v22.16b,v19.16b,v16.16b
+       add     w7,w7,w11
+       ushr    v3.4s,v20.4s,#24
+       add     w8,w8,w12
+       ushr    v7.4s,v21.4s,#24
+       eor     w17,w17,w5
+       ushr    v19.4s,v22.4s,#24
+       eor     w19,w19,w6
+       sli     v3.4s,v20.4s,#8
+       eor     w20,w20,w7
+       sli     v7.4s,v21.4s,#8
+       eor     w21,w21,w8
+       sli     v19.4s,v22.4s,#8
+       ror     w17,w17,#24
+       add     v2.4s,v2.4s,v3.4s
+       ror     w19,w19,#24
+       add     v6.4s,v6.4s,v7.4s
+       ror     w20,w20,#24
+       add     v18.4s,v18.4s,v19.4s
+       ror     w21,w21,#24
+       eor     v20.16b,v1.16b,v2.16b
+       add     w13,w13,w17
+       eor     v21.16b,v5.16b,v6.16b
+       add     w14,w14,w19
+       eor     v22.16b,v17.16b,v18.16b
+       add     w15,w15,w20
+       ushr    v1.4s,v20.4s,#25
+       add     w16,w16,w21
+       ushr    v5.4s,v21.4s,#25
+       eor     w9,w9,w13
+       ushr    v17.4s,v22.4s,#25
+       eor     w10,w10,w14
+       sli     v1.4s,v20.4s,#7
+       eor     w11,w11,w15
+       sli     v5.4s,v21.4s,#7
+       eor     w12,w12,w16
+       sli     v17.4s,v22.4s,#7
+       ror     w9,w9,#25
+       ext     v2.16b,v2.16b,v2.16b,#8
+       ror     w10,w10,#25
+       ext     v6.16b,v6.16b,v6.16b,#8
+       ror     w11,w11,#25
+       ext     v18.16b,v18.16b,v18.16b,#8
+       ror     w12,w12,#25
+       ext     v3.16b,v3.16b,v3.16b,#12
+       ext     v7.16b,v7.16b,v7.16b,#12
+       ext     v19.16b,v19.16b,v19.16b,#12
+       ext     v1.16b,v1.16b,v1.16b,#4
+       ext     v5.16b,v5.16b,v5.16b,#4
+       ext     v17.16b,v17.16b,v17.16b,#4
+       add     v0.4s,v0.4s,v1.4s
+       add     w5,w5,w10
+       add     v4.4s,v4.4s,v5.4s
+       add     w6,w6,w11
+       add     v16.4s,v16.4s,v17.4s
+       add     w7,w7,w12
+       eor     v3.16b,v3.16b,v0.16b
+       add     w8,w8,w9
+       eor     v7.16b,v7.16b,v4.16b
+       eor     w21,w21,w5
+       eor     v19.16b,v19.16b,v16.16b
+       eor     w17,w17,w6
+       rev32   v3.8h,v3.8h
+       eor     w19,w19,w7
+       rev32   v7.8h,v7.8h
+       eor     w20,w20,w8
+       rev32   v19.8h,v19.8h
+       ror     w21,w21,#16
+       add     v2.4s,v2.4s,v3.4s
+       ror     w17,w17,#16
+       add     v6.4s,v6.4s,v7.4s
+       ror     w19,w19,#16
+       add     v18.4s,v18.4s,v19.4s
+       ror     w20,w20,#16
+       eor     v20.16b,v1.16b,v2.16b
+       add     w15,w15,w21
+       eor     v21.16b,v5.16b,v6.16b
+       add     w16,w16,w17
+       eor     v22.16b,v17.16b,v18.16b
+       add     w13,w13,w19
+       ushr    v1.4s,v20.4s,#20
+       add     w14,w14,w20
+       ushr    v5.4s,v21.4s,#20
+       eor     w10,w10,w15
+       ushr    v17.4s,v22.4s,#20
+       eor     w11,w11,w16
+       sli     v1.4s,v20.4s,#12
+       eor     w12,w12,w13
+       sli     v5.4s,v21.4s,#12
+       eor     w9,w9,w14
+       sli     v17.4s,v22.4s,#12
+       ror     w10,w10,#20
+       add     v0.4s,v0.4s,v1.4s
+       ror     w11,w11,#20
+       add     v4.4s,v4.4s,v5.4s
+       ror     w12,w12,#20
+       add     v16.4s,v16.4s,v17.4s
+       ror     w9,w9,#20
+       eor     v20.16b,v3.16b,v0.16b
+       add     w5,w5,w10
+       eor     v21.16b,v7.16b,v4.16b
+       add     w6,w6,w11
+       eor     v22.16b,v19.16b,v16.16b
+       add     w7,w7,w12
+       ushr    v3.4s,v20.4s,#24
+       add     w8,w8,w9
+       ushr    v7.4s,v21.4s,#24
+       eor     w21,w21,w5
+       ushr    v19.4s,v22.4s,#24
+       eor     w17,w17,w6
+       sli     v3.4s,v20.4s,#8
+       eor     w19,w19,w7
+       sli     v7.4s,v21.4s,#8
+       eor     w20,w20,w8
+       sli     v19.4s,v22.4s,#8
+       ror     w21,w21,#24
+       add     v2.4s,v2.4s,v3.4s
+       ror     w17,w17,#24
+       add     v6.4s,v6.4s,v7.4s
+       ror     w19,w19,#24
+       add     v18.4s,v18.4s,v19.4s
+       ror     w20,w20,#24
+       eor     v20.16b,v1.16b,v2.16b
+       add     w15,w15,w21
+       eor     v21.16b,v5.16b,v6.16b
+       add     w16,w16,w17
+       eor     v22.16b,v17.16b,v18.16b
+       add     w13,w13,w19
+       ushr    v1.4s,v20.4s,#25
+       add     w14,w14,w20
+       ushr    v5.4s,v21.4s,#25
+       eor     w10,w10,w15
+       ushr    v17.4s,v22.4s,#25
+       eor     w11,w11,w16
+       sli     v1.4s,v20.4s,#7
+       eor     w12,w12,w13
+       sli     v5.4s,v21.4s,#7
+       eor     w9,w9,w14
+       sli     v17.4s,v22.4s,#7
+       ror     w10,w10,#25
+       ext     v2.16b,v2.16b,v2.16b,#8
+       ror     w11,w11,#25
+       ext     v6.16b,v6.16b,v6.16b,#8
+       ror     w12,w12,#25
+       ext     v18.16b,v18.16b,v18.16b,#8
+       ror     w9,w9,#25
+       ext     v3.16b,v3.16b,v3.16b,#4
+       ext     v7.16b,v7.16b,v7.16b,#4
+       ext     v19.16b,v19.16b,v19.16b,#4
+       ext     v1.16b,v1.16b,v1.16b,#12
+       ext     v5.16b,v5.16b,v5.16b,#12
+       ext     v17.16b,v17.16b,v17.16b,#12
+       cbnz    x4,.Loop_neon
+
+       add     w5,w5,w22               // accumulate key block
+       add     v0.4s,v0.4s,v24.4s
+       add     x6,x6,x22,lsr#32
+       add     v4.4s,v4.4s,v24.4s
+       add     w7,w7,w23
+       add     v16.4s,v16.4s,v24.4s
+       add     x8,x8,x23,lsr#32
+       add     v2.4s,v2.4s,v26.4s
+       add     w9,w9,w24
+       add     v6.4s,v6.4s,v26.4s
+       add     x10,x10,x24,lsr#32
+       add     v18.4s,v18.4s,v26.4s
+       add     w11,w11,w25
+       add     v3.4s,v3.4s,v27.4s
+       add     x12,x12,x25,lsr#32
+       add     w13,w13,w26
+       add     v7.4s,v7.4s,v28.4s
+       add     x14,x14,x26,lsr#32
+       add     w15,w15,w27
+       add     v19.4s,v19.4s,v29.4s
+       add     x16,x16,x27,lsr#32
+       add     w17,w17,w28
+       add     v1.4s,v1.4s,v25.4s
+       add     x19,x19,x28,lsr#32
+       add     w20,w20,w30
+       add     v5.4s,v5.4s,v25.4s
+       add     x21,x21,x30,lsr#32
+       add     v17.4s,v17.4s,v25.4s
+
+       b.lo    .Ltail_neon
+
+       add     x5,x5,x6,lsl#32 // pack
+       add     x7,x7,x8,lsl#32
+       ldp     x6,x8,[x1,#0]           // load input
+       add     x9,x9,x10,lsl#32
+       add     x11,x11,x12,lsl#32
+       ldp     x10,x12,[x1,#16]
+       add     x13,x13,x14,lsl#32
+       add     x15,x15,x16,lsl#32
+       ldp     x14,x16,[x1,#32]
+       add     x17,x17,x19,lsl#32
+       add     x20,x20,x21,lsl#32
+       ldp     x19,x21,[x1,#48]
+       add     x1,x1,#64
+#ifdef __ARMEB__
+       rev     x5,x5
+       rev     x7,x7
+       rev     x9,x9
+       rev     x11,x11
+       rev     x13,x13
+       rev     x15,x15
+       rev     x17,x17
+       rev     x20,x20
+#endif
+       ld1     {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
+       eor     x5,x5,x6
+       eor     x7,x7,x8
+       eor     x9,x9,x10
+       eor     x11,x11,x12
+       eor     x13,x13,x14
+       eor     v0.16b,v0.16b,v20.16b
+       eor     x15,x15,x16
+       eor     v1.16b,v1.16b,v21.16b
+       eor     x17,x17,x19
+       eor     v2.16b,v2.16b,v22.16b
+       eor     x20,x20,x21
+       eor     v3.16b,v3.16b,v23.16b
+       ld1     {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
+
+       stp     x5,x7,[x0,#0]           // store output
+       add     x28,x28,#4                      // increment counter
+       stp     x9,x11,[x0,#16]
+       add     v27.4s,v27.4s,v31.4s            // += 4
+       stp     x13,x15,[x0,#32]
+       add     v28.4s,v28.4s,v31.4s
+       stp     x17,x20,[x0,#48]
+       add     v29.4s,v29.4s,v31.4s
+       add     x0,x0,#64
+
+       st1     {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
+       ld1     {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64
+
+       eor     v4.16b,v4.16b,v20.16b
+       eor     v5.16b,v5.16b,v21.16b
+       eor     v6.16b,v6.16b,v22.16b
+       eor     v7.16b,v7.16b,v23.16b
+       st1     {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
+
+       eor     v16.16b,v16.16b,v0.16b
+       eor     v17.16b,v17.16b,v1.16b
+       eor     v18.16b,v18.16b,v2.16b
+       eor     v19.16b,v19.16b,v3.16b
+       st1     {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64
+
+       b.hi    .Loop_outer_neon
+
+       ldp     x19,x20,[x29,#16]
+       add     sp,sp,#64
+       ldp     x21,x22,[x29,#32]
+       ldp     x23,x24,[x29,#48]
+       ldp     x25,x26,[x29,#64]
+       ldp     x27,x28,[x29,#80]
+       ldp     x29,x30,[sp],#96
+       ret
+
+.Ltail_neon:
+       add     x2,x2,#256
+       cmp     x2,#64
+       b.lo    .Less_than_64
+
+       add     x5,x5,x6,lsl#32 // pack
+       add     x7,x7,x8,lsl#32
+       ldp     x6,x8,[x1,#0]           // load input
+       add     x9,x9,x10,lsl#32
+       add     x11,x11,x12,lsl#32
+       ldp     x10,x12,[x1,#16]
+       add     x13,x13,x14,lsl#32
+       add     x15,x15,x16,lsl#32
+       ldp     x14,x16,[x1,#32]
+       add     x17,x17,x19,lsl#32
+       add     x20,x20,x21,lsl#32
+       ldp     x19,x21,[x1,#48]
+       add     x1,x1,#64
+#ifdef __ARMEB__
+       rev     x5,x5
+       rev     x7,x7
+       rev     x9,x9
+       rev     x11,x11
+       rev     x13,x13
+       rev     x15,x15
+       rev     x17,x17
+       rev     x20,x20
+#endif
+       eor     x5,x5,x6
+       eor     x7,x7,x8
+       eor     x9,x9,x10
+       eor     x11,x11,x12
+       eor     x13,x13,x14
+       eor     x15,x15,x16
+       eor     x17,x17,x19
+       eor     x20,x20,x21
+
+       stp     x5,x7,[x0,#0]           // store output
+       add     x28,x28,#4                      // increment counter
+       stp     x9,x11,[x0,#16]
+       stp     x13,x15,[x0,#32]
+       stp     x17,x20,[x0,#48]
+       add     x0,x0,#64
+       b.eq    .Ldone_neon
+       sub     x2,x2,#64
+       cmp     x2,#64
+       b.lo    .Less_than_128
+
+       ld1     {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
+       eor     v0.16b,v0.16b,v20.16b
+       eor     v1.16b,v1.16b,v21.16b
+       eor     v2.16b,v2.16b,v22.16b
+       eor     v3.16b,v3.16b,v23.16b
+       st1     {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
+       b.eq    .Ldone_neon
+       sub     x2,x2,#64
+       cmp     x2,#64
+       b.lo    .Less_than_192
+
+       ld1     {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
+       eor     v4.16b,v4.16b,v20.16b
+       eor     v5.16b,v5.16b,v21.16b
+       eor     v6.16b,v6.16b,v22.16b
+       eor     v7.16b,v7.16b,v23.16b
+       st1     {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
+       b.eq    .Ldone_neon
+       sub     x2,x2,#64
+
+       st1     {v16.16b,v17.16b,v18.16b,v19.16b},[sp]
+       b       .Last_neon
+
+.Less_than_128:
+       st1     {v0.16b,v1.16b,v2.16b,v3.16b},[sp]
+       b       .Last_neon
+.Less_than_192:
+       st1     {v4.16b,v5.16b,v6.16b,v7.16b},[sp]
+       b       .Last_neon
+
+.align 4
+.Last_neon:
+       sub     x0,x0,#1
+       add     x1,x1,x2
+       add     x0,x0,x2
+       add     x4,sp,x2
+       neg     x2,x2
+
+.Loop_tail_neon:
+       ldrb    w10,[x1,x2]
+       ldrb    w11,[x4,x2]
+       add     x2,x2,#1
+       eor     w10,w10,w11
+       strb    w10,[x0,x2]
+       cbnz    x2,.Loop_tail_neon
+
+       stp     xzr,xzr,[sp,#0]
+       stp     xzr,xzr,[sp,#16]
+       stp     xzr,xzr,[sp,#32]
+       stp     xzr,xzr,[sp,#48]
+
+.Ldone_neon:
+       ldp     x19,x20,[x29,#16]
+       add     sp,sp,#64
+       ldp     x21,x22,[x29,#32]
+       ldp     x23,x24,[x29,#48]
+       ldp     x25,x26,[x29,#64]
+       ldp     x27,x28,[x29,#80]
+       ldp     x29,x30,[sp],#96
+       ret
+.size  ChaCha20_neon,.-ChaCha20_neon
+.type  ChaCha20_512_neon,%function
+.align 5
+ChaCha20_512_neon:
+       stp     x29,x30,[sp,#-96]!
+       add     x29,sp,#0
+
+       adr     x5,.Lsigma
+       stp     x19,x20,[sp,#16]
+       stp     x21,x22,[sp,#32]
+       stp     x23,x24,[sp,#48]
+       stp     x25,x26,[sp,#64]
+       stp     x27,x28,[sp,#80]
+
+.L512_or_more_neon:
+       sub     sp,sp,#128+64
+
+       ldp     x22,x23,[x5]            // load sigma
+       ld1     {v24.4s},[x5],#16
+       ldp     x24,x25,[x3]            // load key
+       ldp     x26,x27,[x3,#16]
+       ld1     {v25.4s,v26.4s},[x3]
+       ldp     x28,x30,[x4]            // load counter
+       ld1     {v27.4s},[x4]
+       ld1     {v31.4s},[x5]
+#ifdef __ARMEB__
+       rev64   v24.4s,v24.4s
+       ror     x24,x24,#32
+       ror     x25,x25,#32
+       ror     x26,x26,#32
+       ror     x27,x27,#32
+       ror     x28,x28,#32
+       ror     x30,x30,#32
+#endif
+       add     v27.4s,v27.4s,v31.4s            // += 1
+       stp     q24,q25,[sp,#0]         // off-load key block, invariant part
+       add     v27.4s,v27.4s,v31.4s            // not typo
+       str     q26,[sp,#32]
+       add     v28.4s,v27.4s,v31.4s
+       add     v29.4s,v28.4s,v31.4s
+       add     v30.4s,v29.4s,v31.4s
+       shl     v31.4s,v31.4s,#2                        // 1 -> 4
+
+       stp     d8,d9,[sp,#128+0]               // meet ABI requirements
+       stp     d10,d11,[sp,#128+16]
+       stp     d12,d13,[sp,#128+32]
+       stp     d14,d15,[sp,#128+48]
+
+       sub     x2,x2,#512                      // not typo
+
+.Loop_outer_512_neon:
+       mov     v0.16b,v24.16b
+       mov     v4.16b,v24.16b
+       mov     v8.16b,v24.16b
+       mov     v12.16b,v24.16b
+       mov     v16.16b,v24.16b
+       mov     v20.16b,v24.16b
+       mov     v1.16b,v25.16b
+       mov     w5,w22                  // unpack key block
+       mov     v5.16b,v25.16b
+       lsr     x6,x22,#32
+       mov     v9.16b,v25.16b
+       mov     w7,w23
+       mov     v13.16b,v25.16b
+       lsr     x8,x23,#32
+       mov     v17.16b,v25.16b
+       mov     w9,w24
+       mov     v21.16b,v25.16b
+       lsr     x10,x24,#32
+       mov     v3.16b,v27.16b
+       mov     w11,w25
+       mov     v7.16b,v28.16b
+       lsr     x12,x25,#32
+       mov     v11.16b,v29.16b
+       mov     w13,w26
+       mov     v15.16b,v30.16b
+       lsr     x14,x26,#32
+       mov     v2.16b,v26.16b
+       mov     w15,w27
+       mov     v6.16b,v26.16b
+       lsr     x16,x27,#32
+       add     v19.4s,v3.4s,v31.4s                     // +4
+       mov     w17,w28
+       add     v23.4s,v7.4s,v31.4s                     // +4
+       lsr     x19,x28,#32
+       mov     v10.16b,v26.16b
+       mov     w20,w30
+       mov     v14.16b,v26.16b
+       lsr     x21,x30,#32
+       mov     v18.16b,v26.16b
+       stp     q27,q28,[sp,#48]                // off-load key block, variable part
+       mov     v22.16b,v26.16b
+       str     q29,[sp,#80]
+
+       mov     x4,#5
+       subs    x2,x2,#512
+.Loop_upper_neon:
+       sub     x4,x4,#1
+       add     v0.4s,v0.4s,v1.4s
+       add     w5,w5,w9
+       add     v4.4s,v4.4s,v5.4s
+       add     w6,w6,w10
+       add     v8.4s,v8.4s,v9.4s
+       add     w7,w7,w11
+       add     v12.4s,v12.4s,v13.4s
+       add     w8,w8,w12
+       add     v16.4s,v16.4s,v17.4s
+       eor     w17,w17,w5
+       add     v20.4s,v20.4s,v21.4s
+       eor     w19,w19,w6
+       eor     v3.16b,v3.16b,v0.16b
+       eor     w20,w20,w7
+       eor     v7.16b,v7.16b,v4.16b
+       eor     w21,w21,w8
+       eor     v11.16b,v11.16b,v8.16b
+       ror     w17,w17,#16
+       eor     v15.16b,v15.16b,v12.16b
+       ror     w19,w19,#16
+       eor     v19.16b,v19.16b,v16.16b
+       ror     w20,w20,#16
+       eor     v23.16b,v23.16b,v20.16b
+       ror     w21,w21,#16
+       rev32   v3.8h,v3.8h
+       add     w13,w13,w17
+       rev32   v7.8h,v7.8h
+       add     w14,w14,w19
+       rev32   v11.8h,v11.8h
+       add     w15,w15,w20
+       rev32   v15.8h,v15.8h
+       add     w16,w16,w21
+       rev32   v19.8h,v19.8h
+       eor     w9,w9,w13
+       rev32   v23.8h,v23.8h
+       eor     w10,w10,w14
+       add     v2.4s,v2.4s,v3.4s
+       eor     w11,w11,w15
+       add     v6.4s,v6.4s,v7.4s
+       eor     w12,w12,w16
+       add     v10.4s,v10.4s,v11.4s
+       ror     w9,w9,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w10,w10,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w11,w11,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w12,w12,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w9
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w10
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w11
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w12
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w17,w17,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w19,w19,w6
+       ushr    v1.4s,v24.4s,#20
+       eor     w20,w20,w7
+       ushr    v5.4s,v25.4s,#20
+       eor     w21,w21,w8
+       ushr    v9.4s,v26.4s,#20
+       ror     w17,w17,#24
+       ushr    v13.4s,v27.4s,#20
+       ror     w19,w19,#24
+       ushr    v17.4s,v28.4s,#20
+       ror     w20,w20,#24
+       ushr    v21.4s,v29.4s,#20
+       ror     w21,w21,#24
+       sli     v1.4s,v24.4s,#12
+       add     w13,w13,w17
+       sli     v5.4s,v25.4s,#12
+       add     w14,w14,w19
+       sli     v9.4s,v26.4s,#12
+       add     w15,w15,w20
+       sli     v13.4s,v27.4s,#12
+       add     w16,w16,w21
+       sli     v17.4s,v28.4s,#12
+       eor     w9,w9,w13
+       sli     v21.4s,v29.4s,#12
+       eor     w10,w10,w14
+       add     v0.4s,v0.4s,v1.4s
+       eor     w11,w11,w15
+       add     v4.4s,v4.4s,v5.4s
+       eor     w12,w12,w16
+       add     v8.4s,v8.4s,v9.4s
+       ror     w9,w9,#25
+       add     v12.4s,v12.4s,v13.4s
+       ror     w10,w10,#25
+       add     v16.4s,v16.4s,v17.4s
+       ror     w11,w11,#25
+       add     v20.4s,v20.4s,v21.4s
+       ror     w12,w12,#25
+       eor     v24.16b,v3.16b,v0.16b
+       add     w5,w5,w10
+       eor     v25.16b,v7.16b,v4.16b
+       add     w6,w6,w11
+       eor     v26.16b,v11.16b,v8.16b
+       add     w7,w7,w12
+       eor     v27.16b,v15.16b,v12.16b
+       add     w8,w8,w9
+       eor     v28.16b,v19.16b,v16.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v23.16b,v20.16b
+       eor     w17,w17,w6
+       ushr    v3.4s,v24.4s,#24
+       eor     w19,w19,w7
+       ushr    v7.4s,v25.4s,#24
+       eor     w20,w20,w8
+       ushr    v11.4s,v26.4s,#24
+       ror     w21,w21,#16
+       ushr    v15.4s,v27.4s,#24
+       ror     w17,w17,#16
+       ushr    v19.4s,v28.4s,#24
+       ror     w19,w19,#16
+       ushr    v23.4s,v29.4s,#24
+       ror     w20,w20,#16
+       sli     v3.4s,v24.4s,#8
+       add     w15,w15,w21
+       sli     v7.4s,v25.4s,#8
+       add     w16,w16,w17
+       sli     v11.4s,v26.4s,#8
+       add     w13,w13,w19
+       sli     v15.4s,v27.4s,#8
+       add     w14,w14,w20
+       sli     v19.4s,v28.4s,#8
+       eor     w10,w10,w15
+       sli     v23.4s,v29.4s,#8
+       eor     w11,w11,w16
+       add     v2.4s,v2.4s,v3.4s
+       eor     w12,w12,w13
+       add     v6.4s,v6.4s,v7.4s
+       eor     w9,w9,w14
+       add     v10.4s,v10.4s,v11.4s
+       ror     w10,w10,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w11,w11,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w12,w12,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w9,w9,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w10
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w11
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w12
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w9
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w17,w17,w6
+       ushr    v1.4s,v24.4s,#25
+       eor     w19,w19,w7
+       ushr    v5.4s,v25.4s,#25
+       eor     w20,w20,w8
+       ushr    v9.4s,v26.4s,#25
+       ror     w21,w21,#24
+       ushr    v13.4s,v27.4s,#25
+       ror     w17,w17,#24
+       ushr    v17.4s,v28.4s,#25
+       ror     w19,w19,#24
+       ushr    v21.4s,v29.4s,#25
+       ror     w20,w20,#24
+       sli     v1.4s,v24.4s,#7
+       add     w15,w15,w21
+       sli     v5.4s,v25.4s,#7
+       add     w16,w16,w17
+       sli     v9.4s,v26.4s,#7
+       add     w13,w13,w19
+       sli     v13.4s,v27.4s,#7
+       add     w14,w14,w20
+       sli     v17.4s,v28.4s,#7
+       eor     w10,w10,w15
+       sli     v21.4s,v29.4s,#7
+       eor     w11,w11,w16
+       ext     v2.16b,v2.16b,v2.16b,#8
+       eor     w12,w12,w13
+       ext     v6.16b,v6.16b,v6.16b,#8
+       eor     w9,w9,w14
+       ext     v10.16b,v10.16b,v10.16b,#8
+       ror     w10,w10,#25
+       ext     v14.16b,v14.16b,v14.16b,#8
+       ror     w11,w11,#25
+       ext     v18.16b,v18.16b,v18.16b,#8
+       ror     w12,w12,#25
+       ext     v22.16b,v22.16b,v22.16b,#8
+       ror     w9,w9,#25
+       ext     v3.16b,v3.16b,v3.16b,#12
+       ext     v7.16b,v7.16b,v7.16b,#12
+       ext     v11.16b,v11.16b,v11.16b,#12
+       ext     v15.16b,v15.16b,v15.16b,#12
+       ext     v19.16b,v19.16b,v19.16b,#12
+       ext     v23.16b,v23.16b,v23.16b,#12
+       ext     v1.16b,v1.16b,v1.16b,#4
+       ext     v5.16b,v5.16b,v5.16b,#4
+       ext     v9.16b,v9.16b,v9.16b,#4
+       ext     v13.16b,v13.16b,v13.16b,#4
+       ext     v17.16b,v17.16b,v17.16b,#4
+       ext     v21.16b,v21.16b,v21.16b,#4
+       add     v0.4s,v0.4s,v1.4s
+       add     w5,w5,w9
+       add     v4.4s,v4.4s,v5.4s
+       add     w6,w6,w10
+       add     v8.4s,v8.4s,v9.4s
+       add     w7,w7,w11
+       add     v12.4s,v12.4s,v13.4s
+       add     w8,w8,w12
+       add     v16.4s,v16.4s,v17.4s
+       eor     w17,w17,w5
+       add     v20.4s,v20.4s,v21.4s
+       eor     w19,w19,w6
+       eor     v3.16b,v3.16b,v0.16b
+       eor     w20,w20,w7
+       eor     v7.16b,v7.16b,v4.16b
+       eor     w21,w21,w8
+       eor     v11.16b,v11.16b,v8.16b
+       ror     w17,w17,#16
+       eor     v15.16b,v15.16b,v12.16b
+       ror     w19,w19,#16
+       eor     v19.16b,v19.16b,v16.16b
+       ror     w20,w20,#16
+       eor     v23.16b,v23.16b,v20.16b
+       ror     w21,w21,#16
+       rev32   v3.8h,v3.8h
+       add     w13,w13,w17
+       rev32   v7.8h,v7.8h
+       add     w14,w14,w19
+       rev32   v11.8h,v11.8h
+       add     w15,w15,w20
+       rev32   v15.8h,v15.8h
+       add     w16,w16,w21
+       rev32   v19.8h,v19.8h
+       eor     w9,w9,w13
+       rev32   v23.8h,v23.8h
+       eor     w10,w10,w14
+       add     v2.4s,v2.4s,v3.4s
+       eor     w11,w11,w15
+       add     v6.4s,v6.4s,v7.4s
+       eor     w12,w12,w16
+       add     v10.4s,v10.4s,v11.4s
+       ror     w9,w9,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w10,w10,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w11,w11,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w12,w12,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w9
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w10
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w11
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w12
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w17,w17,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w19,w19,w6
+       ushr    v1.4s,v24.4s,#20
+       eor     w20,w20,w7
+       ushr    v5.4s,v25.4s,#20
+       eor     w21,w21,w8
+       ushr    v9.4s,v26.4s,#20
+       ror     w17,w17,#24
+       ushr    v13.4s,v27.4s,#20
+       ror     w19,w19,#24
+       ushr    v17.4s,v28.4s,#20
+       ror     w20,w20,#24
+       ushr    v21.4s,v29.4s,#20
+       ror     w21,w21,#24
+       sli     v1.4s,v24.4s,#12
+       add     w13,w13,w17
+       sli     v5.4s,v25.4s,#12
+       add     w14,w14,w19
+       sli     v9.4s,v26.4s,#12
+       add     w15,w15,w20
+       sli     v13.4s,v27.4s,#12
+       add     w16,w16,w21
+       sli     v17.4s,v28.4s,#12
+       eor     w9,w9,w13
+       sli     v21.4s,v29.4s,#12
+       eor     w10,w10,w14
+       add     v0.4s,v0.4s,v1.4s
+       eor     w11,w11,w15
+       add     v4.4s,v4.4s,v5.4s
+       eor     w12,w12,w16
+       add     v8.4s,v8.4s,v9.4s
+       ror     w9,w9,#25
+       add     v12.4s,v12.4s,v13.4s
+       ror     w10,w10,#25
+       add     v16.4s,v16.4s,v17.4s
+       ror     w11,w11,#25
+       add     v20.4s,v20.4s,v21.4s
+       ror     w12,w12,#25
+       eor     v24.16b,v3.16b,v0.16b
+       add     w5,w5,w10
+       eor     v25.16b,v7.16b,v4.16b
+       add     w6,w6,w11
+       eor     v26.16b,v11.16b,v8.16b
+       add     w7,w7,w12
+       eor     v27.16b,v15.16b,v12.16b
+       add     w8,w8,w9
+       eor     v28.16b,v19.16b,v16.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v23.16b,v20.16b
+       eor     w17,w17,w6
+       ushr    v3.4s,v24.4s,#24
+       eor     w19,w19,w7
+       ushr    v7.4s,v25.4s,#24
+       eor     w20,w20,w8
+       ushr    v11.4s,v26.4s,#24
+       ror     w21,w21,#16
+       ushr    v15.4s,v27.4s,#24
+       ror     w17,w17,#16
+       ushr    v19.4s,v28.4s,#24
+       ror     w19,w19,#16
+       ushr    v23.4s,v29.4s,#24
+       ror     w20,w20,#16
+       sli     v3.4s,v24.4s,#8
+       add     w15,w15,w21
+       sli     v7.4s,v25.4s,#8
+       add     w16,w16,w17
+       sli     v11.4s,v26.4s,#8
+       add     w13,w13,w19
+       sli     v15.4s,v27.4s,#8
+       add     w14,w14,w20
+       sli     v19.4s,v28.4s,#8
+       eor     w10,w10,w15
+       sli     v23.4s,v29.4s,#8
+       eor     w11,w11,w16
+       add     v2.4s,v2.4s,v3.4s
+       eor     w12,w12,w13
+       add     v6.4s,v6.4s,v7.4s
+       eor     w9,w9,w14
+       add     v10.4s,v10.4s,v11.4s
+       ror     w10,w10,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w11,w11,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w12,w12,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w9,w9,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w10
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w11
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w12
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w9
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w17,w17,w6
+       ushr    v1.4s,v24.4s,#25
+       eor     w19,w19,w7
+       ushr    v5.4s,v25.4s,#25
+       eor     w20,w20,w8
+       ushr    v9.4s,v26.4s,#25
+       ror     w21,w21,#24
+       ushr    v13.4s,v27.4s,#25
+       ror     w17,w17,#24
+       ushr    v17.4s,v28.4s,#25
+       ror     w19,w19,#24
+       ushr    v21.4s,v29.4s,#25
+       ror     w20,w20,#24
+       sli     v1.4s,v24.4s,#7
+       add     w15,w15,w21
+       sli     v5.4s,v25.4s,#7
+       add     w16,w16,w17
+       sli     v9.4s,v26.4s,#7
+       add     w13,w13,w19
+       sli     v13.4s,v27.4s,#7
+       add     w14,w14,w20
+       sli     v17.4s,v28.4s,#7
+       eor     w10,w10,w15
+       sli     v21.4s,v29.4s,#7
+       eor     w11,w11,w16
+       ext     v2.16b,v2.16b,v2.16b,#8
+       eor     w12,w12,w13
+       ext     v6.16b,v6.16b,v6.16b,#8
+       eor     w9,w9,w14
+       ext     v10.16b,v10.16b,v10.16b,#8
+       ror     w10,w10,#25
+       ext     v14.16b,v14.16b,v14.16b,#8
+       ror     w11,w11,#25
+       ext     v18.16b,v18.16b,v18.16b,#8
+       ror     w12,w12,#25
+       ext     v22.16b,v22.16b,v22.16b,#8
+       ror     w9,w9,#25
+       ext     v3.16b,v3.16b,v3.16b,#4
+       ext     v7.16b,v7.16b,v7.16b,#4
+       ext     v11.16b,v11.16b,v11.16b,#4
+       ext     v15.16b,v15.16b,v15.16b,#4
+       ext     v19.16b,v19.16b,v19.16b,#4
+       ext     v23.16b,v23.16b,v23.16b,#4
+       ext     v1.16b,v1.16b,v1.16b,#12
+       ext     v5.16b,v5.16b,v5.16b,#12
+       ext     v9.16b,v9.16b,v9.16b,#12
+       ext     v13.16b,v13.16b,v13.16b,#12
+       ext     v17.16b,v17.16b,v17.16b,#12
+       ext     v21.16b,v21.16b,v21.16b,#12
+       cbnz    x4,.Loop_upper_neon
+
+       add     w5,w5,w22               // accumulate key block
+       add     x6,x6,x22,lsr#32
+       add     w7,w7,w23
+       add     x8,x8,x23,lsr#32
+       add     w9,w9,w24
+       add     x10,x10,x24,lsr#32
+       add     w11,w11,w25
+       add     x12,x12,x25,lsr#32
+       add     w13,w13,w26
+       add     x14,x14,x26,lsr#32
+       add     w15,w15,w27
+       add     x16,x16,x27,lsr#32
+       add     w17,w17,w28
+       add     x19,x19,x28,lsr#32
+       add     w20,w20,w30
+       add     x21,x21,x30,lsr#32
+
+       add     x5,x5,x6,lsl#32 // pack
+       add     x7,x7,x8,lsl#32
+       ldp     x6,x8,[x1,#0]           // load input
+       add     x9,x9,x10,lsl#32
+       add     x11,x11,x12,lsl#32
+       ldp     x10,x12,[x1,#16]
+       add     x13,x13,x14,lsl#32
+       add     x15,x15,x16,lsl#32
+       ldp     x14,x16,[x1,#32]
+       add     x17,x17,x19,lsl#32
+       add     x20,x20,x21,lsl#32
+       ldp     x19,x21,[x1,#48]
+       add     x1,x1,#64
+#ifdef __ARMEB__
+       rev     x5,x5
+       rev     x7,x7
+       rev     x9,x9
+       rev     x11,x11
+       rev     x13,x13
+       rev     x15,x15
+       rev     x17,x17
+       rev     x20,x20
+#endif
+       eor     x5,x5,x6
+       eor     x7,x7,x8
+       eor     x9,x9,x10
+       eor     x11,x11,x12
+       eor     x13,x13,x14
+       eor     x15,x15,x16
+       eor     x17,x17,x19
+       eor     x20,x20,x21
+
+       stp     x5,x7,[x0,#0]           // store output
+       add     x28,x28,#1                      // increment counter
+       mov     w5,w22                  // unpack key block
+       lsr     x6,x22,#32
+       stp     x9,x11,[x0,#16]
+       mov     w7,w23
+       lsr     x8,x23,#32
+       stp     x13,x15,[x0,#32]
+       mov     w9,w24
+       lsr     x10,x24,#32
+       stp     x17,x20,[x0,#48]
+       add     x0,x0,#64
+       mov     w11,w25
+       lsr     x12,x25,#32
+       mov     w13,w26
+       lsr     x14,x26,#32
+       mov     w15,w27
+       lsr     x16,x27,#32
+       mov     w17,w28
+       lsr     x19,x28,#32
+       mov     w20,w30
+       lsr     x21,x30,#32
+
+       mov     x4,#5
+.Loop_lower_neon:
+       sub     x4,x4,#1
+       add     v0.4s,v0.4s,v1.4s
+       add     w5,w5,w9
+       add     v4.4s,v4.4s,v5.4s
+       add     w6,w6,w10
+       add     v8.4s,v8.4s,v9.4s
+       add     w7,w7,w11
+       add     v12.4s,v12.4s,v13.4s
+       add     w8,w8,w12
+       add     v16.4s,v16.4s,v17.4s
+       eor     w17,w17,w5
+       add     v20.4s,v20.4s,v21.4s
+       eor     w19,w19,w6
+       eor     v3.16b,v3.16b,v0.16b
+       eor     w20,w20,w7
+       eor     v7.16b,v7.16b,v4.16b
+       eor     w21,w21,w8
+       eor     v11.16b,v11.16b,v8.16b
+       ror     w17,w17,#16
+       eor     v15.16b,v15.16b,v12.16b
+       ror     w19,w19,#16
+       eor     v19.16b,v19.16b,v16.16b
+       ror     w20,w20,#16
+       eor     v23.16b,v23.16b,v20.16b
+       ror     w21,w21,#16
+       rev32   v3.8h,v3.8h
+       add     w13,w13,w17
+       rev32   v7.8h,v7.8h
+       add     w14,w14,w19
+       rev32   v11.8h,v11.8h
+       add     w15,w15,w20
+       rev32   v15.8h,v15.8h
+       add     w16,w16,w21
+       rev32   v19.8h,v19.8h
+       eor     w9,w9,w13
+       rev32   v23.8h,v23.8h
+       eor     w10,w10,w14
+       add     v2.4s,v2.4s,v3.4s
+       eor     w11,w11,w15
+       add     v6.4s,v6.4s,v7.4s
+       eor     w12,w12,w16
+       add     v10.4s,v10.4s,v11.4s
+       ror     w9,w9,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w10,w10,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w11,w11,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w12,w12,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w9
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w10
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w11
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w12
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w17,w17,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w19,w19,w6
+       ushr    v1.4s,v24.4s,#20
+       eor     w20,w20,w7
+       ushr    v5.4s,v25.4s,#20
+       eor     w21,w21,w8
+       ushr    v9.4s,v26.4s,#20
+       ror     w17,w17,#24
+       ushr    v13.4s,v27.4s,#20
+       ror     w19,w19,#24
+       ushr    v17.4s,v28.4s,#20
+       ror     w20,w20,#24
+       ushr    v21.4s,v29.4s,#20
+       ror     w21,w21,#24
+       sli     v1.4s,v24.4s,#12
+       add     w13,w13,w17
+       sli     v5.4s,v25.4s,#12
+       add     w14,w14,w19
+       sli     v9.4s,v26.4s,#12
+       add     w15,w15,w20
+       sli     v13.4s,v27.4s,#12
+       add     w16,w16,w21
+       sli     v17.4s,v28.4s,#12
+       eor     w9,w9,w13
+       sli     v21.4s,v29.4s,#12
+       eor     w10,w10,w14
+       add     v0.4s,v0.4s,v1.4s
+       eor     w11,w11,w15
+       add     v4.4s,v4.4s,v5.4s
+       eor     w12,w12,w16
+       add     v8.4s,v8.4s,v9.4s
+       ror     w9,w9,#25
+       add     v12.4s,v12.4s,v13.4s
+       ror     w10,w10,#25
+       add     v16.4s,v16.4s,v17.4s
+       ror     w11,w11,#25
+       add     v20.4s,v20.4s,v21.4s
+       ror     w12,w12,#25
+       eor     v24.16b,v3.16b,v0.16b
+       add     w5,w5,w10
+       eor     v25.16b,v7.16b,v4.16b
+       add     w6,w6,w11
+       eor     v26.16b,v11.16b,v8.16b
+       add     w7,w7,w12
+       eor     v27.16b,v15.16b,v12.16b
+       add     w8,w8,w9
+       eor     v28.16b,v19.16b,v16.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v23.16b,v20.16b
+       eor     w17,w17,w6
+       ushr    v3.4s,v24.4s,#24
+       eor     w19,w19,w7
+       ushr    v7.4s,v25.4s,#24
+       eor     w20,w20,w8
+       ushr    v11.4s,v26.4s,#24
+       ror     w21,w21,#16
+       ushr    v15.4s,v27.4s,#24
+       ror     w17,w17,#16
+       ushr    v19.4s,v28.4s,#24
+       ror     w19,w19,#16
+       ushr    v23.4s,v29.4s,#24
+       ror     w20,w20,#16
+       sli     v3.4s,v24.4s,#8
+       add     w15,w15,w21
+       sli     v7.4s,v25.4s,#8
+       add     w16,w16,w17
+       sli     v11.4s,v26.4s,#8
+       add     w13,w13,w19
+       sli     v15.4s,v27.4s,#8
+       add     w14,w14,w20
+       sli     v19.4s,v28.4s,#8
+       eor     w10,w10,w15
+       sli     v23.4s,v29.4s,#8
+       eor     w11,w11,w16
+       add     v2.4s,v2.4s,v3.4s
+       eor     w12,w12,w13
+       add     v6.4s,v6.4s,v7.4s
+       eor     w9,w9,w14
+       add     v10.4s,v10.4s,v11.4s
+       ror     w10,w10,#20
+       add     v14.4s,v14.4s,v15.4s
+       ror     w11,w11,#20
+       add     v18.4s,v18.4s,v19.4s
+       ror     w12,w12,#20
+       add     v22.4s,v22.4s,v23.4s
+       ror     w9,w9,#20
+       eor     v24.16b,v1.16b,v2.16b
+       add     w5,w5,w10
+       eor     v25.16b,v5.16b,v6.16b
+       add     w6,w6,w11
+       eor     v26.16b,v9.16b,v10.16b
+       add     w7,w7,w12
+       eor     v27.16b,v13.16b,v14.16b
+       add     w8,w8,w9
+       eor     v28.16b,v17.16b,v18.16b
+       eor     w21,w21,w5
+       eor     v29.16b,v21.16b,v22.16b
+       eor     w17,w17,w6
+       ushr    v1.4s,v24.4s,#25
+       eor     w19,w19,w7
+       ushr    v5.4s,v25.4s,#25
+       eor     w20,w20,w8
+       ushr    v9.4s,v26.4s,#25
+       ror     w21,w21,#24
+       ushr    v13.4s,v27.4s,#25
+       ror     w17,w17,#24
+       ushr    v17.4s,v28.4s,#25
+       ror     w19,w19,#24
+       ushr    v21.4s,v29.4s,#25
+       ror     w20,w20,#24
+       sli     v1.4s,v24.4s,#7
+       add     w15,w15,w21
+       sli     v5.4s,v25.4s,#7
+       add     w16,w16,w17
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2.19.0

Part of the diff is not shown here. View the raw message.

#43 lore raw

Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations

Quoted 20 lines · Ard Biesheuvel
Quoted 10 lines · Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation, and are included here in raw form without modification, so that subsequent commits that fix these up for the kernel can see how it has changed. This awkward commit splitting has been requested for the ARM[64] implementations in particular.

While this is CRYPTOGAMS code, the originating code for this happens to be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2

Quoted 8 lines

Based-on-code-from: Andy Polyakov <[email protected]>
Cc: Samuel Neves <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Jean-Philippe Aumasson <[email protected]>
Cc: Andy Polyakov <[email protected]>
Cc: Russell King <[email protected]>
Cc: linux-arm-kernel at lists.infradead.org

Quoted 4 lines

As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.

Note that I am getting bounces from LAKML because the patch is too big.

author maintainer
#44 lore raw

Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations

In reply to Ard Biesheuvel

Hi Ard,

Quoted 3 lines · Ard Biesheuvel

As I mentioned before, I'd prefer this to be based on the original .pl but if I am the only one objecting to this, I guess I can live with it.

We're working on that, actually. It's not obvious when it'll be ready to ship -- perhaps after the initial merge, but perhaps way sooner -- but that is something we're trying to do for arm/arm64/mips64.

Jason

author maintainer
#45 lore raw

PATCH v6 6/23 zinc: port Andy Polyakov's ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These port and prepare Andy Polyakov's implementations for the kernel, but don't actually wire up any of the code yet. The wiring will be done in a subsequent commit, since we'll need to merge these implementations with another one. We make a few small changes to the assembly:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed (prefixed with .L) to fit kernel conventions.
- Constants have been rearranged so that they are closer to the code that is using them. [ARM only]
- The neon code can jump to the scalar code when it makes sense to do

    so.

- The neon_512 function as a separate function has been removed, leaving the decision up to the main neon entry point. [ARM64 only]

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/chacha20/chacha20-arm-cryptogams.S   | 367 +++++++++---------
 lib/zinc/chacha20/chacha20-arm64-cryptogams.S |  57 +--
 2 files changed, 193 insertions(+), 231 deletions(-)
lib/zinc/chacha20/chacha20-arm-cryptogams.S +180 -187
index 05a3a9e6e93f..770bab469171 100644
@@ -1,9 +1,12 @@
 /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
 /*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ *
+ * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
  */
 
-#include "arm_arch.h"
+#include <linux/linkage.h>
 
 .text
 #if defined(__thumb2__) || defined(__clang__)
@@ -24,48 +27,25 @@
 .long	0x61707865,0x3320646e,0x79622d32,0x6b206574	@ endian-neutral
 .Lone:
 .long	1,0,0,0
-.Lrot8:
-.long	0x02010003,0x06050407
-#if __ARM_MAX_ARCH__>=7
-.LOPENSSL_armcap:
-.word   OPENSSL_armcap_P-.LChaCha20_ctr32
-#else
 .word	-1
-#endif
 
-.globl	ChaCha20_ctr32
-.type	ChaCha20_ctr32,%function
 .align	5
-ChaCha20_ctr32:
-.LChaCha20_ctr32:
+ENTRY(chacha20_arm)
 	ldr	r12,[sp,#0]		@ pull pointer to counter and nonce
 	stmdb	sp!,{r0-r2,r4-r11,lr}
-#if __ARM_ARCH__<7 && !defined(__thumb2__)
-	sub	r14,pc,#16		@ ChaCha20_ctr32
-#else
-	adr	r14,.LChaCha20_ctr32
-#endif
 	cmp	r2,#0			@ len==0?
-#ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	eq
 #endif
 	addeq	sp,sp,#4*3
-	beq	.Lno_data
-#if __ARM_MAX_ARCH__>=7
-	cmp	r2,#192			@ test len
-	bls	.Lshort
-	ldr	r4,[r14,#-24]
-	ldr	r4,[r14,r4]
-# ifdef	__APPLE__
-	ldr	r4,[r4]
-# endif
-	tst	r4,#ARMV7_NEON
-	bne	.LChaCha20_neon
-.Lshort:
-#endif
+	beq	.Lno_data_arm
 	ldmia	r12,{r4-r7}		@ load counter and nonce
 	sub	sp,sp,#4*(16)		@ off-load area
-	sub	r14,r14,#64		@ .Lsigma
+#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__)
+	sub	r14,pc,#100		@ .Lsigma
+#else
+	adr	r14,.Lsigma		@ .Lsigma
+#endif
 	stmdb	sp!,{r4-r7}		@ copy counter and nonce
 	ldmia	r3,{r4-r11}		@ load key
 	ldmia	r14,{r0-r3}		@ load sigma
@@ -191,7 +171,7 @@ ChaCha20_ctr32:
 	@ rx and second half@sp+4*(16+8)
 
 	cmp	r11,#64		@ done yet?
-#ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
 #endif
 	addlo	r12,sp,#4*(0)		@ shortcut or ...
@@ -202,49 +182,49 @@ ChaCha20_ctr32:
 	ldr	r8,[sp,#4*(0)]	@ load key material
 	ldr	r9,[sp,#4*(1)]
 
-#if __ARM_ARCH__>=6 || !defined(__ARMEB__)
-# if __ARM_ARCH__<7
+#if __LINUX_ARM_ARCH__ >= 6 || !defined(__ARMEB__)
+#if __LINUX_ARM_ARCH__ < 7
 	orr	r10,r12,r14
 	tst	r10,#3		@ are input and output aligned?
 	ldr	r10,[sp,#4*(2)]
 	bne	.Lunaligned
 	cmp	r11,#64		@ restore flags
-# else
+#else
 	ldr	r10,[sp,#4*(2)]
-# endif
+#endif
 	ldr	r11,[sp,#4*(3)]
 
 	add	r0,r0,r8	@ accumulate key material
 	add	r1,r1,r9
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r8,[r12],#16		@ load input
 	ldrhs	r9,[r12,#-12]
 
 	add	r2,r2,r10
 	add	r3,r3,r11
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r10,[r12,#-8]
 	ldrhs	r11,[r12,#-4]
-# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__)
 	rev	r0,r0
 	rev	r1,r1
 	rev	r2,r2
 	rev	r3,r3
-# endif
-# ifdef	__thumb2__
+#endif
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r0,r0,r8	@ xor with input
 	eorhs	r1,r1,r9
 	 add	r8,sp,#4*(4)
 	str	r0,[r14],#16		@ store output
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r2,r2,r10
 	eorhs	r3,r3,r11
 	 ldmia	r8,{r8-r11}	@ load key material
@@ -254,34 +234,34 @@ ChaCha20_ctr32:
 
 	add	r4,r8,r4,ror#13 @ accumulate key material
 	add	r5,r9,r5,ror#13
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r8,[r12],#16		@ load input
 	ldrhs	r9,[r12,#-12]
 	add	r6,r10,r6,ror#13
 	add	r7,r11,r7,ror#13
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r10,[r12,#-8]
 	ldrhs	r11,[r12,#-4]
-# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__)
 	rev	r4,r4
 	rev	r5,r5
 	rev	r6,r6
 	rev	r7,r7
-# endif
-# ifdef	__thumb2__
+#endif
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r4,r4,r8
 	eorhs	r5,r5,r9
 	 add	r8,sp,#4*(8)
 	str	r4,[r14],#16		@ store output
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r6,r6,r10
 	eorhs	r7,r7,r11
 	str	r5,[r14,#-12]
@@ -294,39 +274,39 @@ ChaCha20_ctr32:
 
 	add	r0,r0,r8	@ accumulate key material
 	add	r1,r1,r9
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r8,[r12],#16		@ load input
 	ldrhs	r9,[r12,#-12]
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hi
-# endif
+#endif
 	 strhi	r10,[sp,#4*(16+10)]	@ copy "rx" while at it
 	 strhi	r11,[sp,#4*(16+11)]	@ copy "rx" while at it
 	add	r2,r2,r10
 	add	r3,r3,r11
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r10,[r12,#-8]
 	ldrhs	r11,[r12,#-4]
-# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__)
 	rev	r0,r0
 	rev	r1,r1
 	rev	r2,r2
 	rev	r3,r3
-# endif
-# ifdef	__thumb2__
+#endif
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r0,r0,r8
 	eorhs	r1,r1,r9
 	 add	r8,sp,#4*(12)
 	str	r0,[r14],#16		@ store output
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r2,r2,r10
 	eorhs	r3,r3,r11
 	str	r1,[r14,#-12]
@@ -336,79 +316,79 @@ ChaCha20_ctr32:
 
 	add	r4,r8,r4,ror#24 @ accumulate key material
 	add	r5,r9,r5,ror#24
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hi
-# endif
+#endif
 	 addhi	r8,r8,#1		@ next counter value
 	 strhi	r8,[sp,#4*(12)]	@ save next counter value
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r8,[r12],#16		@ load input
 	ldrhs	r9,[r12,#-12]
 	add	r6,r10,r6,ror#24
 	add	r7,r11,r7,ror#24
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhs	r10,[r12,#-8]
 	ldrhs	r11,[r12,#-4]
-# if __ARM_ARCH__>=6 && defined(__ARMEB__)
+#if __LINUX_ARM_ARCH__ >= 6 && defined(__ARMEB__)
 	rev	r4,r4
 	rev	r5,r5
 	rev	r6,r6
 	rev	r7,r7
-# endif
-# ifdef	__thumb2__
+#endif
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r4,r4,r8
 	eorhs	r5,r5,r9
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	 it	ne
-# endif
+#endif
 	 ldrne	r8,[sp,#4*(32+2)]	@ re-load len
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	eorhs	r6,r6,r10
 	eorhs	r7,r7,r11
 	str	r4,[r14],#16		@ store output
 	str	r5,[r14,#-12]
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	hs
-# endif
+#endif
 	 subhs	r11,r8,#64		@ len-=64
 	str	r6,[r14,#-8]
 	str	r7,[r14,#-4]
 	bhi	.Loop_outer
 
 	beq	.Ldone
-# if __ARM_ARCH__<7
+#if __LINUX_ARM_ARCH__ < 7
 	b	.Ltail
 
 .align	4
 .Lunaligned:				@ unaligned endian-neutral path
 	cmp	r11,#64		@ restore flags
-# endif
 #endif
-#if __ARM_ARCH__<7
+#endif
+#if __LINUX_ARM_ARCH__ < 7
 	ldr	r11,[sp,#4*(3)]
 	add	r0,r8,r0	@ accumulate key material
 	add	r1,r9,r1
 	add	r2,r10,r2
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r8,r8,r8		@ zero or ...
 	ldrhsb	r8,[r12],#16			@ ... load input
 	eorlo	r9,r9,r9
 	ldrhsb	r9,[r12,#-12]
 
 	add	r3,r11,r3
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r10,r10,r10
 	ldrhsb	r10,[r12,#-8]
 	eorlo	r11,r11,r11
@@ -416,53 +396,53 @@ ChaCha20_ctr32:
 
 	eor	r0,r8,r0		@ xor with input (or zero)
 	eor	r1,r9,r1
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-15]		@ load more input
 	ldrhsb	r9,[r12,#-11]
 	eor	r2,r10,r2
 	 strb	r0,[r14],#16		@ store output
 	eor	r3,r11,r3
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-7]
 	ldrhsb	r11,[r12,#-3]
 	 strb	r1,[r14,#-12]
 	eor	r0,r8,r0,lsr#8
 	 strb	r2,[r14,#-8]
 	eor	r1,r9,r1,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-14]		@ load more input
 	ldrhsb	r9,[r12,#-10]
 	 strb	r3,[r14,#-4]
 	eor	r2,r10,r2,lsr#8
 	 strb	r0,[r14,#-15]
 	eor	r3,r11,r3,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-6]
 	ldrhsb	r11,[r12,#-2]
 	 strb	r1,[r14,#-11]
 	eor	r0,r8,r0,lsr#8
 	 strb	r2,[r14,#-7]
 	eor	r1,r9,r1,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-13]		@ load more input
 	ldrhsb	r9,[r12,#-9]
 	 strb	r3,[r14,#-3]
 	eor	r2,r10,r2,lsr#8
 	 strb	r0,[r14,#-14]
 	eor	r3,r11,r3,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-5]
 	ldrhsb	r11,[r12,#-1]
 	 strb	r1,[r14,#-10]
@@ -482,18 +462,18 @@ ChaCha20_ctr32:
 	add	r4,r8,r4,ror#13	@ accumulate key material
 	add	r5,r9,r5,ror#13
 	add	r6,r10,r6,ror#13
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r8,r8,r8		@ zero or ...
 	ldrhsb	r8,[r12],#16			@ ... load input
 	eorlo	r9,r9,r9
 	ldrhsb	r9,[r12,#-12]
 
 	add	r7,r11,r7,ror#13
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r10,r10,r10
 	ldrhsb	r10,[r12,#-8]
 	eorlo	r11,r11,r11
@@ -501,53 +481,53 @@ ChaCha20_ctr32:
 
 	eor	r4,r8,r4		@ xor with input (or zero)
 	eor	r5,r9,r5
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-15]		@ load more input
 	ldrhsb	r9,[r12,#-11]
 	eor	r6,r10,r6
 	 strb	r4,[r14],#16		@ store output
 	eor	r7,r11,r7
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-7]
 	ldrhsb	r11,[r12,#-3]
 	 strb	r5,[r14,#-12]
 	eor	r4,r8,r4,lsr#8
 	 strb	r6,[r14,#-8]
 	eor	r5,r9,r5,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-14]		@ load more input
 	ldrhsb	r9,[r12,#-10]
 	 strb	r7,[r14,#-4]
 	eor	r6,r10,r6,lsr#8
 	 strb	r4,[r14,#-15]
 	eor	r7,r11,r7,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-6]
 	ldrhsb	r11,[r12,#-2]
 	 strb	r5,[r14,#-11]
 	eor	r4,r8,r4,lsr#8
 	 strb	r6,[r14,#-7]
 	eor	r5,r9,r5,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-13]		@ load more input
 	ldrhsb	r9,[r12,#-9]
 	 strb	r7,[r14,#-3]
 	eor	r6,r10,r6,lsr#8
 	 strb	r4,[r14,#-14]
 	eor	r7,r11,r7,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-5]
 	ldrhsb	r11,[r12,#-1]
 	 strb	r5,[r14,#-10]
@@ -564,26 +544,26 @@ ChaCha20_ctr32:
 	add	r8,sp,#4*(4+4)
 	ldmia	r8,{r8-r11}		@ load key material
 	ldmia	r0,{r0-r7}		@ load second half
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hi
-# endif
+#endif
 	strhi	r10,[sp,#4*(16+10)]		@ copy "rx"
 	strhi	r11,[sp,#4*(16+11)]		@ copy "rx"
 	add	r0,r8,r0	@ accumulate key material
 	add	r1,r9,r1
 	add	r2,r10,r2
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r8,r8,r8		@ zero or ...
 	ldrhsb	r8,[r12],#16			@ ... load input
 	eorlo	r9,r9,r9
 	ldrhsb	r9,[r12,#-12]
 
 	add	r3,r11,r3
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r10,r10,r10
 	ldrhsb	r10,[r12,#-8]
 	eorlo	r11,r11,r11
@@ -591,53 +571,53 @@ ChaCha20_ctr32:
 
 	eor	r0,r8,r0		@ xor with input (or zero)
 	eor	r1,r9,r1
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-15]		@ load more input
 	ldrhsb	r9,[r12,#-11]
 	eor	r2,r10,r2
 	 strb	r0,[r14],#16		@ store output
 	eor	r3,r11,r3
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-7]
 	ldrhsb	r11,[r12,#-3]
 	 strb	r1,[r14,#-12]
 	eor	r0,r8,r0,lsr#8
 	 strb	r2,[r14,#-8]
 	eor	r1,r9,r1,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-14]		@ load more input
 	ldrhsb	r9,[r12,#-10]
 	 strb	r3,[r14,#-4]
 	eor	r2,r10,r2,lsr#8
 	 strb	r0,[r14,#-15]
 	eor	r3,r11,r3,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-6]
 	ldrhsb	r11,[r12,#-2]
 	 strb	r1,[r14,#-11]
 	eor	r0,r8,r0,lsr#8
 	 strb	r2,[r14,#-7]
 	eor	r1,r9,r1,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-13]		@ load more input
 	ldrhsb	r9,[r12,#-9]
 	 strb	r3,[r14,#-3]
 	eor	r2,r10,r2,lsr#8
 	 strb	r0,[r14,#-14]
 	eor	r3,r11,r3,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-5]
 	ldrhsb	r11,[r12,#-1]
 	 strb	r1,[r14,#-10]
@@ -654,25 +634,25 @@ ChaCha20_ctr32:
 	add	r8,sp,#4*(4+8)
 	ldmia	r8,{r8-r11}		@ load key material
 	add	r4,r8,r4,ror#24	@ accumulate key material
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hi
-# endif
+#endif
 	addhi	r8,r8,#1			@ next counter value
 	strhi	r8,[sp,#4*(12)]		@ save next counter value
 	add	r5,r9,r5,ror#24
 	add	r6,r10,r6,ror#24
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r8,r8,r8		@ zero or ...
 	ldrhsb	r8,[r12],#16			@ ... load input
 	eorlo	r9,r9,r9
 	ldrhsb	r9,[r12,#-12]
 
 	add	r7,r11,r7,ror#24
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itete	lo
-# endif
+#endif
 	eorlo	r10,r10,r10
 	ldrhsb	r10,[r12,#-8]
 	eorlo	r11,r11,r11
@@ -680,53 +660,53 @@ ChaCha20_ctr32:
 
 	eor	r4,r8,r4		@ xor with input (or zero)
 	eor	r5,r9,r5
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-15]		@ load more input
 	ldrhsb	r9,[r12,#-11]
 	eor	r6,r10,r6
 	 strb	r4,[r14],#16		@ store output
 	eor	r7,r11,r7
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-7]
 	ldrhsb	r11,[r12,#-3]
 	 strb	r5,[r14,#-12]
 	eor	r4,r8,r4,lsr#8
 	 strb	r6,[r14,#-8]
 	eor	r5,r9,r5,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-14]		@ load more input
 	ldrhsb	r9,[r12,#-10]
 	 strb	r7,[r14,#-4]
 	eor	r6,r10,r6,lsr#8
 	 strb	r4,[r14,#-15]
 	eor	r7,r11,r7,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-6]
 	ldrhsb	r11,[r12,#-2]
 	 strb	r5,[r14,#-11]
 	eor	r4,r8,r4,lsr#8
 	 strb	r6,[r14,#-7]
 	eor	r5,r9,r5,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r8,[r12,#-13]		@ load more input
 	ldrhsb	r9,[r12,#-9]
 	 strb	r7,[r14,#-3]
 	eor	r6,r10,r6,lsr#8
 	 strb	r4,[r14,#-14]
 	eor	r7,r11,r7,lsr#8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	itt	hs
-# endif
+#endif
 	ldrhsb	r10,[r12,#-5]
 	ldrhsb	r11,[r12,#-1]
 	 strb	r5,[r14,#-10]
@@ -740,13 +720,13 @@ ChaCha20_ctr32:
 	eor	r7,r11,r7,lsr#8
 	 strb	r6,[r14,#-5]
 	 strb	r7,[r14,#-1]
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	ne
-# endif
+#endif
 	ldrne	r8,[sp,#4*(32+2)]		@ re-load len
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	hs
-# endif
+#endif
 	subhs	r11,r8,#64			@ len-=64
 	bhi	.Loop_outer
 
@@ -768,20 +748,33 @@ ChaCha20_ctr32:
 
 .Ldone:
 	add	sp,sp,#4*(32+3)
-.Lno_data:
+.Lno_data_arm:
 	ldmia	sp!,{r4-r11,pc}
-.size	ChaCha20_ctr32,.-ChaCha20_ctr32
-#if __ARM_MAX_ARCH__>=7
+ENDPROC(chacha20_arm)
+
+#ifdef CONFIG_KERNEL_MODE_NEON
+.align	5
+.Lsigma2:
+.long	0x61707865,0x3320646e,0x79622d32,0x6b206574	@ endian-neutral
+.Lone2:
+.long	1,0,0,0
+.word	-1
+
 .arch	armv7-a
 .fpu	neon
 
-.type	ChaCha20_neon,%function
 .align	5
-ChaCha20_neon:
+ENTRY(chacha20_neon)
 	ldr		r12,[sp,#0]		@ pull pointer to counter and nonce
 	stmdb		sp!,{r0-r2,r4-r11,lr}
-.LChaCha20_neon:
-	adr		r14,.Lsigma
+	cmp		r2,#0			@ len==0?
+#ifdef	__thumb2__
+	itt		eq
+#endif
+	addeq		sp,sp,#4*3
+	beq		.Lno_data_neon
+.Lchacha20_neon_begin:
+	adr		r14,.Lsigma2
 	vstmdb		sp!,{d8-d15}		@ ABI spec says so
 	stmdb		sp!,{r0-r3}
 
@@ -1121,12 +1114,12 @@ ChaCha20_neon:
 	ldr		r10,[r12,#-8]
 	add		r3,r3,r11
 	ldr		r11,[r12,#-4]
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r0,r0
 	rev		r1,r1
 	rev		r2,r2
 	rev		r3,r3
-# endif
+#endif
 	eor		r0,r0,r8	@ xor with input
 	 add		r8,sp,#4*(4)
 	eor		r1,r1,r9
@@ -1146,12 +1139,12 @@ ChaCha20_neon:
 	ldr		r10,[r12,#-8]
 	add		r7,r11,r7,ror#13
 	ldr		r11,[r12,#-4]
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r4,r4
 	rev		r5,r5
 	rev		r6,r6
 	rev		r7,r7
-# endif
+#endif
 	eor		r4,r4,r8
 	 add		r8,sp,#4*(8)
 	eor		r5,r5,r9
@@ -1170,24 +1163,24 @@ ChaCha20_neon:
 	ldr		r8,[r12],#16		@ load input
 	add		r1,r1,r9
 	ldr		r9,[r12,#-12]
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	hi
-# endif
+#endif
 	 strhi		r10,[sp,#4*(16+10)]	@ copy "rx" while at it
 	add		r2,r2,r10
 	ldr		r10,[r12,#-8]
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	hi
-# endif
+#endif
 	 strhi		r11,[sp,#4*(16+11)]	@ copy "rx" while at it
 	add		r3,r3,r11
 	ldr		r11,[r12,#-4]
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r0,r0
 	rev		r1,r1
 	rev		r2,r2
 	rev		r3,r3
-# endif
+#endif
 	eor		r0,r0,r8
 	 add		r8,sp,#4*(12)
 	eor		r1,r1,r9
@@ -1210,16 +1203,16 @@ ChaCha20_neon:
 	add		r7,r11,r7,ror#24
 	ldr		r10,[r12,#-8]
 	ldr		r11,[r12,#-4]
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r4,r4
 	rev		r5,r5
 	rev		r6,r6
 	rev		r7,r7
-# endif
+#endif
 	eor		r4,r4,r8
-# ifdef	__thumb2__
+#ifdef __thumb2__
 	it	hi
-# endif
+#endif
 	 ldrhi		r8,[sp,#4*(32+2)]	@ re-load len
 	eor		r5,r5,r9
 	eor		r6,r6,r10
@@ -1379,7 +1372,7 @@ ChaCha20_neon:
 	add		r6,r10,r6,ror#13
 	add		r7,r11,r7,ror#13
 	 ldmia		r8,{r8-r11}	@ load key material
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r0,r0
 	rev		r1,r1
 	rev		r2,r2
@@ -1388,7 +1381,7 @@ ChaCha20_neon:
 	rev		r5,r5
 	rev		r6,r6
 	rev		r7,r7
-# endif
+#endif
 	stmia		sp,{r0-r7}
 	 add		r0,sp,#4*(16+8)
 
@@ -1408,7 +1401,7 @@ ChaCha20_neon:
 	add		r6,r10,r6,ror#24
 	add		r7,r11,r7,ror#24
 	 ldr		r11,[sp,#4*(32+2)]	@ re-load len
-# ifdef	__ARMEB__
+#ifdef	__ARMEB__
 	rev		r0,r0
 	rev		r1,r1
 	rev		r2,r2
@@ -1417,7 +1410,7 @@ ChaCha20_neon:
 	rev		r5,r5
 	rev		r6,r6
 	rev		r7,r7
-# endif
+#endif
 	stmia		r8,{r0-r7}
 	 add		r10,sp,#4*(0)
 	 sub		r11,r11,#64*3	@ len-=64*3
@@ -1434,7 +1427,7 @@ ChaCha20_neon:
 	add		sp,sp,#4*(32+4)
 	vldmia		sp,{d8-d15}
 	add		sp,sp,#4*(16+3)
+.Lno_data_neon:
 	ldmia		sp!,{r4-r11,pc}
-.size	ChaCha20_neon,.-ChaCha20_neon
-.comm	OPENSSL_armcap_P,4,4
+ENDPROC(chacha20_neon)
 #endif
lib/zinc/chacha20/chacha20-arm64-cryptogams.S +13 -44

94 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#46 lore raw

PATCH v6 7/23 zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).

This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.

Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.

This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:

- ARM1176JZF-S [ARMv6]

  - Cortex-A7    [ARMv7]
  - Cortex-A8    [ARMv7]
  - Cortex-A9    [ARMv7]
  - Cortex-A17   [ARMv7]
  - Cortex-A53   [ARMv8]
  - Cortex-A55   [ARMv8]
  - Cortex-A73   [ARMv8]
  - Cortex-A75   [ARMv8]

Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                             |   2 +
 lib/zinc/chacha20/chacha20-arm-glue.h         |  88 +++
 ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
 ...20-arm64-cryptogams.S => chacha20-arm64.S} |   0
 lib/zinc/chacha20/chacha20.c                  |   2 +
 5 files changed, 556 insertions(+), 38 deletions(-)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
 rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
 rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)
lib/zinc/Makefile +2 -0
index 223a0816c918..e47f64e12bbd 100644
@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +88 -0

88 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464 -38

527 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0 -0
similarity index 100%
rename from lib/zinc/chacha20/chacha20-arm64-cryptogams.S
rename to lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/chacha20/chacha20.c +2 -0
index 4354b874a6a5..fc4f74fca653 100644
@@ -16,6 +16,8 @@
 
 #if defined(CONFIG_ZINC_ARCH_X86_64)
 #include "chacha20-x86_64-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
+#include "chacha20-arm-glue.h"
 #else
 void __init chacha20_fpu_init(void)
 {
Signature · 1 line
2.19.0
#47 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 113 lines · Jason A. Donenfeld

These wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).

This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.

Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.

This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:

- ARM1176JZF-S [ARMv6]

  - Cortex-A7    [ARMv7]
  - Cortex-A8    [ARMv7]
  - Cortex-A9    [ARMv7]
  - Cortex-A17   [ARMv7]
  - Cortex-A53   [ARMv8]
  - Cortex-A55   [ARMv8]
  - Cortex-A73   [ARMv8]
  - Cortex-A75   [ARMv8]

Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                             |   2 +
 lib/zinc/chacha20/chacha20-arm-glue.h         |  88 +++
 ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
 ...20-arm64-cryptogams.S => chacha20-arm64.S} |   0
 lib/zinc/chacha20/chacha20.c                  |   2 +
 5 files changed, 556 insertions(+), 38 deletions(-)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
 rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
 rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)
lib/zinc/Makefile +2 -0
index 223a0816c918..e47f64e12bbd 100644
@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +43 -0
new file mode 100644
index 000000000000..86cce851ed02
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/hwcap.h>
+#include <asm/neon.h>
+#if defined(CONFIG_ARM)
+#include <asm/system_info.h>
+#include <asm/cputype.h>
+#endif
+
+asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len,
+                            const u32 key[8], const u32 counter[4]);
+#if defined(CONFIG_ARM)
+asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]);
+#endif
+#if defined(CONFIG_KERNEL_MODE_NEON)
+asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len,
+                             const u32 key[8], const u32 counter[4]);
+#endif
+
+static bool chacha20_use_neon __ro_after_init;
+
+static void __init chacha20_fpu_init(void)
+{
+#if defined(CONFIG_ARM64)
+       chacha20_use_neon = elf_hwcap & HWCAP_ASIMD;
+#elif defined(CONFIG_ARM)
+       switch (read_cpuid_part()) {
+       case ARM_CPU_PART_CORTEX_A7:
+       case ARM_CPU_PART_CORTEX_A5:
+               /* The Cortex-A7 and Cortex-A5 do not perform well with the NEON
+                * implementation but do incredibly with the scalar one and use
+                * less power.
+                */
+               break;
+       default:
+               chacha20_use_neon = elf_hwcap & HWCAP_NEON;
+       }
+#endif
+}
+

lib/zinc/chacha20/chacha20-arm-glue.h line 53

Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10 -0
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, size_t len,
+                                simd_context_t *simd_context)
+{
+#if defined(CONFIG_KERNEL_MODE_NEON)
+       if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))
+               chacha20_neon(dst, src, len, state->key, state->counter);
+       else
+#endif

Better to use IS_ENABLED() here:

lib/zinc/chacha20/chacha20-arm-glue.h line 56

Quoted 3 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +3 -0
+       if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) &&
+           chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))

Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.

Quoted 592 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +35 -0
+               chacha20_arm(dst, src, len, state->key, state->counter);
+
+       state->counter[0] += (len + 63) / 64;
+       return true;
+}
+
+static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
+                                 const u8 nonce[HCHACHA20_NONCE_SIZE],
+                                 const u8 key[HCHACHA20_KEY_SIZE],
+                                 simd_context_t *simd_context)
+{
+#if defined(CONFIG_ARM)
+       u32 x[] = { CHACHA20_CONSTANT_EXPA,
+                   CHACHA20_CONSTANT_ND_3,
+                   CHACHA20_CONSTANT_2_BY,
+                   CHACHA20_CONSTANT_TE_K,
+                   get_unaligned_le32(key + 0),
+                   get_unaligned_le32(key + 4),
+                   get_unaligned_le32(key + 8),
+                   get_unaligned_le32(key + 12),
+                   get_unaligned_le32(key + 16),
+                   get_unaligned_le32(key + 20),
+                   get_unaligned_le32(key + 24),
+                   get_unaligned_le32(key + 28),
+                   get_unaligned_le32(nonce + 0),
+                   get_unaligned_le32(nonce + 4),
+                   get_unaligned_le32(nonce + 8),
+                   get_unaligned_le32(nonce + 12)
+       };
+       hchacha20_arm(x, derived_key);
+       return true;
+#else
+       return false;
+#endif
+}
lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464 -38
similarity index 71%
rename from lib/zinc/chacha20/chacha20-arm-cryptogams.S
rename to lib/zinc/chacha20/chacha20-arm.S
index 770bab469171..5abedafcf129 100644
@@ -1,13 +1,475 @@
 /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
 /*
+ * Copyright (C) 2018 Google, Inc.
  * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
- *
- * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
  */
 
 #include <linux/linkage.h>
 
+/*
+ * The following scalar routine was written by Eric Biggers.
+ *
+ * Design notes:
+ *
+ * 16 registers would be needed to hold the state matrix, but only 14 are
+ * available because 'sp' and 'pc' cannot be used.  So we spill the elements
+ * (x8, x9) to the stack and swap them out with (x10, x11).  This adds one
+ * 'ldrd' and one 'strd' instruction per round.
+ *
+ * All rotates are performed using the implicit rotate operand accepted by the
+ * 'add' and 'eor' instructions.  This is faster than using explicit rotate
+ * instructions.  To make this work, we allow the values in the second and last
+ * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the
+ * wrong rotation amount.  The rotation amount is then fixed up just in time
+ * when the values are used.  'brot' is the number of bits the values in row 'b'
+ * need to be rotated right to arrive at the correct values, and 'drot'
+ * similarly for row 'd'.  (brot, drot) start out as (0, 0) but we make it such
+ * that they end up as (25, 24) after every round.
+ */
+
+       // ChaCha state registers
+       X0      .req    r0
+       X1      .req    r1
+       X2      .req    r2
+       X3      .req    r3
+       X4      .req    r4
+       X5      .req    r5
+       X6      .req    r6
+       X7      .req    r7
+       X8_X10  .req    r8      // shared by x8 and x10
+       X9_X11  .req    r9      // shared by x9 and x11
+       X12     .req    r10
+       X13     .req    r11
+       X14     .req    r12
+       X15     .req    r14
+
+.Lexpand_32byte_k:
+       // "expand 32-byte k"
+       .word   0x61707865, 0x3320646e, 0x79622d32, 0x6b206574
+
+#ifdef __thumb2__
+#  define adrl adr
+#endif
+
+.macro __rev           out, in,  t0, t1, t2
+.if __LINUX_ARM_ARCH__ >= 6
+       rev             \out, \in
+.else
+       lsl             \t0, \in, #24
+       and             \t1, \in, #0xff00
+       and             \t2, \in, #0xff0000
+       orr             \out, \t0, \in, lsr #24
+       orr             \out, \out, \t1, lsl #8
+       orr             \out, \out, \t2, lsr #8
+.endif
+.endm
+
+.macro _le32_bswap     x,  t0, t1, t2
+#ifdef __ARMEB__
+       __rev           \x, \x,  \t0, \t1, \t2
+#endif
+.endm
+
+.macro _le32_bswap_4x  a, b, c, d,  t0, t1, t2
+       _le32_bswap     \a,  \t0, \t1, \t2
+       _le32_bswap     \b,  \t0, \t1, \t2
+       _le32_bswap     \c,  \t0, \t1, \t2
+       _le32_bswap     \d,  \t0, \t1, \t2
+.endm
+
+.macro __ldrd          a, b, src, offset
+#if __LINUX_ARM_ARCH__ >= 6
+       ldrd            \a, \b, [\src, #\offset]
+#else
+       ldr             \a, [\src, #\offset]
+       ldr             \b, [\src, #\offset + 4]
+#endif
+.endm
+
+.macro __strd          a, b, dst, offset
+#if __LINUX_ARM_ARCH__ >= 6
+       strd            \a, \b, [\dst, #\offset]
+#else
+       str             \a, [\dst, #\offset]
+       str             \b, [\dst, #\offset + 4]
+#endif
+.endm
+
+.macro _halfround      a1, b1, c1, d1,  a2, b2, c2, d2
+
+       // a += b; d ^= a; d = rol(d, 16);
+       add             \a1, \a1, \b1, ror #brot
+       add             \a2, \a2, \b2, ror #brot
+       eor             \d1, \a1, \d1, ror #drot
+       eor             \d2, \a2, \d2, ror #drot
+       // drot == 32 - 16 == 16
+
+       // c += d; b ^= c; b = rol(b, 12);
+       add             \c1, \c1, \d1, ror #16
+       add             \c2, \c2, \d2, ror #16
+       eor             \b1, \c1, \b1, ror #brot
+       eor             \b2, \c2, \b2, ror #brot
+       // brot == 32 - 12 == 20
+
+       // a += b; d ^= a; d = rol(d, 8);
+       add             \a1, \a1, \b1, ror #20
+       add             \a2, \a2, \b2, ror #20
+       eor             \d1, \a1, \d1, ror #16
+       eor             \d2, \a2, \d2, ror #16
+       // drot == 32 - 8 == 24
+
+       // c += d; b ^= c; b = rol(b, 7);
+       add             \c1, \c1, \d1, ror #24
+       add             \c2, \c2, \d2, ror #24
+       eor             \b1, \c1, \b1, ror #20
+       eor             \b2, \c2, \b2, ror #20
+       // brot == 32 - 7 == 25
+.endm
+
+.macro _doubleround
+
+       // column round
+
+       // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13)
+       _halfround      X0, X4, X8_X10, X12,  X1, X5, X9_X11, X13
+
+       // save (x8, x9); restore (x10, x11)
+       __strd          X8_X10, X9_X11, sp, 0
+       __ldrd          X8_X10, X9_X11, sp, 8
+
+       // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15)
+       _halfround      X2, X6, X8_X10, X14,  X3, X7, X9_X11, X15
+
+       .set brot, 25
+       .set drot, 24
+
+       // diagonal round
+
+       // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12)
+       _halfround      X0, X5, X8_X10, X15,  X1, X6, X9_X11, X12
+
+       // save (x10, x11); restore (x8, x9)
+       __strd          X8_X10, X9_X11, sp, 8
+       __ldrd          X8_X10, X9_X11, sp, 0
+
+       // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14)
+       _halfround      X2, X7, X8_X10, X13,  X3, X4, X9_X11, X14
+.endm
+
+.macro _chacha_permute nrounds
+       .set brot, 0
+       .set drot, 0
+       .rept \nrounds / 2
+        _doubleround
+       .endr
+.endm
+
+.macro _chacha         nrounds
+
+.Lnext_block\@:
+       // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN
+       // Registers contain x0-x9,x12-x15.
+
+       // Do the core ChaCha permutation to update x0-x15.
+       _chacha_permute \nrounds
+
+       add             sp, #8
+       // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers contain x0-x9,x12-x15.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15).
+       push            {X8_X10, X9_X11, X12, X13, X14, X15}
+
+       // Load (OUT, IN, LEN).
+       ldr             r14, [sp, #96]
+       ldr             r12, [sp, #100]
+       ldr             r11, [sp, #104]
+
+       orr             r10, r14, r12
+
+       // Use slow path if fewer than 64 bytes remain.
+       cmp             r11, #64
+       blt             .Lxor_slowpath\@
+
+       // Use slow path if IN and/or OUT isn't 4-byte aligned.  Needed even on
+       // ARMv6+, since ldmia and stmia (used below) still require alignment.
+       tst             r10, #3
+       bne             .Lxor_slowpath\@
+
+       // Fast path: XOR 64 bytes of aligned data.
+
+       // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // x0-x3
+       __ldrd          r8, r9, sp, 32
+       __ldrd          r10, r11, sp, 40
+       add             X0, X0, r8
+       add             X1, X1, r9
+       add             X2, X2, r10
+       add             X3, X3, r11
+       _le32_bswap_4x  X0, X1, X2, X3,  r8, r9, r10
+       ldmia           r12!, {r8-r11}
+       eor             X0, X0, r8
+       eor             X1, X1, r9
+       eor             X2, X2, r10
+       eor             X3, X3, r11
+       stmia           r14!, {X0-X3}
+
+       // x4-x7
+       __ldrd          r8, r9, sp, 48
+       __ldrd          r10, r11, sp, 56
+       add             X4, r8, X4, ror #brot
+       add             X5, r9, X5, ror #brot
+       ldmia           r12!, {X0-X3}
+       add             X6, r10, X6, ror #brot
+       add             X7, r11, X7, ror #brot
+       _le32_bswap_4x  X4, X5, X6, X7,  r8, r9, r10
+       eor             X4, X4, X0
+       eor             X5, X5, X1
+       eor             X6, X6, X2
+       eor             X7, X7, X3
+       stmia           r14!, {X4-X7}
+
+       // x8-x15
+       pop             {r0-r7}                 // (x8-x9,x12-x15,x10-x11)
+       __ldrd          r8, r9, sp, 32
+       __ldrd          r10, r11, sp, 40
+       add             r0, r0, r8              // x8
+       add             r1, r1, r9              // x9
+       add             r6, r6, r10             // x10
+       add             r7, r7, r11             // x11
+       _le32_bswap_4x  r0, r1, r6, r7,  r8, r9, r10
+       ldmia           r12!, {r8-r11}
+       eor             r0, r0, r8              // x8
+       eor             r1, r1, r9              // x9
+       eor             r6, r6, r10             // x10
+       eor             r7, r7, r11             // x11
+       stmia           r14!, {r0,r1,r6,r7}
+       ldmia           r12!, {r0,r1,r6,r7}
+       __ldrd          r8, r9, sp, 48
+       __ldrd          r10, r11, sp, 56
+       add             r2, r8, r2, ror #drot   // x12
+       add             r3, r9, r3, ror #drot   // x13
+       add             r4, r10, r4, ror #drot  // x14
+       add             r5, r11, r5, ror #drot  // x15
+       _le32_bswap_4x  r2, r3, r4, r5,  r9, r10, r11
+         ldr           r9, [sp, #72]           // load LEN
+       eor             r2, r2, r0              // x12
+       eor             r3, r3, r1              // x13
+       eor             r4, r4, r6              // x14
+       eor             r5, r5, r7              // x15
+         subs          r9, #64                 // decrement and check LEN
+       stmia           r14!, {r2-r5}
+
+       beq             .Ldone\@
+
+.Lprepare_for_next_block\@:
+
+       // Stack: x0-x15 OUT IN LEN
+
+       // Increment block counter (x12)
+       add             r8, #1
+
+       // Store updated (OUT, IN, LEN)
+       str             r14, [sp, #64]
+       str             r12, [sp, #68]
+       str             r9, [sp, #72]
+
+         mov           r14, sp
+
+       // Store updated block counter (x12)
+       str             r8, [sp, #48]
+
+         sub           sp, #16
+
+       // Reload state and do next block
+       ldmia           r14!, {r0-r11}          // load x0-x11
+       __strd          r10, r11, sp, 8         // store x10-x11 before state
+       ldmia           r14, {r10-r12,r14}      // load x12-x15
+       b               .Lnext_block\@
+
+.Lxor_slowpath\@:
+       // Slow path: < 64 bytes remaining, or unaligned input or output buffer.
+       // We handle it by storing the 64 bytes of keystream to the stack, then
+       // XOR-ing the needed portion with the data.
+
+       // Allocate keystream buffer
+       sub             sp, #64
+       mov             r14, sp
+
+       // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // Save keystream for x0-x3
+       __ldrd          r8, r9, sp, 96
+       __ldrd          r10, r11, sp, 104
+       add             X0, X0, r8
+       add             X1, X1, r9
+       add             X2, X2, r10
+       add             X3, X3, r11
+       _le32_bswap_4x  X0, X1, X2, X3,  r8, r9, r10
+       stmia           r14!, {X0-X3}
+
+       // Save keystream for x4-x7
+       __ldrd          r8, r9, sp, 112
+       __ldrd          r10, r11, sp, 120
+       add             X4, r8, X4, ror #brot
+       add             X5, r9, X5, ror #brot
+       add             X6, r10, X6, ror #brot
+       add             X7, r11, X7, ror #brot
+       _le32_bswap_4x  X4, X5, X6, X7,  r8, r9, r10
+         add           r8, sp, #64
+       stmia           r14!, {X4-X7}
+
+       // Save keystream for x8-x15
+       ldm             r8, {r0-r7}             // (x8-x9,x12-x15,x10-x11)
+       __ldrd          r8, r9, sp, 128
+       __ldrd          r10, r11, sp, 136
+       add             r0, r0, r8              // x8
+       add             r1, r1, r9              // x9
+       add             r6, r6, r10             // x10
+       add             r7, r7, r11             // x11
+       _le32_bswap_4x  r0, r1, r6, r7,  r8, r9, r10
+       stmia           r14!, {r0,r1,r6,r7}
+       __ldrd          r8, r9, sp, 144
+       __ldrd          r10, r11, sp, 152
+       add             r2, r8, r2, ror #drot   // x12
+       add             r3, r9, r3, ror #drot   // x13
+       add             r4, r10, r4, ror #drot  // x14
+       add             r5, r11, r5, ror #drot  // x15
+       _le32_bswap_4x  r2, r3, r4, r5,  r9, r10, r11
+       stmia           r14, {r2-r5}
+
+       // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN
+       // Registers: r8 is block counter, r12 is IN.
+
+       ldr             r9, [sp, #168]          // LEN
+       ldr             r14, [sp, #160]         // OUT
+       cmp             r9, #64
+         mov           r0, sp
+       movle           r1, r9
+       movgt           r1, #64
+       // r1 is number of bytes to XOR, in range [1, 64]
+
+.if __LINUX_ARM_ARCH__ < 6
+       orr             r2, r12, r14
+       tst             r2, #3                  // IN or OUT misaligned?
+       bne             .Lxor_next_byte\@
+.endif
+
+       // XOR a word at a time
+.rept 16
+       subs            r1, #4
+       blt             .Lxor_words_done\@
+       ldr             r2, [r12], #4
+       ldr             r3, [r0], #4
+       eor             r2, r2, r3
+       str             r2, [r14], #4
+.endr
+       b               .Lxor_slowpath_done\@
+.Lxor_words_done\@:
+       ands            r1, r1, #3
+       beq             .Lxor_slowpath_done\@
+
+       // XOR a byte at a time
+.Lxor_next_byte\@:
+       ldrb            r2, [r12], #1
+       ldrb            r3, [r0], #1
+       eor             r2, r2, r3
+       strb            r2, [r14], #1
+       subs            r1, #1
+       bne             .Lxor_next_byte\@
+
+.Lxor_slowpath_done\@:
+       subs            r9, #64
+       add             sp, #96
+       bgt             .Lprepare_for_next_block\@
+
+.Ldone\@:
+.endm  // _chacha
+
+/*
+ * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8],
+ *                  const u32 iv[4]);
+ */
+ENTRY(chacha20_arm)
+       cmp             r2, #0                  // len == 0?
+       bxeq            lr
+
+       push            {r0-r2,r4-r11,lr}
+
+       // Push state x0-x15 onto stack.
+       // Also store an extra copy of x10-x11 just before the state.
+
+       ldr             r4, [sp, #48]           // iv
+       mov             r0, sp
+       sub             sp, #80
+
+       // iv: x12-x15
+       ldm             r4, {X12,X13,X14,X15}
+       stmdb           r0!, {X12,X13,X14,X15}
+
+       // key: x4-x11
+       __ldrd          X8_X10, X9_X11, r3, 24
+       __strd          X8_X10, X9_X11, sp, 8
+       stmdb           r0!, {X8_X10, X9_X11}
+       ldm             r3, {X4-X9_X11}
+       stmdb           r0!, {X4-X9_X11}
+
+       // constants: x0-x3
+       adrl            X3, .Lexpand_32byte_k
+       ldm             X3, {X0-X3}
+       __strd          X0, X1, sp, 16
+       __strd          X2, X3, sp, 24
+
+       _chacha         20
+
+       add             sp, #76
+       pop             {r4-r11, pc}
+ENDPROC(chacha20_arm)
+
+/*
+ * void hchacha20_arm(const u32 state[16], u32 out[8]);
+ */
+ENTRY(hchacha20_arm)
+       push            {r1,r4-r11,lr}
+
+       mov             r14, r0
+       ldmia           r14!, {r0-r11}          // load x0-x11
+       push            {r10-r11}               // store x10-x11 to stack
+       ldm             r14, {r10-r12,r14}      // load x12-x15
+       sub             sp, #8
+
+       _chacha_permute 20
+
+       // Skip over (unused0-unused1, x10-x11)
+       add             sp, #16
+
+       // Fix up rotations of x12-x15
+       ror             X12, X12, #drot
+       ror             X13, X13, #drot
+         pop           {r4}                    // load 'out'
+       ror             X14, X14, #drot
+       ror             X15, X15, #drot
+
+       // Store (x0-x3,x12-x15) to 'out'
+       stm             r4, {X0,X1,X2,X3,X12,X13,X14,X15}
+
+       pop             {r4-r11,pc}
+ENDPROC(hchacha20_arm)
+
+#ifdef CONFIG_KERNEL_MODE_NEON
+/*
+ * This following NEON routine was ported from Andy Polyakov's implementation
+ * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine,
+ * since certain NEON code paths actually branch to it.
+ */
+
 .text
 #if defined(__thumb2__) || defined(__clang__)
 .syntax        unified
@@ -22,39 +484,6 @@
 #define ldrhsb ldrbhs
 #endif
 
-.align 5
-.Lsigma:
-.long  0x61707865,0x3320646e,0x79622d32,0x6b206574     @ endian-neutral
-.Lone:
-.long  1,0,0,0
-.word  -1
-
-.align 5
-ENTRY(chacha20_arm)
-       ldr     r12,[sp,#0]             @ pull pointer to counter and nonce
-       stmdb   sp!,{r0-r2,r4-r11,lr}
-       cmp     r2,#0                   @ len==0?
-#ifdef __thumb2__
-       itt     eq
-#endif
-       addeq   sp,sp,#4*3
-       beq     .Lno_data_arm
-       ldmia   r12,{r4-r7}             @ load counter and nonce
-       sub     sp,sp,#4*(16)           @ off-load area
-#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__)
-       sub     r14,pc,#100             @ .Lsigma
-#else
-       adr     r14,.Lsigma             @ .Lsigma
-#endif
-       stmdb   sp!,{r4-r7}             @ copy counter and nonce
-       ldmia   r3,{r4-r11}             @ load key
-       ldmia   r14,{r0-r3}             @ load sigma
-       stmdb   sp!,{r4-r11}            @ copy key
-       stmdb   sp!,{r0-r3}             @ copy sigma
-       str     r10,[sp,#4*(16+10)]     @ off-load "rx"
-       str     r11,[sp,#4*(16+11)]     @ off-load "rx"
-       b       .Loop_outer_enter
-
 .align 4
 .Loop_outer:
        ldmia   sp,{r0-r9}              @ load key material
@@ -748,11 +1177,8 @@ ENTRY(chacha20_arm)
 
 .Ldone:
        add     sp,sp,#4*(32+3)
-.Lno_data_arm:
        ldmia   sp!,{r4-r11,pc}
-ENDPROC(chacha20_arm)
 
-#ifdef CONFIG_KERNEL_MODE_NEON
 .align 5
 .Lsigma2:
 .long  0x61707865,0x3320646e,0x79622d32,0x6b206574     @ endian-neutral
lib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0 -0
similarity index 100%
rename from lib/zinc/chacha20/chacha20-arm64-cryptogams.S
rename to lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/chacha20/chacha20.c +2 -1
index 4354b874a6a5..fc4f74fca653 100644
@@ -16,6 +16,8 @@
 
 #if defined(CONFIG_ZINC_ARCH_X86_64)
 #include "chacha20-x86_64-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
+#include "chacha20-arm-glue.h"
 #else
 void __init chacha20_fpu_init(void)
 {
--

2.19.0

author maintainer
#48 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Hi Ard,

Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4 -0
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, size_t len,
+                                simd_context_t *simd_context)
+{
Quoted 12 lines

lib/zinc/chacha20/chacha20-arm-glue.h line 53

Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6 -0
+#if defined(CONFIG_KERNEL_MODE_NEON)
+       if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))
+               chacha20_neon(dst, src, len, state->key, state->counter);
+       else
+#endif

Better to use IS_ENABLED() here:

Quoted 3 lines
(quoted diff) +3 -0
+       if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) &&
+           chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))

Good idea. I'll fix that up.

Quoted 4 lines

Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.

The vast majority of crypto routines in arch/*/crypto/ follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle. I made a change so that in chacha20poly1305.c, it calls simd_relax after handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now. And if it becomes one, this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.

Jason

#49 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

(+ Herbert, Thomas)

Quoted 25 lines · Jason A. Donenfeld

Hi Ard,

Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4 -0
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, size_t len,
+                                simd_context_t *simd_context)
+{
Quoted 12 lines

lib/zinc/chacha20/chacha20-arm-glue.h line 53

Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6 -0
+#if defined(CONFIG_KERNEL_MODE_NEON)
+       if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))
+               chacha20_neon(dst, src, len, state->key, state->counter);
+       else
+#endif

Better to use IS_ENABLED() here:

Quoted 3 lines
(quoted diff) +3 -0
+       if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) &&
+           chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))

Good idea. I'll fix that up.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.

The vast majority of crypto routines in arch/*/crypto/ follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle.

Anything that uses the scatterwalk API (AEADs and skciphers) will handle at most a page at a time. Hashes are different, which is why some of them have to handle it explicitly.

Quoted 7 lines

I made a change so that in chacha20poly1305.c, it calls simd_relax after handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now.

The state of the tree does not capture all relevant context or history. The scheduling latency issue was brought up very recently by the -rt folks on the mailing lists.

Quoted 5 lines

And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.

You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.

Part of the [justified] criticism on the current state of the crypto API is on its complexity, and so I don't think it makes sense to keep it simple now and add the complexity later (and the same concern applies to async support btw).

author maintainer
#50 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 3 lines · Ard Biesheuvel

I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).

Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.

#51 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 7 lines · Jason A. Donenfeld
Quoted 3 lines · Ard Biesheuvel

I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).

Quoted 10 lines

Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.

Framing it as /needless/ complexity does not help at all. The changes you are proposing are very useful, but nobody wants two crypto subsystems with two different maintainers in the kernel, so I would like to understand where this is going in the future. I am not saying it should block these patches though.

Also, I have spent a *lot* of time looking at your code, and trying to make it better, especially for use cases that weren't on your radar to begin with (e.g., 'pet projects' [your words] like the Cortex-A7 which will be in almost every new 32-bit Android phone). So characterizing my feedback as some kind of sabotage is not very productive either.

Contrary to what you seem to think, I am not deeply invested in the crypto API. What I do care about is that the ARM crypto pieces in the kernel are maintained, supported and improved by someone who understands the use cases Linaro's members care about, and is willing to make an effort to gain such understanding if he doesn't. I have no doubt that your involvement in the kernel's crypto subsystem will have a significant positive impact when it comes to code quality, robustness and usability. I'd just like to see a bit more consideration for other aspects of kernel programming, e.g., preemption under -rt, stack size constraints, coding style, importing code from other projects etc. - please try to be less dismissive of feedback first time around, but try to understand why people are raising these issues; I'm sure you will appreciate it when future contributors to zinc will do the same.

author maintainer
#52 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 5 lines · Ard Biesheuvel

Framing it as /needless/ complexity does not help at all. The changes you are proposing are very useful, but nobody wants two crypto subsystems with two different maintainers in the kernel, so I would like to understand where this is going in the future. I am not saying it should block these patches though.

Thanks for clarifying. I understood you to be intending to block the patches until they were converted to an async interface, which is not what Zinc's about. Seeing as you're just curious about future directions, that seems much more tenable.

Quoted 3 lines

Also, I have spent a *lot* of time looking at your code, and trying to make it better, especially for use cases that weren't on your radar to begin with

I am extremely grateful for a good portion of your reviews indeed. As I mentioned earlier, much is very useful. But in other places, I fear you're steering this in a direction I really am hesitant to go.

Quoted 1 line

(e.g., 'pet projects' [your words]

Taken out of context.

Quoted 3 lines

consideration for other aspects of kernel programming, e.g., preemption under -rt, stack size constraints, coding style, importing code from other projects etc.

And indeed all of these concerns I've been pretty amenable to, and continue to do so. What I'm commenting on are things outside of these.

Quoted 3 lines

- please try to be less dismissive of
feedback first time around, but try to understand why people are raising these issues

Apologies, and duly noted. I'll give you the benefit of the doubt.

Jason

#53 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld
Quoted 6 lines · Jason A. Donenfeld
Quoted 3 lines · Ard Biesheuvel

I don't think it makes sense to keep
it simple now and add the complexity later (and the same concern applies to async support btw).

Quoted 10 lines

Ugh, no. I don't want to add needless complexity, period. Zinc is synchronous, not asynchronous. It provides software implementations. That's what it does. While many of your reviews have been useful, many of your comments indicate some desire to change and mold the purpose and focus of Zinc away from Zinc's intents. Stop that. It's not going to become a bloated mess of "things Ard wanted and quipped about on LKML." Things like these only serve to filibuster the patchset indefinitely. But maybe that's what you'd like all along? Hard to tell, honestly. So, no, sorry, Zinc isn't gaining an async interface right now.

Can you please stop accusing Ard of "filibustering" your patchset? Spending too long in non-preemptible code is a real problem even on non-RT systems. syzkaller has been reporting bugs where the kernel spins too long without any preemption points, both in crypto-related code and elsewhere in the kernel. So we've had to add explicit preemption points to address those, as otherwise users can lock up all CPUs for tens of seconds. The issue being discussed here is basically the same except here preemption is being explicitly disabled via kernel_neon_begin(), so it becomes a problem even on non-CONFIG_PREEMPT kernels.

It's much better to address this problem (which is a regression from the current skcipher API which only maps a page at a time) up front rather than have to rush to patch it after the fact once the syzbot reports start coming in.

- Eric

author maintainer
#54 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 8 lines · Eric Biggers

Can you please stop accusing Ard of "filibustering" your patchset? Spending too long in non-preemptible code is a real problem even on non-RT systems. syzkaller has been reporting bugs where the kernel spins too long without any preemption points, both in crypto-related code and elsewhere in the kernel. So we've had to add explicit preemption points to address those, as otherwise users can lock up all CPUs for tens of seconds. The issue being discussed here is basically the same except here preemption is being explicitly disabled via kernel_neon_begin(), so it becomes a problem even on non-CONFIG_PREEMPT kernels.

The async distraction (re:filibustering) and the preempt concern are two totally different things. I've already posted some code elsewhere in this thread that addresses the preempt issue that looked good to Ard. This will be part of v7.

#55 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Ard Biesheuvel
Quoted 60 lines · Ard Biesheuvel

(+ Herbert, Thomas)

Quoted 25 lines · Jason A. Donenfeld

Hi Ard,

Quoted 4 lines · Ard Biesheuvel
Quoted 4 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +4 -0
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, size_t len,
+                                simd_context_t *simd_context)
+{
Quoted 12 lines

lib/zinc/chacha20/chacha20-arm-glue.h line 53

Quoted 6 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +6 -0
+#if defined(CONFIG_KERNEL_MODE_NEON)
+       if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))
+               chacha20_neon(dst, src, len, state->key, state->counter);
+       else
+#endif

Better to use IS_ENABLED() here:

Quoted 3 lines
(quoted diff) +3 -0
+       if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) &&
+           chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))

Good idea. I'll fix that up.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.

The vast majority of crypto routines in arch/*/crypto/ follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle.

Anything that uses the scatterwalk API (AEADs and skciphers) will handle at most a page at a time. Hashes are different, which is why some of them have to handle it explicitly.

Quoted 7 lines

I made a change so that in chacha20poly1305.c, it calls simd_relax after handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now.

The state of the tree does not capture all relevant context or history. The scheduling latency issue was brought up very recently by the -rt folks on the mailing lists.

Quoted 5 lines

And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.

Quoted 4 lines

You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.

Actually, looking at the code again, the abstraction does appear to be fine, it is just the chacha20 code that does not make use of it.

author maintainer
#56 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 4 lines · Ard Biesheuvel

On Wed, 26 Sep 2018 at 16:02, Ard Biesheuvel <[email protected]> wrote:
Actually, looking at the code again, the abstraction does appear to be fine, it is just the chacha20 code that does not make use of it.

So what you have in mind is something like calling simd_relax() every 4096 bytes or so?

author maintainer
#57 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 2 lines · Jason A. Donenfeld

So what you have in mind is something like calling simd_relax() every 4096 bytes or so?

That was actually pretty easy, putting together both of your suggestions:

static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
u8 *src, size_t len,
simd_context_t *simd_context)

{
    while (len > PAGE_SIZE) {
        chacha20_arch(state, dst, src, PAGE_SIZE, simd_context);
        len -= PAGE_SIZE;
        src += PAGE_SIZE;
        dst += PAGE_SIZE;
        simd_relax(simd_context);
    }
    if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon &&
        len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context))
        chacha20_neon(dst, src, len, state->key, state->counter);
    else
        chacha20_arm(dst, src, len, state->key, state->counter);

    state->counter[0] += (len + 63) / 64;
    return true;
}
#58 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 19 lines · Jason A. Donenfeld
Quoted 2 lines · Jason A. Donenfeld

So what you have in mind is something like calling simd_relax() every 4096 bytes or so?

That was actually pretty easy, putting together both of your suggestions:

static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
u8 *src, size_t len,
simd_context_t *simd_context)

{
    while (len > PAGE_SIZE) {
        chacha20_arch(state, dst, src, PAGE_SIZE, simd_context);
        len -= PAGE_SIZE;
        src += PAGE_SIZE;
        dst += PAGE_SIZE;
        simd_relax(simd_context);
    }
    if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon &&
Quoted 8 lines
        len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context))
        chacha20_neon(dst, src, len, state->key, state->counter);
    else
        chacha20_arm(dst, src, len, state->key, state->counter);

    state->counter[0] += (len + 63) / 64;
    return true;
}

Nice one :-)

This works for me (but perhaps add a comment as well)

author maintainer
#59 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 3 lines · Ard Biesheuvel

Nice one :-)

This works for me (but perhaps add a comment as well)

Sure. Just a prototype; it'll be clean for v7.

author maintainer
#60 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Ard Biesheuvel
Quoted 26 lines · Ard Biesheuvel
Quoted 24 lines · Jason A. Donenfeld
Quoted 2 lines · Jason A. Donenfeld

So what you have in mind is something like calling simd_relax() every 4096 bytes or so?

That was actually pretty easy, putting together both of your suggestions:

static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
u8 *src, size_t len,
simd_context_t *simd_context)

{
    while (len > PAGE_SIZE) {
        chacha20_arch(state, dst, src, PAGE_SIZE, simd_context);
        len -= PAGE_SIZE;
        src += PAGE_SIZE;
        dst += PAGE_SIZE;
        simd_relax(simd_context);
    }
    if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && chacha20_use_neon &&
        len >= CHACHA20_BLOCK_SIZE * 3 && simd_use(simd_context))
        chacha20_neon(dst, src, len, state->key, state->counter);
    else
        chacha20_arm(dst, src, len, state->key, state->counter);
Quoted 7 lines
Quoted 3 lines
    state->counter[0] += (len + 63) / 64;
    return true;
}

Nice one :-)

This works for me (but perhaps add a comment as well)

As elegant as my quick recursive solution was, gcc produced kind of bad code from it, as you might expect. So I've implemented this using a boring old loop that works the way it's supposed to. This is marked for v7.

author maintainer
#61 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Hi Thomas,

I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:

preempt_disable();
asm volatile(".fill N, 1, 0x90;");
preempt_enable();

What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?

Thanks,
Jason

author maintainer
#62 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Hey again Thomas,

Quoted 18 lines · Jason A. Donenfeld

Hi Thomas,

I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:

preempt_disable();
asm volatile(".fill N, 1, 0x90;");
preempt_enable();

Quoted 7 lines

What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?

Thanks,
Jason

From talking to Peter (now CC'd) on IRC, it sounds like what you're

mostly interested in is clocktime latency on reasonable hardware, with a goal of around ~20?s as a maximum upper bound? I don't expect to get anywhere near this value at all, but if you can confirm that's a decent ballpark, it would make for some interesting calculations.

Regards,
Jason

#63 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 28 lines · Jason A. Donenfeld

Hey again Thomas,

Quoted 22 lines · Jason A. Donenfeld

Hi Thomas,

I'm trying to optimize this for crypto performance while still taking into account preemption concerns. I'm having a bit of trouble figuring out a way to determine numerically what the upper bounds for this stuff looks like. I'm sure I could pick a pretty sane number that's arguably okay -- and way under the limit -- but I still am interested in determining what that limit actually is. I was hoping there'd be a debugging option called, "warn if preemption is disabled for too long", or something, but I couldn't find anything like that. I'm also not quite sure what the latency limits are, to just compute this with a formula. Essentially what I'm trying to determine is:

preempt_disable();
asm volatile(".fill N, 1, 0x90;");
preempt_enable();

What is the maximum value of N for which the above is okay? What technique would you generally use in measuring this?

Quoted 10 lines
Quoted 2 lines

Thanks,
Jason

From talking to Peter (now CC'd) on IRC, it sounds like what you're mostly interested in is clocktime latency on reasonable hardware, with a goal of around ~20?s as a maximum upper bound? I don't expect to get anywhere near this value at all, but if you can confirm that's a decent ballpark, it would make for some interesting calculations.

I would add another consideration: if you can get better latency with negligible overhead (0.1%? 0.05%), then that might make sense too. For example, it seems plausible that checking need_resched() every few blocks adds basically no overhead, and the SIMD helpers could do this themselves or perhaps only ever do a block at a time.

need_resched() costs a cacheline access, but it?s usually a hot cacheline, and the actual check is just whether a certain bit in memory is set.

author maintainer
#64 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 3 lines · Andy Lutomirski

I would add another consideration: if you can get better latency with negligible overhead (0.1%? 0.05%), then that might make sense too. For example, it seems plausible that checking need_resched() every few blocks adds basically no overhead, and the SIMD helpers could do this themselves or perhaps only ever do a block at a time.

need_resched() costs a cacheline access, but it?s usually a hot cacheline, and the actual check is just whether a certain bit in memory is set.

Yes you're right, I do plan to check quite often, rather than seldom, for this reason. I've been toying with the idea of instead processing 65k (maximum size of a UDP packet) at a time before checking need_resched(), but armed with the 20?s figure, this isn't remotely possible on most hardware. So I'll stick with the original conservative plan of checking very often, and not making things different from the aspects worked out by the present crypto API in this regard.

#65 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Ard Biesheuvel
Quoted 17 lines · Ard Biesheuvel

(+ Herbert, Thomas)

Quoted 10 lines · Jason A. Donenfeld

Hi Ard,
.

And if it becomes one,
this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.

Quoted 8 lines

You are introducing a very useful SIMD abstraction, but it lets code run with preemption disabled for unbounded amounts of time, and so now is the time to ensure we get it right.

Part of the [justified] criticism on the current state of the crypto API is on its complexity, and so I don't think it makes sense to keep it simple now and add the complexity later (and the same concern applies to async support btw).

Are, is what you?re saying that the Zinc chacha20 functions should call simd_relax() every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.

What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an __might_sleep or similar. Or simd_relax should learn to *not* schedule if the result of preempt_enable() leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.

As for async, ISTM a really good WireGuard accelerator would expose a different interface than crypto API supports, and it probably makes sense to wait for such hardware to show up before figuring out how to use it. And no matter what form it takes, I don?t think it should complicate the basic Zinc crypto entry points.

author maintainer
#66 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 3 lines · Andy Lutomirski

Are, is what you?re saying that the Zinc chacha20 functions should call simd_relax() every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.

What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an __might_sleep or similar. Or simd_relax should learn to *not* schedule if the result of preempt_enable() leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.

I'm not sure this is actually a problem. Namely:

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
schedule(); <--- bug!

Calling kernel_fpu_end() disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use of preempt_disable won't actually do anything until the outer preempt enable is called:

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
preempt_enable();
schedule(); <--- works!

Or am I missing some more subtle point?

Jason

#67 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 18 lines · Jason A. Donenfeld
Quoted 3 lines · Andy Lutomirski

Are, is what you?re saying that the Zinc chacha20 functions should call simd_relax() every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.

What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an __might_sleep or similar. Or simd_relax should learn to *not* schedule if the result of preempt_enable() leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.

I'm not sure this is actually a problem. Namely:

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
schedule(); <--- bug!

Calling kernel_fpu_end() disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use of preempt_disable won't actually do anything until the outer preempt enable is called:

Quoted 8 lines

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
preempt_enable();
schedule(); <--- works!

Or am I missing some more subtle point?

No that seems accurate to me.

#68 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld
Quoted 21 lines · Jason A. Donenfeld
Quoted 5 lines · Andy Lutomirski

Are, is what you?re saying that the Zinc chacha20 functions should call simd_relax() every n bytes automatically for some reasonable value of n? If so, seems sensible, except that some care might be needed to make sure they interact with preemption correctly.

What I mean is: the public Zinc entry points should either be callable in an atomic context or they should not be. I think this should be checked at runtime in an appropriate place with an __might_sleep or similar. Or simd_relax should learn to *not* schedule if the result of preempt_enable() leaves it atomic. (And the latter needs to be done in a way that works even on non-preempt kernels, and I don?t remember whether that?s possible.). And this should happen regardless of how many bytes are processed. IOW, calling into Zinc should be equally not atomic-safe for 100 bytes and for 10 MB.

I'm not sure this is actually a problem. Namely:

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
schedule(); <--- bug!

Calling kernel_fpu_end() disables preemption, but AFAIK, preemption enabling/disabling is recursive, so kernel_fpu_end's use of preempt_disable won't actually do anything until the outer preempt enable is called:

Quoted 8 lines

preempt_disable();
kernel_fpu_begin();
kernel_fpu_end();
preempt_enable();
schedule(); <--- works!

Or am I missing some more subtle point?

No, I think you?re right. I was mid-remembering precisely how simd_relax() worked.

#69 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld
Quoted 4 lines
Quoted 3 lines

Also, this still has unbounded worst case scheduling latency, given that the outer library function passes its entire input straight into the NEON routine.

Quoted 13 lines

The vast majority of crypto routines in arch/*/crypto/ follow this same exact pattern, actually. I realize a few don't -- probably the ones you had a hand in :) -- but I think this is up to the caller to handle. I made a change so that in chacha20poly1305.c, it calls simd_relax after handling each scatter-gather element, so a "construction" will handle this gracefully. But I believe it's up to the caller to decide on what sizes of information it wants to pass to primitives. Put differently, this also hasn't ever been an issue before -- the existing state of the tree indicates this -- and so I don't anticipate this will be a real issue now. And if it becomes one, this is something we can address *later*, but certainly there's no use of adding additional complexity to the initial patchset to do this now.

Hi Jason

This is not my area of expertise, so you should verify what i'm say here...

My guess is, IPSEC will mostly ask the crypto code to work on 1500 byte full MTU packets and 64 byte TCP ACK packets. Disk encryption i guess works on 4K blocks. So these requests are all quite small, keeping the latency reasonably bounded.

The wireguard interface claims it is GSO capable. This means the network stack will pass it big chunks of data and leave it to the network interface to perform the segmentation into 1500 byte MTU frames on the wire. I've not looked at how wireguard actually handles these big chunks. But to get maximum performance, it should try to keep them whole, just add a header and/or trailer. Will wireguard pass these big chunks of data to the crypto code? Do we now have 64K blocks being worked on? Does the latency jump from 4K to 64K? That might be new, so the existing state of the tree does not help you here.

Andrew

author maintainer
#70 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 9 lines · Andrew Lunn

The wireguard interface claims it is GSO capable. This means the network stack will pass it big chunks of data and leave it to the network interface to perform the segmentation into 1500 byte MTU frames on the wire. I've not looked at how wireguard actually handles these big chunks. But to get maximum performance, it should try to keep them whole, just add a header and/or trailer. Will wireguard pass these big chunks of data to the crypto code? Do we now have 64K blocks being worked on? Does the latency jump from 4K to 64K? That might be new, so the existing state of the tree does not help you here.

No, it only requests GSO superpackets so that it can group the pieces and encrypt them on the same core. But they're each encrypted separately (broken up immediately after ndo_start_xmit), and so they wind up being ~1420 bytes each to encrypt. I spoke about this at netdev2.2 if you're interested in the architecture; there's a paper:

https://www.wireguard.com/papers/wireguard-netdev22.pdf
https://www.youtube.com/watch?v=54orFwtQ1XY
https://www.wireguard.com/talks/netdev2017-slides.pdf

#71 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld
Quoted 11 lines · Jason A. Donenfeld

These wire Andy Polyakov's implementations up to the kernel for ARMv7,8 NEON, and introduce Eric Biggers' ultra-fast scalar implementation for CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).

This commit does the following:
- Adds the glue code for the assembly implementations.
- Renames the ARMv8 code into place, since it can at this point be used wholesale.
- Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7 scalar code.

Quoted 3 lines

Commit note: Eric Biggers' scalar code is brand new, and quite possibly prematurely added to this commit, and so it may require a bit of revision.

Please put comments like this below the ---

Quoted 43 lines

This commit delivers approximately the same or much better performance than the existing crypto API's code and has been measured to do as such on:

- ARM1176JZF-S [ARMv6]

  - Cortex-A7    [ARMv7]
  - Cortex-A8    [ARMv7]
  - Cortex-A9    [ARMv7]
  - Cortex-A17   [ARMv7]
  - Cortex-A53   [ARMv8]
  - Cortex-A55   [ARMv8]
  - Cortex-A73   [ARMv8]
  - Cortex-A75   [ARMv8]

Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers', but is also significantly shorter. This has the advantage that it does not evict other code from L1 cache -- particularly on ARM11 chips -- and so in certain circumstances it can actually be faster. However, it wasn't found that this had an affect on any code existing in the kernel today.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                             |   2 +
 lib/zinc/chacha20/chacha20-arm-glue.h         |  88 +++
 ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
 ...20-arm64-cryptogams.S => chacha20-arm64.S} |   0
 lib/zinc/chacha20/chacha20.c                  |   2 +
 5 files changed, 556 insertions(+), 38 deletions(-)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
 rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
 rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)
lib/zinc/Makefile +0 -0
index 223a0816c918..e47f64e12bbd 100644

lib/zinc/Makefile ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG · line 8

Quoted 5 lines
lib/zinc/Makefile +2 -0
@@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o

Are these CONFIG_ symbols defined anywhere at this point?

In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases, and the dependency on !CPU_32v3 you introduce (looking at the version of Kconfig at the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel but ARCH_RPC is built in v3 mode because of historical reasons while the actual core is a v4)

Quoted 19 lines
lib/zinc/Makefile +0 -0
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +12 -0
new file mode 100644
index 000000000000..86cce851ed02
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/hwcap.h>
+#include <asm/neon.h>
+#if defined(CONFIG_ARM)
+#include <asm/system_info.h>
+#include <asm/cputype.h>
+#endif
+

lib/zinc/chacha20/chacha20-arm-glue.h line 22

Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10 -0
+asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len,
+                            const u32 key[8], const u32 counter[4]);
+#if defined(CONFIG_ARM)
+asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]);
+#endif
+#if defined(CONFIG_KERNEL_MODE_NEON)
+asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len,
+                             const u32 key[8], const u32 counter[4]);
+#endif
+

No need to make asmlinkage declarations conditional

Quoted 16 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +16 -0
+static bool chacha20_use_neon __ro_after_init;
+
+static void __init chacha20_fpu_init(void)
+{
+#if defined(CONFIG_ARM64)
+       chacha20_use_neon = elf_hwcap & HWCAP_ASIMD;
+#elif defined(CONFIG_ARM)
+       switch (read_cpuid_part()) {
+       case ARM_CPU_PART_CORTEX_A7:
+       case ARM_CPU_PART_CORTEX_A5:
+               /* The Cortex-A7 and Cortex-A5 do not perform well with the NEON
+                * implementation but do incredibly with the scalar one and use
+                * less power.
+                */
+               break;
+       default:

lib/zinc/chacha20/chacha20-arm-glue.h line 48

Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10 -0
+               chacha20_use_neon = elf_hwcap & HWCAP_NEON;
+       }
+#endif
+}
+
+static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
+                                const u8 *src, size_t len,
+                                simd_context_t *simd_context)
+{
+#if defined(CONFIG_KERNEL_MODE_NEON)

if (IS_ENABLED())

Quoted 25 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +25 -0
+       if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
+           simd_use(simd_context))
+               chacha20_neon(dst, src, len, state->key, state->counter);
+       else
+#endif
+               chacha20_arm(dst, src, len, state->key, state->counter);
+
+       state->counter[0] += (len + 63) / 64;
+       return true;
+}
+
+static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
+                                 const u8 nonce[HCHACHA20_NONCE_SIZE],
+                                 const u8 key[HCHACHA20_KEY_SIZE],
+                                 simd_context_t *simd_context)
+{
+#if defined(CONFIG_ARM)
+       u32 x[] = { CHACHA20_CONSTANT_EXPA,
+                   CHACHA20_CONSTANT_ND_3,
+                   CHACHA20_CONSTANT_2_BY,
+                   CHACHA20_CONSTANT_TE_K,
+                   get_unaligned_le32(key + 0),
+                   get_unaligned_le32(key + 4),
+                   get_unaligned_le32(key + 8),
+                   get_unaligned_le32(key + 12),

lib/zinc/chacha20/chacha20-arm-glue.h line 83

Quoted 10 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +10 -0
+                   get_unaligned_le32(key + 16),
+                   get_unaligned_le32(key + 20),
+                   get_unaligned_le32(key + 24),
+                   get_unaligned_le32(key + 28),
+                   get_unaligned_le32(nonce + 0),
+                   get_unaligned_le32(nonce + 4),
+                   get_unaligned_le32(nonce + 8),
+                   get_unaligned_le32(nonce + 12)
+       };
+       hchacha20_arm(x, derived_key);

"""
if (!IS_ENABLED(CONFIG_ARM))
return false;

hchacha20_arm(x, derived_key);
return true;
"""

and drop the #ifdefs

Quoted 551 lines
lib/zinc/chacha20/chacha20-arm-glue.h new file +5 -0
+       return true;
+#else
+       return false;
+#endif
+}
lib/zinc/chacha20/chacha20-arm.S renamed from lib/zinc/chacha20/chacha20-arm-cryptogams.S +464 -38
similarity index 71%
rename from lib/zinc/chacha20/chacha20-arm-cryptogams.S
rename to lib/zinc/chacha20/chacha20-arm.S
index 770bab469171..5abedafcf129 100644
@@ -1,13 +1,475 @@
 /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
 /*
+ * Copyright (C) 2018 Google, Inc.
  * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
- *
- * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
  */
 
 #include <linux/linkage.h>
 
+/*
+ * The following scalar routine was written by Eric Biggers.
+ *
+ * Design notes:
+ *
+ * 16 registers would be needed to hold the state matrix, but only 14 are
+ * available because 'sp' and 'pc' cannot be used.  So we spill the elements
+ * (x8, x9) to the stack and swap them out with (x10, x11).  This adds one
+ * 'ldrd' and one 'strd' instruction per round.
+ *
+ * All rotates are performed using the implicit rotate operand accepted by the
+ * 'add' and 'eor' instructions.  This is faster than using explicit rotate
+ * instructions.  To make this work, we allow the values in the second and last
+ * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the
+ * wrong rotation amount.  The rotation amount is then fixed up just in time
+ * when the values are used.  'brot' is the number of bits the values in row 'b'
+ * need to be rotated right to arrive at the correct values, and 'drot'
+ * similarly for row 'd'.  (brot, drot) start out as (0, 0) but we make it such
+ * that they end up as (25, 24) after every round.
+ */
+
+       // ChaCha state registers
+       X0      .req    r0
+       X1      .req    r1
+       X2      .req    r2
+       X3      .req    r3
+       X4      .req    r4
+       X5      .req    r5
+       X6      .req    r6
+       X7      .req    r7
+       X8_X10  .req    r8      // shared by x8 and x10
+       X9_X11  .req    r9      // shared by x9 and x11
+       X12     .req    r10
+       X13     .req    r11
+       X14     .req    r12
+       X15     .req    r14
+
+.Lexpand_32byte_k:
+       // "expand 32-byte k"
+       .word   0x61707865, 0x3320646e, 0x79622d32, 0x6b206574
+
+#ifdef __thumb2__
+#  define adrl adr
+#endif
+
+.macro __rev           out, in,  t0, t1, t2
+.if __LINUX_ARM_ARCH__ >= 6
+       rev             \out, \in
+.else
+       lsl             \t0, \in, #24
+       and             \t1, \in, #0xff00
+       and             \t2, \in, #0xff0000
+       orr             \out, \t0, \in, lsr #24
+       orr             \out, \out, \t1, lsl #8
+       orr             \out, \out, \t2, lsr #8
+.endif
+.endm
+
+.macro _le32_bswap     x,  t0, t1, t2
+#ifdef __ARMEB__
+       __rev           \x, \x,  \t0, \t1, \t2
+#endif
+.endm
+
+.macro _le32_bswap_4x  a, b, c, d,  t0, t1, t2
+       _le32_bswap     \a,  \t0, \t1, \t2
+       _le32_bswap     \b,  \t0, \t1, \t2
+       _le32_bswap     \c,  \t0, \t1, \t2
+       _le32_bswap     \d,  \t0, \t1, \t2
+.endm
+
+.macro __ldrd          a, b, src, offset
+#if __LINUX_ARM_ARCH__ >= 6
+       ldrd            \a, \b, [\src, #\offset]
+#else
+       ldr             \a, [\src, #\offset]
+       ldr             \b, [\src, #\offset + 4]
+#endif
+.endm
+
+.macro __strd          a, b, dst, offset
+#if __LINUX_ARM_ARCH__ >= 6
+       strd            \a, \b, [\dst, #\offset]
+#else
+       str             \a, [\dst, #\offset]
+       str             \b, [\dst, #\offset + 4]
+#endif
+.endm
+
+.macro _halfround      a1, b1, c1, d1,  a2, b2, c2, d2
+
+       // a += b; d ^= a; d = rol(d, 16);
+       add             \a1, \a1, \b1, ror #brot
+       add             \a2, \a2, \b2, ror #brot
+       eor             \d1, \a1, \d1, ror #drot
+       eor             \d2, \a2, \d2, ror #drot
+       // drot == 32 - 16 == 16
+
+       // c += d; b ^= c; b = rol(b, 12);
+       add             \c1, \c1, \d1, ror #16
+       add             \c2, \c2, \d2, ror #16
+       eor             \b1, \c1, \b1, ror #brot
+       eor             \b2, \c2, \b2, ror #brot
+       // brot == 32 - 12 == 20
+
+       // a += b; d ^= a; d = rol(d, 8);
+       add             \a1, \a1, \b1, ror #20
+       add             \a2, \a2, \b2, ror #20
+       eor             \d1, \a1, \d1, ror #16
+       eor             \d2, \a2, \d2, ror #16
+       // drot == 32 - 8 == 24
+
+       // c += d; b ^= c; b = rol(b, 7);
+       add             \c1, \c1, \d1, ror #24
+       add             \c2, \c2, \d2, ror #24
+       eor             \b1, \c1, \b1, ror #20
+       eor             \b2, \c2, \b2, ror #20
+       // brot == 32 - 7 == 25
+.endm
+
+.macro _doubleround
+
+       // column round
+
+       // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13)
+       _halfround      X0, X4, X8_X10, X12,  X1, X5, X9_X11, X13
+
+       // save (x8, x9); restore (x10, x11)
+       __strd          X8_X10, X9_X11, sp, 0
+       __ldrd          X8_X10, X9_X11, sp, 8
+
+       // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15)
+       _halfround      X2, X6, X8_X10, X14,  X3, X7, X9_X11, X15
+
+       .set brot, 25
+       .set drot, 24
+
+       // diagonal round
+
+       // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12)
+       _halfround      X0, X5, X8_X10, X15,  X1, X6, X9_X11, X12
+
+       // save (x10, x11); restore (x8, x9)
+       __strd          X8_X10, X9_X11, sp, 8
+       __ldrd          X8_X10, X9_X11, sp, 0
+
+       // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14)
+       _halfround      X2, X7, X8_X10, X13,  X3, X4, X9_X11, X14
+.endm
+
+.macro _chacha_permute nrounds
+       .set brot, 0
+       .set drot, 0
+       .rept \nrounds / 2
+        _doubleround
+       .endr
+.endm
+
+.macro _chacha         nrounds
+
+.Lnext_block\@:
+       // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN
+       // Registers contain x0-x9,x12-x15.
+
+       // Do the core ChaCha permutation to update x0-x15.
+       _chacha_permute \nrounds
+
+       add             sp, #8
+       // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers contain x0-x9,x12-x15.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15).
+       push            {X8_X10, X9_X11, X12, X13, X14, X15}
+
+       // Load (OUT, IN, LEN).
+       ldr             r14, [sp, #96]
+       ldr             r12, [sp, #100]
+       ldr             r11, [sp, #104]
+
+       orr             r10, r14, r12
+
+       // Use slow path if fewer than 64 bytes remain.
+       cmp             r11, #64
+       blt             .Lxor_slowpath\@
+
+       // Use slow path if IN and/or OUT isn't 4-byte aligned.  Needed even on
+       // ARMv6+, since ldmia and stmia (used below) still require alignment.
+       tst             r10, #3
+       bne             .Lxor_slowpath\@
+
+       // Fast path: XOR 64 bytes of aligned data.
+
+       // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // x0-x3
+       __ldrd          r8, r9, sp, 32
+       __ldrd          r10, r11, sp, 40
+       add             X0, X0, r8
+       add             X1, X1, r9
+       add             X2, X2, r10
+       add             X3, X3, r11
+       _le32_bswap_4x  X0, X1, X2, X3,  r8, r9, r10
+       ldmia           r12!, {r8-r11}
+       eor             X0, X0, r8
+       eor             X1, X1, r9
+       eor             X2, X2, r10
+       eor             X3, X3, r11
+       stmia           r14!, {X0-X3}
+
+       // x4-x7
+       __ldrd          r8, r9, sp, 48
+       __ldrd          r10, r11, sp, 56
+       add             X4, r8, X4, ror #brot
+       add             X5, r9, X5, ror #brot
+       ldmia           r12!, {X0-X3}
+       add             X6, r10, X6, ror #brot
+       add             X7, r11, X7, ror #brot
+       _le32_bswap_4x  X4, X5, X6, X7,  r8, r9, r10
+       eor             X4, X4, X0
+       eor             X5, X5, X1
+       eor             X6, X6, X2
+       eor             X7, X7, X3
+       stmia           r14!, {X4-X7}
+
+       // x8-x15
+       pop             {r0-r7}                 // (x8-x9,x12-x15,x10-x11)
+       __ldrd          r8, r9, sp, 32
+       __ldrd          r10, r11, sp, 40
+       add             r0, r0, r8              // x8
+       add             r1, r1, r9              // x9
+       add             r6, r6, r10             // x10
+       add             r7, r7, r11             // x11
+       _le32_bswap_4x  r0, r1, r6, r7,  r8, r9, r10
+       ldmia           r12!, {r8-r11}
+       eor             r0, r0, r8              // x8
+       eor             r1, r1, r9              // x9
+       eor             r6, r6, r10             // x10
+       eor             r7, r7, r11             // x11
+       stmia           r14!, {r0,r1,r6,r7}
+       ldmia           r12!, {r0,r1,r6,r7}
+       __ldrd          r8, r9, sp, 48
+       __ldrd          r10, r11, sp, 56
+       add             r2, r8, r2, ror #drot   // x12
+       add             r3, r9, r3, ror #drot   // x13
+       add             r4, r10, r4, ror #drot  // x14
+       add             r5, r11, r5, ror #drot  // x15
+       _le32_bswap_4x  r2, r3, r4, r5,  r9, r10, r11
+         ldr           r9, [sp, #72]           // load LEN
+       eor             r2, r2, r0              // x12
+       eor             r3, r3, r1              // x13
+       eor             r4, r4, r6              // x14
+       eor             r5, r5, r7              // x15
+         subs          r9, #64                 // decrement and check LEN
+       stmia           r14!, {r2-r5}
+
+       beq             .Ldone\@
+
+.Lprepare_for_next_block\@:
+
+       // Stack: x0-x15 OUT IN LEN
+
+       // Increment block counter (x12)
+       add             r8, #1
+
+       // Store updated (OUT, IN, LEN)
+       str             r14, [sp, #64]
+       str             r12, [sp, #68]
+       str             r9, [sp, #72]
+
+         mov           r14, sp
+
+       // Store updated block counter (x12)
+       str             r8, [sp, #48]
+
+         sub           sp, #16
+
+       // Reload state and do next block
+       ldmia           r14!, {r0-r11}          // load x0-x11
+       __strd          r10, r11, sp, 8         // store x10-x11 before state
+       ldmia           r14, {r10-r12,r14}      // load x12-x15
+       b               .Lnext_block\@
+
+.Lxor_slowpath\@:
+       // Slow path: < 64 bytes remaining, or unaligned input or output buffer.
+       // We handle it by storing the 64 bytes of keystream to the stack, then
+       // XOR-ing the needed portion with the data.
+
+       // Allocate keystream buffer
+       sub             sp, #64
+       mov             r14, sp
+
+       // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
+       // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0.
+       // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
+
+       // Save keystream for x0-x3
+       __ldrd          r8, r9, sp, 96
+       __ldrd          r10, r11, sp, 104
+       add             X0, X0, r8
+       add             X1, X1, r9
+       add             X2, X2, r10
+       add             X3, X3, r11
+       _le32_bswap_4x  X0, X1, X2, X3,  r8, r9, r10
+       stmia           r14!, {X0-X3}
+
+       // Save keystream for x4-x7
+       __ldrd          r8, r9, sp, 112
+       __ldrd          r10, r11, sp, 120
+       add             X4, r8, X4, ror #brot
+       add             X5, r9, X5, ror #brot
+       add             X6, r10, X6, ror #brot
+       add             X7, r11, X7, ror #brot
+       _le32_bswap_4x  X4, X5, X6, X7,  r8, r9, r10
+         add           r8, sp, #64
+       stmia           r14!, {X4-X7}
+
+       // Save keystream for x8-x15
+       ldm             r8, {r0-r7}             // (x8-x9,x12-x15,x10-x11)
+       __ldrd          r8, r9, sp, 128
+       __ldrd          r10, r11, sp, 136
+       add             r0, r0, r8              // x8
+       add             r1, r1, r9              // x9
+       add             r6, r6, r10             // x10
+       add             r7, r7, r11             // x11
+       _le32_bswap_4x  r0, r1, r6, r7,  r8, r9, r10
+       stmia           r14!, {r0,r1,r6,r7}
+       __ldrd          r8, r9, sp, 144
+       __ldrd          r10, r11, sp, 152
+       add             r2, r8, r2, ror #drot   // x12
+       add             r3, r9, r3, ror #drot   // x13
+       add             r4, r10, r4, ror #drot  // x14
+       add             r5, r11, r5, ror #drot  // x15
+       _le32_bswap_4x  r2, r3, r4, r5,  r9, r10, r11
+       stmia           r14, {r2-r5}
+
+       // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN
+       // Registers: r8 is block counter, r12 is IN.
+
+       ldr             r9, [sp, #168]          // LEN
+       ldr             r14, [sp, #160]         // OUT
+       cmp             r9, #64
+         mov           r0, sp
+       movle           r1, r9
+       movgt           r1, #64
+       // r1 is number of bytes to XOR, in range [1, 64]
+
+.if __LINUX_ARM_ARCH__ < 6
+       orr             r2, r12, r14
+       tst             r2, #3                  // IN or OUT misaligned?
+       bne             .Lxor_next_byte\@
+.endif
+
+       // XOR a word at a time
+.rept 16
+       subs            r1, #4
+       blt             .Lxor_words_done\@
+       ldr             r2, [r12], #4
+       ldr             r3, [r0], #4
+       eor             r2, r2, r3
+       str             r2, [r14], #4
+.endr
+       b               .Lxor_slowpath_done\@
+.Lxor_words_done\@:
+       ands            r1, r1, #3
+       beq             .Lxor_slowpath_done\@
+
+       // XOR a byte at a time
+.Lxor_next_byte\@:
+       ldrb            r2, [r12], #1
+       ldrb            r3, [r0], #1
+       eor             r2, r2, r3
+       strb            r2, [r14], #1
+       subs            r1, #1
+       bne             .Lxor_next_byte\@
+
+.Lxor_slowpath_done\@:
+       subs            r9, #64
+       add             sp, #96
+       bgt             .Lprepare_for_next_block\@
+
+.Ldone\@:
+.endm  // _chacha
+
+/*
+ * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8],
+ *                  const u32 iv[4]);
+ */
+ENTRY(chacha20_arm)
+       cmp             r2, #0                  // len == 0?
+       bxeq            lr
+
+       push            {r0-r2,r4-r11,lr}
+
+       // Push state x0-x15 onto stack.
+       // Also store an extra copy of x10-x11 just before the state.
+
+       ldr             r4, [sp, #48]           // iv
+       mov             r0, sp
+       sub             sp, #80
+
+       // iv: x12-x15
+       ldm             r4, {X12,X13,X14,X15}
+       stmdb           r0!, {X12,X13,X14,X15}
+
+       // key: x4-x11
+       __ldrd          X8_X10, X9_X11, r3, 24
+       __strd          X8_X10, X9_X11, sp, 8
+       stmdb           r0!, {X8_X10, X9_X11}
+       ldm             r3, {X4-X9_X11}
+       stmdb           r0!, {X4-X9_X11}
+
+       // constants: x0-x3
+       adrl            X3, .Lexpand_32byte_k
+       ldm             X3, {X0-X3}
+       __strd          X0, X1, sp, 16
+       __strd          X2, X3, sp, 24
+
+       _chacha         20
+
+       add             sp, #76
+       pop             {r4-r11, pc}
+ENDPROC(chacha20_arm)
+
+/*
+ * void hchacha20_arm(const u32 state[16], u32 out[8]);
+ */
+ENTRY(hchacha20_arm)
+       push            {r1,r4-r11,lr}
+
+       mov             r14, r0
+       ldmia           r14!, {r0-r11}          // load x0-x11
+       push            {r10-r11}               // store x10-x11 to stack
+       ldm             r14, {r10-r12,r14}      // load x12-x15
+       sub             sp, #8
+
+       _chacha_permute 20
+
+       // Skip over (unused0-unused1, x10-x11)
+       add             sp, #16
+
+       // Fix up rotations of x12-x15
+       ror             X12, X12, #drot
+       ror             X13, X13, #drot
+         pop           {r4}                    // load 'out'
+       ror             X14, X14, #drot
+       ror             X15, X15, #drot
+
+       // Store (x0-x3,x12-x15) to 'out'
+       stm             r4, {X0,X1,X2,X3,X12,X13,X14,X15}
+
+       pop             {r4-r11,pc}
+ENDPROC(hchacha20_arm)
+
+#ifdef CONFIG_KERNEL_MODE_NEON
+/*
+ * This following NEON routine was ported from Andy Polyakov's implementation
+ * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine,
+ * since certain NEON code paths actually branch to it.
+ */
+
 .text
 #if defined(__thumb2__) || defined(__clang__)
 .syntax        unified
@@ -22,39 +484,6 @@
 #define ldrhsb ldrbhs
 #endif
 
-.align 5
-.Lsigma:
-.long  0x61707865,0x3320646e,0x79622d32,0x6b206574     @ endian-neutral
-.Lone:
-.long  1,0,0,0
-.word  -1
-
-.align 5
-ENTRY(chacha20_arm)
-       ldr     r12,[sp,#0]             @ pull pointer to counter and nonce
-       stmdb   sp!,{r0-r2,r4-r11,lr}
-       cmp     r2,#0                   @ len==0?
-#ifdef __thumb2__
-       itt     eq
-#endif
-       addeq   sp,sp,#4*3
-       beq     .Lno_data_arm
-       ldmia   r12,{r4-r7}             @ load counter and nonce
-       sub     sp,sp,#4*(16)           @ off-load area
-#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__)
-       sub     r14,pc,#100             @ .Lsigma
-#else
-       adr     r14,.Lsigma             @ .Lsigma
-#endif
-       stmdb   sp!,{r4-r7}             @ copy counter and nonce
-       ldmia   r3,{r4-r11}             @ load key
-       ldmia   r14,{r0-r3}             @ load sigma
-       stmdb   sp!,{r4-r11}            @ copy key
-       stmdb   sp!,{r0-r3}             @ copy sigma
-       str     r10,[sp,#4*(16+10)]     @ off-load "rx"
-       str     r11,[sp,#4*(16+11)]     @ off-load "rx"
-       b       .Loop_outer_enter
-
 .align 4
 .Loop_outer:
        ldmia   sp,{r0-r9}              @ load key material
@@ -748,11 +1177,8 @@ ENTRY(chacha20_arm)
 
 .Ldone:
        add     sp,sp,#4*(32+3)
-.Lno_data_arm:
        ldmia   sp!,{r4-r11,pc}
-ENDPROC(chacha20_arm)
 
-#ifdef CONFIG_KERNEL_MODE_NEON
 .align 5
 .Lsigma2:
 .long  0x61707865,0x3320646e,0x79622d32,0x6b206574     @ endian-neutral
lib/zinc/chacha20/chacha20-arm64.S renamed from lib/zinc/chacha20/chacha20-arm64-cryptogams.S +0 -0
similarity index 100%
rename from lib/zinc/chacha20/chacha20-arm64-cryptogams.S
rename to lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/chacha20/chacha20.c +0 -0
index 4354b874a6a5..fc4f74fca653 100644

lib/zinc/chacha20/chacha20.c line 19

Quoted 4 lines
lib/zinc/chacha20/chacha20.c +1 -0
@@ -16,6 +16,8 @@
 
 #if defined(CONFIG_ZINC_ARCH_X86_64)
 #include "chacha20-x86_64-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)

As above, just use CONFIG_ARM / CONFIG_ARM64 directly

Quoted 7 lines
lib/zinc/chacha20/chacha20.c +1 -1
+#include "chacha20-arm-glue.h"
 #else
 void __init chacha20_fpu_init(void)
 {
--

2.19.0

author maintainer
#72 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Hi Ard,

Quoted 1 line · Ard Biesheuvel

Please put comments like this below the ---

git-notes is nice for this indeed.

Quoted 1 line

Are these CONFIG_ symbols defined anywhere at this point?

Yes, they're introduced in the first zinc commit. There's no git-blame on git.kernel.org, presumably because it's expensive to compute, but there is on my personal instance, so this might help:
https://git.zx2c4.com/linux-dev/blame/lib/zinc/Kconfig?h=jd/wireguard

Quoted 2 lines

In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases

You mean to say that since these nobs are def_bool y and are essentially "depends on ARM", then I should just straight up use CONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.

Quoted 6 lines

and the
dependency on !CPU_32v3 you introduce (looking at the version of Kconfig at the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel but ARCH_RPC is built in v3 mode because of historical reasons while the actual core is a v4)

I added the !CPU_32v3 in my development tree after posting v6, so good to hear you're just looking straight at the updated tree. If you see things jump out in there prior to me posting v7, don't hesitate to let me know.

The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected, ARCH_RPC. Have a better suggestion than !CPU_32v3? It seems to me like so long as the kernel has CPU_32v3 as a thing in any form, I should mark Zinc as not supporting it, since we'll certainly be at least v4 and up. (Do you guys have any old Acorn ARM610 boxes sitting around for old time's sake at LinaroHQ? ;-)

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+#endif
+

No need to make asmlinkage declarations conditional

Yep, addressed in the IS_ENABLED cleanup.

Quoted 2 lines

if (IS_ENABLED())

Sorted.

Quoted 10 lines

"""
if (!IS_ENABLED(CONFIG_ARM))
return false;

hchacha20_arm(x, derived_key);
return true;
"""

and drop the #ifdefs

Also sorted.

Regards,
Jason

#73 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Quoted 18 lines · Jason A. Donenfeld

Hi Ard,

Quoted 1 line · Ard Biesheuvel

Please put comments like this below the ---

git-notes is nice for this indeed.

Quoted 1 line

Are these CONFIG_ symbols defined anywhere at this point?

Yes, they're introduced in the first zinc commit. There's no git-blame on git.kernel.org, presumably because it's expensive to compute, but there is on my personal instance, so this might help:
https://git.zx2c4.com/linux-dev/blame/lib/zinc/Kconfig?h=jd/wireguard

Quoted 2 lines

In any case, I don't think these is a reason for these, at least not on ARM/arm64. The 64-bitness is implied in both cases

Quoted 9 lines

You mean to say that since these nobs are def_bool y and are essentially "depends on ARM", then I should just straight up use CONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.

Sure. But probably better to be consistent then, and stop using CONFIG_ARM directly in your code.

Quoted 12 lines
Quoted 6 lines

and the
dependency on !CPU_32v3 you introduce (looking at the version of Kconfig at the end of the series) seems spurious to me. Was that added because of some kbuild robot report? (we don't support ARMv3 in the kernel but ARCH_RPC is built in v3 mode because of historical reasons while the actual core is a v4)

I added the !CPU_32v3 in my development tree after posting v6, so good to hear you're just looking straight at the updated tree. If you see things jump out in there prior to me posting v7, don't hesitate to let me know.

Quoted 4 lines

The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected, ARCH_RPC. Have a better suggestion than !CPU_32v3?

Yes, you could just add

asflags-$(CONFIG_CPU_32v3) += -march=armv4

with a comment stating that we don't actually support ARMv3 but only use it as a code generation target for reasons unrelated to the ISA

Quoted 5 lines

It seems to me like so long as the kernel has CPU_32v3 as a thing in any form, I should mark Zinc as not supporting it, since we'll certainly be at least v4 and up. (Do you guys have any old Acorn ARM610 boxes sitting around for old time's sake at LinaroHQ? ;-)

AFAIK we only support the StrongARM flavor of RiscPC which is ARMv4. But yes, some people do care deeply about these antiquated platforms, including Russell, and there is no particular to leave this one behind.

Quoted 27 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+#endif
+

No need to make asmlinkage declarations conditional

Yep, addressed in the IS_ENABLED cleanup.

Quoted 2 lines

if (IS_ENABLED())

Sorted.

Quoted 10 lines

"""
if (!IS_ENABLED(CONFIG_ARM))
return false;

hchacha20_arm(x, derived_key);
return true;
"""

and drop the #ifdefs

Also sorted.

Regards,
Jason

author maintainer
#74 lore raw

Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations

Hi Ard,

Quoted 11 lines · Ard Biesheuvel
Quoted 8 lines

You mean to say that since these nobs are def_bool y and are essentially "depends on ARM", then I should just straight up use CONFIG_ARM? I had thought about this, but figured this would make it easier to later make these optional or have other options block them need be, or even if the dependencies and requirements for having them changes (for example, with UML on x86). I think doing it this way gives us some flexibility later on. So if that's a compelling enough reason, I'd like to keep those.

Sure. But probably better to be consistent then, and stop using CONFIG_ARM directly in your code.

Ack.

Quoted 11 lines
Quoted 4 lines

The reason it was added was indeed because of:
https://lists.01.org/pipermail/kbuild-all/2018-September/053114.html
-- exactly what you suspected, ARCH_RPC. Have a better suggestion than !CPU_32v3?

Yes, you could just add

asflags-$(CONFIG_CPU_32v3) += -march=armv4

with a comment stating that we don't actually support ARMv3 but only use it as a code generation target for reasons unrelated to the ISA

Alright, I'll do exactly that. Though, if the rationale for this has to do only with codegen -- with what the C compiler does -- then shouldn't this be set globally for CONFIG_CPU_32v3? I couldn't find any macros that test against __LINUX_ARM_ARCH__ being 3 in the assembly, so this shouldn't be a problem I don't think. Maybe I'll send a patch.

Jason