Skip to contentSkip to conversation

WireGuard: Secure Network Tunnel

PATCH Superseded Jason A. Donenfeld 4 participants 38 messages Sep 14, 2018 – Sep 17, 2018 1 message not in the archive

Subsystems (30)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

v4 cover WireGuard: Secure Network Tunnel

Changes v3->v4:
- Remove mistaken double 07/17 patch.
- Fix whitespace issues in blake2s assembly.
- It's not possible to put compound literals into __initconst, so we now instead just use boring fixed size struct members.
- Move away from makefile ifdef maze and instead prefer kconfig values, which also makes the design a bit more modular too, which could help in the future.
- Port old crypto API implementations (ChaCha20 and Poly1305) to Zinc.
- Port security/keys/big_key to Zinc as second example of a good usage of Zinc.
- Document precisely what is different between the kernel code and CRYPTOGAMS code when the CRYPTOGAMS code is used.
- Move changelog to top of 00/20 message so that people can actually find it.

-----------------------------------------------------------

This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:

* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

-----------------------------------------------------------

WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.

Ample information, including documentation, installation instructions, and project details, is available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:

* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/

The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:

* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf

The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:

* https://www.wireguard.com/formal-verification/

This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.

It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.

Enjoy,
Jason

author maintainer
#1 lore raw

PATCH v4 1/20 asm: simd context helper API

Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:

    simd_context_t simd_context = simd_get();
    while ((item = get_item_from_queue()) != NULL) {
        encrypt_item(item, simd_context);
        simd_context = simd_relax(simd_context);
    }
    simd_put(simd_context);

The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.

---
 arch/alpha/include/asm/Kbuild      |  5 ++--
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/simd.h        | 42 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/simd.h      | 37 +++++++++++++++++++++-----
 arch/c6x/include/asm/Kbuild        |  3 ++-
 arch/h8300/include/asm/Kbuild      |  3 ++-
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/nds32/include/asm/Kbuild      |  7 ++---
 arch/nios2/include/asm/Kbuild      |  1 +
 arch/openrisc/include/asm/Kbuild   |  7 ++---
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  3 ++-
 arch/riscv/include/asm/Kbuild      |  3 ++-
 arch/s390/include/asm/Kbuild       |  3 ++-
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/um/include/asm/Kbuild         |  3 ++-
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/simd.h        | 30 ++++++++++++++++++++-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/simd.h         | 15 +++++++++++
 include/linux/simd.h               | 28 ++++++++++++++++++++
 26 files changed, 180 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm/include/asm/simd.h
 create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +3 -2
index 0580cb8c84b2..07b2c1025d34 100644
@@ -2,14 +2,15 @@
 
 
 generic-y += compat.h
+generic-y += current.h
 generic-y += exec.h
 generic-y += export.h
 generic-y += fb.h
 generic-y += irq_work.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h
arch/arc/include/asm/Kbuild +1 -0
index feed50ce89fa..a7f4255f1649 100644
@@ -22,6 +22,7 @@ generic-y += parport.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/arm/include/asm/simd.h new file +42 -0

42 lines not on this page · raw message

arch/arm64/include/asm/simd.h +31 -6
index 6495cc51246f..058c336de38d 100644
@@ -1,11 +1,10 @@
-/*
- * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+/* SPDX-License-Identifier: GPL-2.0
  *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published
- * by the Free Software Foundation.
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  */
 
+#include <linux/simd.h>
 #ifndef __ASM_SIMD_H
 #define __ASM_SIMD_H
 
@@ -16,6 +15,8 @@
 #include <linux/types.h>
 
 #ifdef CONFIG_KERNEL_MODE_NEON
+#include <asm/neon.h>
+#include <asm/simd.h>
 
 DECLARE_PER_CPU(bool, kernel_neon_busy);
 
@@ -40,12 +41,36 @@ static __must_check inline bool may_use_simd(void)
 		!this_cpu_read(kernel_neon_busy);
 }
 
+static inline simd_context_t simd_get(void)
+{
+	bool have_simd = may_use_simd();
+	if (have_simd)
+		kernel_neon_begin();
+	return have_simd ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t prior_context)
+{
+	if (prior_context != HAVE_NO_SIMD)
+		kernel_neon_end();
+}
+
 #else /* ! CONFIG_KERNEL_MODE_NEON */
 
-static __must_check inline bool may_use_simd(void) {
+static __must_check inline bool may_use_simd(void)
+{
 	return false;
 }
 
+static inline simd_context_t simd_get(void)
+{
+	return HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t prior_context)
+{
+}
+
 #endif /* ! CONFIG_KERNEL_MODE_NEON */
 
 #endif
arch/c6x/include/asm/Kbuild +2 -1
index 33a2c94fed0d..22f3d8333c74 100644
@@ -5,8 +5,8 @@ generic-y += compat.h
 generic-y += current.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h
 generic-y += preempt.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += tlbflush.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
index a5d0b2991f47..f5c2f12d593e 100644
@@ -8,8 +8,8 @@ generic-y += current.h
 generic-y += delay.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h
 generic-y += scatterlist.h
 generic-y += sections.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += spinlock.h
 generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
index dd2fd9c0d292..217d4695fd8a 100644
@@ -29,6 +29,7 @@ generic-y += rwsem.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
index 557bbc8ba9f5..41c5ebdf79e5 100644
@@ -4,6 +4,7 @@ generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += vtime.h
 generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
index a4b8d3331a9e..73898dd1a4d0 100644
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
index 569ba9e670c1..7a877eea99d3 100644
@@ -25,6 +25,7 @@ generic-y += parport.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += syscalls.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
index 58351e48421e..e8868e0fb2c3 100644
@@ -16,6 +16,7 @@ generic-y += qrwlock.h
 generic-y += qspinlock.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
index dbc4e5422550..603c1d020620 100644
@@ -7,14 +7,14 @@ generic-y += bug.h
 generic-y += bugs.h
 generic-y += checksum.h
 generic-y += clkdev.h
-generic-y += cmpxchg.h
 generic-y += cmpxchg-local.h
+generic-y += cmpxchg.h
 generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
 generic-y += shmbuf.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += stat.h
 generic-y += switch_to.h
 generic-y += timex.h
 generic-y += topology.h
 generic-y += trace_clock.h
-generic-y += xor.h
 generic-y += unaligned.h
 generic-y += user.h
 generic-y += vga.h
 generic-y += word-at-a-time.h
+generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
index 8fde4fa2c34f..571a9d9ad107 100644
@@ -33,6 +33,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
index eb87cd8327c8..5e9f2f4c4d39 100644
@@ -28,12 +28,13 @@ generic-y += module.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
-generic-y += qspinlock_types.h
-generic-y += qspinlock.h
-generic-y += qrwlock_types.h
 generic-y += qrwlock.h
+generic-y += qrwlock_types.h
+generic-y += qspinlock.h
+generic-y += qspinlock_types.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += string.h
 generic-y += switch_to.h
 generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
index 2013d639e735..97970b4d05ab 100644
@@ -17,6 +17,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += seccomp.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
index 3196d227e351..64290f48e733 100644
@@ -4,7 +4,8 @@ generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
+generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += vtime.h
-generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
index efdbe311e936..6669b7374c0a 100644
@@ -5,9 +5,9 @@ generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h
 generic-y += shmbuf.h
 generic-y += shmparam.h
 generic-y += signal.h
+generic-y += simd.h
 generic-y += socket.h
 generic-y += sockios.h
 generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
index e3239772887a..7a26dc6ce815 100644
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h
 generic-y += asm-offsets.h
 generic-y += cacheflush.h
 generic-y += device.h
+generic-y += div64.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
-generic-y += div64.h
 generic-y += emergency-restart.h
 generic-y += export.h
 generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
index 6a5609a55965..8e64ff35a933 100644
@@ -16,6 +16,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += trace_clock.h
 generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
index 410b263ef5c8..72b9e08fb350 100644
@@ -17,5 +17,6 @@ generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
index b10dde6cb793..d37288b08dd2 100644
@@ -16,15 +16,16 @@ generic-y += io.h
 generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += kdebug.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += param.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += switch_to.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
 generic-y += xor.h
-generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
index bfc7abe77905..98a908720bbd 100644
@@ -27,6 +27,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += syscalls.h
 generic-y += topology.h
arch/x86/include/asm/simd.h +29 -1
index a341c878e977..79411178988a 100644
@@ -1,4 +1,11 @@
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
 
 #include <asm/fpu/api.h>
 
@@ -10,3 +17,24 @@ static __must_check inline bool may_use_simd(void)
 {
 	return irq_fpu_usable();
 }
+
+static inline simd_context_t simd_get(void)
+{
+	bool have_simd = false;
+#if !defined(CONFIG_UML)
+	have_simd = may_use_simd();
+	if (have_simd)
+		kernel_fpu_begin();
+#endif
+	return have_simd ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t prior_context)
+{
+#if !defined(CONFIG_UML)
+	if (prior_context != HAVE_NO_SIMD)
+		kernel_fpu_end();
+#endif
+}
+
+#endif /* _ASM_SIMD_H */
arch/xtensa/include/asm/Kbuild +1 -0
index 82c756431b49..7950f359649d 100644
@@ -24,6 +24,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
include/asm-generic/simd.h +15 -0
index d0343d58a74a..fad899a5a92d 100644
@@ -1,5 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
+
 #include <linux/hardirq.h>
 
 /*
@@ -13,3 +17,14 @@ static __must_check inline bool may_use_simd(void)
 {
 	return !in_interrupt();
 }
+
+static inline simd_context_t simd_get(void)
+{
+	return HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t prior_context)
+{
+}
+
+#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +28 -0
new file mode 100644
index 000000000000..f62d047188bf
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _SIMD_H
+#define _SIMD_H
+
+typedef enum {
+	HAVE_NO_SIMD,
+	HAVE_FULL_SIMD
+} simd_context_t;
+
+#include <linux/sched.h>
+#include <asm/simd.h>
+
+static inline simd_context_t simd_relax(simd_context_t prior_context)
+{
+#ifdef CONFIG_PREEMPT
+	if (prior_context != HAVE_NO_SIMD && need_resched()) {
+		simd_put(prior_context);
+		return simd_get();
+	}
+#endif
+	return prior_context;
+}
+
+#endif /* _SIMD_H */
Signature · 1 line
2.19.0
author maintainer
#2 lore raw

PATCH v4 2/20 zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld

Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.

Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:

* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.

* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.

* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.

This last observation has seen itself play out several times over and over again within the kernel:

* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.

* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.

* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.

Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.

It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.

All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].

Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.

SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.

More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.

Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.

Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.

Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.

As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.

[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 MAINTAINERS       |  8 ++++++++
 lib/Kconfig       |  2 ++
 lib/Makefile      |  2 ++
 lib/zinc/Kconfig  | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 lib/zinc/Makefile |  8 ++++++++
 lib/zinc/main.c   | 31 +++++++++++++++++++++++++++++++
 6 files changed, 97 insertions(+)
 create mode 100644 lib/zinc/Kconfig
 create mode 100644 lib/zinc/Makefile
 create mode 100644 lib/zinc/main.c
MAINTAINERS +8 -0
index 2ef884b883c3..d2092e52320d 100644
@@ -16160,6 +16160,14 @@ Q:	https://patchwork.linuxtv.org/project/linux-media/list/
 S:	Maintained
 F:	drivers/media/dvb-frontends/zd1301_demod*
 
+ZINC CRYPTOGRAPHY LIBRARY
+M:	Jason A. Donenfeld <[email protected]>
+M:	Samuel Neves <[email protected]>
+S:	Maintained
+F:	lib/zinc/
+F:	include/zinc/
+L:	[email protected]
+
 ZPOOL COMPRESSED PAGE STORAGE API
 M:	Dan Streetman <[email protected]>
 L:	[email protected]
lib/Kconfig +2 -0
index a3928d4438b5..3e6848269c66 100644
@@ -485,6 +485,8 @@ config GLOB_SELFTEST
 	  module load) by a small amount, so you're welcome to play with
 	  it, but you probably don't need it.
 
+source "lib/zinc/Kconfig"
+
 #
 # Netlink attribute parsing support is select'ed if needed
 #
lib/Makefile +2 -0
index ca3f7ebb900d..3f16e35d2c11 100644
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o
 
 obj-$(CONFIG_ASN1) += asn1_decoder.o
 
+obj-$(CONFIG_ZINC) += zinc/
+
 obj-$(CONFIG_FONT_SUPPORT) += fonts/
 
 obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +46 -0

46 lines not on this page · raw message

lib/zinc/Makefile new file +8 -0
new file mode 100644
index 000000000000..dad47573de42
@@ -0,0 +1,8 @@
+ccflags-y := -O3
+ccflags-y += -Wframe-larger-than=8192
+ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
+
+zinc-y += main.o
+
+obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/main.c new file +31 -0
new file mode 100644
index 000000000000..ceece33ff5a7
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <linux/init.h>
+#include <linux/module.h>
+
+#ifdef DEBUG
+#define selftest(which) do { \
+	if (!which ## _selftest()) \
+		return -ENOTRECOVERABLE; \
+} while (0)
+#else
+#define selftest(which)
+#endif
+
+static int __init mod_init(void)
+{
+	return 0;
+}
+
+static void __exit mod_exit(void)
+{
+}
+
+module_init(mod_init);
+module_exit(mod_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Zinc cryptography library");
+MODULE_AUTHOR("Jason A. Donenfeld <[email protected]>");
Signature · 1 line
2.19.0
author maintainer
#3 lore raw

PATCH v4 3/20 zinc: ChaCha20 generic C implementation

In reply to Jason A. Donenfeld

This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.

Information: https://cr.yp.to/chacha.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20.h      |  54 +++++++++++
 lib/zinc/Kconfig             |   5 ++
 lib/zinc/Makefile            |   4 +
 lib/zinc/chacha20/chacha20.c | 168 +++++++++++++++++++++++++++++++++++
 lib/zinc/main.c              |   5 ++
 5 files changed, 236 insertions(+)
 create mode 100644 include/zinc/chacha20.h
 create mode 100644 lib/zinc/chacha20/chacha20.c
include/zinc/chacha20.h new file +54 -0
new file mode 100644
index 000000000000..3c2c2f72d88a
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20_H
+#define _ZINC_CHACHA20_H
+
+#include <asm/unaligned.h>
+#include <linux/simd.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+enum {
+	CHACHA20_IV_SIZE = 16,
+	CHACHA20_KEY_SIZE = 32,
+	CHACHA20_BLOCK_SIZE = 64,
+	CHACHA20_BLOCK_WORDS = CHACHA20_BLOCK_SIZE / sizeof(u32),
+	HCHACHA20_KEY_SIZE = 32,
+	HCHACHA20_NONCE_SIZE = 16
+};
+
+struct chacha20_ctx {
+	u32 key[8];
+	u32 counter[4];
+} __aligned(32);
+
+void chacha20_fpu_init(void);
+
+static inline void chacha20_init(struct chacha20_ctx *state,
+				 const u8 key[CHACHA20_KEY_SIZE],
+				 const u64 nonce)
+{
+	state->key[0] = get_unaligned_le32(key + 0);
+	state->key[1] = get_unaligned_le32(key + 4);
+	state->key[2] = get_unaligned_le32(key + 8);
+	state->key[3] = get_unaligned_le32(key + 12);
+	state->key[4] = get_unaligned_le32(key + 16);
+	state->key[5] = get_unaligned_le32(key + 20);
+	state->key[6] = get_unaligned_le32(key + 24);
+	state->key[7] = get_unaligned_le32(key + 28);
+	state->counter[0] = state->counter[1] = 0;
+	state->counter[2] = nonce & U32_MAX;
+	state->counter[3] = nonce >> 32;
+}
+void chacha20(struct chacha20_ctx *state, u8 *dst, const u8 *src, u32 len,
+	      simd_context_t simd_context);
+
+/* Derived key should be 32-bit aligned */
+void hchacha20(u8 derived_key[CHACHA20_KEY_SIZE],
+	       const u8 nonce[HCHACHA20_NONCE_SIZE],
+	       const u8 key[HCHACHA20_KEY_SIZE], simd_context_t simd_context);
+
+#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +5 -0
index 5980c411af0d..e7d396d61607 100644
@@ -1,6 +1,11 @@
 config ZINC
 	tristate
 
+config ZINC_CHACHA20
+	bool
+	select ZINC
+	select CRYPTO_ALGAPI
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	depends on ZINC
lib/zinc/Makefile +4 -0
index dad47573de42..0b5a964bfba6 100644
@@ -3,6 +3,10 @@ ccflags-y += -Wframe-larger-than=8192
 ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
+ifeq ($(CONFIG_ZINC_CHACHA20),y)
+zinc-y += chacha20/chacha20.o
+endif
+
 zinc-y += main.o
 
 obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/chacha20/chacha20.c new file +168 -0

168 lines not on this page · raw message

lib/zinc/main.c +5 -0
index ceece33ff5a7..7e8e84b706b7 100644
@@ -3,6 +3,8 @@
  * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  */
 
+#include <zinc/chacha20.h>
+
 #include <linux/init.h>
 #include <linux/module.h>
 
@@ -17,6 +19,9 @@
 
 static int __init mod_init(void)
 {
+#ifdef CONFIG_ZINC_CHACHA20
+	chacha20_fpu_init();
+#endif
 	return 0;
 }
 
Signature · 1 line
2.19.0
author maintainer
#4 lore raw

PATCH v4 4/20 zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed (prefixed with .L) to fit kernel conventions.
- Constants have been rearranged so that they are closer to the code that is using them. [ARM only]
- The neon code can jump to the scalar code when it makes sense to do so.
- The neon_512 function as a separate function has been removed, leaving the decision up to the main neon entry point. [ARM64 only]

After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.

ARM:

-ChaCha20_ctr32:
-.LChaCha20_ctr32:
+ENTRY(chacha20_arm)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
- sub r14,pc,#16 @ ChaCha20_ctr32
- adr r14,.LChaCha20_ctr32
cmp r2,#0 @ len==0?
itt eq
addeq sp,sp,#4*3
- beq .Lno_data
- cmp r2,#192 @ test len
- bls .Lshort
- ldr r4,[r14,#-32]
- ldr r4,[r14,r4]
- ldr r4,[r4]
- tst r4,#ARMV7_NEON
- bne .LChaCha20_neon
+ beq .Lno_data_arm
.Lshort:
ldmia r12,{r4-r7} @ load counter and nonce
sub sp,sp,#4*(16) @ off-load area
- sub r14,r14,#64 @ .Lsigma
+ sub r14,pc,#100 @ .Lsigma
+ adr r14,.Lsigma @ .Lsigma
stmdb sp!,{r4-r7} @ copy counter and nonce
ldmia r3,{r4-r11} @ load key
ldmia r14,{r0-r3} @ load sigma
@@ -617,14 +615,25 @@

.Ldone:
add sp,sp,#4*(32+3)
-.Lno_data:
+.Lno_data_arm:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_arm)

-ChaCha20_neon:
+ENTRY(chacha20_neon)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
-.LChaCha20_neon:
- adr r14,.Lsigma
+ cmp r2,#0 @ len==0?
+ itt eq
+ addeq sp,sp,#4*3
+ beq .Lno_data_neon
+ cmp r2,#192 @ test len
+ bls .Lshort
+.Lchacha20_neon_begin:
+ adr r14,.Lsigma2
vstmdb sp!,{d8-d15} @ ABI spec says so
stmdb sp!,{r0-r3}

@@ -1265,4 +1274,6 @@
add sp,sp,#4*(32+4)
vldmia sp,{d8-d15}
add sp,sp,#4*(16+3)
+.Lno_data_neon:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_neon)

ARM64:

-ChaCha20_ctr32:
+ENTRY(chacha20_arm)
cbz x2,.Labort
- adr x5,.LOPENSSL_armcap_P
- cmp x2,#192
- b.lo .Lshort
- ldrsw x6,[x5]
- ldr x6,[x5]
- ldr w17,[x6,x5]
- tst w17,#ARMV7_NEON
- b.ne ChaCha20_neon
-
.Lshort:
stp x29,x30,[sp,#-96]!
add x29,sp,#0
@@ -279,8 +274,13 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
+ENDPROC(chacha20_arm)
+
+ENTRY(chacha20_neon)
+ cbz x2,.Labort_neon
+ cmp x2,#192
+ b.lo .Lshort

-ChaCha20_neon:
stp x29,x30,[sp,#-96]!
add x29,sp,#0

@@ -763,16 +763,6 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
-ChaCha20_512_neon:
- stp x29,x30,[sp,#-96]!
- add x29,sp,#0
-
- adr x5,.Lsigma
- stp x19,x20,[sp,#16]
- stp x21,x22,[sp,#32]
- stp x23,x24,[sp,#48]
- stp x25,x26,[sp,#64]
- stp x27,x28,[sp,#80]

.L512_or_more_neon:
sub sp,sp,#128+64
@@ -1920,4 +1910,6 @@
ldp x25,x26,[x29,#64]
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
+.Labort_neon:
ret
+ENDPROC(chacha20_neon)

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                     |    8 +
 lib/zinc/chacha20/chacha20-arm-glue.h |   50 +
 lib/zinc/chacha20/chacha20-arm.S      | 1473 +++++++++++++++++++
 lib/zinc/chacha20/chacha20-arm64.S    | 1942 +++++++++++++++++++++++++
 4 files changed, 3473 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-arm.S
 create mode 100644 lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/Makefile +8 -0
index 0b5a964bfba6..8d14cb13349a 100644
@@ -5,6 +5,14 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 ifeq ($(CONFIG_ZINC_CHACHA20),y)
 zinc-y += chacha20/chacha20.o
+ifeq ($(CONFIG_ZINC_ARCH_ARM),y)
+zinc-y += chacha20/chacha20-arm.o
+CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h
+endif
+ifeq ($(CONFIG_ZINC_ARCH_ARM64),y)
+zinc-y += chacha20/chacha20-arm64.o
+CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h
+endif
 endif
 
 zinc-y += main.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +50 -0

50 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm.S new file +1473 -0

1,473 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm64.S new file +1942 -0

1,463 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#5 lore raw

PATCH v4 5/20 zinc: ChaCha20 x86_64 implementation

In reply to Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.

- When aligning the stack, the original code was saving %rsp to %r9.

    To keep objtool happy, we use instead the DRAP idiom to save %rsp
    to %r10:

      leaq    8(%rsp),%r10
      ... code here ...
      leaq    -8(%r10),%rsp

- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                        |    4 +
 lib/zinc/chacha20/chacha20-x86_64-glue.h |  102 +
 lib/zinc/chacha20/chacha20-x86_64.S      | 2632 ++++++++++++++++++++++
 3 files changed, 2738 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +4 -0
index 8d14cb13349a..32e4bd94ea0b 100644
@@ -5,6 +5,10 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 ifeq ($(CONFIG_ZINC_CHACHA20),y)
 zinc-y += chacha20/chacha20.o
+ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
+zinc-y += chacha20/chacha20-x86_64.o
+CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-x86_64-glue.h
+endif
 ifeq ($(CONFIG_ZINC_ARCH_ARM),y)
 zinc-y += chacha20/chacha20-arm.o
 CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +102 -0

102 lines not on this page · raw message

lib/zinc/chacha20/chacha20-x86_64.S new file +2632 -0

2,632 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#6 lore raw

PATCH v4 6/20 zinc: ChaCha20 MIPS32r2 implementation

In reply to Jason A. Donenfeld

This MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Ralf Baechle <[email protected]>
Cc Paul Burton <[email protected]>
Cc James Hogan <[email protected]>
---
 lib/zinc/Makefile                      |   4 +
 lib/zinc/chacha20/chacha20-mips-glue.h |  28 ++
 lib/zinc/chacha20/chacha20-mips.S      | 474 +++++++++++++++++++++++++
 3 files changed, 506 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-mips-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-mips.S
lib/zinc/Makefile +4 -0
index 32e4bd94ea0b..9f6a5e65d729 100644
@@ -17,6 +17,10 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM64),y)
 zinc-y += chacha20/chacha20-arm64.o
 CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-arm-glue.h
 endif
+ifeq ($(CONFIG_ZINC_ARCH_MIPS)$(CONFIG_CPU_MIPS32_R2),yy)
+zinc-y += chacha20/chacha20-mips.o
+CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-mips-glue.h
+endif
 endif
 
 zinc-y += main.o
lib/zinc/chacha20/chacha20-mips-glue.h new file +28 -0
new file mode 100644
index 000000000000..5b2c8cec36c8
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <zinc/chacha20.h>
+
+asmlinkage void chacha20_mips(u8 *out, const u8 *in, const size_t len,
+			      const u32 key[8], const u32 counter[4]);
+void __init chacha20_fpu_init(void)
+{
+}
+
+static inline bool chacha20_arch(u8 *dst, const u8 *src, const size_t len,
+				 const u32 key[8], const u32 counter[4],
+				 simd_context_t simd_context)
+{
+	chacha20_mips(dst, src, len, key, counter);
+	return true;
+}
+
+static inline bool hchacha20_arch(u8 *derived_key, const u8 *nonce,
+				  const u8 *key, simd_context_t simd_context)
+{
+	return false;
+}
+
+#define HAVE_CHACHA20_ARCH_IMPLEMENTATION
lib/zinc/chacha20/chacha20-mips.S new file +474 -0

474 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#7 lore raw

PATCH v4 7/20 zinc: Poly1305 generic C implementations and selftest

In reply to Jason A. Donenfeld

These two C implementations -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel and for usage with accelerated primitives.

Information: https://cr.yp.to/mac.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/poly1305.h              |  38 ++
 lib/zinc/Kconfig                     |   4 +
 lib/zinc/Makefile                    |   4 +
 lib/zinc/main.c                      |   5 +
 lib/zinc/poly1305/poly1305-donna32.h | 205 +++++++
 lib/zinc/poly1305/poly1305-donna64.h | 182 ++++++
 lib/zinc/poly1305/poly1305.c         | 131 ++++
 lib/zinc/selftest/poly1305.h         | 876 +++++++++++++++++++++++++++
 8 files changed, 1445 insertions(+)
 create mode 100644 include/zinc/poly1305.h
 create mode 100644 lib/zinc/poly1305/poly1305-donna32.h
 create mode 100644 lib/zinc/poly1305/poly1305-donna64.h
 create mode 100644 lib/zinc/poly1305/poly1305.c
 create mode 100644 lib/zinc/selftest/poly1305.h
include/zinc/poly1305.h new file +38 -0
new file mode 100644
index 000000000000..338430c8477a
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_POLY1305_H
+#define _ZINC_POLY1305_H
+
+#include <linux/simd.h>
+#include <linux/types.h>
+
+enum poly1305_lengths {
+	POLY1305_BLOCK_SIZE = 16,
+	POLY1305_KEY_SIZE = 32,
+	POLY1305_MAC_SIZE = 16
+};
+
+struct poly1305_ctx {
+	u8 opaque[24 * sizeof(u64)];
+	u32 nonce[4];
+	u8 data[POLY1305_BLOCK_SIZE];
+	size_t num;
+} __aligned(8);
+
+void poly1305_fpu_init(void);
+
+void poly1305_init(struct poly1305_ctx *ctx, const u8 key[POLY1305_KEY_SIZE],
+		   simd_context_t simd_context);
+void poly1305_update(struct poly1305_ctx *ctx, const u8 *input, size_t len,
+		     simd_context_t simd_context);
+void poly1305_final(struct poly1305_ctx *ctx, u8 mac[POLY1305_MAC_SIZE],
+		    simd_context_t simd_context);
+
+#ifdef DEBUG
+bool poly1305_selftest(void);
+#endif
+
+#endif /* _ZINC_POLY1305_H */
lib/zinc/Kconfig +4 -0
index e7d396d61607..bc8c61334362 100644
@@ -6,6 +6,10 @@ config ZINC_CHACHA20
 	select ZINC
 	select CRYPTO_ALGAPI
 
+config ZINC_POLY1305
+	bool
+	select ZINC
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	depends on ZINC
lib/zinc/Makefile +4 -0
index 9f6a5e65d729..d1e3892e06d9 100644
@@ -23,6 +23,10 @@ CFLAGS_chacha20.o += -include $(srctree)/$(src)/chacha20/chacha20-mips-glue.h
 endif
 endif
 
+ifeq ($(CONFIG_ZINC_POLY1305),y)
+zinc-y += poly1305/poly1305.o
+endif
+
 zinc-y += main.o
 
 obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/main.c +5 -0
index 7e8e84b706b7..d871dd406a5c 100644
@@ -4,6 +4,7 @@
  */
 
 #include <zinc/chacha20.h>
+#include <zinc/poly1305.h>
 
 #include <linux/init.h>
 #include <linux/module.h>
@@ -21,6 +22,10 @@ static int __init mod_init(void)
 {
 #ifdef CONFIG_ZINC_CHACHA20
 	chacha20_fpu_init();
+#endif
+#ifdef CONFIG_ZINC_POLY1305
+	poly1305_fpu_init();
+	selftest(poly1305);
 #endif
 	return 0;
 }
lib/zinc/poly1305/poly1305-donna32.h new file +205 -0

205 lines not on this page · raw message

lib/zinc/poly1305/poly1305-donna64.h new file +182 -0

182 lines not on this page · raw message

lib/zinc/poly1305/poly1305.c new file +131 -0

131 lines not on this page · raw message

lib/zinc/selftest/poly1305.h new file +876 -0

876 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#8 lore raw

PATCH v4 8/20 zinc: Poly1305 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed to fit kernel conventions.
- The neon code can jump to the scalar code when it makes sense to do so.

After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.

ARM:

-poly1305_init:
-.Lpoly1305_init:
+ENTRY(poly1305_init_arm)
stmdb sp!,{r4-r11}

eor r3,r3,r3
@@ -18,8 +25,6 @@
moveq r0,#0
beq .Lno_key

- adr r11,.Lpoly1305_init
- ldr r12,.LOPENSSL_armcap
ldrb r4,[r1,#0]
mov r10,#0x0fffffff
ldrb r5,[r1,#1]
@@ -34,8 +39,6 @@
ldrb r7,[r1,#6]
and r4,r4,r10

- ldr r12,[r11,r12] @ OPENSSL_armcap_P
- ldr r12,[r12]
ldrb r8,[r1,#7]
orr r5,r5,r6,lsl#8
ldrb r6,[r1,#8]
@@ -45,22 +48,6 @@
ldrb r8,[r1,#10]
and r5,r5,r3

- tst r12,#ARMV7_NEON @ check for NEON
- adr r9,poly1305_blocks_neon
- adr r11,poly1305_blocks
- it ne
- movne r11,r9
- adr r12,poly1305_emit
- adr r10,poly1305_emit_neon
- it ne
- movne r12,r10
- itete eq
- addeq r12,r11,#(poly1305_emit-.Lpoly1305_init)
- addne r12,r11,#(poly1305_emit_neon-.Lpoly1305_init)
- addeq r11,r11,#(poly1305_blocks-.Lpoly1305_init)
- addne r11,r11,#(poly1305_blocks_neon-.Lpoly1305_init)
- orr r12,r12,#1 @ thumb-ify address
- orr r11,r11,#1
ldrb r9,[r1,#11]
orr r6,r6,r7,lsl#8
ldrb r7,[r1,#12]
@@ -79,17 +66,16 @@
str r6,[r0,#8]
and r7,r7,r3
str r7,[r0,#12]
- stmia r2,{r11,r12} @ fill functions table
- mov r0,#1
- mov r0,#0
.Lno_key:
ldmia sp!,{r4-r11}
bx lr @ bx lr
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_blocks:
-.Lpoly1305_blocks:
+ENDPROC(poly1305_init_arm)
+
+ENTRY(poly1305_blocks_arm)
+.Lpoly1305_blocks_arm:
stmdb sp!,{r3-r11,lr}

ands r2,r2,#-16
@@ -231,10 +217,11 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_emit:
+ENDPROC(poly1305_blocks_arm)
+
+ENTRY(poly1305_emit_arm)
stmdb sp!,{r4-r11}
.Lpoly1305_emit_enter:
-
ldmia r0,{r3-r7}
adds r8,r3,#5 @ compare to modulus
adcs r9,r4,#0
@@ -305,8 +292,12 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
+ENDPROC(poly1305_emit_arm)
+
+

-poly1305_init_neon:
+ENTRY(poly1305_init_neon)
+.Lpoly1305_init_neon:
ldr r4,[r0,#20] @ load key base 2^32
ldr r5,[r0,#24]
ldr r6,[r0,#28]
@@ -515,8 +506,9 @@
vst1.32 {d8[1]},[r7]

bx lr @ bx lr
+ENDPROC(poly1305_init_neon)

-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr ip,[r0,#36] @ is_base2_26
ands r2,r2,#-16
beq .Lno_data_neon
@@ -524,7 +516,7 @@
cmp r2,#64
bhs .Lenter_neon
tst ip,ip @ is_base2_26?
- beq .Lpoly1305_blocks
+ beq .Lpoly1305_blocks_arm

.Lenter_neon:
stmdb sp!,{r4-r7}
@@ -534,7 +526,7 @@
bne .Lbase2_26_neon

stmdb sp!,{r1-r3,lr}
- bl poly1305_init_neon
+ bl .Lpoly1305_init_neon

ldr r4,[r0,#0] @ load hash value base 2^32
ldr r5,[r0,#4]
@@ -989,8 +981,9 @@
ldmia sp!,{r4-r7}
.Lno_data_neon:
bx lr @ bx lr
+ENDPROC(poly1305_blocks_neon)

-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr ip,[r0,#36] @ is_base2_26

stmdb sp!,{r4-r11}
@@ -1055,6 +1048,6 @@

ldmia sp!,{r4-r11}
bx lr @ bx lr
+ENDPROC(poly1305_emit_neon)

ARM64:

-poly1305_init:
+ENTRY(poly1305_init_arm)
cmp x1,xzr
stp xzr,xzr,[x0] // zero hash value
stp xzr,xzr,[x0,#16] // [along with is_base2_26]
@@ -11,14 +15,9 @@
csel x0,xzr,x0,eq
b.eq .Lno_key

- ldrsw x11,.LOPENSSL_armcap_P
- ldr x11,.LOPENSSL_armcap_P
- adr x10,.LOPENSSL_armcap_P
-
ldp x7,x8,[x1] // load key
mov x9,#0xfffffffc0fffffff
movk x9,#0x0fff,lsl#48
- ldr w17,[x10,x11]
rev x7,x7 // flip bytes
rev x8,x8
and x7,x7,x9 // &=0ffffffc0fffffff
@@ -26,24 +25,11 @@
and x8,x8,x9 // &=0ffffffc0ffffffc
stp x7,x8,[x0,#32] // save key value

- tst w17,#ARMV7_NEON
-
- adr x12,poly1305_blocks
- adr x7,poly1305_blocks_neon
- adr x13,poly1305_emit
- adr x8,poly1305_emit_neon
-
- csel x12,x12,x7,eq
- csel x13,x13,x8,eq
-
- stp w12,w13,[x2]
- stp x12,x13,[x2]
-
- mov x0,#1
.Lno_key:
ret
+ENDPROC(poly1305_init_arm)

-poly1305_blocks:
+ENTRY(poly1305_blocks_arm)
ands x2,x2,#-16
b.eq .Lno_data

@@ -100,8 +86,9 @@

.Lno_data:
ret
+ENDPROC(poly1305_blocks_arm)

-poly1305_emit:
+ENTRY(poly1305_emit_arm)
ldp x4,x5,[x0] // load hash base 2^64
ldr x6,[x0,#16]
ldp x10,x11,[x2] // load nonce
@@ -124,7 +111,9 @@
stp x4,x5,[x1] // write result

ret
-poly1305_mult:
+ENDPROC(poly1305_emit_arm)
+
+__poly1305_mult:
mul x12,x4,x7 // h0*r0
umulh x13,x4,x7

@@ -158,7 +147,7 @@

ret

-poly1305_splat:
+__poly1305_splat:
and x12,x4,#0x03ffffff // base 2^64 -> base 2^26
ubfx x13,x4,#26,#26
extr x14,x5,x4,#52
@@ -182,11 +171,11 @@

ret

-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr x17,[x0,#24]
cmp x2,#128
b.hs .Lblocks_neon
- cbz x17,poly1305_blocks
+ cbz x17,poly1305_blocks_arm

.Lblocks_neon:
stp x29,x30,[sp,#-80]!
@@ -232,7 +221,7 @@
adcs x5,x5,x13
adc x6,x6,x3

- bl poly1305_mult
+ bl __poly1305_mult
ldr x30,[sp,#8]

cbz x3,.Lstore_base2_64_neon
@@ -274,7 +263,7 @@
adcs x5,x5,x13
adc x6,x6,x3

- bl poly1305_mult
+ bl __poly1305_mult

.Linit_neon:
and x10,x4,#0x03ffffff // base 2^64 -> base 2^26
@@ -301,19 +290,19 @@
mov x5,x8
mov x6,xzr
add x0,x0,#48+12
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^2
+ bl __poly1305_mult // r^2
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^3
+ bl __poly1305_mult // r^3
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^4
+ bl __poly1305_mult // r^4
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat
ldr x30,[sp,#8]

add x16,x1,#32
@@ -743,10 +732,11 @@
.Lno_data_neon:
ldr x29,[sp],#80
ret
+ENDPROC(poly1305_blocks_neon)

-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr x17,[x0,#24]
- cbz x17,poly1305_emit
+ cbz x17,poly1305_emit_arm

ldp w10,w11,[x0] // load hash value base 2^26
ldp w12,w13,[x0,#8]
@@ -788,6 +778,6 @@
stp x4,x5,[x1] // write result

ret
+ENDPROC(poly1305_emit_neon)

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                     |    8 +
 lib/zinc/poly1305/poly1305-arm-glue.h |   69 ++
 lib/zinc/poly1305/poly1305-arm.S      | 1117 +++++++++++++++++++++++++
 lib/zinc/poly1305/poly1305-arm64.S    |  822 ++++++++++++++++++
 4 files changed, 2016 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-arm.S
 create mode 100644 lib/zinc/poly1305/poly1305-arm64.S
lib/zinc/Makefile +8 -0
index d1e3892e06d9..f37df89a3f87 100644
@@ -25,6 +25,14 @@ endif
 
 ifeq ($(CONFIG_ZINC_POLY1305),y)
 zinc-y += poly1305/poly1305.o
+ifeq ($(CONFIG_ZINC_ARCH_ARM),y)
+zinc-y += poly1305/poly1305-arm.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
+endif
+ifeq ($(CONFIG_ZINC_ARCH_ARM64),y)
+zinc-y += poly1305/poly1305-arm64.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
+endif
 endif
 
 zinc-y += main.o
lib/zinc/poly1305/poly1305-arm-glue.h new file +69 -0

69 lines not on this page · raw message

lib/zinc/poly1305/poly1305-arm.S new file +1117 -0

1,117 lines not on this page · raw message

lib/zinc/poly1305/poly1305-arm64.S new file +822 -0

822 lines not on this page · raw message

Signature · 1 line
2.19.0
#9 lore raw

Re: [PATCH net-next v4 08/20] zinc: Poly1305 ARM and ARM64 implementations

Quoted 12 lines · Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed to fit kernel conventions.
- The neon code can jump to the scalar code when it makes sense to do so.

Quoted 3 lines

After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.

As I asked in response to v3, could we please have this as a separate patch on top? The diff below is corrupted.

Also, both Andy and Eric have offered to get involved in upstreaming these changes to OpenSSL, so there is no delta to begin with.

Quoted 153 lines

ARM:

(quoted diff) +20 -34
-poly1305_init:
-.Lpoly1305_init:
+ENTRY(poly1305_init_arm)
        stmdb   sp!,{r4-r11}
 
        eor     r3,r3,r3
@@ -18,8 +25,6 @@
        moveq   r0,#0
        beq     .Lno_key
 
-       adr     r11,.Lpoly1305_init
-       ldr     r12,.LOPENSSL_armcap
        ldrb    r4,[r1,#0]
        mov     r10,#0x0fffffff
        ldrb    r5,[r1,#1]
@@ -34,8 +39,6 @@
        ldrb    r7,[r1,#6]
        and     r4,r4,r10
 
-       ldr     r12,[r11,r12]           @ OPENSSL_armcap_P
-       ldr     r12,[r12]
        ldrb    r8,[r1,#7]
        orr     r5,r5,r6,lsl#8
        ldrb    r6,[r1,#8]
@@ -45,22 +48,6 @@
        ldrb    r8,[r1,#10]
        and     r5,r5,r3
 
-       tst     r12,#ARMV7_NEON         @ check for NEON
-       adr     r9,poly1305_blocks_neon
-       adr     r11,poly1305_blocks
-       it      ne
-       movne   r11,r9
-       adr     r12,poly1305_emit
-       adr     r10,poly1305_emit_neon
-       it      ne
-       movne   r12,r10
-       itete   eq
-       addeq   r12,r11,#(poly1305_emit-.Lpoly1305_init)
-       addne   r12,r11,#(poly1305_emit_neon-.Lpoly1305_init)
-       addeq   r11,r11,#(poly1305_blocks-.Lpoly1305_init)
-       addne   r11,r11,#(poly1305_blocks_neon-.Lpoly1305_init)
-       orr     r12,r12,#1      @ thumb-ify address
-       orr     r11,r11,#1
        ldrb    r9,[r1,#11]
        orr     r6,r6,r7,lsl#8
        ldrb    r7,[r1,#12]
@@ -79,17 +66,16 @@
        str     r6,[r0,#8]
        and     r7,r7,r3
        str     r7,[r0,#12]
-       stmia   r2,{r11,r12}            @ fill functions table
-       mov     r0,#1
-       mov     r0,#0
 .Lno_key:
        ldmia   sp!,{r4-r11}
        bx      lr                              @ bx    lr
        tst     lr,#1
        moveq   pc,lr                   @ be binary compatible with V4, yet
        .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
-poly1305_blocks:
-.Lpoly1305_blocks:
+ENDPROC(poly1305_init_arm)
+
+ENTRY(poly1305_blocks_arm)
+.Lpoly1305_blocks_arm:
        stmdb   sp!,{r3-r11,lr}
 
        ands    r2,r2,#-16
@@ -231,10 +217,11 @@
        tst     lr,#1
        moveq   pc,lr                   @ be binary compatible with V4, yet
        .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
-poly1305_emit:
+ENDPROC(poly1305_blocks_arm)
+
+ENTRY(poly1305_emit_arm)
        stmdb   sp!,{r4-r11}
 .Lpoly1305_emit_enter:
-
        ldmia   r0,{r3-r7}
        adds    r8,r3,#5                @ compare to modulus
        adcs    r9,r4,#0
@@ -305,8 +292,12 @@
        tst     lr,#1
        moveq   pc,lr                   @ be binary compatible with V4, yet
        .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
+ENDPROC(poly1305_emit_arm)
+
+
 
-poly1305_init_neon:
+ENTRY(poly1305_init_neon)
+.Lpoly1305_init_neon:
        ldr     r4,[r0,#20]             @ load key base 2^32
        ldr     r5,[r0,#24]
        ldr     r6,[r0,#28]
@@ -515,8 +506,9 @@
        vst1.32         {d8[1]},[r7]
 
        bx      lr                              @ bx    lr
+ENDPROC(poly1305_init_neon)
 
-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
        ldr     ip,[r0,#36]             @ is_base2_26
        ands    r2,r2,#-16
        beq     .Lno_data_neon
@@ -524,7 +516,7 @@
        cmp     r2,#64
        bhs     .Lenter_neon
        tst     ip,ip                   @ is_base2_26?
-       beq     .Lpoly1305_blocks
+       beq     .Lpoly1305_blocks_arm
 
 .Lenter_neon:
        stmdb   sp!,{r4-r7}
@@ -534,7 +526,7 @@
        bne     .Lbase2_26_neon
 
        stmdb   sp!,{r1-r3,lr}
-       bl      poly1305_init_neon
+       bl      .Lpoly1305_init_neon
 
        ldr     r4,[r0,#0]              @ load hash value base 2^32
        ldr     r5,[r0,#4]
@@ -989,8 +981,9 @@
        ldmia   sp!,{r4-r7}
 .Lno_data_neon:
        bx      lr                                      @ bx    lr
+ENDPROC(poly1305_blocks_neon)
 
-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
        ldr     ip,[r0,#36]             @ is_base2_26
 
        stmdb   sp!,{r4-r11}
@@ -1055,6 +1048,6 @@
 
        ldmia   sp!,{r4-r11}
        bx      lr                              @ bx    lr
+ENDPROC(poly1305_emit_neon)
 

ARM64:

(quoted diff) +1 -1
-poly1305_init:
+ENTRY(poly1305_init_arm)
        cmp     x1,xzr
        stp     xzr,xzr,[x0]            // zero hash value
        stp     xzr,xzr,[x0,#16]        // [along with is_base2_26]

Review line 15

Quoted 5 lines
(quoted diff) +0 -2
@@ -11,14 +15,9 @@
        csel    x0,xzr,x0,eq
        b.eq    .Lno_key
 
-       ldrsw   x11,.LOPENSSL_armcap_P
-       ldr     x11,.LOPENSSL_armcap_P

In the original, this looks like

#ifdef __ILP32__
ldrsw $t1,.LOPENSSL_armcap_P
#else
ldr $t1,.LOPENSSL_armcap_P
#endif

so I guess git commit ate those lines.

Quoted 172 lines
(quoted diff) +23 -34
-       adr     x10,.LOPENSSL_armcap_P
-
        ldp     x7,x8,[x1]              // load key
        mov     x9,#0xfffffffc0fffffff
        movk    x9,#0x0fff,lsl#48
-       ldr     w17,[x10,x11]
        rev     x7,x7                   // flip bytes
        rev     x8,x8
        and     x7,x7,x9                // &=0ffffffc0fffffff
@@ -26,24 +25,11 @@
        and     x8,x8,x9                // &=0ffffffc0ffffffc
        stp     x7,x8,[x0,#32]  // save key value
 
-       tst     w17,#ARMV7_NEON
-
-       adr     x12,poly1305_blocks
-       adr     x7,poly1305_blocks_neon
-       adr     x13,poly1305_emit
-       adr     x8,poly1305_emit_neon
-
-       csel    x12,x12,x7,eq
-       csel    x13,x13,x8,eq
-
-       stp     w12,w13,[x2]
-       stp     x12,x13,[x2]
-
-       mov     x0,#1
 .Lno_key:
        ret
+ENDPROC(poly1305_init_arm)
 
-poly1305_blocks:
+ENTRY(poly1305_blocks_arm)
        ands    x2,x2,#-16
        b.eq    .Lno_data
 
@@ -100,8 +86,9 @@
 
 .Lno_data:
        ret
+ENDPROC(poly1305_blocks_arm)
 
-poly1305_emit:
+ENTRY(poly1305_emit_arm)
        ldp     x4,x5,[x0]              // load hash base 2^64
        ldr     x6,[x0,#16]
        ldp     x10,x11,[x2]    // load nonce
@@ -124,7 +111,9 @@
        stp     x4,x5,[x1]              // write result
 
        ret
-poly1305_mult:
+ENDPROC(poly1305_emit_arm)
+
+__poly1305_mult:
        mul     x12,x4,x7               // h0*r0
        umulh   x13,x4,x7
 
@@ -158,7 +147,7 @@
 
        ret
 
-poly1305_splat:
+__poly1305_splat:
        and     x12,x4,#0x03ffffff      // base 2^64 -> base 2^26
        ubfx    x13,x4,#26,#26
        extr    x14,x5,x4,#52
@@ -182,11 +171,11 @@
 
        ret
 
-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
        ldr     x17,[x0,#24]
        cmp     x2,#128
        b.hs    .Lblocks_neon
-       cbz     x17,poly1305_blocks
+       cbz     x17,poly1305_blocks_arm
 
 .Lblocks_neon:
        stp     x29,x30,[sp,#-80]!
@@ -232,7 +221,7 @@
        adcs    x5,x5,x13
        adc     x6,x6,x3
 
-       bl      poly1305_mult
+       bl      __poly1305_mult
        ldr     x30,[sp,#8]
 
        cbz     x3,.Lstore_base2_64_neon
@@ -274,7 +263,7 @@
        adcs    x5,x5,x13
        adc     x6,x6,x3
 
-       bl      poly1305_mult
+       bl      __poly1305_mult
 
 .Linit_neon:
        and     x10,x4,#0x03ffffff      // base 2^64 -> base 2^26
@@ -301,19 +290,19 @@
        mov     x5,x8
        mov     x6,xzr
        add     x0,x0,#48+12
-       bl      poly1305_splat
+       bl      __poly1305_splat
 
-       bl      poly1305_mult           // r^2
+       bl      __poly1305_mult         // r^2
        sub     x0,x0,#4
-       bl      poly1305_splat
+       bl      __poly1305_splat
 
-       bl      poly1305_mult           // r^3
+       bl      __poly1305_mult         // r^3
        sub     x0,x0,#4
-       bl      poly1305_splat
+       bl      __poly1305_splat
 
-       bl      poly1305_mult           // r^4
+       bl      __poly1305_mult         // r^4
        sub     x0,x0,#4
-       bl      poly1305_splat
+       bl      __poly1305_splat
        ldr     x30,[sp,#8]
 
        add     x16,x1,#32
@@ -743,10 +732,11 @@
 .Lno_data_neon:
        ldr     x29,[sp],#80
        ret
+ENDPROC(poly1305_blocks_neon)
 
-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
        ldr     x17,[x0,#24]
-       cbz     x17,poly1305_emit
+       cbz     x17,poly1305_emit_arm
 
        ldp     w10,w11,[x0]            // load hash value base 2^26
        ldp     w12,w13,[x0,#8]
@@ -788,6 +778,6 @@
        stp     x4,x5,[x1]              // write result
 
        ret
+ENDPROC(poly1305_emit_neon)
 
Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                     |    8 +
 lib/zinc/poly1305/poly1305-arm-glue.h |   69 ++
 lib/zinc/poly1305/poly1305-arm.S      | 1117 +++++++++++++++++++++++++
 lib/zinc/poly1305/poly1305-arm64.S    |  822 ++++++++++++++++++
 4 files changed, 2016 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-arm.S
 create mode 100644 lib/zinc/poly1305/poly1305-arm64.S
lib/zinc/Makefile +0 -0
index d1e3892e06d9..f37df89a3f87 100644
@@ -25,6 +25,14 @@ endif
 
 ifeq ($(CONFIG_ZINC_POLY1305),y)
 zinc-y += poly1305/poly1305.o

lib/zinc/Makefile endif · line 37

Quoted 10 lines
lib/zinc/Makefile +8 -0
+ifeq ($(CONFIG_ZINC_ARCH_ARM),y)
+zinc-y += poly1305/poly1305-arm.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
+endif
+ifeq ($(CONFIG_ZINC_ARCH_ARM64),y)
+zinc-y += poly1305/poly1305-arm64.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
+endif
 endif
 

I still don't like the GCC -includes, especially because these .h files contain function and variable definitions so they are not actually header files to begin with.

Also, you mentioned in the commit log that you got rid of defines and made the code more modular, but as far as I can tell, libzinc is still a single monolithic binary that is essentially always builtin once we move random.c to it.

Quoted 66 lines
lib/zinc/Makefile +0 -0
 zinc-y += main.o
lib/zinc/poly1305/poly1305-arm-glue.h new file +59 -0
new file mode 100644
index 000000000000..53f8fec7f858
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <zinc/poly1305.h>
+#include <asm/hwcap.h>
+#include <asm/neon.h>
+
+asmlinkage void poly1305_init_arm(void *ctx, const u8 key[16]);
+asmlinkage void poly1305_blocks_arm(void *ctx, const u8 *inp, const size_t len,
+                                   const u32 padbit);
+asmlinkage void poly1305_emit_arm(void *ctx, u8 mac[16], const u32 nonce[4]);
+#if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&                                     \
+       (defined(CONFIG_64BIT) || __LINUX_ARM_ARCH__ >= 7)
+#define ARM_USE_NEON
+asmlinkage void poly1305_blocks_neon(void *ctx, const u8 *inp, const size_t len,
+                                    const u32 padbit);
+asmlinkage void poly1305_emit_neon(void *ctx, u8 mac[16], const u32 nonce[4]);
+#endif
+
+static bool poly1305_use_neon __ro_after_init;
+
+void __init poly1305_fpu_init(void)
+{
+#if defined(CONFIG_ARM64)
+       poly1305_use_neon = elf_hwcap & HWCAP_ASIMD;
+#elif defined(CONFIG_ARM)
+       poly1305_use_neon = elf_hwcap & HWCAP_NEON;
+#endif
+}
+
+static inline bool poly1305_init_arch(void *ctx,
+                                     const u8 key[POLY1305_KEY_SIZE],
+                                     simd_context_t simd_context)
+{
+       poly1305_init_arm(ctx, key);
+       return true;
+}
+
+static inline bool poly1305_blocks_arch(void *ctx, const u8 *inp,
+                                       const size_t len, const u32 padbit,
+                                       simd_context_t simd_context)
+{
+#if defined(ARM_USE_NEON)
+       if (simd_context == HAVE_FULL_SIMD && poly1305_use_neon) {
+               poly1305_blocks_neon(ctx, inp, len, padbit);
+               return true;
+       }
+#endif
+       poly1305_blocks_arm(ctx, inp, len, padbit);
+       return true;
+}
+
+static inline bool poly1305_emit_arch(void *ctx, u8 mac[POLY1305_MAC_SIZE],
+                                     const u32 nonce[4],
+                                     simd_context_t simd_context)
+{
+#if defined(ARM_USE_NEON)

lib/zinc/poly1305/poly1305-arm-glue.h line 69

Quoted 10 lines
lib/zinc/poly1305/poly1305-arm-glue.h new file +10 -0
+       if (simd_context == HAVE_FULL_SIMD && poly1305_use_neon) {
+               poly1305_emit_neon(ctx, mac, nonce);
+               return true;
+       }
+#endif
+       poly1305_emit_arm(ctx, mac, nonce);
+       return true;
+}
+
+#define HAVE_POLY1305_ARCH_IMPLEMENTATION

We shouldn't #define HAVE_xxx constants in code but only in Kconfig.

Quoted 1,954 lines
lib/zinc/poly1305/poly1305-arm.S new file +1117 -0
new file mode 100644
index 000000000000..110f4317b5d7
@@ -0,0 +1,1117 @@
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ *
+ * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
+ */
+
+#include <linux/linkage.h>
+
+.text
+#if defined(__thumb2__)
+.syntax        unified
+.thumb
+#else
+.code  32
+#endif
+
+.align 5
+ENTRY(poly1305_init_arm)
+       stmdb   sp!,{r4-r11}
+
+       eor     r3,r3,r3
+       cmp     r1,#0
+       str     r3,[r0,#0]              @ zero hash value
+       str     r3,[r0,#4]
+       str     r3,[r0,#8]
+       str     r3,[r0,#12]
+       str     r3,[r0,#16]
+       str     r3,[r0,#36]             @ is_base2_26
+       add     r0,r0,#20
+
+#ifdef __thumb2__
+       it      eq
+#endif
+       moveq   r0,#0
+       beq     .Lno_key
+
+       ldrb    r4,[r1,#0]
+       mov     r10,#0x0fffffff
+       ldrb    r5,[r1,#1]
+       and     r3,r10,#-4              @ 0x0ffffffc
+       ldrb    r6,[r1,#2]
+       ldrb    r7,[r1,#3]
+       orr     r4,r4,r5,lsl#8
+       ldrb    r5,[r1,#4]
+       orr     r4,r4,r6,lsl#16
+       ldrb    r6,[r1,#5]
+       orr     r4,r4,r7,lsl#24
+       ldrb    r7,[r1,#6]
+       and     r4,r4,r10
+
+       ldrb    r8,[r1,#7]
+       orr     r5,r5,r6,lsl#8
+       ldrb    r6,[r1,#8]
+       orr     r5,r5,r7,lsl#16
+       ldrb    r7,[r1,#9]
+       orr     r5,r5,r8,lsl#24
+       ldrb    r8,[r1,#10]
+       and     r5,r5,r3
+
+       ldrb    r9,[r1,#11]
+       orr     r6,r6,r7,lsl#8
+       ldrb    r7,[r1,#12]
+       orr     r6,r6,r8,lsl#16
+       ldrb    r8,[r1,#13]
+       orr     r6,r6,r9,lsl#24
+       ldrb    r9,[r1,#14]
+       and     r6,r6,r3
+
+       ldrb    r10,[r1,#15]
+       orr     r7,r7,r8,lsl#8
+       str     r4,[r0,#0]
+       orr     r7,r7,r9,lsl#16
+       str     r5,[r0,#4]
+       orr     r7,r7,r10,lsl#24
+       str     r6,[r0,#8]
+       and     r7,r7,r3
+       str     r7,[r0,#12]
+.Lno_key:
+       ldmia   sp!,{r4-r11}
+#if __LINUX_ARM_ARCH__ >= 5
+       bx      lr                              @ bx    lr
+#else
+       tst     lr,#1
+       moveq   pc,lr                   @ be binary compatible with V4, yet
+       .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
+#endif
+ENDPROC(poly1305_init_arm)
+
+.align 5
+ENTRY(poly1305_blocks_arm)
+.Lpoly1305_blocks_arm:
+       stmdb   sp!,{r3-r11,lr}
+
+       ands    r2,r2,#-16
+       beq     .Lno_data
+
+       cmp     r3,#0
+       add     r2,r2,r1                @ end pointer
+       sub     sp,sp,#32
+
+       ldmia   r0,{r4-r12}             @ load context
+
+       str     r0,[sp,#12]             @ offload stuff
+       mov     lr,r1
+       str     r2,[sp,#16]
+       str     r10,[sp,#20]
+       str     r11,[sp,#24]
+       str     r12,[sp,#28]
+       b       .Loop
+
+.Loop:
+#if __LINUX_ARM_ARCH__ < 7
+       ldrb    r0,[lr],#16             @ load input
+#ifdef __thumb2__
+       it      hi
+#endif
+       addhi   r8,r8,#1                @ 1<<128
+       ldrb    r1,[lr,#-15]
+       ldrb    r2,[lr,#-14]
+       ldrb    r3,[lr,#-13]
+       orr     r1,r0,r1,lsl#8
+       ldrb    r0,[lr,#-12]
+       orr     r2,r1,r2,lsl#16
+       ldrb    r1,[lr,#-11]
+       orr     r3,r2,r3,lsl#24
+       ldrb    r2,[lr,#-10]
+       adds    r4,r4,r3                @ accumulate input
+
+       ldrb    r3,[lr,#-9]
+       orr     r1,r0,r1,lsl#8
+       ldrb    r0,[lr,#-8]
+       orr     r2,r1,r2,lsl#16
+       ldrb    r1,[lr,#-7]
+       orr     r3,r2,r3,lsl#24
+       ldrb    r2,[lr,#-6]
+       adcs    r5,r5,r3
+
+       ldrb    r3,[lr,#-5]
+       orr     r1,r0,r1,lsl#8
+       ldrb    r0,[lr,#-4]
+       orr     r2,r1,r2,lsl#16
+       ldrb    r1,[lr,#-3]
+       orr     r3,r2,r3,lsl#24
+       ldrb    r2,[lr,#-2]
+       adcs    r6,r6,r3
+
+       ldrb    r3,[lr,#-1]
+       orr     r1,r0,r1,lsl#8
+       str     lr,[sp,#8]              @ offload input pointer
+       orr     r2,r1,r2,lsl#16
+       add     r10,r10,r10,lsr#2
+       orr     r3,r2,r3,lsl#24
+#else
+       ldr     r0,[lr],#16             @ load input
+#ifdef __thumb2__
+       it      hi
+#endif
+       addhi   r8,r8,#1                @ padbit
+       ldr     r1,[lr,#-12]
+       ldr     r2,[lr,#-8]
+       ldr     r3,[lr,#-4]
+#ifdef __ARMEB__
+       rev     r0,r0
+       rev     r1,r1
+       rev     r2,r2
+       rev     r3,r3
+#endif
+       adds    r4,r4,r0                @ accumulate input
+       str     lr,[sp,#8]              @ offload input pointer
+       adcs    r5,r5,r1
+       add     r10,r10,r10,lsr#2
+       adcs    r6,r6,r2
+#endif
+       add     r11,r11,r11,lsr#2
+       adcs    r7,r7,r3
+       add     r12,r12,r12,lsr#2
+
+       umull   r2,r3,r5,r9
+        adc    r8,r8,#0
+       umull   r0,r1,r4,r9
+       umlal   r2,r3,r8,r10
+       umlal   r0,r1,r7,r10
+       ldr     r10,[sp,#20]            @ reload r10
+       umlal   r2,r3,r6,r12
+       umlal   r0,r1,r5,r12
+       umlal   r2,r3,r7,r11
+       umlal   r0,r1,r6,r11
+       umlal   r2,r3,r4,r10
+       str     r0,[sp,#0]              @ future r4
+        mul    r0,r11,r8
+       ldr     r11,[sp,#24]            @ reload r11
+       adds    r2,r2,r1                @ d1+=d0>>32
+        eor    r1,r1,r1
+       adc     lr,r3,#0                @ future r6
+       str     r2,[sp,#4]              @ future r5
+
+       mul     r2,r12,r8
+       eor     r3,r3,r3
+       umlal   r0,r1,r7,r12
+       ldr     r12,[sp,#28]            @ reload r12
+       umlal   r2,r3,r7,r9
+       umlal   r0,r1,r6,r9
+       umlal   r2,r3,r6,r10
+       umlal   r0,r1,r5,r10
+       umlal   r2,r3,r5,r11
+       umlal   r0,r1,r4,r11
+       umlal   r2,r3,r4,r12
+       ldr     r4,[sp,#0]
+       mul     r8,r9,r8
+       ldr     r5,[sp,#4]
+
+       adds    r6,lr,r0                @ d2+=d1>>32
+       ldr     lr,[sp,#8]              @ reload input pointer
+       adc     r1,r1,#0
+       adds    r7,r2,r1                @ d3+=d2>>32
+       ldr     r0,[sp,#16]             @ reload end pointer
+       adc     r3,r3,#0
+       add     r8,r8,r3                @ h4+=d3>>32
+
+       and     r1,r8,#-4
+       and     r8,r8,#3
+       add     r1,r1,r1,lsr#2          @ *=5
+       adds    r4,r4,r1
+       adcs    r5,r5,#0
+       adcs    r6,r6,#0
+       adcs    r7,r7,#0
+       adc     r8,r8,#0
+
+       cmp     r0,lr                   @ done yet?
+       bhi     .Loop
+
+       ldr     r0,[sp,#12]
+       add     sp,sp,#32
+       stmia   r0,{r4-r8}              @ store the result
+
+.Lno_data:
+#if __LINUX_ARM_ARCH__ >= 5
+       ldmia   sp!,{r3-r11,pc}
+#else
+       ldmia   sp!,{r3-r11,lr}
+       tst     lr,#1
+       moveq   pc,lr                   @ be binary compatible with V4, yet
+       .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
+#endif
+ENDPROC(poly1305_blocks_arm)
+
+.align 5
+ENTRY(poly1305_emit_arm)
+       stmdb   sp!,{r4-r11}
+.Lpoly1305_emit_enter:
+       ldmia   r0,{r3-r7}
+       adds    r8,r3,#5                @ compare to modulus
+       adcs    r9,r4,#0
+       adcs    r10,r5,#0
+       adcs    r11,r6,#0
+       adc     r7,r7,#0
+       tst     r7,#4                   @ did it carry/borrow?
+
+#ifdef __thumb2__
+       it      ne
+#endif
+       movne   r3,r8
+       ldr     r8,[r2,#0]
+#ifdef __thumb2__
+       it      ne
+#endif
+       movne   r4,r9
+       ldr     r9,[r2,#4]
+#ifdef __thumb2__
+       it      ne
+#endif
+       movne   r5,r10
+       ldr     r10,[r2,#8]
+#ifdef __thumb2__
+       it      ne
+#endif
+       movne   r6,r11
+       ldr     r11,[r2,#12]
+
+       adds    r3,r3,r8
+       adcs    r4,r4,r9
+       adcs    r5,r5,r10
+       adc     r6,r6,r11
+
+#if __LINUX_ARM_ARCH__ >= 7
+#ifdef __ARMEB__
+       rev     r3,r3
+       rev     r4,r4
+       rev     r5,r5
+       rev     r6,r6
+#endif
+       str     r3,[r1,#0]
+       str     r4,[r1,#4]
+       str     r5,[r1,#8]
+       str     r6,[r1,#12]
+#else
+       strb    r3,[r1,#0]
+       mov     r3,r3,lsr#8
+       strb    r4,[r1,#4]
+       mov     r4,r4,lsr#8
+       strb    r5,[r1,#8]
+       mov     r5,r5,lsr#8
+       strb    r6,[r1,#12]
+       mov     r6,r6,lsr#8
+
+       strb    r3,[r1,#1]
+       mov     r3,r3,lsr#8
+       strb    r4,[r1,#5]
+       mov     r4,r4,lsr#8
+       strb    r5,[r1,#9]
+       mov     r5,r5,lsr#8
+       strb    r6,[r1,#13]
+       mov     r6,r6,lsr#8
+
+       strb    r3,[r1,#2]
+       mov     r3,r3,lsr#8
+       strb    r4,[r1,#6]
+       mov     r4,r4,lsr#8
+       strb    r5,[r1,#10]
+       mov     r5,r5,lsr#8
+       strb    r6,[r1,#14]
+       mov     r6,r6,lsr#8
+
+       strb    r3,[r1,#3]
+       strb    r4,[r1,#7]
+       strb    r5,[r1,#11]
+       strb    r6,[r1,#15]
+#endif
+       ldmia   sp!,{r4-r11}
+#if __LINUX_ARM_ARCH__ >= 5
+       bx      lr                              @ bx    lr
+#else
+       tst     lr,#1
+       moveq   pc,lr                   @ be binary compatible with V4, yet
+       .word   0xe12fff1e                      @ interoperable with Thumb ISA:-)
+#endif
+ENDPROC(poly1305_emit_arm)
+
+
+#if __LINUX_ARM_ARCH__ >= 7
+.fpu   neon
+
+.align 5
+ENTRY(poly1305_init_neon)
+.Lpoly1305_init_neon:
+       ldr     r4,[r0,#20]             @ load key base 2^32
+       ldr     r5,[r0,#24]
+       ldr     r6,[r0,#28]
+       ldr     r7,[r0,#32]
+
+       and     r2,r4,#0x03ffffff       @ base 2^32 -> base 2^26
+       mov     r3,r4,lsr#26
+       mov     r4,r5,lsr#20
+       orr     r3,r3,r5,lsl#6
+       mov     r5,r6,lsr#14
+       orr     r4,r4,r6,lsl#12
+       mov     r6,r7,lsr#8
+       orr     r5,r5,r7,lsl#18
+       and     r3,r3,#0x03ffffff
+       and     r4,r4,#0x03ffffff
+       and     r5,r5,#0x03ffffff
+
+       vdup.32 d0,r2                   @ r^1 in both lanes
+       add     r2,r3,r3,lsl#2          @ *5
+       vdup.32 d1,r3
+       add     r3,r4,r4,lsl#2
+       vdup.32 d2,r2
+       vdup.32 d3,r4
+       add     r4,r5,r5,lsl#2
+       vdup.32 d4,r3
+       vdup.32 d5,r5
+       add     r5,r6,r6,lsl#2
+       vdup.32 d6,r4
+       vdup.32 d7,r6
+       vdup.32 d8,r5
+
+       mov     r5,#2           @ counter
+
+.Lsquare_neon:
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ d0 = h0*r0 + h4*5*r1 + h3*5*r2 + h2*5*r3 + h1*5*r4
+       @ d1 = h1*r0 + h0*r1   + h4*5*r2 + h3*5*r3 + h2*5*r4
+       @ d2 = h2*r0 + h1*r1   + h0*r2   + h4*5*r3 + h3*5*r4
+       @ d3 = h3*r0 + h2*r1   + h1*r2   + h0*r3   + h4*5*r4
+       @ d4 = h4*r0 + h3*r1   + h2*r2   + h1*r3   + h0*r4
+
+       vmull.u32       q5,d0,d0[1]
+       vmull.u32       q6,d1,d0[1]
+       vmull.u32       q7,d3,d0[1]
+       vmull.u32       q8,d5,d0[1]
+       vmull.u32       q9,d7,d0[1]
+
+       vmlal.u32       q5,d7,d2[1]
+       vmlal.u32       q6,d0,d1[1]
+       vmlal.u32       q7,d1,d1[1]
+       vmlal.u32       q8,d3,d1[1]
+       vmlal.u32       q9,d5,d1[1]
+
+       vmlal.u32       q5,d5,d4[1]
+       vmlal.u32       q6,d7,d4[1]
+       vmlal.u32       q8,d1,d3[1]
+       vmlal.u32       q7,d0,d3[1]
+       vmlal.u32       q9,d3,d3[1]
+
+       vmlal.u32       q5,d3,d6[1]
+       vmlal.u32       q8,d0,d5[1]
+       vmlal.u32       q6,d5,d6[1]
+       vmlal.u32       q7,d7,d6[1]
+       vmlal.u32       q9,d1,d5[1]
+
+       vmlal.u32       q8,d7,d8[1]
+       vmlal.u32       q5,d1,d8[1]
+       vmlal.u32       q6,d3,d8[1]
+       vmlal.u32       q7,d5,d8[1]
+       vmlal.u32       q9,d0,d7[1]
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ lazy reduction as discussed in "NEON crypto" by D.J. Bernstein
+       @ and P. Schwabe
+       @
+       @ H0>>+H1>>+H2>>+H3>>+H4
+       @ H3>>+H4>>*5+H0>>+H1
+       @
+       @ Trivia.
+       @
+       @ Result of multiplication of n-bit number by m-bit number is
+       @ n+m bits wide. However! Even though 2^n is a n+1-bit number,
+       @ m-bit number multiplied by 2^n is still n+m bits wide.
+       @
+       @ Sum of two n-bit numbers is n+1 bits wide, sum of three - n+2,
+       @ and so is sum of four. Sum of 2^m n-m-bit numbers and n-bit
+       @ one is n+1 bits wide.
+       @
+       @ >>+ denotes Hnext += Hn>>26, Hn &= 0x3ffffff. This means that
+       @ H0, H2, H3 are guaranteed to be 26 bits wide, while H1 and H4
+       @ can be 27. However! In cases when their width exceeds 26 bits
+       @ they are limited by 2^26+2^6. This in turn means that *sum*
+       @ of the products with these values can still be viewed as sum
+       @ of 52-bit numbers as long as the amount of addends is not a
+       @ power of 2. For example,
+       @
+       @ H4 = H4*R0 + H3*R1 + H2*R2 + H1*R3 + H0 * R4,
+       @
+       @ which can't be larger than 5 * (2^26 + 2^6) * (2^26 + 2^6), or
+       @ 5 * (2^52 + 2*2^32 + 2^12), which in turn is smaller than
+       @ 8 * (2^52) or 2^55. However, the value is then multiplied by
+       @ by 5, so we should be looking at 5 * 5 * (2^52 + 2^33 + 2^12),
+       @ which is less than 32 * (2^52) or 2^57. And when processing
+       @ data we are looking at triple as many addends...
+       @
+       @ In key setup procedure pre-reduced H0 is limited by 5*4+1 and
+       @ 5*H4 - by 5*5 52-bit addends, or 57 bits. But when hashing the
+       @ input H0 is limited by (5*4+1)*3 addends, or 58 bits, while
+       @ 5*H4 by 5*5*3, or 59[!] bits. How is this relevant? vmlal.u32
+       @ instruction accepts 2x32-bit input and writes 2x64-bit result.
+       @ This means that result of reduction have to be compressed upon
+       @ loop wrap-around. This can be done in the process of reduction
+       @ to minimize amount of instructions [as well as amount of
+       @ 128-bit instructions, which benefits low-end processors], but
+       @ one has to watch for H2 (which is narrower than H0) and 5*H4
+       @ not being wider than 58 bits, so that result of right shift
+       @ by 26 bits fits in 32 bits. This is also useful on x86,
+       @ because it allows to use paddd in place for paddq, which
+       @ benefits Atom, where paddq is ridiculously slow.
+
+       vshr.u64        q15,q8,#26
+       vmovn.i64       d16,q8
+        vshr.u64       q4,q5,#26
+        vmovn.i64      d10,q5
+       vadd.i64        q9,q9,q15               @ h3 -> h4
+       vbic.i32        d16,#0xfc000000 @ &=0x03ffffff
+        vadd.i64       q6,q6,q4                @ h0 -> h1
+        vbic.i32       d10,#0xfc000000
+
+       vshrn.u64       d30,q9,#26
+       vmovn.i64       d18,q9
+        vshr.u64       q4,q6,#26
+        vmovn.i64      d12,q6
+        vadd.i64       q7,q7,q4                @ h1 -> h2
+       vbic.i32        d18,#0xfc000000
+        vbic.i32       d12,#0xfc000000
+
+       vadd.i32        d10,d10,d30
+       vshl.u32        d30,d30,#2
+        vshrn.u64      d8,q7,#26
+        vmovn.i64      d14,q7
+       vadd.i32        d10,d10,d30     @ h4 -> h0
+        vadd.i32       d16,d16,d8      @ h2 -> h3
+        vbic.i32       d14,#0xfc000000
+
+       vshr.u32        d30,d10,#26
+       vbic.i32        d10,#0xfc000000
+        vshr.u32       d8,d16,#26
+        vbic.i32       d16,#0xfc000000
+       vadd.i32        d12,d12,d30     @ h0 -> h1
+        vadd.i32       d18,d18,d8      @ h3 -> h4
+
+       subs            r5,r5,#1
+       beq             .Lsquare_break_neon
+
+       add             r6,r0,#(48+0*9*4)
+       add             r7,r0,#(48+1*9*4)
+
+       vtrn.32         d0,d10          @ r^2:r^1
+       vtrn.32         d3,d14
+       vtrn.32         d5,d16
+       vtrn.32         d1,d12
+       vtrn.32         d7,d18
+
+       vshl.u32        d4,d3,#2                @ *5
+       vshl.u32        d6,d5,#2
+       vshl.u32        d2,d1,#2
+       vshl.u32        d8,d7,#2
+       vadd.i32        d4,d4,d3
+       vadd.i32        d2,d2,d1
+       vadd.i32        d6,d6,d5
+       vadd.i32        d8,d8,d7
+
+       vst4.32         {d0[0],d1[0],d2[0],d3[0]},[r6]!
+       vst4.32         {d0[1],d1[1],d2[1],d3[1]},[r7]!
+       vst4.32         {d4[0],d5[0],d6[0],d7[0]},[r6]!
+       vst4.32         {d4[1],d5[1],d6[1],d7[1]},[r7]!
+       vst1.32         {d8[0]},[r6,:32]
+       vst1.32         {d8[1]},[r7,:32]
+
+       b               .Lsquare_neon
+
+.align 4
+.Lsquare_break_neon:
+       add             r6,r0,#(48+2*4*9)
+       add             r7,r0,#(48+3*4*9)
+
+       vmov            d0,d10          @ r^4:r^3
+       vshl.u32        d2,d12,#2               @ *5
+       vmov            d1,d12
+       vshl.u32        d4,d14,#2
+       vmov            d3,d14
+       vshl.u32        d6,d16,#2
+       vmov            d5,d16
+       vshl.u32        d8,d18,#2
+       vmov            d7,d18
+       vadd.i32        d2,d2,d12
+       vadd.i32        d4,d4,d14
+       vadd.i32        d6,d6,d16
+       vadd.i32        d8,d8,d18
+
+       vst4.32         {d0[0],d1[0],d2[0],d3[0]},[r6]!
+       vst4.32         {d0[1],d1[1],d2[1],d3[1]},[r7]!
+       vst4.32         {d4[0],d5[0],d6[0],d7[0]},[r6]!
+       vst4.32         {d4[1],d5[1],d6[1],d7[1]},[r7]!
+       vst1.32         {d8[0]},[r6]
+       vst1.32         {d8[1]},[r7]
+
+       bx      lr                              @ bx    lr
+ENDPROC(poly1305_init_neon)
+
+.align 5
+ENTRY(poly1305_blocks_neon)
+       ldr     ip,[r0,#36]             @ is_base2_26
+       ands    r2,r2,#-16
+       beq     .Lno_data_neon
+
+       cmp     r2,#64
+       bhs     .Lenter_neon
+       tst     ip,ip                   @ is_base2_26?
+       beq     .Lpoly1305_blocks_arm
+
+.Lenter_neon:
+       stmdb   sp!,{r4-r7}
+       vstmdb  sp!,{d8-d15}            @ ABI specification says so
+
+       tst     ip,ip                   @ is_base2_26?
+       bne     .Lbase2_26_neon
+
+       stmdb   sp!,{r1-r3,lr}
+       bl      .Lpoly1305_init_neon
+
+       ldr     r4,[r0,#0]              @ load hash value base 2^32
+       ldr     r5,[r0,#4]
+       ldr     r6,[r0,#8]
+       ldr     r7,[r0,#12]
+       ldr     ip,[r0,#16]
+
+       and     r2,r4,#0x03ffffff       @ base 2^32 -> base 2^26
+       mov     r3,r4,lsr#26
+        veor   d10,d10,d10
+       mov     r4,r5,lsr#20
+       orr     r3,r3,r5,lsl#6
+        veor   d12,d12,d12
+       mov     r5,r6,lsr#14
+       orr     r4,r4,r6,lsl#12
+        veor   d14,d14,d14
+       mov     r6,r7,lsr#8
+       orr     r5,r5,r7,lsl#18
+        veor   d16,d16,d16
+       and     r3,r3,#0x03ffffff
+       orr     r6,r6,ip,lsl#24
+        veor   d18,d18,d18
+       and     r4,r4,#0x03ffffff
+       mov     r1,#1
+       and     r5,r5,#0x03ffffff
+       str     r1,[r0,#36]             @ is_base2_26
+
+       vmov.32 d10[0],r2
+       vmov.32 d12[0],r3
+       vmov.32 d14[0],r4
+       vmov.32 d16[0],r5
+       vmov.32 d18[0],r6
+       adr     r5,.Lzeros
+
+       ldmia   sp!,{r1-r3,lr}
+       b       .Lbase2_32_neon
+
+.align 4
+.Lbase2_26_neon:
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ load hash value
+
+       veor            d10,d10,d10
+       veor            d12,d12,d12
+       veor            d14,d14,d14
+       veor            d16,d16,d16
+       veor            d18,d18,d18
+       vld4.32         {d10[0],d12[0],d14[0],d16[0]},[r0]!
+       adr             r5,.Lzeros
+       vld1.32         {d18[0]},[r0]
+       sub             r0,r0,#16               @ rewind
+
+.Lbase2_32_neon:
+       add             r4,r1,#32
+       mov             r3,r3,lsl#24
+       tst             r2,#31
+       beq             .Leven
+
+       vld4.32         {d20[0],d22[0],d24[0],d26[0]},[r1]!
+       vmov.32         d28[0],r3
+       sub             r2,r2,#16
+       add             r4,r1,#32
+
+#ifdef __ARMEB__
+       vrev32.8        q10,q10
+       vrev32.8        q13,q13
+       vrev32.8        q11,q11
+       vrev32.8        q12,q12
+#endif
+       vsri.u32        d28,d26,#8      @ base 2^32 -> base 2^26
+       vshl.u32        d26,d26,#18
+
+       vsri.u32        d26,d24,#14
+       vshl.u32        d24,d24,#12
+       vadd.i32        d29,d28,d18     @ add hash value and move to #hi
+
+       vbic.i32        d26,#0xfc000000
+       vsri.u32        d24,d22,#20
+       vshl.u32        d22,d22,#6
+
+       vbic.i32        d24,#0xfc000000
+       vsri.u32        d22,d20,#26
+       vadd.i32        d27,d26,d16
+
+       vbic.i32        d20,#0xfc000000
+       vbic.i32        d22,#0xfc000000
+       vadd.i32        d25,d24,d14
+
+       vadd.i32        d21,d20,d10
+       vadd.i32        d23,d22,d12
+
+       mov             r7,r5
+       add             r6,r0,#48
+
+       cmp             r2,r2
+       b               .Long_tail
+
+.align 4
+.Leven:
+       subs            r2,r2,#64
+       it              lo
+       movlo           r4,r5
+
+       vmov.i32        q14,#1<<24              @ padbit, yes, always
+       vld4.32         {d20,d22,d24,d26},[r1]  @ inp[0:1]
+       add             r1,r1,#64
+       vld4.32         {d21,d23,d25,d27},[r4]  @ inp[2:3] (or 0)
+       add             r4,r4,#64
+       itt             hi
+       addhi           r7,r0,#(48+1*9*4)
+       addhi           r6,r0,#(48+3*9*4)
+
+#ifdef __ARMEB__
+       vrev32.8        q10,q10
+       vrev32.8        q13,q13
+       vrev32.8        q11,q11
+       vrev32.8        q12,q12
+#endif
+       vsri.u32        q14,q13,#8              @ base 2^32 -> base 2^26
+       vshl.u32        q13,q13,#18
+
+       vsri.u32        q13,q12,#14
+       vshl.u32        q12,q12,#12
+
+       vbic.i32        q13,#0xfc000000
+       vsri.u32        q12,q11,#20
+       vshl.u32        q11,q11,#6
+
+       vbic.i32        q12,#0xfc000000
+       vsri.u32        q11,q10,#26
+
+       vbic.i32        q10,#0xfc000000
+       vbic.i32        q11,#0xfc000000
+
+       bls             .Lskip_loop
+
+       vld4.32         {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^2
+       vld4.32         {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^4
+       vld4.32         {d4[1],d5[1],d6[1],d7[1]},[r7]!
+       vld4.32         {d4[0],d5[0],d6[0],d7[0]},[r6]!
+       b               .Loop_neon
+
+.align 5
+.Loop_neon:
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2
+       @ ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^3+inp[7]*r
+       @   ___________________/
+       @ ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2+inp[8])*r^2
+       @ ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^4+inp[7]*r^2+inp[9])*r
+       @   ___________________/ ____________________/
+       @
+       @ Note that we start with inp[2:3]*r^2. This is because it
+       @ doesn't depend on reduction in previous iteration.
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ d4 = h4*r0 + h3*r1   + h2*r2   + h1*r3   + h0*r4
+       @ d3 = h3*r0 + h2*r1   + h1*r2   + h0*r3   + h4*5*r4
+       @ d2 = h2*r0 + h1*r1   + h0*r2   + h4*5*r3 + h3*5*r4
+       @ d1 = h1*r0 + h0*r1   + h4*5*r2 + h3*5*r3 + h2*5*r4
+       @ d0 = h0*r0 + h4*5*r1 + h3*5*r2 + h2*5*r3 + h1*5*r4
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ inp[2:3]*r^2
+
+       vadd.i32        d24,d24,d14     @ accumulate inp[0:1]
+       vmull.u32       q7,d25,d0[1]
+       vadd.i32        d20,d20,d10
+       vmull.u32       q5,d21,d0[1]
+       vadd.i32        d26,d26,d16
+       vmull.u32       q8,d27,d0[1]
+       vmlal.u32       q7,d23,d1[1]
+       vadd.i32        d22,d22,d12
+       vmull.u32       q6,d23,d0[1]
+
+       vadd.i32        d28,d28,d18
+       vmull.u32       q9,d29,d0[1]
+       subs            r2,r2,#64
+       vmlal.u32       q5,d29,d2[1]
+       it              lo
+       movlo           r4,r5
+       vmlal.u32       q8,d25,d1[1]
+       vld1.32         d8[1],[r7,:32]
+       vmlal.u32       q6,d21,d1[1]
+       vmlal.u32       q9,d27,d1[1]
+
+       vmlal.u32       q5,d27,d4[1]
+       vmlal.u32       q8,d23,d3[1]
+       vmlal.u32       q9,d25,d3[1]
+       vmlal.u32       q6,d29,d4[1]
+       vmlal.u32       q7,d21,d3[1]
+
+       vmlal.u32       q8,d21,d5[1]
+       vmlal.u32       q5,d25,d6[1]
+       vmlal.u32       q9,d23,d5[1]
+       vmlal.u32       q6,d27,d6[1]
+       vmlal.u32       q7,d29,d6[1]
+
+       vmlal.u32       q8,d29,d8[1]
+       vmlal.u32       q5,d23,d8[1]
+       vmlal.u32       q9,d21,d7[1]
+       vmlal.u32       q6,d25,d8[1]
+       vmlal.u32       q7,d27,d8[1]
+
+       vld4.32         {d21,d23,d25,d27},[r4]  @ inp[2:3] (or 0)
+       add             r4,r4,#64
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ (hash+inp[0:1])*r^4 and accumulate
+
+       vmlal.u32       q8,d26,d0[0]
+       vmlal.u32       q5,d20,d0[0]
+       vmlal.u32       q9,d28,d0[0]
+       vmlal.u32       q6,d22,d0[0]
+       vmlal.u32       q7,d24,d0[0]
+       vld1.32         d8[0],[r6,:32]
+
+       vmlal.u32       q8,d24,d1[0]
+       vmlal.u32       q5,d28,d2[0]
+       vmlal.u32       q9,d26,d1[0]
+       vmlal.u32       q6,d20,d1[0]
+       vmlal.u32       q7,d22,d1[0]
+
+       vmlal.u32       q8,d22,d3[0]
+       vmlal.u32       q5,d26,d4[0]
+       vmlal.u32       q9,d24,d3[0]
+       vmlal.u32       q6,d28,d4[0]
+       vmlal.u32       q7,d20,d3[0]
+
+       vmlal.u32       q8,d20,d5[0]
+       vmlal.u32       q5,d24,d6[0]
+       vmlal.u32       q9,d22,d5[0]
+       vmlal.u32       q6,d26,d6[0]
+       vmlal.u32       q8,d28,d8[0]
+
+       vmlal.u32       q7,d28,d6[0]
+       vmlal.u32       q5,d22,d8[0]
+       vmlal.u32       q9,d20,d7[0]
+       vmov.i32        q14,#1<<24              @ padbit, yes, always
+       vmlal.u32       q6,d24,d8[0]
+       vmlal.u32       q7,d26,d8[0]
+
+       vld4.32         {d20,d22,d24,d26},[r1]  @ inp[0:1]
+       add             r1,r1,#64
+#ifdef __ARMEB__
+       vrev32.8        q10,q10
+       vrev32.8        q11,q11
+       vrev32.8        q12,q12
+       vrev32.8        q13,q13
+#endif
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ lazy reduction interleaved with base 2^32 -> base 2^26 of
+       @ inp[0:3] previously loaded to q10-q13 and smashed to q10-q14.
+
+       vshr.u64        q15,q8,#26
+       vmovn.i64       d16,q8
+        vshr.u64       q4,q5,#26
+        vmovn.i64      d10,q5
+       vadd.i64        q9,q9,q15               @ h3 -> h4
+       vbic.i32        d16,#0xfc000000
+         vsri.u32      q14,q13,#8              @ base 2^32 -> base 2^26
+        vadd.i64       q6,q6,q4                @ h0 -> h1
+         vshl.u32      q13,q13,#18
+        vbic.i32       d10,#0xfc000000
+
+       vshrn.u64       d30,q9,#26
+       vmovn.i64       d18,q9
+        vshr.u64       q4,q6,#26
+        vmovn.i64      d12,q6
+        vadd.i64       q7,q7,q4                @ h1 -> h2
+         vsri.u32      q13,q12,#14
+       vbic.i32        d18,#0xfc000000
+         vshl.u32      q12,q12,#12
+        vbic.i32       d12,#0xfc000000
+
+       vadd.i32        d10,d10,d30
+       vshl.u32        d30,d30,#2
+         vbic.i32      q13,#0xfc000000
+        vshrn.u64      d8,q7,#26
+        vmovn.i64      d14,q7
+       vaddl.u32       q5,d10,d30      @ h4 -> h0 [widen for a sec]
+         vsri.u32      q12,q11,#20
+        vadd.i32       d16,d16,d8      @ h2 -> h3
+         vshl.u32      q11,q11,#6
+        vbic.i32       d14,#0xfc000000
+         vbic.i32      q12,#0xfc000000
+
+       vshrn.u64       d30,q5,#26              @ re-narrow
+       vmovn.i64       d10,q5
+         vsri.u32      q11,q10,#26
+         vbic.i32      q10,#0xfc000000
+        vshr.u32       d8,d16,#26
+        vbic.i32       d16,#0xfc000000
+       vbic.i32        d10,#0xfc000000
+       vadd.i32        d12,d12,d30     @ h0 -> h1
+        vadd.i32       d18,d18,d8      @ h3 -> h4
+         vbic.i32      q11,#0xfc000000
+
+       bhi             .Loop_neon
+
+.Lskip_loop:
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ multiply (inp[0:1]+hash) or inp[2:3] by r^2:r^1
+
+       add             r7,r0,#(48+0*9*4)
+       add             r6,r0,#(48+1*9*4)
+       adds            r2,r2,#32
+       it              ne
+       movne           r2,#0
+       bne             .Long_tail
+
+       vadd.i32        d25,d24,d14     @ add hash value and move to #hi
+       vadd.i32        d21,d20,d10
+       vadd.i32        d27,d26,d16
+       vadd.i32        d23,d22,d12
+       vadd.i32        d29,d28,d18
+
+.Long_tail:
+       vld4.32         {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^1
+       vld4.32         {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^2
+
+       vadd.i32        d24,d24,d14     @ can be redundant
+       vmull.u32       q7,d25,d0
+       vadd.i32        d20,d20,d10
+       vmull.u32       q5,d21,d0
+       vadd.i32        d26,d26,d16
+       vmull.u32       q8,d27,d0
+       vadd.i32        d22,d22,d12
+       vmull.u32       q6,d23,d0
+       vadd.i32        d28,d28,d18
+       vmull.u32       q9,d29,d0
+
+       vmlal.u32       q5,d29,d2
+       vld4.32         {d4[1],d5[1],d6[1],d7[1]},[r7]!
+       vmlal.u32       q8,d25,d1
+       vld4.32         {d4[0],d5[0],d6[0],d7[0]},[r6]!
+       vmlal.u32       q6,d21,d1
+       vmlal.u32       q9,d27,d1
+       vmlal.u32       q7,d23,d1
+
+       vmlal.u32       q8,d23,d3
+       vld1.32         d8[1],[r7,:32]
+       vmlal.u32       q5,d27,d4
+       vld1.32         d8[0],[r6,:32]
+       vmlal.u32       q9,d25,d3
+       vmlal.u32       q6,d29,d4
+       vmlal.u32       q7,d21,d3
+
+       vmlal.u32       q8,d21,d5
+        it             ne
+        addne          r7,r0,#(48+2*9*4)
+       vmlal.u32       q5,d25,d6
+        it             ne
+        addne          r6,r0,#(48+3*9*4)
+       vmlal.u32       q9,d23,d5
+       vmlal.u32       q6,d27,d6
+       vmlal.u32       q7,d29,d6
+
+       vmlal.u32       q8,d29,d8
+        vorn           q0,q0,q0        @ all-ones, can be redundant
+       vmlal.u32       q5,d23,d8
+        vshr.u64       q0,q0,#38
+       vmlal.u32       q9,d21,d7
+       vmlal.u32       q6,d25,d8
+       vmlal.u32       q7,d27,d8
+
+       beq             .Lshort_tail
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ (hash+inp[0:1])*r^4:r^3 and accumulate
+
+       vld4.32         {d0[1],d1[1],d2[1],d3[1]},[r7]! @ load r^3
+       vld4.32         {d0[0],d1[0],d2[0],d3[0]},[r6]! @ load r^4
+
+       vmlal.u32       q7,d24,d0
+       vmlal.u32       q5,d20,d0
+       vmlal.u32       q8,d26,d0
+       vmlal.u32       q6,d22,d0
+       vmlal.u32       q9,d28,d0
+
+       vmlal.u32       q5,d28,d2
+       vld4.32         {d4[1],d5[1],d6[1],d7[1]},[r7]!
+       vmlal.u32       q8,d24,d1
+       vld4.32         {d4[0],d5[0],d6[0],d7[0]},[r6]!
+       vmlal.u32       q6,d20,d1
+       vmlal.u32       q9,d26,d1
+       vmlal.u32       q7,d22,d1
+
+       vmlal.u32       q8,d22,d3
+       vld1.32         d8[1],[r7,:32]
+       vmlal.u32       q5,d26,d4
+       vld1.32         d8[0],[r6,:32]
+       vmlal.u32       q9,d24,d3
+       vmlal.u32       q6,d28,d4
+       vmlal.u32       q7,d20,d3
+
+       vmlal.u32       q8,d20,d5
+       vmlal.u32       q5,d24,d6
+       vmlal.u32       q9,d22,d5
+       vmlal.u32       q6,d26,d6
+       vmlal.u32       q7,d28,d6
+
+       vmlal.u32       q8,d28,d8
+        vorn           q0,q0,q0        @ all-ones
+       vmlal.u32       q5,d22,d8
+        vshr.u64       q0,q0,#38
+       vmlal.u32       q9,d20,d7
+       vmlal.u32       q6,d24,d8
+       vmlal.u32       q7,d26,d8
+
+.Lshort_tail:
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ horizontal addition
+
+       vadd.i64        d16,d16,d17
+       vadd.i64        d10,d10,d11
+       vadd.i64        d18,d18,d19
+       vadd.i64        d12,d12,d13
+       vadd.i64        d14,d14,d15
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ lazy reduction, but without narrowing
+
+       vshr.u64        q15,q8,#26
+       vand.i64        q8,q8,q0
+        vshr.u64       q4,q5,#26
+        vand.i64       q5,q5,q0
+       vadd.i64        q9,q9,q15               @ h3 -> h4
+        vadd.i64       q6,q6,q4                @ h0 -> h1
+
+       vshr.u64        q15,q9,#26
+       vand.i64        q9,q9,q0
+        vshr.u64       q4,q6,#26
+        vand.i64       q6,q6,q0
+        vadd.i64       q7,q7,q4                @ h1 -> h2
+
+       vadd.i64        q5,q5,q15
+       vshl.u64        q15,q15,#2
+        vshr.u64       q4,q7,#26
+        vand.i64       q7,q7,q0
+       vadd.i64        q5,q5,q15               @ h4 -> h0
+        vadd.i64       q8,q8,q4                @ h2 -> h3
+
+       vshr.u64        q15,q5,#26
+       vand.i64        q5,q5,q0
+        vshr.u64       q4,q8,#26
+        vand.i64       q8,q8,q0
+       vadd.i64        q6,q6,q15               @ h0 -> h1
+        vadd.i64       q9,q9,q4                @ h3 -> h4
+
+       cmp             r2,#0
+       bne             .Leven
+
+       @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+       @ store hash value
+
+       vst4.32         {d10[0],d12[0],d14[0],d16[0]},[r0]!
+       vst1.32         {d18[0]},[r0]
+
+       vldmia  sp!,{d8-d15}                    @ epilogue
+       ldmia   sp!,{r4-r7}
+.Lno_data_neon:
+       bx      lr                                      @ bx    lr
+ENDPROC(poly1305_blocks_neon)
+
+.align 5
+ENTRY(poly1305_emit_neon)
+       ldr     ip,[r0,#36]             @ is_base2_26
+
+       stmdb   sp!,{r4-r11}
+
+       tst     ip,ip
+       beq     .Lpoly1305_emit_enter
+
+       ldmia   r0,{r3-r7}
+       eor     r8,r8,r8
+
+       adds    r3,r3,r4,lsl#26 @ base 2^26 -> base 2^32
+       mov     r4,r4,lsr#6
+       adcs    r4,r4,r5,lsl#20
+       mov     r5,r5,lsr#12
+       adcs    r5,r5,r6,lsl#14
+       mov     r6,r6,lsr#18
+       adcs    r6,r6,r7,lsl#8
+       adc     r7,r8,r7,lsr#24 @ can be partially reduced ...
+
+       and     r8,r7,#-4               @ ... so reduce
+       and     r7,r6,#3
+       add     r8,r8,r8,lsr#2  @ *= 5
+       adds    r3,r3,r8
+       adcs    r4,r4,#0
+       adcs    r5,r5,#0
+       adcs    r6,r6,#0
+       adc     r7,r7,#0
+
+       adds    r8,r3,#5                @ compare to modulus
+       adcs    r9,r4,#0
+       adcs    r10,r5,#0
+       adcs    r11,r6,#0
+       adc     r7,r7,#0
+       tst     r7,#4                   @ did it carry/borrow?
+
+       it      ne
+       movne   r3,r8
+       ldr     r8,[r2,#0]
+       it      ne
+       movne   r4,r9
+       ldr     r9,[r2,#4]
+       it      ne
+       movne   r5,r10
+       ldr     r10,[r2,#8]
+       it      ne
+       movne   r6,r11
+       ldr     r11,[r2,#12]
+
+       adds    r3,r3,r8                @ accumulate nonce
+       adcs    r4,r4,r9
+       adcs    r5,r5,r10
+       adc     r6,r6,r11
+
+#ifdef __ARMEB__
+       rev     r3,r3
+       rev     r4,r4
+       rev     r5,r5
+       rev     r6,r6
+#endif
+       str     r3,[r1,#0]              @ store the result
+       str     r4,[r1,#4]
+       str     r5,[r1,#8]
+       str     r6,[r1,#12]
+
+       ldmia   sp!,{r4-r11}
+       bx      lr                              @ bx    lr
+ENDPROC(poly1305_emit_neon)
+
+.align 5
+.Lzeros:
+.long  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+#endif
lib/zinc/poly1305/poly1305-arm64.S new file +822 -1
new file mode 100644
index 000000000000..c20023544183
@@ -0,0 +1,822 @@
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ * Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+ *
+ * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
+ */
+
+#include <linux/linkage.h>
+.text
+
+.align 5
+ENTRY(poly1305_init_arm)
+       cmp     x1,xzr
+       stp     xzr,xzr,[x0]            // zero hash value
+       stp     xzr,xzr,[x0,#16]        // [along with is_base2_26]
+
+       csel    x0,xzr,x0,eq
+       b.eq    .Lno_key
+
+       ldp     x7,x8,[x1]              // load key
+       mov     x9,#0xfffffffc0fffffff
+       movk    x9,#0x0fff,lsl#48
+#ifdef __ARMEB__
+       rev     x7,x7                   // flip bytes
+       rev     x8,x8
+#endif
+       and     x7,x7,x9                // &=0ffffffc0fffffff
+       and     x9,x9,#-4
+       and     x8,x8,x9                // &=0ffffffc0ffffffc
+       stp     x7,x8,[x0,#32]  // save key value
+
+.Lno_key:
+       ret
+ENDPROC(poly1305_init_arm)
+
+.align 5
+ENTRY(poly1305_blocks_arm)
+       ands    x2,x2,#-16
+       b.eq    .Lno_data
+
+       ldp     x4,x5,[x0]              // load hash value
+       ldp     x7,x8,[x0,#32]  // load key value
+       ldr     x6,[x0,#16]
+       add     x9,x8,x8,lsr#2  // s1 = r1 + (r1 >> 2)
+       b       .Loop
+
+.align 5
+.Loop:
+       ldp     x10,x11,[x1],#16        // load input
+       sub     x2,x2,#16
+#ifdef __ARMEB__
+       rev     x10,x10
+       rev     x11,x11
+#endif
+       adds    x4,x4,x10               // accumulate input
+       adcs    x5,x5,x11
+
+       mul     x12,x4,x7               // h0*r0
+       adc     x6,x6,x3
+       umulh   x13,x4,x7
+
+       mul     x10,x5,x9               // h1*5*r1
+       umulh   x11,x5,x9
+
+       adds    x12,x12,x10
+       mul     x10,x4,x8               // h0*r1
+       adc     x13,x13,x11
+       umulh   x14,x4,x8
+
+       adds    x13,x13,x10
+       mul     x10,x5,x7               // h1*r0
+       adc     x14,x14,xzr
+       umulh   x11,x5,x7
+
+       adds    x13,x13,x10
+       mul     x10,x6,x9               // h2*5*r1
+       adc     x14,x14,x11
+       mul     x11,x6,x7               // h2*r0
+
+       adds    x13,x13,x10
+       adc     x14,x14,x11
+
+       and     x10,x14,#-4             // final reduction
+       and     x6,x14,#3
+       add     x10,x10,x14,lsr#2
+       adds    x4,x12,x10
+       adcs    x5,x13,xzr
+       adc     x6,x6,xzr
+
+       cbnz    x2,.Loop
+
+       stp     x4,x5,[x0]              // store hash value
+       str     x6,[x0,#16]
+
+.Lno_data:
+       ret
+ENDPROC(poly1305_blocks_arm)
+
+.align 5
+ENTRY(poly1305_emit_arm)
+       ldp     x4,x5,[x0]              // load hash base 2^64
+       ldr     x6,[x0,#16]
+       ldp     x10,x11,[x2]    // load nonce
+
+       adds    x12,x4,#5               // compare to modulus
+       adcs    x13,x5,xzr
+       adc     x14,x6,xzr
+
+       tst     x14,#-4                 // see if it's carried/borrowed
+
+       csel    x4,x4,x12,eq
+       csel    x5,x5,x13,eq
+
+#ifdef __ARMEB__
+       ror     x10,x10,#32             // flip nonce words
+       ror     x11,x11,#32
+#endif
+       adds    x4,x4,x10               // accumulate nonce
+       adc     x5,x5,x11
+#ifdef __ARMEB__
+       rev     x4,x4                   // flip output bytes
+       rev     x5,x5
+#endif
+       stp     x4,x5,[x1]              // write result
+
+       ret
+ENDPROC(poly1305_emit_arm)
+
+.align 5
+__poly1305_mult:
+       mul     x12,x4,x7               // h0*r0
+       umulh   x13,x4,x7
+
+       mul     x10,x5,x9               // h1*5*r1
+       umulh   x11,x5,x9
+
+       adds    x12,x12,x10
+       mul     x10,x4,x8               // h0*r1
+       adc     x13,x13,x11
+       umulh   x14,x4,x8
+
+       adds    x13,x13,x10
+       mul     x10,x5,x7               // h1*r0
+       adc     x14,x14,xzr
+       umulh   x11,x5,x7
+
+       adds    x13,x13,x10
+       mul     x10,x6,x9               // h2*5*r1
+       adc     x14,x14,x11
+       mul     x11,x6,x7               // h2*r0
+
+       adds    x13,x13,x10
+       adc     x14,x14,x11
+
+       and     x10,x14,#-4             // final reduction
+       and     x6,x14,#3
+       add     x10,x10,x14,lsr#2
+       adds    x4,x12,x10
+       adcs    x5,x13,xzr
+       adc     x6,x6,xzr
+
+       ret
+
+__poly1305_splat:
+       and     x12,x4,#0x03ffffff      // base 2^64 -> base 2^26
+       ubfx    x13,x4,#26,#26
+       extr    x14,x5,x4,#52
+       and     x14,x14,#0x03ffffff
+       ubfx    x15,x5,#14,#26
+       extr    x16,x6,x5,#40
+
+       str     w12,[x0,#16*0]  // r0
+       add     w12,w13,w13,lsl#2       // r1*5
+       str     w13,[x0,#16*1]  // r1
+       add     w13,w14,w14,lsl#2       // r2*5
+       str     w12,[x0,#16*2]  // s1
+       str     w14,[x0,#16*3]  // r2
+       add     w14,w15,w15,lsl#2       // r3*5
+       str     w13,[x0,#16*4]  // s2
+       str     w15,[x0,#16*5]  // r3
+       add     w15,w16,w16,lsl#2       // r4*5
+       str     w14,[x0,#16*6]  // s3
+       str     w16,[x0,#16*7]  // r4
+       str     w15,[x0,#16*8]  // s4
+
+       ret
+
+.align 5
+ENTRY(poly1305_blocks_neon)
+       ldr     x17,[x0,#24]
+       cmp     x2,#128
+       b.hs    .Lblocks_neon
+       cbz     x17,poly1305_blocks_arm
+
+.Lblocks_neon:
+       stp     x29,x30,[sp,#-80]!
+       add     x29,sp,#0
+
+       ands    x2,x2,#-16
+       b.eq    .Lno_data_neon
+
+       cbz     x17,.Lbase2_64_neon
+
+       ldp     w10,w11,[x0]            // load hash value base 2^26
+       ldp     w12,w13,[x0,#8]
+       ldr     w14,[x0,#16]
+
+       tst     x2,#31
+       b.eq    .Leven_neon
+
+       ldp     x7,x8,[x0,#32]  // load key value
+
+       add     x4,x10,x11,lsl#26       // base 2^26 -> base 2^64
+       lsr     x5,x12,#12
+       adds    x4,x4,x12,lsl#52
+       add     x5,x5,x13,lsl#14
+       adc     x5,x5,xzr
+       lsr     x6,x14,#24
+       adds    x5,x5,x14,lsl#40
+       adc     x14,x6,xzr              // can be partially reduced...
+
+       ldp     x12,x13,[x1],#16        // load input
+       sub     x2,x2,#16
+       add     x9,x8,x8,lsr#2  // s1 = r1 + (r1 >> 2)
+
+       and     x10,x14,#-4             // ... so reduce
+       and     x6,x14,#3
+       add     x10,x10,x14,lsr#2
+       adds    x4,x4,x10
+       adcs    x5,x5,xzr
+       adc     x6,x6,xzr
+
+#ifdef __ARMEB__
+       rev     x12,x12
+       rev     x13,x13
+#endif
+       adds    x4,x4,x12               // accumulate input
+       adcs    x5,x5,x13
+       adc     x6,x6,x3
+
+       bl      __poly1305_mult
+       ldr     x30,[sp,#8]
+
+       cbz     x3,.Lstore_base2_64_neon
+
+       and     x10,x4,#0x03ffffff      // base 2^64 -> base 2^26
+       ubfx    x11,x4,#26,#26
+       extr    x12,x5,x4,#52
+       and     x12,x12,#0x03ffffff
+       ubfx    x13,x5,#14,#26
+       extr    x14,x6,x5,#40
+
+       cbnz    x2,.Leven_neon
+
+       stp     w10,w11,[x0]            // store hash value base 2^26
+       stp     w12,w13,[x0,#8]
+       str     w14,[x0,#16]
+       b       .Lno_data_neon
+
+.align 4
+.Lstore_base2_64_neon:
+       stp     x4,x5,[x0]              // store hash value base 2^64
+       stp     x6,xzr,[x0,#16] // note that is_base2_26 is zeroed
+       b       .Lno_data_neon
+
+.align 4
+.Lbase2_64_neon:
+       ldp     x7,x8,[x0,#32]  // load key value
+
+       ldp     x4,x5,[x0]              // load hash value base 2^64
+       ldr     x6,[x0,#16]
+
+       tst     x2,#31
+       b.eq    .Linit_neon
+
+       ldp     x12,x13,[x1],#16        // load input
+       sub     x2,x2,#16
+       add     x9,x8,x8,lsr#2  // s1 = r1 + (r1 >> 2)
+#ifdef __ARMEB__
+       rev     x12,x12
+       rev     x13,x13
+#endif
+       adds    x4,x4,x12               // accumulate input
+       adcs    x5,x5,x13
+       adc     x6,x6,x3
+
+       bl      __poly1305_mult
+
+.Linit_neon:
+       and     x10,x4,#0x03ffffff      // base 2^64 -> base 2^26
+       ubfx    x11,x4,#26,#26
+       extr    x12,x5,x4,#52
+       and     x12,x12,#0x03ffffff
+       ubfx    x13,x5,#14,#26
+       extr    x14,x6,x5,#40
+
+       stp     d8,d9,[sp,#16]          // meet ABI requirements
+       stp     d10,d11,[sp,#32]
+       stp     d12,d13,[sp,#48]
+       stp     d14,d15,[sp,#64]
+
+       fmov    d24,x10
+       fmov    d25,x11
+       fmov    d26,x12
+       fmov    d27,x13
+       fmov    d28,x14
+
+       ////////////////////////////////// initialize r^n table
+       mov     x4,x7                   // r^1
+       add     x9,x8,x8,lsr#2  // s1 = r1 + (r1 >> 2)
+       mov     x5,x8
+       mov     x6,xzr
+       add     x0,x0,#48+12
+       bl      __poly1305_splat
+
+       bl      __poly1305_mult         // r^2
+       sub     x0,x0,#4
+       bl      __poly1305_splat
+
+       bl      __poly1305_mult         // r^3
+       sub     x0,x0,#4
+       bl      __poly1305_splat
+
+       bl      __poly1305_mult         // r^4
+       sub     x0,x0,#4
+       bl      __poly1305_splat
+       ldr     x30,[sp,#8]
+
+       add     x16,x1,#32
+       adr     x17,.Lzeros
+       subs    x2,x2,#64
+       csel    x16,x17,x16,lo
+
+       mov     x4,#1
+       str     x4,[x0,#-24]            // set is_base2_26
+       sub     x0,x0,#48               // restore original x0
+       b       .Ldo_neon
+
+.align 4
+.Leven_neon:
+       add     x16,x1,#32
+       adr     x17,.Lzeros
+       subs    x2,x2,#64
+       csel    x16,x17,x16,lo
+
+       stp     d8,d9,[sp,#16]          // meet ABI requirements
+       stp     d10,d11,[sp,#32]
+       stp     d12,d13,[sp,#48]
+       stp     d14,d15,[sp,#64]
+
+       fmov    d24,x10
+       fmov    d25,x11
+       fmov    d26,x12
+       fmov    d27,x13
+       fmov    d28,x14
+
+.Ldo_neon:
+       ldp     x8,x12,[x16],#16        // inp[2:3] (or zero)
+       ldp     x9,x13,[x16],#48
+
+       lsl     x3,x3,#24
+       add     x15,x0,#48
+
+#ifdef __ARMEB__
+       rev     x8,x8
+       rev     x12,x12
+       rev     x9,x9
+       rev     x13,x13
+#endif
+       and     x4,x8,#0x03ffffff       // base 2^64 -> base 2^26
+       and     x5,x9,#0x03ffffff
+       ubfx    x6,x8,#26,#26
+       ubfx    x7,x9,#26,#26
+       add     x4,x4,x5,lsl#32         // bfi  x4,x5,#32,#32
+       extr    x8,x12,x8,#52
+       extr    x9,x13,x9,#52
+       add     x6,x6,x7,lsl#32         // bfi  x6,x7,#32,#32
+       fmov    d14,x4
+       and     x8,x8,#0x03ffffff
+       and     x9,x9,#0x03ffffff
+       ubfx    x10,x12,#14,#26
+       ubfx    x11,x13,#14,#26
+       add     x12,x3,x12,lsr#40
+       add     x13,x3,x13,lsr#40
+       add     x8,x8,x9,lsl#32         // bfi  x8,x9,#32,#32
+       fmov    d15,x6
+       add     x10,x10,x11,lsl#32      // bfi  x10,x11,#32,#32
+       add     x12,x12,x13,lsl#32      // bfi  x12,x13,#32,#32
+       fmov    d16,x8
+       fmov    d17,x10
+       fmov    d18,x12
+
+       ldp     x8,x12,[x1],#16 // inp[0:1]
+       ldp     x9,x13,[x1],#48
+
+       ld1     {v0.4s,v1.4s,v2.4s,v3.4s},[x15],#64
+       ld1     {v4.4s,v5.4s,v6.4s,v7.4s},[x15],#64
+       ld1     {v8.4s},[x15]
+
+#ifdef __ARMEB__
+       rev     x8,x8
+       rev     x12,x12
+       rev     x9,x9
+       rev     x13,x13
+#endif
+       and     x4,x8,#0x03ffffff       // base 2^64 -> base 2^26
+       and     x5,x9,#0x03ffffff
+       ubfx    x6,x8,#26,#26
+       ubfx    x7,x9,#26,#26
+       add     x4,x4,x5,lsl#32         // bfi  x4,x5,#32,#32
+       extr    x8,x12,x8,#52
+       extr    x9,x13,x9,#52
+       add     x6,x6,x7,lsl#32         // bfi  x6,x7,#32,#32
+       fmov    d9,x4
+       and     x8,x8,#0x03ffffff
+       and     x9,x9,#0x03ffffff
+       ubfx    x10,x12,#14,#26
+       ubfx    x11,x13,#14,#26
+       add     x12,x3,x12,lsr#40
+       add     x13,x3,x13,lsr#40
+       add     x8,x8,x9,lsl#32         // bfi  x8,x9,#32,#32
+       fmov    d10,x6
+       add     x10,x10,x11,lsl#32      // bfi  x10,x11,#32,#32
+       add     x12,x12,x13,lsl#32      // bfi  x12,x13,#32,#32
+       movi    v31.2d,#-1
+       fmov    d11,x8
+       fmov    d12,x10
+       fmov    d13,x12
+       ushr    v31.2d,v31.2d,#38
+
+       b.ls    .Lskip_loop
+
+.align 4
+.Loop_neon:
+       ////////////////////////////////////////////////////////////////
+       // ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2
+       // ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^3+inp[7]*r
+       //   ___________________/
+       // ((inp[0]*r^4+inp[2]*r^2+inp[4])*r^4+inp[6]*r^2+inp[8])*r^2
+       // ((inp[1]*r^4+inp[3]*r^2+inp[5])*r^4+inp[7]*r^2+inp[9])*r
+       //   ___________________/ ____________________/
+       //
+       // Note that we start with inp[2:3]*r^2. This is because it
+       // doesn't depend on reduction in previous iteration.
+       ////////////////////////////////////////////////////////////////
+       // d4 = h0*r4 + h1*r3   + h2*r2   + h3*r1   + h4*r0
+       // d3 = h0*r3 + h1*r2   + h2*r1   + h3*r0   + h4*5*r4
+       // d2 = h0*r2 + h1*r1   + h2*r0   + h3*5*r4 + h4*5*r3
+       // d1 = h0*r1 + h1*r0   + h2*5*r4 + h3*5*r3 + h4*5*r2
+       // d0 = h0*r0 + h1*5*r4 + h2*5*r3 + h3*5*r2 + h4*5*r1
+
+       subs    x2,x2,#64
+       umull   v23.2d,v14.2s,v7.s[2]
+       csel    x16,x17,x16,lo
+       umull   v22.2d,v14.2s,v5.s[2]
+       umull   v21.2d,v14.2s,v3.s[2]
+       ldp     x8,x12,[x16],#16        // inp[2:3] (or zero)
+       umull   v20.2d,v14.2s,v1.s[2]
+       ldp     x9,x13,[x16],#48
+       umull   v19.2d,v14.2s,v0.s[2]
+#ifdef __ARMEB__
+       rev     x8,x8
+       rev     x12,x12
+       rev     x9,x9
+       rev     x13,x13
+#endif
+
+       umlal   v23.2d,v15.2s,v5.s[2]
+       and     x4,x8,#0x03ffffff       // base 2^64 -> base 2^26
+       umlal   v22.2d,v15.2s,v3.s[2]
+       and     x5,x9,#0x03ffffff
+       umlal   v21.2d,v15.2s,v1.s[2]
+       ubfx    x6,x8,#26,#26
+       umlal   v20.2d,v15.2s,v0.s[2]
+       ubfx    x7,x9,#26,#26
+       umlal   v19.2d,v15.2s,v8.s[2]
+       add     x4,x4,x5,lsl#32         // bfi  x4,x5,#32,#32
+
+       umlal   v23.2d,v16.2s,v3.s[2]
+       extr    x8,x12,x8,#52
+       umlal   v22.2d,v16.2s,v1.s[2]
+       extr    x9,x13,x9,#52
+       umlal   v21.2d,v16.2s,v0.s[2]
+       add     x6,x6,x7,lsl#32         // bfi  x6,x7,#32,#32
+       umlal   v20.2d,v16.2s,v8.s[2]
+       fmov    d14,x4
+       umlal   v19.2d,v16.2s,v6.s[2]
+       and     x8,x8,#0x03ffffff
+
+       umlal   v23.2d,v17.2s,v1.s[2]
+       and     x9,x9,#0x03ffffff
+       umlal   v22.2d,v17.2s,v0.s[2]
+       ubfx    x10,x12,#14,#26
+       umlal   v21.2d,v17.2s,v8.s[2]
+       ubfx    x11,x13,#14,#26
+       umlal   v20.2d,v17.2s,v6.s[2]
+       add     x8,x8,x9,lsl#32         // bfi  x8,x9,#32,#32
+       umlal   v19.2d,v17.2s,v4.s[2]
+       fmov    d15,x6
+
+       add     v11.2s,v11.2s,v26.2s
+       add     x12,x3,x12,lsr#40
+       umlal   v23.2d,v18.2s,v0.s[2]
+       add     x13,x3,x13,lsr#40
+       umlal   v22.2d,v18.2s,v8.s[2]
+       add     x10,x10,x11,lsl#32      // bfi  x10,x11,#32,#32
+       umlal   v21.2d,v18.2s,v6.s[2]
+       add     x12,x12,x13,lsl#32      // bfi  x12,x13,#32,#32
+       umlal   v20.2d,v18.2s,v4.s[2]
+       fmov    d16,x8
+       umlal   v19.2d,v18.2s,v2.s[2]
+       fmov    d17,x10
+
+       ////////////////////////////////////////////////////////////////
+       // (hash+inp[0:1])*r^4 and accumulate
+
+       add     v9.2s,v9.2s,v24.2s
+       fmov    d18,x12
+       umlal   v22.2d,v11.2s,v1.s[0]
+       ldp     x8,x12,[x1],#16 // inp[0:1]
+       umlal   v19.2d,v11.2s,v6.s[0]
+       ldp     x9,x13,[x1],#48
+       umlal   v23.2d,v11.2s,v3.s[0]
+       umlal   v20.2d,v11.2s,v8.s[0]
+       umlal   v21.2d,v11.2s,v0.s[0]
+#ifdef __ARMEB__
+       rev     x8,x8
+       rev     x12,x12
+       rev     x9,x9
+       rev     x13,x13
+#endif
+
+       add     v10.2s,v10.2s,v25.2s
+       umlal   v22.2d,v9.2s,v5.s[0]
+       umlal   v23.2d,v9.2s,v7.s[0]
+       and     x4,x8,#0x03ffffff       // base 2^64 -> base 2^26
+       umlal   v21.2d,v9.2s,v3.s[0]
+       and     x5,x9,#0x03ffffff
+       umlal   v19.2d,v9.2s,v0.s[0]
+       ubfx    x6,x8,#26,#26
+       umlal   v20.2d,v9.2s,v1.s[0]
+       ubfx    x7,x9,#26,#26
+
+       add     v12.2s,v12.2s,v27.2s
+       add     x4,x4,x5,lsl#32         // bfi  x4,x5,#32,#32
+       umlal   v22.2d,v10.2s,v3.s[0]
+       extr    x8,x12,x8,#52
+       umlal   v23.2d,v10.2s,v5.s[0]
+       extr    x9,x13,x9,#52
+       umlal   v19.2d,v10.2s,v8.s[0]
+       add     x6,x6,x7,lsl#32         // bfi  x6,x7,#32,#32
+       umlal   v21.2d,v10.2s,v1.s[0]
+       fmov    d9,x4
+       umlal   v20.2d,v10.2s,v0.s[0]
+       and     x8,x8,#0x03ffffff
+
+       add     v13.2s,v13.2s,v28.2s
+       and     x9,x9,#0x03ffffff
+       umlal   v22.2d,v12.2s,v0.s[0]
+       ubfx    x10,x12,#14,#26
+       umlal   v19.2d,v12.2s,v4.s[0]
+       ubfx    x11,x13,#14,#26
+       umlal   v23.2d,v12.2s,v1.s[0]
+       add     x8,x8,x9,lsl#32         // bfi  x8,x9,#32,#32
+       umlal   v20.2d,v12.2s,v6.s[0]
+       fmov    d10,x6
+       umlal   v21.2d,v12.2s,v8.s[0]
+       add     x12,x3,x12,lsr#40
+
+       umlal   v22.2d,v13.2s,v8.s[0]
+       add     x13,x3,x13,lsr#40
+       umlal   v19.2d,v13.2s,v2.s[0]
+       add     x10,x10,x11,lsl#32      // bfi  x10,x11,#32,#32
+       umlal   v23.2d,v13.2s,v0.s[0]
+       add     x12,x12,x13,lsl#32      // bfi  x12,x13,#32,#32
+       umlal   v20.2d,v13.2s,v4.s[0]
+       fmov    d11,x8
+       umlal   v21.2d,v13.2s,v6.s[0]
+       fmov    d12,x10
+       fmov    d13,x12
+
+       /////////////////////////////////////////////////////////////////
+       // lazy reduction as discussed in "NEON crypto" by D.J. Bernstein
+       // and P. Schwabe
+       //
+       // [see discussion in poly1305-armv4 module]
+
+       ushr    v29.2d,v22.2d,#26
+       xtn     v27.2s,v22.2d
+       ushr    v30.2d,v19.2d,#26
+       and     v19.16b,v19.16b,v31.16b
+       add     v23.2d,v23.2d,v29.2d    // h3 -> h4
+       bic     v27.2s,#0xfc,lsl#24     // &=0x03ffffff
+       add     v20.2d,v20.2d,v30.2d    // h0 -> h1
+
+       ushr    v29.2d,v23.2d,#26
+       xtn     v28.2s,v23.2d
+       ushr    v30.2d,v20.2d,#26
+       xtn     v25.2s,v20.2d
+       bic     v28.2s,#0xfc,lsl#24
+       add     v21.2d,v21.2d,v30.2d    // h1 -> h2
+
+       add     v19.2d,v19.2d,v29.2d
+       shl     v29.2d,v29.2d,#2
+       shrn    v30.2s,v21.2d,#26
+       xtn     v26.2s,v21.2d
+       add     v19.2d,v19.2d,v29.2d    // h4 -> h0
+       bic     v25.2s,#0xfc,lsl#24
+       add     v27.2s,v27.2s,v30.2s            // h2 -> h3
+       bic     v26.2s,#0xfc,lsl#24
+
+       shrn    v29.2s,v19.2d,#26
+       xtn     v24.2s,v19.2d
+       ushr    v30.2s,v27.2s,#26
+       bic     v27.2s,#0xfc,lsl#24
+       bic     v24.2s,#0xfc,lsl#24
+       add     v25.2s,v25.2s,v29.2s            // h0 -> h1
+       add     v28.2s,v28.2s,v30.2s            // h3 -> h4
+
+       b.hi    .Loop_neon
+
+.Lskip_loop:
+       dup     v16.2d,v16.d[0]
+       add     v11.2s,v11.2s,v26.2s
+
+       ////////////////////////////////////////////////////////////////
+       // multiply (inp[0:1]+hash) or inp[2:3] by r^2:r^1
+
+       adds    x2,x2,#32
+       b.ne    .Long_tail
+
+       dup     v16.2d,v11.d[0]
+       add     v14.2s,v9.2s,v24.2s
+       add     v17.2s,v12.2s,v27.2s
+       add     v15.2s,v10.2s,v25.2s
+       add     v18.2s,v13.2s,v28.2s
+
+.Long_tail:
+       dup     v14.2d,v14.d[0]
+       umull2  v19.2d,v16.4s,v6.4s
+       umull2  v22.2d,v16.4s,v1.4s
+       umull2  v23.2d,v16.4s,v3.4s
+       umull2  v21.2d,v16.4s,v0.4s
+       umull2  v20.2d,v16.4s,v8.4s
+
+       dup     v15.2d,v15.d[0]
+       umlal2  v19.2d,v14.4s,v0.4s
+       umlal2  v21.2d,v14.4s,v3.4s
+       umlal2  v22.2d,v14.4s,v5.4s
+       umlal2  v23.2d,v14.4s,v7.4s
+       umlal2  v20.2d,v14.4s,v1.4s
+
+       dup     v17.2d,v17.d[0]
+       umlal2  v19.2d,v15.4s,v8.4s
+       umlal2  v22.2d,v15.4s,v3.4s
+       umlal2  v21.2d,v15.4s,v1.4s
+       umlal2  v23.2d,v15.4s,v5.4s
+       umlal2  v20.2d,v15.4s,v0.4s
+
+       dup     v18.2d,v18.d[0]
+       umlal2  v22.2d,v17.4s,v0.4s
+       umlal2  v23.2d,v17.4s,v1.4s
+       umlal2  v19.2d,v17.4s,v4.4s
+       umlal2  v20.2d,v17.4s,v6.4s
+       umlal2  v21.2d,v17.4s,v8.4s
+
+       umlal2  v22.2d,v18.4s,v8.4s
+       umlal2  v19.2d,v18.4s,v2.4s
+       umlal2  v23.2d,v18.4s,v0.4s
+       umlal2  v20.2d,v18.4s,v4.4s
+       umlal2  v21.2d,v18.4s,v6.4s
+
+       b.eq    .Lshort_tail
+
+       ////////////////////////////////////////////////////////////////
+       // (hash+inp[0:1])*r^4:r^3 and accumulate
+
+       add     v9.2s,v9.2s,v24.2s
+       umlal   v22.2d,v11.2s,v1.2s
+       umlal   v19.2d,v11.2s,v6.2s
+       umlal   v23.2d,v11.2s,v3.2s
+       umlal   v20.2d,v11.2s,v8.2s
+       umlal   v21.2d,v11.2s,v0.2s
+
+       add     v10.2s,v10.2s,v25.2s
+       umlal   v22.2d,v9.2s,v5.2s
+       umlal   v19.2d,v9.2s,v0.2s
+       umlal   v23.2d,v9.2s,v7.2s
+       umlal   v20.2d,v9.2s,v1.2s
+       umlal   v21.2d,v9.2s,v3.2s
+
+       add     v12.2s,v12.2s,v27.2s
+       umlal   v22.2d,v10.2s,v3.2s
+       umlal   v19.2d,v10.2s,v8.2s
+       umlal   v23.2d,v10.2s,v5.2s
+       umlal   v20.2d,v10.2s,v0.2s
+       umlal   v21.2d,v10.2s,v1.2s
+
+       add     v13.2s,v13.2s,v28.2s
+       umlal   v22.2d,v12.2s,v0.2s
+       umlal   v19.2d,v12.2s,v4.2s
+       umlal   v23.2d,v12.2s,v1.2s
+       umlal   v20.2d,v12.2s,v6.2s
+       umlal   v21.2d,v12.2s,v8.2s
+
+       umlal   v22.2d,v13.2s,v8.2s
+       umlal   v19.2d,v13.2s,v2.2s
+       umlal   v23.2d,v13.2s,v0.2s
+       umlal   v20.2d,v13.2s,v4.2s
+       umlal   v21.2d,v13.2s,v6.2s
+
+.Lshort_tail:
+       ////////////////////////////////////////////////////////////////
+       // horizontal add
+
+       addp    v22.2d,v22.2d,v22.2d
+       ldp     d8,d9,[sp,#16]          // meet ABI requirements
+       addp    v19.2d,v19.2d,v19.2d
+       ldp     d10,d11,[sp,#32]
+       addp    v23.2d,v23.2d,v23.2d
+       ldp     d12,d13,[sp,#48]
+       addp    v20.2d,v20.2d,v20.2d
+       ldp     d14,d15,[sp,#64]
+       addp    v21.2d,v21.2d,v21.2d
+
+       ////////////////////////////////////////////////////////////////
+       // lazy reduction, but without narrowing
+
+       ushr    v29.2d,v22.2d,#26
+       and     v22.16b,v22.16b,v31.16b
+       ushr    v30.2d,v19.2d,#26
+       and     v19.16b,v19.16b,v31.16b
+
+       add     v23.2d,v23.2d,v29.2d    // h3 -> h4
+       add     v20.2d,v20.2d,v30.2d    // h0 -> h1
+
+       ushr    v29.2d,v23.2d,#26
+       and     v23.16b,v23.16b,v31.16b
+       ushr    v30.2d,v20.2d,#26
+       and     v20.16b,v20.16b,v31.16b
+       add     v21.2d,v21.2d,v30.2d    // h1 -> h2
+
+       add     v19.2d,v19.2d,v29.2d
+       shl     v29.2d,v29.2d,#2
+       ushr    v30.2d,v21.2d,#26
+       and     v21.16b,v21.16b,v31.16b
+       add     v19.2d,v19.2d,v29.2d    // h4 -> h0
+       add     v22.2d,v22.2d,v30.2d    // h2 -> h3
+
+       ushr    v29.2d,v19.2d,#26
+       and     v19.16b,v19.16b,v31.16b
+       ushr    v30.2d,v22.2d,#26
+       and     v22.16b,v22.16b,v31.16b
+       add     v20.2d,v20.2d,v29.2d    // h0 -> h1
+       add     v23.2d,v23.2d,v30.2d    // h3 -> h4
+
+       ////////////////////////////////////////////////////////////////
+       // write the result, can be partially reduced
+
+       st4     {v19.s,v20.s,v21.s,v22.s}[0],[x0],#16
+       st1     {v23.s}[0],[x0]
+
+.Lno_data_neon:
+       ldr     x29,[sp],#80
+       ret
+ENDPROC(poly1305_blocks_neon)
+
+.align 5
+ENTRY(poly1305_emit_neon)
+       ldr     x17,[x0,#24]
+       cbz     x17,poly1305_emit_arm
+
+       ldp     w10,w11,[x0]            // load hash value base 2^26
+       ldp     w12,w13,[x0,#8]
+       ldr     w14,[x0,#16]
+
+       add     x4,x10,x11,lsl#26       // base 2^26 -> base 2^64
+       lsr     x5,x12,#12
+       adds    x4,x4,x12,lsl#52
+       add     x5,x5,x13,lsl#14
+       adc     x5,x5,xzr
+       lsr     x6,x14,#24
+       adds    x5,x5,x14,lsl#40
+       adc     x6,x6,xzr               // can be partially reduced...
+
+       ldp     x10,x11,[x2]    // load nonce
+
+       and     x12,x6,#-4              // ... so reduce
+       add     x12,x12,x6,lsr#2
+       and     x6,x6,#3
+       adds    x4,x4,x12
+       adcs    x5,x5,xzr
+       adc     x6,x6,xzr
+
+       adds    x12,x4,#5               // compare to modulus
+       adcs    x13,x5,xzr
+       adc     x14,x6,xzr
+
+       tst     x14,#-4                 // see if it's carried/borrowed
+
+       csel    x4,x4,x12,eq
+       csel    x5,x5,x13,eq
+
+#ifdef __ARMEB__
+       ror     x10,x10,#32             // flip nonce words
+       ror     x11,x11,#32
+#endif
+       adds    x4,x4,x10               // accumulate nonce
+       adc     x5,x5,x11
+#ifdef __ARMEB__
+       rev     x4,x4                   // flip output bytes
+       rev     x5,x5
+#endif
+       stp     x4,x5,[x1]              // write result
+
+       ret
+ENDPROC(poly1305_emit_neon)
+
+.align 5
+.Lzeros:
+.long  0,0,0,0,0,0,0,0
--

2.19.0

author maintainer
#10 lore raw

Re: [PATCH net-next v4 08/20] zinc: Poly1305 ARM and ARM64 implementations

Hi Ard,

Quoted 2 lines · Ard Biesheuvel

As I asked in response to v3, could we please have this as a separate patch on top? The diff below is corrupted.

I had played with that originally, but thought it made things actually harder to review, whereas here you have the changes presented pretty straight forwardly, and I'd appreciate your review of them. If you and Eric both prefer I split this into two commits, with the first one just plopping down the CRYPTOGAMS code as is and the second one bringing it up to kernel-snuff, I can do that.

Quoted 2 lines

Also, both Andy and Eric have offered to get involved in upstreaming these changes to OpenSSL, so there is no delta to begin with.

Yes, I think this is probably a good long-term plan, which we can act on sometime after Zinc is merged.

Quoted 3 lines

I still don't like the GCC -includes, especially because these .h files contain function and variable definitions so they are not actually header files to begin with.

I very very strongly disagree with you here. I think doing it via -include is significantly cleaner than any of the alternatives, and allows the code to be cleanly expressed as conditionals that the optimizer trivially compiles out in the case of stub functions returning false and branch optimizes when the stub functions return true. It is extremely important that these compile together as one compilation unit. Yes, this is a different design than the crypto API's approach, but I believe the approach presented here poses significant improvements and is a lot cleaner.

Quoted 4 lines

Also, you mentioned in the commit log that you got rid of defines and made the code more modular, but as far as I can tell, libzinc is still a single monolithic binary that is essentially always builtin once we move random.c to it.

Yes, it's still monolithic, but it's now trivial to split up when the time comes to do that. If you and AndyL think that it should be split into multiple modules _now_, then I can go ahead and do that for v5. But if it's not essential, it seems simpler to keep it as is. I'll wait for word from you two on this.

Jason

author maintainer
#11 lore raw

PATCH v4 9/20 zinc: Poly1305 x86_64 implementation

In reply to Jason A. Donenfeld

This provides AVX, AVX-2, and AVX-512F implementations for Poly1305. The AVX-512F implementation is disabled on Skylake, due to throttling. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- poly1305_blocks_avx512 jumped to the middle of the poly1305_blocks_avx2 for the final blocks. To appease objtool, the relevant tail avx2 code was duplicated for the avx512 function.

- The original uses %rbp as a scratch register. However, the kernel expects %rbp to be a valid frame pointer at any given time in order to do proper unwinding. Thus we need to alter the code in order to preserve it. The most straightforward manner in which this was accomplished was by replacing $d3, formerly %r10, by %rdi, and replacing %rbp by %r10. Because %rdi, a pointer to the context structure, does not change and is not used by poly1305_iteration, it is safe to use it here, and the overhead of saving and restoring it should be minimal.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                        |    4 +
 lib/zinc/poly1305/poly1305-x86_64-glue.h |  109 +
 lib/zinc/poly1305/poly1305-x86_64.S      | 2792 ++++++++++++++++++++++
 3 files changed, 2905 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-x86_64-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-x86_64.S
lib/zinc/Makefile +4 -0
index f37df89a3f87..72112f8ffba1 100644
@@ -25,6 +25,10 @@ endif
 
 ifeq ($(CONFIG_ZINC_POLY1305),y)
 zinc-y += poly1305/poly1305.o
+ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
+zinc-y += poly1305/poly1305-x86_64.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-x86_64-glue.h
+endif
 ifeq ($(CONFIG_ZINC_ARCH_ARM),y)
 zinc-y += poly1305/poly1305-arm.o
 CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
lib/zinc/poly1305/poly1305-x86_64-glue.h new file +109 -0

109 lines not on this page · raw message

lib/zinc/poly1305/poly1305-x86_64.S new file +2792 -0

2,792 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#12 lore raw

PATCH v4 10/20 zinc: Poly1305 MIPS32r2 and MIPS64 implementations

In reply to Jason A. Donenfeld

This MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets. The MIPS64 implementation comes from Andy Polyakov results in a nice speedup on commodity Octeon hardware, and has been modified slightly from the original:

- The function names have been renamed to fit kernel conventions.
- A comment has been added.

No changes have been made to the actual instructions.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Ralf Baechle <[email protected]>
Cc Paul Burton <[email protected]>
Cc James Hogan <[email protected]>
---
 lib/zinc/Makefile                      |   8 +
 lib/zinc/poly1305/poly1305-mips-glue.h |  40 +++
 lib/zinc/poly1305/poly1305-mips.S      | 417 +++++++++++++++++++++++++
 lib/zinc/poly1305/poly1305-mips64.S    | 359 +++++++++++++++++++++
 4 files changed, 824 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-mips-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-mips.S
 create mode 100644 lib/zinc/poly1305/poly1305-mips64.S
lib/zinc/Makefile +8 -0
index 72112f8ffba1..c8c49d59794b 100644
@@ -37,6 +37,14 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM64),y)
 zinc-y += poly1305/poly1305-arm64.o
 CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-arm-glue.h
 endif
+ifeq ($(CONFIG_ZINC_ARCH_MIPS)$(CONFIG_CPU_MIPS32_R2),yy)
+zinc-y += poly1305/poly1305-mips.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h
+endif
+ifeq ($(CONFIG_ZINC_ARCH_MIPS64),y)
+zinc-y += poly1305/poly1305-mips64.o
+CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h
+endif
 endif
 
 zinc-y += main.o
lib/zinc/poly1305/poly1305-mips-glue.h new file +40 -0
new file mode 100644
index 000000000000..e29f85915eec
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <zinc/poly1305.h>
+
+asmlinkage void poly1305_init_mips(void *ctx, const u8 key[16]);
+asmlinkage void poly1305_blocks_mips(void *ctx, const u8 *inp, const size_t len,
+				     const u32 padbit);
+asmlinkage void poly1305_emit_mips(void *ctx, u8 mac[16], const u32 nonce[4]);
+void __init poly1305_fpu_init(void)
+{
+}
+
+static inline bool poly1305_init_arch(void *ctx,
+				      const u8 key[POLY1305_KEY_SIZE],
+				      simd_context_t simd_context)
+{
+	poly1305_init_mips(ctx, key);
+	return true;
+}
+
+static inline bool poly1305_blocks_arch(void *ctx, const u8 *inp,
+					const size_t len, const u32 padbit,
+					simd_context_t simd_context)
+{
+	poly1305_blocks_mips(ctx, inp, len, padbit);
+	return true;
+}
+
+static inline bool poly1305_emit_arch(void *ctx, u8 mac[POLY1305_MAC_SIZE],
+				      const u32 nonce[4],
+				      simd_context_t simd_context)
+{
+	poly1305_emit_mips(ctx, mac, nonce);
+	return true;
+}
+
+#define HAVE_POLY1305_ARCH_IMPLEMENTATION
lib/zinc/poly1305/poly1305-mips.S new file +417 -0

417 lines not on this page · raw message

lib/zinc/poly1305/poly1305-mips64.S new file +359 -0

359 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#13 lore raw

PATCH v4 11/20 zinc: ChaCha20Poly1305 construction and selftest

In reply to Jason A. Donenfeld

This is an implementation of the ChaCha20Poly1305 AEAD, with an easy API for encrypting either contiguous buffers or scatter gather lists (such as those created from skb_to_sgvec).

Information: https://tools.ietf.org/html/rfc8439

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20poly1305.h      |   54 +
 lib/zinc/Kconfig                     |    7 +
 lib/zinc/Makefile                    |    4 +
 lib/zinc/chacha20poly1305.c          |  333 ++
 lib/zinc/main.c                      |    4 +
 lib/zinc/selftest/chacha20poly1305.h | 7852 ++++++++++++++++++++++++++
 6 files changed, 8254 insertions(+)
 create mode 100644 include/zinc/chacha20poly1305.h
 create mode 100644 lib/zinc/chacha20poly1305.c
 create mode 100644 lib/zinc/selftest/chacha20poly1305.h
include/zinc/chacha20poly1305.h new file +54 -0
new file mode 100644
index 000000000000..b607c76f665d
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20POLY1305_H
+#define _ZINC_CHACHA20POLY1305_H
+
+#include <linux/simd.h>
+#include <linux/types.h>
+
+struct scatterlist;
+
+enum chacha20poly1305_lengths {
+	XCHACHA20POLY1305_NONCELEN = 24,
+	CHACHA20POLY1305_KEYLEN = 32,
+	CHACHA20POLY1305_AUTHTAGLEN = 16
+};
+
+void chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
+			      const u8 *ad, const size_t ad_len,
+			      const u64 nonce,
+			      const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check chacha20poly1305_encrypt_sg(
+	struct scatterlist *dst, struct scatterlist *src, const size_t src_len,
+	const u8 *ad, const size_t ad_len, const u64 nonce,
+	const u8 key[CHACHA20POLY1305_KEYLEN], simd_context_t simd_context);
+
+bool __must_check
+chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
+			 const u8 *ad, const size_t ad_len, const u64 nonce,
+			 const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check chacha20poly1305_decrypt_sg(
+	struct scatterlist *dst, struct scatterlist *src, const size_t src_len,
+	const u8 *ad, const size_t ad_len, const u64 nonce,
+	const u8 key[CHACHA20POLY1305_KEYLEN], simd_context_t simd_context);
+
+void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
+			       const u8 *ad, const size_t ad_len,
+			       const u8 nonce[XCHACHA20POLY1305_NONCELEN],
+			       const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check xchacha20poly1305_decrypt(
+	u8 *dst, const u8 *src, const size_t src_len, const u8 *ad,
+	const size_t ad_len, const u8 nonce[XCHACHA20POLY1305_NONCELEN],
+	const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+#ifdef DEBUG
+bool chacha20poly1305_selftest(void);
+#endif
+
+#endif /* _ZINC_CHACHA20POLY1305_H */
lib/zinc/Kconfig +7 -0
index bc8c61334362..98d095ed2418 100644
@@ -10,6 +10,13 @@ config ZINC_POLY1305
 	bool
 	select ZINC
 
+config ZINC_CHACHA20POLY1305
+	bool
+	select ZINC
+	select ZINC_CHACHA20
+	select ZINC_POLY1305
+	select CRYPTO_BLKCIPHER
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	depends on ZINC
lib/zinc/Makefile +4 -0
index c8c49d59794b..1664871aaf71 100644
@@ -47,6 +47,10 @@ CFLAGS_poly1305.o += -include $(srctree)/$(src)/poly1305/poly1305-mips-glue.h
 endif
 endif
 
+ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y)
+zinc-y += chacha20poly1305.o
+endif
+
 zinc-y += main.o
 
 obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/chacha20poly1305.c new file +333 -0

333 lines not on this page · raw message

lib/zinc/main.c +4 -0
index d871dd406a5c..a15b27fd0e4c 100644
@@ -3,6 +3,7 @@
  * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
  */
 
+#include <zinc/chacha20poly1305.h>
 #include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
 
@@ -26,6 +27,9 @@ static int __init mod_init(void)
 #ifdef CONFIG_ZINC_POLY1305
 	poly1305_fpu_init();
 	selftest(poly1305);
+#endif
+#ifdef CONFIG_ZINC_CHACHA20POLY1305
+	selftest(chacha20poly1305);
 #endif
 	return 0;
 }
lib/zinc/selftest/chacha20poly1305.h new file +7852 -0

2,574 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#14 lore raw

PATCH v4 12/20 zinc: BLAKE2s generic C implementation and selftest

In reply to Jason A. Donenfeld

The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.

Information: https://blake2.net/

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/blake2s.h      |  101 ++
 lib/zinc/Kconfig            |    4 +
 lib/zinc/Makefile           |    4 +
 lib/zinc/blake2s/blake2s.c  |  274 +++++
 lib/zinc/main.c             |    5 +
 lib/zinc/selftest/blake2s.h | 2095 +++++++++++++++++++++++++++++++++++
 6 files changed, 2483 insertions(+)
 create mode 100644 include/zinc/blake2s.h
 create mode 100644 lib/zinc/blake2s/blake2s.c
 create mode 100644 lib/zinc/selftest/blake2s.h
include/zinc/blake2s.h new file +101 -0
new file mode 100644
index 000000000000..5e32d576e86a
@@ -0,0 +1,101 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_BLAKE2S_H
+#define _ZINC_BLAKE2S_H
+
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <crypto/algapi.h>
+
+enum blake2s_lengths {
+	BLAKE2S_BLOCKBYTES = 64,
+	BLAKE2S_OUTBYTES = 32,
+	BLAKE2S_KEYBYTES = 32
+};
+
+struct blake2s_state {
+	u32 h[8];
+	u32 t[2];
+	u32 f[2];
+	u8 buf[BLAKE2S_BLOCKBYTES];
+	size_t buflen;
+	u8 last_node;
+};
+
+void blake2s_init(struct blake2s_state *state, const size_t outlen);
+void blake2s_init_key(struct blake2s_state *state, const size_t outlen,
+		      const void *key, const size_t keylen);
+void blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen);
+void __blake2s_final(struct blake2s_state *state);
+static inline void blake2s_final(struct blake2s_state *state, u8 *out,
+				 const size_t outlen)
+{
+	int i;
+
+#ifdef DEBUG
+	BUG_ON(!out || !outlen || outlen > BLAKE2S_OUTBYTES);
+#endif
+	__blake2s_final(state);
+
+	if (__builtin_constant_p(outlen) && !(outlen % sizeof(u32))) {
+		if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+		    IS_ALIGNED((unsigned long)out, __alignof__(u32))) {
+			__le32 *outwords = (__le32 *)out;
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				outwords[i] = cpu_to_le32(state->h[i]);
+		} else {
+			__le32 buffer[BLAKE2S_OUTBYTES];
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				buffer[i] = cpu_to_le32(state->h[i]);
+			memcpy(out, buffer, outlen);
+			memzero_explicit(buffer, sizeof(buffer));
+		}
+	} else {
+		u8 buffer[BLAKE2S_OUTBYTES] __aligned(__alignof__(u32));
+		__le32 *outwords = (__le32 *)buffer;
+
+		for (i = 0; i < 8; ++i)
+			outwords[i] = cpu_to_le32(state->h[i]);
+		memcpy(out, buffer, outlen);
+		memzero_explicit(buffer, sizeof(buffer));
+	}
+
+	memzero_explicit(state, sizeof(*state));
+}
+
+static inline void blake2s(u8 *out, const u8 *in, const u8 *key,
+			   const size_t outlen, const size_t inlen,
+			   const size_t keylen)
+{
+	struct blake2s_state state;
+
+#ifdef DEBUG
+	BUG_ON((!in && inlen > 0) || !out || !outlen ||
+	       outlen > BLAKE2S_OUTBYTES || keylen > BLAKE2S_KEYBYTES ||
+	       (!key && keylen));
+#endif
+
+	if (keylen)
+		blake2s_init_key(&state, outlen, key, keylen);
+	else
+		blake2s_init(&state, outlen);
+
+	blake2s_update(&state, in, inlen);
+	blake2s_final(&state, out, outlen);
+}
+
+void blake2s_hmac(u8 *out, const u8 *in, const u8 *key, const size_t outlen,
+		  const size_t inlen, const size_t keylen);
+
+void blake2s_fpu_init(void);
+
+#ifdef DEBUG
+bool blake2s_selftest(void);
+#endif
+
+#endif /* _ZINC_BLAKE2S_H */
lib/zinc/Kconfig +4 -0
index 98d095ed2418..4db30012c2d6 100644
@@ -17,6 +17,10 @@ config ZINC_CHACHA20POLY1305
 	select ZINC_POLY1305
 	select CRYPTO_BLKCIPHER
 
+config ZINC_BLAKE2S
+	bool
+	select ZINC
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	depends on ZINC
lib/zinc/Makefile +4 -0
index 1664871aaf71..45817ec5539c 100644
@@ -51,6 +51,10 @@ ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y)
 zinc-y += chacha20poly1305.o
 endif
 
+ifeq ($(CONFIG_ZINC_BLAKE2S),y)
+zinc-y += blake2s/blake2s.o
+endif
+
 zinc-y += main.o
 
 obj-$(CONFIG_ZINC) := zinc.o
lib/zinc/blake2s/blake2s.c new file +274 -0

274 lines not on this page · raw message

lib/zinc/main.c +5 -0
index a15b27fd0e4c..b1d650d82edb 100644
@@ -6,6 +6,7 @@
 #include <zinc/chacha20poly1305.h>
 #include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
+#include <zinc/blake2s.h>
 
 #include <linux/init.h>
 #include <linux/module.h>
@@ -30,6 +31,10 @@ static int __init mod_init(void)
 #endif
 #ifdef CONFIG_ZINC_CHACHA20POLY1305
 	selftest(chacha20poly1305);
+#endif
+#ifdef CONFIG_ZINC_BLAKE2S
+	blake2s_fpu_init();
+	selftest(blake2s);
 #endif
 	return 0;
 }
lib/zinc/selftest/blake2s.h new file +2095 -0

2,095 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#15 lore raw

PATCH v4 13/20 zinc: BLAKE2s x86_64 implementation

In reply to Jason A. Donenfeld

These implementations from Samuel Neves support AVX and AVX-512VL. Originally this used AVX-512F, but Skylake thermal throttling made AVX-512VL more attractive and possible to do with negligable difference.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                      |   4 +
 lib/zinc/blake2s/blake2s-x86_64-glue.h |  62 +++
 lib/zinc/blake2s/blake2s-x86_64.S      | 685 +++++++++++++++++++++++++
 3 files changed, 751 insertions(+)
 create mode 100644 lib/zinc/blake2s/blake2s-x86_64-glue.h
 create mode 100644 lib/zinc/blake2s/blake2s-x86_64.S
lib/zinc/Makefile +4 -0
index 45817ec5539c..49455abfdf5e 100644
@@ -53,6 +53,10 @@ endif
 
 ifeq ($(CONFIG_ZINC_BLAKE2S),y)
 zinc-y += blake2s/blake2s.o
+ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
+zinc-y += blake2s/blake2s-x86_64.o
+CFLAGS_blake2s.o += -include $(srctree)/$(src)/blake2s/blake2s-x86_64-glue.h
+endif
 endif
 
 zinc-y += main.o
lib/zinc/blake2s/blake2s-x86_64-glue.h new file +62 -0

62 lines not on this page · raw message

lib/zinc/blake2s/blake2s-x86_64.S new file +685 -0

685 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#16 lore raw

PATCH v4 14/20 zinc: Curve25519 generic C implementations and selftest

In reply to Jason A. Donenfeld

This contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic.

The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star
The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto

Information: https://cr.yp.to/ecdh.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Karthikeyan Bhargavan <[email protected]>
---
 include/zinc/curve25519.h               |   28 +
 lib/zinc/Kconfig                        |    5 +
 lib/zinc/Makefile                       |    4 +
 lib/zinc/curve25519/curve25519-fiat32.h |  862 +++++++++++++++
 lib/zinc/curve25519/curve25519-hacl64.h |  785 ++++++++++++++
 lib/zinc/curve25519/curve25519.c        |   83 ++
 lib/zinc/main.c                         |    5 +
 lib/zinc/selftest/curve25519.h          | 1321 +++++++++++++++++++++++
 8 files changed, 3093 insertions(+)
 create mode 100644 include/zinc/curve25519.h
 create mode 100644 lib/zinc/curve25519/curve25519-fiat32.h
 create mode 100644 lib/zinc/curve25519/curve25519-hacl64.h
 create mode 100644 lib/zinc/curve25519/curve25519.c
 create mode 100644 lib/zinc/selftest/curve25519.h
include/zinc/curve25519.h new file +28 -0
new file mode 100644
index 000000000000..0e1caf02f1d8
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CURVE25519_H
+#define _ZINC_CURVE25519_H
+
+#include <linux/types.h>
+
+enum curve25519_lengths {
+	CURVE25519_POINT_SIZE = 32
+};
+
+bool __must_check curve25519(u8 mypublic[CURVE25519_POINT_SIZE],
+			     const u8 secret[CURVE25519_POINT_SIZE],
+			     const u8 basepoint[CURVE25519_POINT_SIZE]);
+void curve25519_generate_secret(u8 secret[CURVE25519_POINT_SIZE]);
+bool __must_check curve25519_generate_public(
+	u8 pub[CURVE25519_POINT_SIZE], const u8 secret[CURVE25519_POINT_SIZE]);
+
+void curve25519_fpu_init(void);
+
+#ifdef DEBUG
+bool curve25519_selftest(void);
+#endif
+
+#endif /* _ZINC_CURVE25519_H */
lib/zinc/Kconfig +5 -0
index 4db30012c2d6..e4e10a1f4027 100644
@@ -21,6 +21,11 @@ config ZINC_BLAKE2S
 	bool
 	select ZINC
 
+config ZINC_CURVE25519
+	bool
+	select ZINC
+	select CONFIG_CRYPTO
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	depends on ZINC
lib/zinc/Makefile +4 -0
index 49455abfdf5e..25826d3eb74a 100644
@@ -51,6 +51,10 @@ ifeq ($(CONFIG_ZINC_CHACHA20POLY1305),y)
 zinc-y += chacha20poly1305.o
 endif
 
+ifeq ($(CONFIG_ZINC_CURVE25519),y)
+zinc-y += curve25519/curve25519.o
+endif
+
 ifeq ($(CONFIG_ZINC_BLAKE2S),y)
 zinc-y += blake2s/blake2s.o
 ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
lib/zinc/curve25519/curve25519-fiat32.h new file +862 -0

862 lines not on this page · raw message

lib/zinc/curve25519/curve25519-hacl64.h new file +785 -0

785 lines not on this page · raw message

lib/zinc/curve25519/curve25519.c new file +83 -0

83 lines not on this page · raw message

lib/zinc/main.c +5 -0
index b1d650d82edb..8138b6d99754 100644
@@ -7,6 +7,7 @@
 #include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
 #include <zinc/blake2s.h>
+#include <zinc/curve25519.h>
 
 #include <linux/init.h>
 #include <linux/module.h>
@@ -35,6 +36,10 @@ static int __init mod_init(void)
 #ifdef CONFIG_ZINC_BLAKE2S
 	blake2s_fpu_init();
 	selftest(blake2s);
+#endif
+#ifdef CONFIG_ZINC_CURVE25519
+	curve25519_fpu_init();
+	selftest(curve25519);
 #endif
 	return 0;
 }
lib/zinc/selftest/curve25519.h new file +1321 -0

1,204 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#17 lore raw

PATCH v4 15/20 zinc: Curve25519 ARM implementation

In reply to Jason A. Donenfeld

This comes from Dan Bernstein and Peter Schwabe's public domain NEON code, and has been modified to be friendly for kernel space, as well as removing some qhasm strangeness to be more idiomatic.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                         |    4 +
 lib/zinc/curve25519/curve25519-arm-glue.h |   46 +
 lib/zinc/curve25519/curve25519-arm.S      | 2095 +++++++++++++++++++++
 3 files changed, 2145 insertions(+)
 create mode 100644 lib/zinc/curve25519/curve25519-arm-glue.h
 create mode 100644 lib/zinc/curve25519/curve25519-arm.S
lib/zinc/Makefile +4 -0
index 25826d3eb74a..0a9d97146c70 100644
@@ -53,6 +53,10 @@ endif
 
 ifeq ($(CONFIG_ZINC_CURVE25519),y)
 zinc-y += curve25519/curve25519.o
+ifeq ($(CONFIG_ZINC_ARCH_ARM)$(CONFIG_KERNEL_MODE_NEON),yy)
+zinc-y += curve25519/curve25519-arm.o
+CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-arm-glue.h
+endif
 endif
 
 ifeq ($(CONFIG_ZINC_BLAKE2S),y)
lib/zinc/curve25519/curve25519-arm-glue.h new file +46 -0

46 lines not on this page · raw message

lib/zinc/curve25519/curve25519-arm.S new file +2095 -0

2,095 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#18 lore raw

PATCH v4 16/20 zinc: Curve25519 x86_64 implementation

In reply to Jason A. Donenfeld

This implementation is the fastest available x86_64 implementation, and unlike Sandy2x, it doesn't requie use of the floating point registers at all. Instead it makes use of BMI2 and ADX, available on recent microarchitectures. The implementation was written by Armando Faz-Hernández with contributions (upstream) from Samuel Neves and me, in addition to further changes in the kernel implementation from us.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Armando Faz-Hernández <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                            |    3 +
 lib/zinc/curve25519/curve25519-x86_64-glue.h |   49 +
 lib/zinc/curve25519/curve25519-x86_64.h      | 2333 ++++++++++++++++++
 3 files changed, 2385 insertions(+)
 create mode 100644 lib/zinc/curve25519/curve25519-x86_64-glue.h
 create mode 100644 lib/zinc/curve25519/curve25519-x86_64.h
lib/zinc/Makefile +3 -0
index 0a9d97146c70..adb3acda630a 100644
@@ -57,6 +57,9 @@ ifeq ($(CONFIG_ZINC_ARCH_ARM)$(CONFIG_KERNEL_MODE_NEON),yy)
 zinc-y += curve25519/curve25519-arm.o
 CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-arm-glue.h
 endif
+ifeq ($(CONFIG_ZINC_ARCH_X86_64),y)
+CFLAGS_curve25519.o += -include $(srctree)/$(src)/curve25519/curve25519-x86_64-glue.h
+endif
 endif
 
 ifeq ($(CONFIG_ZINC_BLAKE2S),y)
lib/zinc/curve25519/curve25519-x86_64-glue.h new file +49 -0

49 lines not on this page · raw message

lib/zinc/curve25519/curve25519-x86_64.h new file +2333 -0

2,333 lines not on this page · raw message

Signature · 1 line
2.19.0
author maintainer
#19 lore raw

PATCH v4 17/20 crypto: port Poly1305 to Zinc

In reply to Jason A. Donenfeld

Now that Poly1305 is in Zinc, we can have the crypto API code simply call into it. We have to do a little bit of book keeping here, because the crypto API receives the key in the first few calls to update.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
---
 arch/x86/crypto/Makefile               |   3 -
 arch/x86/crypto/poly1305-avx2-x86_64.S | 388 ----------------
 arch/x86/crypto/poly1305-sse2-x86_64.S | 584 -------------------------
 arch/x86/crypto/poly1305_glue.c        | 205 ---------
 crypto/Kconfig                         |  15 +-
 crypto/Makefile                        |   2 +-
 crypto/chacha20poly1305.c              |  12 +-
 crypto/poly1305_generic.c              | 304 -------------
 crypto/poly1305_zinc.c                 |  92 ++++
 include/crypto/poly1305.h              |  40 --
 10 files changed, 101 insertions(+), 1544 deletions(-)
 delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305_glue.c
 delete mode 100644 crypto/poly1305_generic.c
 create mode 100644 crypto/poly1305_zinc.c
 delete mode 100644 include/crypto/poly1305.h
arch/x86/crypto/Makefile +0 -3
index a450ad573dcb..cf830219846b 100644
@@ -34,7 +34,6 @@ obj-$(CONFIG_CRYPTO_CRC32_PCLMUL) += crc32-pclmul.o
 obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o
 obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
 obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o
-obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o
 obj-$(CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2) += aegis128l-aesni.o
@@ -110,10 +109,8 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
-poly1305-x86_64-y := poly1305-sse2-x86_64.o poly1305_glue.o
 ifeq ($(avx2_supported),yes)
 sha1-ssse3-y += sha1_avx2_x86_64_asm.o
-poly1305-x86_64-y += poly1305-avx2-x86_64.o
 endif
 ifeq ($(sha1_ni_supported),yes)
 sha1-ssse3-y += sha1_ni_asm.o
arch/x86/crypto/poly1305-avx2-x86_64.S deleted +0 -388

388 lines not on this page · raw message

arch/x86/crypto/poly1305-sse2-x86_64.S deleted +0 -584

584 lines not on this page · raw message

arch/x86/crypto/poly1305_glue.c deleted +0 -205

205 lines not on this page · raw message

crypto/Kconfig +2 -13
index f3e40ac56d93..47859a0f8052 100644
@@ -656,24 +656,13 @@ config CRYPTO_GHASH
 config CRYPTO_POLY1305
 	tristate "Poly1305 authenticator algorithm"
 	select CRYPTO_HASH
+	select ZINC_POLY1305
 	help
 	  Poly1305 authenticator algorithm, RFC7539.
 
 	  Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein.
 	  It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use
-	  in IETF protocols. This is the portable C implementation of Poly1305.
-
-config CRYPTO_POLY1305_X86_64
-	tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)"
-	depends on X86 && 64BIT
-	select CRYPTO_POLY1305
-	help
-	  Poly1305 authenticator algorithm, RFC7539.
-
-	  Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein.
-	  It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use
-	  in IETF protocols. This is the x86_64 assembler implementation using SIMD
-	  instructions.
+	  in IETF protocols.
 
 config CRYPTO_MD4
 	tristate "MD4 digest algorithm"
crypto/Makefile +1 -1
index 6d1d40eeb964..5e60348d02e2 100644
@@ -118,7 +118,7 @@ obj-$(CONFIG_CRYPTO_SEED) += seed.o
 obj-$(CONFIG_CRYPTO_SPECK) += speck.o
 obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o
 obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o
-obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o
+obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
 obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o
crypto/chacha20poly1305.c +6 -6
index 600afa99941f..bf523797bef3 100644
@@ -14,7 +14,7 @@
 #include <crypto/internal/skcipher.h>
 #include <crypto/scatterwalk.h>
 #include <crypto/chacha20.h>
-#include <crypto/poly1305.h>
+#include <zinc/poly1305.h>
 #include <linux/err.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
@@ -62,7 +62,7 @@ struct chachapoly_req_ctx {
 	/* the key we generate for Poly1305 using Chacha20 */
 	u8 key[POLY1305_KEY_SIZE];
 	/* calculated Poly1305 tag */
-	u8 tag[POLY1305_DIGEST_SIZE];
+	u8 tag[POLY1305_MAC_SIZE];
 	/* length of data to en/decrypt, without ICV */
 	unsigned int cryptlen;
 	/* Actual AD, excluding IV */
@@ -471,7 +471,7 @@ static int chachapoly_decrypt(struct aead_request *req)
 {
 	struct chachapoly_req_ctx *rctx = aead_request_ctx(req);
 
-	rctx->cryptlen = req->cryptlen - POLY1305_DIGEST_SIZE;
+	rctx->cryptlen = req->cryptlen - POLY1305_MAC_SIZE;
 
 	/* decrypt call chain:
 	 * - poly_genkey/done()
@@ -513,7 +513,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 static int chachapoly_setauthsize(struct crypto_aead *tfm,
 				  unsigned int authsize)
 {
-	if (authsize != POLY1305_DIGEST_SIZE)
+	if (authsize != POLY1305_MAC_SIZE)
 		return -EINVAL;
 
 	return 0;
@@ -613,7 +613,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
 	poly_hash = __crypto_hash_alg_common(poly);
 
 	err = -EINVAL;
-	if (poly_hash->digestsize != POLY1305_DIGEST_SIZE)
+	if (poly_hash->digestsize != POLY1305_MAC_SIZE)
 		goto out_put_poly;
 
 	err = -ENOMEM;
@@ -666,7 +666,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
 				     ctx->saltlen;
 	inst->alg.ivsize = ivsize;
 	inst->alg.chunksize = crypto_skcipher_alg_chunksize(chacha);
-	inst->alg.maxauthsize = POLY1305_DIGEST_SIZE;
+	inst->alg.maxauthsize = POLY1305_MAC_SIZE;
 	inst->alg.init = chachapoly_init;
 	inst->alg.exit = chachapoly_exit;
 	inst->alg.encrypt = chachapoly_encrypt;
crypto/poly1305_generic.c deleted +0 -304

304 lines not on this page · raw message

crypto/poly1305_zinc.c new file +92 -0

92 lines not on this page · raw message

include/crypto/poly1305.h deleted +0 -40
deleted file mode 100644
index f718a19da82f..000000000000
@@ -1,40 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Common values for the Poly1305 algorithm
- */
-
-#ifndef _CRYPTO_POLY1305_H
-#define _CRYPTO_POLY1305_H
-
-#include <linux/types.h>
-#include <linux/crypto.h>
-
-#define POLY1305_BLOCK_SIZE	16
-#define POLY1305_KEY_SIZE	32
-#define POLY1305_DIGEST_SIZE	16
-
-struct poly1305_desc_ctx {
-	/* key */
-	u32 r[5];
-	/* finalize key */
-	u32 s[4];
-	/* accumulator */
-	u32 h[5];
-	/* partial buffer */
-	u8 buf[POLY1305_BLOCK_SIZE];
-	/* bytes used in partial buffer */
-	unsigned int buflen;
-	/* r key has been set */
-	bool rset;
-	/* s key has been set */
-	bool sset;
-};
-
-int crypto_poly1305_init(struct shash_desc *desc);
-unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx,
-					const u8 *src, unsigned int srclen);
-int crypto_poly1305_update(struct shash_desc *desc,
-			   const u8 *src, unsigned int srclen);
-int crypto_poly1305_final(struct shash_desc *desc, u8 *dst);
-
-#endif
Signature · 1 line
2.19.0
#20 lore raw

Re: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc

I love your patch! Yet something to improve:

Hi Jason,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64

All errors (new ones prefixed by >>):

fs/sysfs/Kconfig:1:error: recursive dependency detected! fs/sysfs/Kconfig:1: symbol SYSFS is selected by CONFIGFS_FS fs/configfs/Kconfig:1: symbol CONFIGFS_FS is selected by ACPI_CONFIGFS drivers/acpi/Kconfig:542: symbol ACPI_CONFIGFS depends on ACPI

>> drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI
>> drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI
>> arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON
>> arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM
>> lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC

lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_POLY1305 lib/zinc/Kconfig:9: symbol ZINC_POLY1305 is selected by CRYPTO_POLY1305 crypto/Kconfig:656: symbol CRYPTO_POLY1305 depends on CRYPTO crypto/Kconfig:16: symbol CRYPTO is selected by IP_SCTP net/sctp/Kconfig:5: symbol IP_SCTP is selected by DLM fs/dlm/Kconfig:1: symbol DLM depends on SYSFS
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"

vim +9 drivers/acpi/Kconfig

^1da177e4 Linus Torvalds    2005-04-16   5  
f5d707ede Arnd Bergmann     2018-08-21  @6  config ARCH_SUPPORTS_ACPI
f5d707ede Arnd Bergmann     2018-08-21   7  	bool
f5d707ede Arnd Bergmann     2018-08-21   8  
3f2c48c9b Jan Engelhardt    2007-07-03  @9  menuconfig ACPI
355ee5eb6 Frans Pop         2007-10-29  10  	bool "ACPI (Advanced Configuration and Power Interface) Support"
2c870e611 Arnd Bergmann     2018-07-24  11  	depends on ARCH_SUPPORTS_ACPI
1300124f6 Adrian Bunk       2006-03-28  12  	depends on PCI
243b66e76 Len Brown         2007-02-15  13  	select PNP
2c870e611 Arnd Bergmann     2018-07-24  14  	default y if X86
1c48aa36e Bjorn Helgaas     2009-02-19  15  	help
^1da177e4 Linus Torvalds    2005-04-16  16  	  Advanced Configuration and Power Interface (ACPI) support for 
1c48aa36e Bjorn Helgaas     2009-02-19  17  	  Linux requires an ACPI-compliant platform (hardware/firmware),
^1da177e4 Linus Torvalds    2005-04-16  18  	  and assumes the presence of OS-directed configuration and power
^1da177e4 Linus Torvalds    2005-04-16  19  	  management (OSPM) software.  This option will enlarge your 
^1da177e4 Linus Torvalds    2005-04-16  20  	  kernel by about 70K.
^1da177e4 Linus Torvalds    2005-04-16  21  
^1da177e4 Linus Torvalds    2005-04-16  22  	  Linux ACPI provides a robust functional replacement for several 
^1da177e4 Linus Torvalds    2005-04-16  23  	  legacy configuration and power management interfaces, including
^1da177e4 Linus Torvalds    2005-04-16  24  	  the Plug-and-Play BIOS specification (PnP BIOS), the 
^1da177e4 Linus Torvalds    2005-04-16  25  	  MultiProcessor Specification (MPS), and the Advanced Power 
^1da177e4 Linus Torvalds    2005-04-16  26  	  Management (APM) specification.  If both ACPI and APM support 
1c48aa36e Bjorn Helgaas     2009-02-19  27  	  are configured, ACPI is used.
^1da177e4 Linus Torvalds    2005-04-16  28  
1c48aa36e Bjorn Helgaas     2009-02-19  29  	  The project home page for the Linux ACPI subsystem is here:
aaf3d29fe Rafael J. Wysocki 2013-10-10  30  	  <https://01.org/linux-acpi>
^1da177e4 Linus Torvalds    2005-04-16  31  
^1da177e4 Linus Torvalds    2005-04-16  32  	  Linux support for ACPI is based on Intel Corporation's ACPI
1c48aa36e Bjorn Helgaas     2009-02-19  33  	  Component Architecture (ACPI CA).  For more information on the
1c48aa36e Bjorn Helgaas     2009-02-19  34  	  ACPI CA, see:
1c48aa36e Bjorn Helgaas     2009-02-19  35  	  <http://acpica.org/>
^1da177e4 Linus Torvalds    2005-04-16  36  
c7f5220d0 Hanjun Guo        2014-04-08  37  	  ACPI is an open industry specification originally co-developed by
c7f5220d0 Hanjun Guo        2014-04-08  38  	  Hewlett-Packard, Intel, Microsoft, Phoenix, and Toshiba. Currently,
c7f5220d0 Hanjun Guo        2014-04-08  39  	  it is developed by the ACPI Specification Working Group (ASWG) under
c7f5220d0 Hanjun Guo        2014-04-08  40  	  the UEFI Forum and any UEFI member can join the ASWG and contribute
c7f5220d0 Hanjun Guo        2014-04-08  41  	  to the ACPI specification.
1c48aa36e Bjorn Helgaas     2009-02-19  42  	  The specification is available at:
^1da177e4 Linus Torvalds    2005-04-16  43  	  <http://www.acpi.info>
c7f5220d0 Hanjun Guo        2014-04-08  44  	  <http://www.uefi.org/acpi/specs>
^1da177e4 Linus Torvalds    2005-04-16  45  

:::::: The code at line 9 was first introduced by commit
:::::: 3f2c48c9b48423d1411695da066d525cca2a27db ACPI: Use menuconfig objects

:::::: TO: Jan Engelhardt <[email protected]> :::::: CC: Len Brown <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#21 lore raw

Re: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc

Greetings Mr. Ro Bot,

Another one of your robot friends also caught this, and the offending code has been removed for v5.

Thanks for botting,
Jason

#22 lore raw

Re: [PATCH net-next v4 17/20] crypto: port Poly1305 to Zinc

In reply to Jason A. Donenfeld
I love your patch! Yet something to improve:

Hi Jason,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64

All errors (new ones prefixed by >>):

>> fs/sysfs/Kconfig:1:error: recursive dependency detected!
>> fs/sysfs/Kconfig:1: symbol SYSFS is selected by CONFIGFS_FS
>> fs/configfs/Kconfig:1: symbol CONFIGFS_FS is selected by ACPI_CONFIGFS
>> drivers/acpi/Kconfig:542: symbol ACPI_CONFIGFS depends on ACPI
   drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI
   drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI
   arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON
   arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM
   lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC
>> lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_POLY1305
>> lib/zinc/Kconfig:9: symbol ZINC_POLY1305 is selected by CRYPTO_POLY1305
>> crypto/Kconfig:656: symbol CRYPTO_POLY1305 depends on CRYPTO
>> crypto/Kconfig:16: symbol CRYPTO is selected by IP_SCTP
>> net/sctp/Kconfig:5: symbol IP_SCTP is selected by DLM
>> fs/dlm/Kconfig:1: symbol DLM depends on SYSFS

For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"

vim +1 lib/zinc/Kconfig

32bbe22e Jason A. Donenfeld 2018-09-14 @1 config ZINC

32bbe22e Jason A. Donenfeld 2018-09-14   2  	tristate
32bbe22e Jason A. Donenfeld 2018-09-14   3  
35f45248 Jason A. Donenfeld 2018-09-14   4  config ZINC_CHACHA20
35f45248 Jason A. Donenfeld 2018-09-14   5  	bool
35f45248 Jason A. Donenfeld 2018-09-14   6  	select ZINC
35f45248 Jason A. Donenfeld 2018-09-14   7  	select CRYPTO_ALGAPI
35f45248 Jason A. Donenfeld 2018-09-14   8  
0a36c146 Jason A. Donenfeld 2018-09-14  @9  config ZINC_POLY1305
0a36c146 Jason A. Donenfeld 2018-09-14  10  	bool
0a36c146 Jason A. Donenfeld 2018-09-14  11  	select ZINC
0a36c146 Jason A. Donenfeld 2018-09-14  12  
1b5dbb86 Jason A. Donenfeld 2018-09-14  13  config ZINC_CHACHA20POLY1305
1b5dbb86 Jason A. Donenfeld 2018-09-14  14  	bool
1b5dbb86 Jason A. Donenfeld 2018-09-14  15  	select ZINC
1b5dbb86 Jason A. Donenfeld 2018-09-14  16  	select ZINC_CHACHA20
1b5dbb86 Jason A. Donenfeld 2018-09-14  17  	select ZINC_POLY1305
1b5dbb86 Jason A. Donenfeld 2018-09-14  18  	select CRYPTO_BLKCIPHER
1b5dbb86 Jason A. Donenfeld 2018-09-14  19  
a740374c Jason A. Donenfeld 2018-09-14  20  config ZINC_BLAKE2S
a740374c Jason A. Donenfeld 2018-09-14  21  	bool
a740374c Jason A. Donenfeld 2018-09-14  22  	select ZINC
a740374c Jason A. Donenfeld 2018-09-14  23  
cec5aa7c Jason A. Donenfeld 2018-09-14  24  config ZINC_CURVE25519
cec5aa7c Jason A. Donenfeld 2018-09-14  25  	bool
cec5aa7c Jason A. Donenfeld 2018-09-14  26  	select ZINC
cec5aa7c Jason A. Donenfeld 2018-09-14  27  	select CONFIG_CRYPTO
cec5aa7c Jason A. Donenfeld 2018-09-14  28  
32bbe22e Jason A. Donenfeld 2018-09-14  29  config ZINC_DEBUG
32bbe22e Jason A. Donenfeld 2018-09-14  30  	bool "Zinc cryptography library debugging and self-tests"
32bbe22e Jason A. Donenfeld 2018-09-14  31  	depends on ZINC
32bbe22e Jason A. Donenfeld 2018-09-14  32  	help
32bbe22e Jason A. Donenfeld 2018-09-14  33  	  This builds a series of self-tests for the Zinc crypto library, which
32bbe22e Jason A. Donenfeld 2018-09-14  34  	  help diagnose any cryptographic algorithm implementation issues that
32bbe22e Jason A. Donenfeld 2018-09-14  35  	  might be at the root cause of potential bugs. It also adds various
32bbe22e Jason A. Donenfeld 2018-09-14  36  	  debugging traps.
32bbe22e Jason A. Donenfeld 2018-09-14  37  
32bbe22e Jason A. Donenfeld 2018-09-14  38  	  Unless you're developing and testing cryptographic routines, or are
32bbe22e Jason A. Donenfeld 2018-09-14  39  	  especially paranoid about correctness on your hardware, you may say
32bbe22e Jason A. Donenfeld 2018-09-14  40  	  N here.
32bbe22e Jason A. Donenfeld 2018-09-14  41  
32bbe22e Jason A. Donenfeld 2018-09-14 @42  config ZINC_ARCH_ARM
32bbe22e Jason A. Donenfeld 2018-09-14  43  	def_bool y
32bbe22e Jason A. Donenfeld 2018-09-14  44  	depends on ARM
32bbe22e Jason A. Donenfeld 2018-09-14  45  	depends on ZINC
32bbe22e Jason A. Donenfeld 2018-09-14  46  	imply VFP
32bbe22e Jason A. Donenfeld 2018-09-14  47  	imply VFPv3 if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14  48  	imply NEON if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14  49  	imply KERNEL_MODE_NEON if CPU_V7

32bbe22e Jason A. Donenfeld 2018-09-14 50

:::::: The code at line 1 was first introduced by commit
:::::: 32bbe22ec6fddc21b70274aab7d2830d69d51578 zinc: introduce minimal cryptography library

:::::: TO: Jason A. Donenfeld <[email protected]> :::::: CC: 0day robot <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#23 lore raw

PATCH v4 18/20 crypto: port ChaCha20 to Zinc

In reply to Jason A. Donenfeld

Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it. The crypto API expects to have a stored key per instance and independent nonces, so we follow suite and store the key and initialize the nonce independently.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
---
 arch/arm/configs/exynos_defconfig       |   1 -
 arch/arm/configs/multi_v7_defconfig     |   1 -
 arch/arm/configs/omap2plus_defconfig    |   1 -
 arch/arm/crypto/Kconfig                 |   6 -
 arch/arm/crypto/Makefile                |   2 -
 arch/arm/crypto/chacha20-neon-core.S    | 521 --------------------
 arch/arm/crypto/chacha20-neon-glue.c    | 127 -----
 arch/arm64/configs/defconfig            |   1 -
 arch/arm64/crypto/Kconfig               |   6 -
 arch/arm64/crypto/Makefile              |   3 -
 arch/arm64/crypto/chacha20-neon-core.S  | 450 -----------------
 arch/arm64/crypto/chacha20-neon-glue.c  | 133 -----
 arch/x86/crypto/Makefile                |   3 -
 arch/x86/crypto/chacha20-avx2-x86_64.S  | 448 -----------------
 arch/x86/crypto/chacha20-ssse3-x86_64.S | 630 ------------------------
 arch/x86/crypto/chacha20_glue.c         | 146 ------
 crypto/Kconfig                          |  16 -
 crypto/Makefile                         |   2 +-
 crypto/chacha20_generic.c               | 136 -----
 crypto/chacha20_zinc.c                  | 100 ++++
 crypto/chacha20poly1305.c               |   2 +-
 include/crypto/chacha20.h               |  12 -
 22 files changed, 102 insertions(+), 2645 deletions(-)
 delete mode 100644 arch/arm/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/arm64/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm64/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20_glue.c
 delete mode 100644 crypto/chacha20_generic.c
 create mode 100644 crypto/chacha20_zinc.c
arch/arm/configs/exynos_defconfig +0 -1
index 27ea6dfcf2f2..95929b5e7b10 100644
@@ -350,7 +350,6 @@ CONFIG_CRYPTO_SHA1_ARM_NEON=m
 CONFIG_CRYPTO_SHA256_ARM=m
 CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_FONTS=y
 CONFIG_FONT_7x14=y
arch/arm/configs/multi_v7_defconfig +0 -1
index fc33444e94f0..63be07724db3 100644
@@ -1000,4 +1000,3 @@ CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_AES_ARM_CE=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
 CONFIG_CRYPTO_CRC32_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/omap2plus_defconfig +0 -1
index 6491419b1dad..f585a8ecc336 100644
@@ -547,7 +547,6 @@ CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_CRC_T10DIF=y
 CONFIG_CRC_ITU_T=y
arch/arm/crypto/Kconfig +0 -6
index 925d1364727a..fb80fd89f0e7 100644
@@ -115,12 +115,6 @@ config CRYPTO_CRC32_ARM_CE
 	depends on KERNEL_MODE_NEON && CRC32
 	select CRYPTO_HASH
 
-config CRYPTO_CHACHA20_NEON
-	tristate "NEON accelerated ChaCha20 symmetric cipher"
-	depends on KERNEL_MODE_NEON
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-
 config CRYPTO_SPECK_NEON
 	tristate "NEON accelerated Speck cipher algorithms"
 	depends on KERNEL_MODE_NEON
arch/arm/crypto/Makefile +0 -2
index 8de542c48ade..bbfa98447063 100644
@@ -9,7 +9,6 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o
 obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o
 obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 
 ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -53,7 +52,6 @@ aes-arm-ce-y	:= aes-ce-core.o aes-ce-glue.o
 ghash-arm-ce-y	:= ghash-ce-core.o ghash-ce-glue.o
 crct10dif-arm-ce-y	:= crct10dif-ce-core.o crct10dif-ce-glue.o
 crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
 ifdef REGENERATE_ARM_CRYPTO
arch/arm/crypto/chacha20-neon-core.S deleted +0 -521

521 lines not on this page · raw message

arch/arm/crypto/chacha20-neon-glue.c deleted +0 -127

127 lines not on this page · raw message

arch/arm64/configs/defconfig +0 -1
index db8d364f8476..6cc3c8a0ad88 100644
@@ -709,5 +709,4 @@ CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m
 CONFIG_CRYPTO_CRC32_ARM64_CE=m
 CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
 CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRYPTO_AES_ARM64_BS=m
arch/arm64/crypto/Kconfig +0 -6
index e3fdb0fd6f70..9db6d775a880 100644
@@ -105,12 +105,6 @@ config CRYPTO_AES_ARM64_NEON_BLK
 	select CRYPTO_AES
 	select CRYPTO_SIMD
 
-config CRYPTO_CHACHA20_NEON
-	tristate "NEON accelerated ChaCha20 symmetric cipher"
-	depends on KERNEL_MODE_NEON
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-
 config CRYPTO_AES_ARM64_BS
 	tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm"
 	depends on KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +0 -3
index bcafd016618e..507c4bfb86e3 100644
@@ -53,9 +53,6 @@ sha256-arm64-y := sha256-glue.o sha256-core.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o
 sha512-arm64-y := sha512-glue.o sha512-core.o
 
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
-
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
arch/arm64/crypto/chacha20-neon-core.S deleted +0 -450

450 lines not on this page · raw message

arch/arm64/crypto/chacha20-neon-glue.c deleted +0 -133

133 lines not on this page · raw message

arch/x86/crypto/Makefile +0 -3
index cf830219846b..419212c31246 100644
@@ -23,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o
 obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o
-obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o
 obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
@@ -76,7 +75,6 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o
 blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o
 twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o
 twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o
-chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o
 serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o
 
 aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -99,7 +97,6 @@ endif
 
 ifeq ($(avx2_supported),yes)
 	camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o
-	chacha20-x86_64-y += chacha20-avx2-x86_64.o
 	serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o
 
 	morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
arch/x86/crypto/chacha20-avx2-x86_64.S deleted +0 -448

448 lines not on this page · raw message

arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0 -630

630 lines not on this page · raw message

arch/x86/crypto/chacha20_glue.c deleted +0 -146

146 lines not on this page · raw message

crypto/Kconfig +0 -16
index 47859a0f8052..93cd4d199447 100644
@@ -1433,22 +1433,6 @@ config CRYPTO_CHACHA20
 
 	  ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
 	  Bernstein and further specified in RFC7539 for use in IETF protocols.
-	  This is the portable C implementation of ChaCha20.
-
-	  See also:
-	  <http://cr.yp.to/chacha/chacha-20080128.pdf>
-
-config CRYPTO_CHACHA20_X86_64
-	tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)"
-	depends on X86 && 64BIT
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-	help
-	  ChaCha20 cipher algorithm, RFC7539.
-
-	  ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
-	  Bernstein and further specified in RFC7539 for use in IETF protocols.
-	  This is the x86_64 assembler implementation using SIMD instructions.
 
 	  See also:
 	  <http://cr.yp.to/chacha/chacha-20080128.pdf>
crypto/Makefile +1 -1
index 5e60348d02e2..587103b87890 100644
@@ -117,7 +117,7 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o
 obj-$(CONFIG_CRYPTO_SEED) += seed.o
 obj-$(CONFIG_CRYPTO_SPECK) += speck.o
 obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o
-obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o
+obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_zinc.o
 obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
crypto/chacha20_generic.c deleted +0 -136

136 lines not on this page · raw message

crypto/chacha20_zinc.c new file +100 -0

100 lines not on this page · raw message

crypto/chacha20poly1305.c +1 -1
index bf523797bef3..b26adb9ed898 100644
@@ -13,7 +13,7 @@
 #include <crypto/internal/hash.h>
 #include <crypto/internal/skcipher.h>
 #include <crypto/scatterwalk.h>
-#include <crypto/chacha20.h>
+#include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
 #include <linux/err.h>
 #include <linux/init.h>
include/crypto/chacha20.h +0 -12
index b83d66073db0..3b92f58f3891 100644
@@ -6,23 +6,11 @@
 #ifndef _CRYPTO_CHACHA20_H
 #define _CRYPTO_CHACHA20_H
 
-#include <crypto/skcipher.h>
-#include <linux/types.h>
-#include <linux/crypto.h>
-
 #define CHACHA20_IV_SIZE	16
 #define CHACHA20_KEY_SIZE	32
 #define CHACHA20_BLOCK_SIZE	64
 #define CHACHA20_BLOCK_WORDS	(CHACHA20_BLOCK_SIZE / sizeof(u32))
 
-struct chacha20_ctx {
-	u32 key[8];
-};
-
 void chacha20_block(u32 *state, u32 *stream);
-void crypto_chacha20_init(u32 *state, struct chacha20_ctx *ctx, u8 *iv);
-int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key,
-			   unsigned int keysize);
-int crypto_chacha20_crypt(struct skcipher_request *req);
 
 #endif
Signature · 1 line
2.19.0
#24 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

Quoted 17 lines · Jason A. Donenfeld

Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it. The crypto API expects to have a stored key per instance and independent nonces, so we follow suite and store the key and initialize the nonce independently.

From our exchange re v3:

Quoted 8 lines

Then there is the performance claim. We know for instance that the OpenSSL ARM NEON code for ChaCha20 is faster on cores that happen to possess a micro-architectural property that ALU instructions are essentially free when they are interleaved with SIMD instructions. But we also know that a) Cortex-A7, which is a relevant target, is not one of those cores, and b) that chip designers are not likely to optimize for that particular usage pattern so relying on it in generic code is unwise in general.

Quoted 6 lines

That's interesting. I'll bring this up with AndyP. FWIW, if you think you have a real and compelling claim here, I'd be much more likely to accept a different ChaCha20 implementation than I would be to accept a different Poly1305 implementation. (It's a *lot* harder to screw up ChaCha20 than it is to screw up Poly1305.)

so could we please bring that discussion to a close before we drop the ARM code?

I am fine with dropping the arm64 code btw.

Quoted 3,008 lines
Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
---
 arch/arm/configs/exynos_defconfig       |   1 -
 arch/arm/configs/multi_v7_defconfig     |   1 -
 arch/arm/configs/omap2plus_defconfig    |   1 -
 arch/arm/crypto/Kconfig                 |   6 -
 arch/arm/crypto/Makefile                |   2 -
 arch/arm/crypto/chacha20-neon-core.S    | 521 --------------------
 arch/arm/crypto/chacha20-neon-glue.c    | 127 -----
 arch/arm64/configs/defconfig            |   1 -
 arch/arm64/crypto/Kconfig               |   6 -
 arch/arm64/crypto/Makefile              |   3 -
 arch/arm64/crypto/chacha20-neon-core.S  | 450 -----------------
 arch/arm64/crypto/chacha20-neon-glue.c  | 133 -----
 arch/x86/crypto/Makefile                |   3 -
 arch/x86/crypto/chacha20-avx2-x86_64.S  | 448 -----------------
 arch/x86/crypto/chacha20-ssse3-x86_64.S | 630 ------------------------
 arch/x86/crypto/chacha20_glue.c         | 146 ------
 crypto/Kconfig                          |  16 -
 crypto/Makefile                         |   2 +-
 crypto/chacha20_generic.c               | 136 -----
 crypto/chacha20_zinc.c                  | 100 ++++
 crypto/chacha20poly1305.c               |   2 +-
 include/crypto/chacha20.h               |  12 -
 22 files changed, 102 insertions(+), 2645 deletions(-)
 delete mode 100644 arch/arm/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/arm64/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm64/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20_glue.c
 delete mode 100644 crypto/chacha20_generic.c
 create mode 100644 crypto/chacha20_zinc.c
arch/arm/configs/exynos_defconfig +0 -1
index 27ea6dfcf2f2..95929b5e7b10 100644
@@ -350,7 +350,6 @@ CONFIG_CRYPTO_SHA1_ARM_NEON=m
 CONFIG_CRYPTO_SHA256_ARM=m
 CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_FONTS=y
 CONFIG_FONT_7x14=y
arch/arm/configs/multi_v7_defconfig +0 -1
index fc33444e94f0..63be07724db3 100644
@@ -1000,4 +1000,3 @@ CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_AES_ARM_CE=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
 CONFIG_CRYPTO_CRC32_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/omap2plus_defconfig +0 -1
index 6491419b1dad..f585a8ecc336 100644
@@ -547,7 +547,6 @@ CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_CRC_T10DIF=y
 CONFIG_CRC_ITU_T=y
arch/arm/crypto/Kconfig +0 -6
index 925d1364727a..fb80fd89f0e7 100644
@@ -115,12 +115,6 @@ config CRYPTO_CRC32_ARM_CE
        depends on KERNEL_MODE_NEON && CRC32
        select CRYPTO_HASH
 
-config CRYPTO_CHACHA20_NEON
-       tristate "NEON accelerated ChaCha20 symmetric cipher"
-       depends on KERNEL_MODE_NEON
-       select CRYPTO_BLKCIPHER
-       select CRYPTO_CHACHA20
-
 config CRYPTO_SPECK_NEON
        tristate "NEON accelerated Speck cipher algorithms"
        depends on KERNEL_MODE_NEON
arch/arm/crypto/Makefile +0 -2
index 8de542c48ade..bbfa98447063 100644
@@ -9,7 +9,6 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o
 obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o
 obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 
 ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -53,7 +52,6 @@ aes-arm-ce-y  := aes-ce-core.o aes-ce-glue.o
 ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o
 crct10dif-arm-ce-y     := crct10dif-ce-core.o crct10dif-ce-glue.o
 crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
 ifdef REGENERATE_ARM_CRYPTO
arch/arm/crypto/chacha20-neon-core.S deleted +0 -521
deleted file mode 100644
index 451a849ad518..000000000000
@@ -1,521 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions
- *
- * Copyright (C) 2016 Linaro, Ltd. <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Based on:
- * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSE3 functions
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/linkage.h>
-
-       .text
-       .fpu            neon
-       .align          5
-
-ENTRY(chacha20_block_xor_neon)
-       // r0: Input state matrix, s
-       // r1: 1 data block output, o
-       // r2: 1 data block input, i
-
-       //
-       // This function encrypts one ChaCha20 block by loading the state matrix
-       // in four NEON registers. It performs matrix operation on four words in
-       // parallel, but requireds shuffling to rearrange the words after each
-       // round.
-       //
-
-       // x0..3 = s0..3
-       add             ip, r0, #0x20
-       vld1.32         {q0-q1}, [r0]
-       vld1.32         {q2-q3}, [ip]
-
-       vmov            q8, q0
-       vmov            q9, q1
-       vmov            q10, q2
-       vmov            q11, q3
-
-       mov             r3, #10
-
-.Ldoubleround:
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       vadd.i32        q0, q0, q1
-       veor            q3, q3, q0
-       vrev32.16       q3, q3
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       vadd.i32        q2, q2, q3
-       veor            q4, q1, q2
-       vshl.u32        q1, q4, #12
-       vsri.u32        q1, q4, #20
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       vadd.i32        q0, q0, q1
-       veor            q4, q3, q0
-       vshl.u32        q3, q4, #8
-       vsri.u32        q3, q4, #24
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       vadd.i32        q2, q2, q3
-       veor            q4, q1, q2
-       vshl.u32        q1, q4, #7
-       vsri.u32        q1, q4, #25
-
-       // x1 = shuffle32(x1, MASK(0, 3, 2, 1))
-       vext.8          q1, q1, q1, #4
-       // x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       vext.8          q2, q2, q2, #8
-       // x3 = shuffle32(x3, MASK(2, 1, 0, 3))
-       vext.8          q3, q3, q3, #12
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       vadd.i32        q0, q0, q1
-       veor            q3, q3, q0
-       vrev32.16       q3, q3
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       vadd.i32        q2, q2, q3
-       veor            q4, q1, q2
-       vshl.u32        q1, q4, #12
-       vsri.u32        q1, q4, #20
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       vadd.i32        q0, q0, q1
-       veor            q4, q3, q0
-       vshl.u32        q3, q4, #8
-       vsri.u32        q3, q4, #24
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       vadd.i32        q2, q2, q3
-       veor            q4, q1, q2
-       vshl.u32        q1, q4, #7
-       vsri.u32        q1, q4, #25
-
-       // x1 = shuffle32(x1, MASK(2, 1, 0, 3))
-       vext.8          q1, q1, q1, #12
-       // x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       vext.8          q2, q2, q2, #8
-       // x3 = shuffle32(x3, MASK(0, 3, 2, 1))
-       vext.8          q3, q3, q3, #4
-
-       subs            r3, r3, #1
-       bne             .Ldoubleround
-
-       add             ip, r2, #0x20
-       vld1.8          {q4-q5}, [r2]
-       vld1.8          {q6-q7}, [ip]
-
-       // o0 = i0 ^ (x0 + s0)
-       vadd.i32        q0, q0, q8
-       veor            q0, q0, q4
-
-       // o1 = i1 ^ (x1 + s1)
-       vadd.i32        q1, q1, q9
-       veor            q1, q1, q5
-
-       // o2 = i2 ^ (x2 + s2)
-       vadd.i32        q2, q2, q10
-       veor            q2, q2, q6
-
-       // o3 = i3 ^ (x3 + s3)
-       vadd.i32        q3, q3, q11
-       veor            q3, q3, q7
-
-       add             ip, r1, #0x20
-       vst1.8          {q0-q1}, [r1]
-       vst1.8          {q2-q3}, [ip]
-
-       bx              lr
-ENDPROC(chacha20_block_xor_neon)
-
-       .align          5
-ENTRY(chacha20_4block_xor_neon)
-       push            {r4-r6, lr}
-       mov             ip, sp                  // preserve the stack pointer
-       sub             r3, sp, #0x20           // allocate a 32 byte buffer
-       bic             r3, r3, #0x1f           // aligned to 32 bytes
-       mov             sp, r3
-
-       // r0: Input state matrix, s
-       // r1: 4 data blocks output, o
-       // r2: 4 data blocks input, i
-
-       //
-       // This function encrypts four consecutive ChaCha20 blocks by loading
-       // the state matrix in NEON registers four times. The algorithm performs
-       // each operation on the corresponding word of each state matrix, hence
-       // requires no word shuffling. For final XORing step we transpose the
-       // matrix by interleaving 32- and then 64-bit words, which allows us to
-       // do XOR in NEON registers.
-       //
-
-       // x0..15[0-3] = s0..3[0..3]
-       add             r3, r0, #0x20
-       vld1.32         {q0-q1}, [r0]
-       vld1.32         {q2-q3}, [r3]
-
-       adr             r3, CTRINC
-       vdup.32         q15, d7[1]
-       vdup.32         q14, d7[0]
-       vld1.32         {q11}, [r3, :128]
-       vdup.32         q13, d6[1]
-       vdup.32         q12, d6[0]
-       vadd.i32        q12, q12, q11           // x12 += counter values 0-3
-       vdup.32         q11, d5[1]
-       vdup.32         q10, d5[0]
-       vdup.32         q9, d4[1]
-       vdup.32         q8, d4[0]
-       vdup.32         q7, d3[1]
-       vdup.32         q6, d3[0]
-       vdup.32         q5, d2[1]
-       vdup.32         q4, d2[0]
-       vdup.32         q3, d1[1]
-       vdup.32         q2, d1[0]
-       vdup.32         q1, d0[1]
-       vdup.32         q0, d0[0]
-
-       mov             r3, #10
-
-.Ldoubleround4:
-       // x0 += x4, x12 = rotl32(x12 ^ x0, 16)
-       // x1 += x5, x13 = rotl32(x13 ^ x1, 16)
-       // x2 += x6, x14 = rotl32(x14 ^ x2, 16)
-       // x3 += x7, x15 = rotl32(x15 ^ x3, 16)
-       vadd.i32        q0, q0, q4
-       vadd.i32        q1, q1, q5
-       vadd.i32        q2, q2, q6
-       vadd.i32        q3, q3, q7
-
-       veor            q12, q12, q0
-       veor            q13, q13, q1
-       veor            q14, q14, q2
-       veor            q15, q15, q3
-
-       vrev32.16       q12, q12
-       vrev32.16       q13, q13
-       vrev32.16       q14, q14
-       vrev32.16       q15, q15
-
-       // x8 += x12, x4 = rotl32(x4 ^ x8, 12)
-       // x9 += x13, x5 = rotl32(x5 ^ x9, 12)
-       // x10 += x14, x6 = rotl32(x6 ^ x10, 12)
-       // x11 += x15, x7 = rotl32(x7 ^ x11, 12)
-       vadd.i32        q8, q8, q12
-       vadd.i32        q9, q9, q13
-       vadd.i32        q10, q10, q14
-       vadd.i32        q11, q11, q15
-
-       vst1.32         {q8-q9}, [sp, :256]
-
-       veor            q8, q4, q8
-       veor            q9, q5, q9
-       vshl.u32        q4, q8, #12
-       vshl.u32        q5, q9, #12
-       vsri.u32        q4, q8, #20
-       vsri.u32        q5, q9, #20
-
-       veor            q8, q6, q10
-       veor            q9, q7, q11
-       vshl.u32        q6, q8, #12
-       vshl.u32        q7, q9, #12
-       vsri.u32        q6, q8, #20
-       vsri.u32        q7, q9, #20
-
-       // x0 += x4, x12 = rotl32(x12 ^ x0, 8)
-       // x1 += x5, x13 = rotl32(x13 ^ x1, 8)
-       // x2 += x6, x14 = rotl32(x14 ^ x2, 8)
-       // x3 += x7, x15 = rotl32(x15 ^ x3, 8)
-       vadd.i32        q0, q0, q4
-       vadd.i32        q1, q1, q5
-       vadd.i32        q2, q2, q6
-       vadd.i32        q3, q3, q7
-
-       veor            q8, q12, q0
-       veor            q9, q13, q1
-       vshl.u32        q12, q8, #8
-       vshl.u32        q13, q9, #8
-       vsri.u32        q12, q8, #24
-       vsri.u32        q13, q9, #24
-
-       veor            q8, q14, q2
-       veor            q9, q15, q3
-       vshl.u32        q14, q8, #8
-       vshl.u32        q15, q9, #8
-       vsri.u32        q14, q8, #24
-       vsri.u32        q15, q9, #24
-
-       vld1.32         {q8-q9}, [sp, :256]
-
-       // x8 += x12, x4 = rotl32(x4 ^ x8, 7)
-       // x9 += x13, x5 = rotl32(x5 ^ x9, 7)
-       // x10 += x14, x6 = rotl32(x6 ^ x10, 7)
-       // x11 += x15, x7 = rotl32(x7 ^ x11, 7)
-       vadd.i32        q8, q8, q12
-       vadd.i32        q9, q9, q13
-       vadd.i32        q10, q10, q14
-       vadd.i32        q11, q11, q15
-
-       vst1.32         {q8-q9}, [sp, :256]
-
-       veor            q8, q4, q8
-       veor            q9, q5, q9
-       vshl.u32        q4, q8, #7
-       vshl.u32        q5, q9, #7
-       vsri.u32        q4, q8, #25
-       vsri.u32        q5, q9, #25
-
-       veor            q8, q6, q10
-       veor            q9, q7, q11
-       vshl.u32        q6, q8, #7
-       vshl.u32        q7, q9, #7
-       vsri.u32        q6, q8, #25
-       vsri.u32        q7, q9, #25
-
-       vld1.32         {q8-q9}, [sp, :256]
-
-       // x0 += x5, x15 = rotl32(x15 ^ x0, 16)
-       // x1 += x6, x12 = rotl32(x12 ^ x1, 16)
-       // x2 += x7, x13 = rotl32(x13 ^ x2, 16)
-       // x3 += x4, x14 = rotl32(x14 ^ x3, 16)
-       vadd.i32        q0, q0, q5
-       vadd.i32        q1, q1, q6
-       vadd.i32        q2, q2, q7
-       vadd.i32        q3, q3, q4
-
-       veor            q15, q15, q0
-       veor            q12, q12, q1
-       veor            q13, q13, q2
-       veor            q14, q14, q3
-
-       vrev32.16       q15, q15
-       vrev32.16       q12, q12
-       vrev32.16       q13, q13
-       vrev32.16       q14, q14
-
-       // x10 += x15, x5 = rotl32(x5 ^ x10, 12)
-       // x11 += x12, x6 = rotl32(x6 ^ x11, 12)
-       // x8 += x13, x7 = rotl32(x7 ^ x8, 12)
-       // x9 += x14, x4 = rotl32(x4 ^ x9, 12)
-       vadd.i32        q10, q10, q15
-       vadd.i32        q11, q11, q12
-       vadd.i32        q8, q8, q13
-       vadd.i32        q9, q9, q14
-
-       vst1.32         {q8-q9}, [sp, :256]
-
-       veor            q8, q7, q8
-       veor            q9, q4, q9
-       vshl.u32        q7, q8, #12
-       vshl.u32        q4, q9, #12
-       vsri.u32        q7, q8, #20
-       vsri.u32        q4, q9, #20
-
-       veor            q8, q5, q10
-       veor            q9, q6, q11
-       vshl.u32        q5, q8, #12
-       vshl.u32        q6, q9, #12
-       vsri.u32        q5, q8, #20
-       vsri.u32        q6, q9, #20
-
-       // x0 += x5, x15 = rotl32(x15 ^ x0, 8)
-       // x1 += x6, x12 = rotl32(x12 ^ x1, 8)
-       // x2 += x7, x13 = rotl32(x13 ^ x2, 8)
-       // x3 += x4, x14 = rotl32(x14 ^ x3, 8)
-       vadd.i32        q0, q0, q5
-       vadd.i32        q1, q1, q6
-       vadd.i32        q2, q2, q7
-       vadd.i32        q3, q3, q4
-
-       veor            q8, q15, q0
-       veor            q9, q12, q1
-       vshl.u32        q15, q8, #8
-       vshl.u32        q12, q9, #8
-       vsri.u32        q15, q8, #24
-       vsri.u32        q12, q9, #24
-
-       veor            q8, q13, q2
-       veor            q9, q14, q3
-       vshl.u32        q13, q8, #8
-       vshl.u32        q14, q9, #8
-       vsri.u32        q13, q8, #24
-       vsri.u32        q14, q9, #24
-
-       vld1.32         {q8-q9}, [sp, :256]
-
-       // x10 += x15, x5 = rotl32(x5 ^ x10, 7)
-       // x11 += x12, x6 = rotl32(x6 ^ x11, 7)
-       // x8 += x13, x7 = rotl32(x7 ^ x8, 7)
-       // x9 += x14, x4 = rotl32(x4 ^ x9, 7)
-       vadd.i32        q10, q10, q15
-       vadd.i32        q11, q11, q12
-       vadd.i32        q8, q8, q13
-       vadd.i32        q9, q9, q14
-
-       vst1.32         {q8-q9}, [sp, :256]
-
-       veor            q8, q7, q8
-       veor            q9, q4, q9
-       vshl.u32        q7, q8, #7
-       vshl.u32        q4, q9, #7
-       vsri.u32        q7, q8, #25
-       vsri.u32        q4, q9, #25
-
-       veor            q8, q5, q10
-       veor            q9, q6, q11
-       vshl.u32        q5, q8, #7
-       vshl.u32        q6, q9, #7
-       vsri.u32        q5, q8, #25
-       vsri.u32        q6, q9, #25
-
-       subs            r3, r3, #1
-       beq             0f
-
-       vld1.32         {q8-q9}, [sp, :256]
-       b               .Ldoubleround4
-
-       // x0[0-3] += s0[0]
-       // x1[0-3] += s0[1]
-       // x2[0-3] += s0[2]
-       // x3[0-3] += s0[3]
-0:     ldmia           r0!, {r3-r6}
-       vdup.32         q8, r3
-       vdup.32         q9, r4
-       vadd.i32        q0, q0, q8
-       vadd.i32        q1, q1, q9
-       vdup.32         q8, r5
-       vdup.32         q9, r6
-       vadd.i32        q2, q2, q8
-       vadd.i32        q3, q3, q9
-
-       // x4[0-3] += s1[0]
-       // x5[0-3] += s1[1]
-       // x6[0-3] += s1[2]
-       // x7[0-3] += s1[3]
-       ldmia           r0!, {r3-r6}
-       vdup.32         q8, r3
-       vdup.32         q9, r4
-       vadd.i32        q4, q4, q8
-       vadd.i32        q5, q5, q9
-       vdup.32         q8, r5
-       vdup.32         q9, r6
-       vadd.i32        q6, q6, q8
-       vadd.i32        q7, q7, q9
-
-       // interleave 32-bit words in state n, n+1
-       vzip.32         q0, q1
-       vzip.32         q2, q3
-       vzip.32         q4, q5
-       vzip.32         q6, q7
-
-       // interleave 64-bit words in state n, n+2
-       vswp            d1, d4
-       vswp            d3, d6
-       vswp            d9, d12
-       vswp            d11, d14
-
-       // xor with corresponding input, write to output
-       vld1.8          {q8-q9}, [r2]!
-       veor            q8, q8, q0
-       veor            q9, q9, q4
-       vst1.8          {q8-q9}, [r1]!
-
-       vld1.32         {q8-q9}, [sp, :256]
-
-       // x8[0-3] += s2[0]
-       // x9[0-3] += s2[1]
-       // x10[0-3] += s2[2]
-       // x11[0-3] += s2[3]
-       ldmia           r0!, {r3-r6}
-       vdup.32         q0, r3
-       vdup.32         q4, r4
-       vadd.i32        q8, q8, q0
-       vadd.i32        q9, q9, q4
-       vdup.32         q0, r5
-       vdup.32         q4, r6
-       vadd.i32        q10, q10, q0
-       vadd.i32        q11, q11, q4
-
-       // x12[0-3] += s3[0]
-       // x13[0-3] += s3[1]
-       // x14[0-3] += s3[2]
-       // x15[0-3] += s3[3]
-       ldmia           r0!, {r3-r6}
-       vdup.32         q0, r3
-       vdup.32         q4, r4
-       adr             r3, CTRINC
-       vadd.i32        q12, q12, q0
-       vld1.32         {q0}, [r3, :128]
-       vadd.i32        q13, q13, q4
-       vadd.i32        q12, q12, q0            // x12 += counter values 0-3
-
-       vdup.32         q0, r5
-       vdup.32         q4, r6
-       vadd.i32        q14, q14, q0
-       vadd.i32        q15, q15, q4
-
-       // interleave 32-bit words in state n, n+1
-       vzip.32         q8, q9
-       vzip.32         q10, q11
-       vzip.32         q12, q13
-       vzip.32         q14, q15
-
-       // interleave 64-bit words in state n, n+2
-       vswp            d17, d20
-       vswp            d19, d22
-       vswp            d25, d28
-       vswp            d27, d30
-
-       vmov            q4, q1
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q8
-       veor            q1, q1, q12
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q2
-       veor            q1, q1, q6
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q10
-       veor            q1, q1, q14
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q4
-       veor            q1, q1, q5
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q9
-       veor            q1, q1, q13
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]!
-       veor            q0, q0, q3
-       veor            q1, q1, q7
-       vst1.8          {q0-q1}, [r1]!
-
-       vld1.8          {q0-q1}, [r2]
-       veor            q0, q0, q11
-       veor            q1, q1, q15
-       vst1.8          {q0-q1}, [r1]
-
-       mov             sp, ip
-       pop             {r4-r6, pc}
-ENDPROC(chacha20_4block_xor_neon)
-
-       .align          4
-CTRINC:        .word           0, 1, 2, 3
arch/arm/crypto/chacha20-neon-glue.c deleted +0 -127
deleted file mode 100644
index 59a7be08e80c..000000000000
@@ -1,127 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, ARM NEON functions
- *
- * Copyright (C) 2016 Linaro, Ltd. <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Based on:
- * ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <crypto/algapi.h>
-#include <crypto/chacha20.h>
-#include <crypto/internal/skcipher.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-#include <asm/hwcap.h>
-#include <asm/neon.h>
-#include <asm/simd.h>
-
-asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src);
-asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src);
-
-static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,
-                           unsigned int bytes)
-{
-       u8 buf[CHACHA20_BLOCK_SIZE];
-
-       while (bytes >= CHACHA20_BLOCK_SIZE * 4) {
-               chacha20_4block_xor_neon(state, dst, src);
-               bytes -= CHACHA20_BLOCK_SIZE * 4;
-               src += CHACHA20_BLOCK_SIZE * 4;
-               dst += CHACHA20_BLOCK_SIZE * 4;
-               state[12] += 4;
-       }
-       while (bytes >= CHACHA20_BLOCK_SIZE) {
-               chacha20_block_xor_neon(state, dst, src);
-               bytes -= CHACHA20_BLOCK_SIZE;
-               src += CHACHA20_BLOCK_SIZE;
-               dst += CHACHA20_BLOCK_SIZE;
-               state[12]++;
-       }
-       if (bytes) {
-               memcpy(buf, src, bytes);
-               chacha20_block_xor_neon(state, buf, buf);
-               memcpy(dst, buf, bytes);
-       }
-}
-
-static int chacha20_neon(struct skcipher_request *req)
-{
-       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
-       struct chacha20_ctx *ctx = crypto_skcipher_ctx(tfm);
-       struct skcipher_walk walk;
-       u32 state[16];
-       int err;
-
-       if (req->cryptlen <= CHACHA20_BLOCK_SIZE || !may_use_simd())
-               return crypto_chacha20_crypt(req);
-
-       err = skcipher_walk_virt(&walk, req, true);
-
-       crypto_chacha20_init(state, ctx, walk.iv);
-
-       kernel_neon_begin();
-       while (walk.nbytes > 0) {
-               unsigned int nbytes = walk.nbytes;
-
-               if (nbytes < walk.total)
-                       nbytes = round_down(nbytes, walk.stride);
-
-               chacha20_doneon(state, walk.dst.virt.addr, walk.src.virt.addr,
-                               nbytes);
-               err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-       }
-       kernel_neon_end();
-
-       return err;
-}
-
-static struct skcipher_alg alg = {
-       .base.cra_name          = "chacha20",
-       .base.cra_driver_name   = "chacha20-neon",
-       .base.cra_priority      = 300,
-       .base.cra_blocksize     = 1,
-       .base.cra_ctxsize       = sizeof(struct chacha20_ctx),
-       .base.cra_module        = THIS_MODULE,
-
-       .min_keysize            = CHACHA20_KEY_SIZE,
-       .max_keysize            = CHACHA20_KEY_SIZE,
-       .ivsize                 = CHACHA20_IV_SIZE,
-       .chunksize              = CHACHA20_BLOCK_SIZE,
-       .walksize               = 4 * CHACHA20_BLOCK_SIZE,
-       .setkey                 = crypto_chacha20_setkey,
-       .encrypt                = chacha20_neon,
-       .decrypt                = chacha20_neon,
-};
-
-static int __init chacha20_simd_mod_init(void)
-{
-       if (!(elf_hwcap & HWCAP_NEON))
-               return -ENODEV;
-
-       return crypto_register_skcipher(&alg);
-}
-
-static void __exit chacha20_simd_mod_fini(void)
-{
-       crypto_unregister_skcipher(&alg);
-}
-
-module_init(chacha20_simd_mod_init);
-module_exit(chacha20_simd_mod_fini);
-
-MODULE_AUTHOR("Ard Biesheuvel <[email protected]>");
-MODULE_LICENSE("GPL v2");
-MODULE_ALIAS_CRYPTO("chacha20");
arch/arm64/configs/defconfig +0 -1
index db8d364f8476..6cc3c8a0ad88 100644
@@ -709,5 +709,4 @@ CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m
 CONFIG_CRYPTO_CRC32_ARM64_CE=m
 CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
 CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRYPTO_AES_ARM64_BS=m
arch/arm64/crypto/Kconfig +0 -6
index e3fdb0fd6f70..9db6d775a880 100644
@@ -105,12 +105,6 @@ config CRYPTO_AES_ARM64_NEON_BLK
        select CRYPTO_AES
        select CRYPTO_SIMD
 
-config CRYPTO_CHACHA20_NEON
-       tristate "NEON accelerated ChaCha20 symmetric cipher"
-       depends on KERNEL_MODE_NEON
-       select CRYPTO_BLKCIPHER
-       select CRYPTO_CHACHA20
-
 config CRYPTO_AES_ARM64_BS
        tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm"
        depends on KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +0 -3
index bcafd016618e..507c4bfb86e3 100644
@@ -53,9 +53,6 @@ sha256-arm64-y := sha256-glue.o sha256-core.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o
 sha512-arm64-y := sha512-glue.o sha512-core.o
 
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
-
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
arch/arm64/crypto/chacha20-neon-core.S deleted +0 -450
deleted file mode 100644
index 13c85e272c2a..000000000000
@@ -1,450 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, arm64 NEON functions
- *
- * Copyright (C) 2016 Linaro, Ltd. <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Based on:
- * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSSE3 functions
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/linkage.h>
-
-       .text
-       .align          6
-
-ENTRY(chacha20_block_xor_neon)
-       // x0: Input state matrix, s
-       // x1: 1 data block output, o
-       // x2: 1 data block input, i
-
-       //
-       // This function encrypts one ChaCha20 block by loading the state matrix
-       // in four NEON registers. It performs matrix operation on four words in
-       // parallel, but requires shuffling to rearrange the words after each
-       // round.
-       //
-
-       // x0..3 = s0..3
-       adr             x3, ROT8
-       ld1             {v0.4s-v3.4s}, [x0]
-       ld1             {v8.4s-v11.4s}, [x0]
-       ld1             {v12.4s}, [x3]
-
-       mov             x3, #10
-
-.Ldoubleround:
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       add             v0.4s, v0.4s, v1.4s
-       eor             v3.16b, v3.16b, v0.16b
-       rev32           v3.8h, v3.8h
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       add             v2.4s, v2.4s, v3.4s
-       eor             v4.16b, v1.16b, v2.16b
-       shl             v1.4s, v4.4s, #12
-       sri             v1.4s, v4.4s, #20
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       add             v0.4s, v0.4s, v1.4s
-       eor             v3.16b, v3.16b, v0.16b
-       tbl             v3.16b, {v3.16b}, v12.16b
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       add             v2.4s, v2.4s, v3.4s
-       eor             v4.16b, v1.16b, v2.16b
-       shl             v1.4s, v4.4s, #7
-       sri             v1.4s, v4.4s, #25
-
-       // x1 = shuffle32(x1, MASK(0, 3, 2, 1))
-       ext             v1.16b, v1.16b, v1.16b, #4
-       // x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       ext             v2.16b, v2.16b, v2.16b, #8
-       // x3 = shuffle32(x3, MASK(2, 1, 0, 3))
-       ext             v3.16b, v3.16b, v3.16b, #12
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       add             v0.4s, v0.4s, v1.4s
-       eor             v3.16b, v3.16b, v0.16b
-       rev32           v3.8h, v3.8h
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       add             v2.4s, v2.4s, v3.4s
-       eor             v4.16b, v1.16b, v2.16b
-       shl             v1.4s, v4.4s, #12
-       sri             v1.4s, v4.4s, #20
-
-       // x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       add             v0.4s, v0.4s, v1.4s
-       eor             v3.16b, v3.16b, v0.16b
-       tbl             v3.16b, {v3.16b}, v12.16b
-
-       // x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       add             v2.4s, v2.4s, v3.4s
-       eor             v4.16b, v1.16b, v2.16b
-       shl             v1.4s, v4.4s, #7
-       sri             v1.4s, v4.4s, #25
-
-       // x1 = shuffle32(x1, MASK(2, 1, 0, 3))
-       ext             v1.16b, v1.16b, v1.16b, #12
-       // x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       ext             v2.16b, v2.16b, v2.16b, #8
-       // x3 = shuffle32(x3, MASK(0, 3, 2, 1))
-       ext             v3.16b, v3.16b, v3.16b, #4
-
-       subs            x3, x3, #1
-       b.ne            .Ldoubleround
-
-       ld1             {v4.16b-v7.16b}, [x2]
-
-       // o0 = i0 ^ (x0 + s0)
-       add             v0.4s, v0.4s, v8.4s
-       eor             v0.16b, v0.16b, v4.16b
-
-       // o1 = i1 ^ (x1 + s1)
-       add             v1.4s, v1.4s, v9.4s
-       eor             v1.16b, v1.16b, v5.16b
-
-       // o2 = i2 ^ (x2 + s2)
-       add             v2.4s, v2.4s, v10.4s
-       eor             v2.16b, v2.16b, v6.16b
-
-       // o3 = i3 ^ (x3 + s3)
-       add             v3.4s, v3.4s, v11.4s
-       eor             v3.16b, v3.16b, v7.16b
-
-       st1             {v0.16b-v3.16b}, [x1]
-
-       ret
-ENDPROC(chacha20_block_xor_neon)
-
-       .align          6
-ENTRY(chacha20_4block_xor_neon)
-       // x0: Input state matrix, s
-       // x1: 4 data blocks output, o
-       // x2: 4 data blocks input, i
-
-       //
-       // This function encrypts four consecutive ChaCha20 blocks by loading
-       // the state matrix in NEON registers four times. The algorithm performs
-       // each operation on the corresponding word of each state matrix, hence
-       // requires no word shuffling. For final XORing step we transpose the
-       // matrix by interleaving 32- and then 64-bit words, which allows us to
-       // do XOR in NEON registers.
-       //
-       adr             x3, CTRINC              // ... and ROT8
-       ld1             {v30.4s-v31.4s}, [x3]
-
-       // x0..15[0-3] = s0..3[0..3]
-       mov             x4, x0
-       ld4r            { v0.4s- v3.4s}, [x4], #16
-       ld4r            { v4.4s- v7.4s}, [x4], #16
-       ld4r            { v8.4s-v11.4s}, [x4], #16
-       ld4r            {v12.4s-v15.4s}, [x4]
-
-       // x12 += counter values 0-3
-       add             v12.4s, v12.4s, v30.4s
-
-       mov             x3, #10
-
-.Ldoubleround4:
-       // x0 += x4, x12 = rotl32(x12 ^ x0, 16)
-       // x1 += x5, x13 = rotl32(x13 ^ x1, 16)
-       // x2 += x6, x14 = rotl32(x14 ^ x2, 16)
-       // x3 += x7, x15 = rotl32(x15 ^ x3, 16)
-       add             v0.4s, v0.4s, v4.4s
-       add             v1.4s, v1.4s, v5.4s
-       add             v2.4s, v2.4s, v6.4s
-       add             v3.4s, v3.4s, v7.4s
-
-       eor             v12.16b, v12.16b, v0.16b
-       eor             v13.16b, v13.16b, v1.16b
-       eor             v14.16b, v14.16b, v2.16b
-       eor             v15.16b, v15.16b, v3.16b
-
-       rev32           v12.8h, v12.8h
-       rev32           v13.8h, v13.8h
-       rev32           v14.8h, v14.8h
-       rev32           v15.8h, v15.8h
-
-       // x8 += x12, x4 = rotl32(x4 ^ x8, 12)
-       // x9 += x13, x5 = rotl32(x5 ^ x9, 12)
-       // x10 += x14, x6 = rotl32(x6 ^ x10, 12)
-       // x11 += x15, x7 = rotl32(x7 ^ x11, 12)
-       add             v8.4s, v8.4s, v12.4s
-       add             v9.4s, v9.4s, v13.4s
-       add             v10.4s, v10.4s, v14.4s
-       add             v11.4s, v11.4s, v15.4s
-
-       eor             v16.16b, v4.16b, v8.16b
-       eor             v17.16b, v5.16b, v9.16b
-       eor             v18.16b, v6.16b, v10.16b
-       eor             v19.16b, v7.16b, v11.16b
-
-       shl             v4.4s, v16.4s, #12
-       shl             v5.4s, v17.4s, #12
-       shl             v6.4s, v18.4s, #12
-       shl             v7.4s, v19.4s, #12
-
-       sri             v4.4s, v16.4s, #20
-       sri             v5.4s, v17.4s, #20
-       sri             v6.4s, v18.4s, #20
-       sri             v7.4s, v19.4s, #20
-
-       // x0 += x4, x12 = rotl32(x12 ^ x0, 8)
-       // x1 += x5, x13 = rotl32(x13 ^ x1, 8)
-       // x2 += x6, x14 = rotl32(x14 ^ x2, 8)
-       // x3 += x7, x15 = rotl32(x15 ^ x3, 8)
-       add             v0.4s, v0.4s, v4.4s
-       add             v1.4s, v1.4s, v5.4s
-       add             v2.4s, v2.4s, v6.4s
-       add             v3.4s, v3.4s, v7.4s
-
-       eor             v12.16b, v12.16b, v0.16b
-       eor             v13.16b, v13.16b, v1.16b
-       eor             v14.16b, v14.16b, v2.16b
-       eor             v15.16b, v15.16b, v3.16b
-
-       tbl             v12.16b, {v12.16b}, v31.16b
-       tbl             v13.16b, {v13.16b}, v31.16b
-       tbl             v14.16b, {v14.16b}, v31.16b
-       tbl             v15.16b, {v15.16b}, v31.16b
-
-       // x8 += x12, x4 = rotl32(x4 ^ x8, 7)
-       // x9 += x13, x5 = rotl32(x5 ^ x9, 7)
-       // x10 += x14, x6 = rotl32(x6 ^ x10, 7)
-       // x11 += x15, x7 = rotl32(x7 ^ x11, 7)
-       add             v8.4s, v8.4s, v12.4s
-       add             v9.4s, v9.4s, v13.4s
-       add             v10.4s, v10.4s, v14.4s
-       add             v11.4s, v11.4s, v15.4s
-
-       eor             v16.16b, v4.16b, v8.16b
-       eor             v17.16b, v5.16b, v9.16b
-       eor             v18.16b, v6.16b, v10.16b
-       eor             v19.16b, v7.16b, v11.16b
-
-       shl             v4.4s, v16.4s, #7
-       shl             v5.4s, v17.4s, #7
-       shl             v6.4s, v18.4s, #7
-       shl             v7.4s, v19.4s, #7
-
-       sri             v4.4s, v16.4s, #25
-       sri             v5.4s, v17.4s, #25
-       sri             v6.4s, v18.4s, #25
-       sri             v7.4s, v19.4s, #25
-
-       // x0 += x5, x15 = rotl32(x15 ^ x0, 16)
-       // x1 += x6, x12 = rotl32(x12 ^ x1, 16)
-       // x2 += x7, x13 = rotl32(x13 ^ x2, 16)
-       // x3 += x4, x14 = rotl32(x14 ^ x3, 16)
-       add             v0.4s, v0.4s, v5.4s
-       add             v1.4s, v1.4s, v6.4s
-       add             v2.4s, v2.4s, v7.4s
-       add             v3.4s, v3.4s, v4.4s
-
-       eor             v15.16b, v15.16b, v0.16b
-       eor             v12.16b, v12.16b, v1.16b
-       eor             v13.16b, v13.16b, v2.16b
-       eor             v14.16b, v14.16b, v3.16b
-
-       rev32           v15.8h, v15.8h
-       rev32           v12.8h, v12.8h
-       rev32           v13.8h, v13.8h
-       rev32           v14.8h, v14.8h
-
-       // x10 += x15, x5 = rotl32(x5 ^ x10, 12)
-       // x11 += x12, x6 = rotl32(x6 ^ x11, 12)
-       // x8 += x13, x7 = rotl32(x7 ^ x8, 12)
-       // x9 += x14, x4 = rotl32(x4 ^ x9, 12)
-       add             v10.4s, v10.4s, v15.4s
-       add             v11.4s, v11.4s, v12.4s
-       add             v8.4s, v8.4s, v13.4s
-       add             v9.4s, v9.4s, v14.4s
-
-       eor             v16.16b, v5.16b, v10.16b
-       eor             v17.16b, v6.16b, v11.16b
-       eor             v18.16b, v7.16b, v8.16b
-       eor             v19.16b, v4.16b, v9.16b
-
-       shl             v5.4s, v16.4s, #12
-       shl             v6.4s, v17.4s, #12
-       shl             v7.4s, v18.4s, #12
-       shl             v4.4s, v19.4s, #12
-
-       sri             v5.4s, v16.4s, #20
-       sri             v6.4s, v17.4s, #20
-       sri             v7.4s, v18.4s, #20
-       sri             v4.4s, v19.4s, #20
-
-       // x0 += x5, x15 = rotl32(x15 ^ x0, 8)
-       // x1 += x6, x12 = rotl32(x12 ^ x1, 8)
-       // x2 += x7, x13 = rotl32(x13 ^ x2, 8)
-       // x3 += x4, x14 = rotl32(x14 ^ x3, 8)
-       add             v0.4s, v0.4s, v5.4s
-       add             v1.4s, v1.4s, v6.4s
-       add             v2.4s, v2.4s, v7.4s
-       add             v3.4s, v3.4s, v4.4s
-
-       eor             v15.16b, v15.16b, v0.16b
-       eor             v12.16b, v12.16b, v1.16b
-       eor             v13.16b, v13.16b, v2.16b
-       eor             v14.16b, v14.16b, v3.16b
-
-       tbl             v15.16b, {v15.16b}, v31.16b
-       tbl             v12.16b, {v12.16b}, v31.16b
-       tbl             v13.16b, {v13.16b}, v31.16b
-       tbl             v14.16b, {v14.16b}, v31.16b
-
-       // x10 += x15, x5 = rotl32(x5 ^ x10, 7)
-       // x11 += x12, x6 = rotl32(x6 ^ x11, 7)
-       // x8 += x13, x7 = rotl32(x7 ^ x8, 7)
-       // x9 += x14, x4 = rotl32(x4 ^ x9, 7)
-       add             v10.4s, v10.4s, v15.4s
-       add             v11.4s, v11.4s, v12.4s
-       add             v8.4s, v8.4s, v13.4s
-       add             v9.4s, v9.4s, v14.4s
-
-       eor             v16.16b, v5.16b, v10.16b
-       eor             v17.16b, v6.16b, v11.16b
-       eor             v18.16b, v7.16b, v8.16b
-       eor             v19.16b, v4.16b, v9.16b
-
-       shl             v5.4s, v16.4s, #7
-       shl             v6.4s, v17.4s, #7
-       shl             v7.4s, v18.4s, #7
-       shl             v4.4s, v19.4s, #7
-
-       sri             v5.4s, v16.4s, #25
-       sri             v6.4s, v17.4s, #25
-       sri             v7.4s, v18.4s, #25
-       sri             v4.4s, v19.4s, #25
-
-       subs            x3, x3, #1
-       b.ne            .Ldoubleround4
-
-       ld4r            {v16.4s-v19.4s}, [x0], #16
-       ld4r            {v20.4s-v23.4s}, [x0], #16
-
-       // x12 += counter values 0-3
-       add             v12.4s, v12.4s, v30.4s
-
-       // x0[0-3] += s0[0]
-       // x1[0-3] += s0[1]
-       // x2[0-3] += s0[2]
-       // x3[0-3] += s0[3]
-       add             v0.4s, v0.4s, v16.4s
-       add             v1.4s, v1.4s, v17.4s
-       add             v2.4s, v2.4s, v18.4s
-       add             v3.4s, v3.4s, v19.4s
-
-       ld4r            {v24.4s-v27.4s}, [x0], #16
-       ld4r            {v28.4s-v31.4s}, [x0]
-
-       // x4[0-3] += s1[0]
-       // x5[0-3] += s1[1]
-       // x6[0-3] += s1[2]
-       // x7[0-3] += s1[3]
-       add             v4.4s, v4.4s, v20.4s
-       add             v5.4s, v5.4s, v21.4s
-       add             v6.4s, v6.4s, v22.4s
-       add             v7.4s, v7.4s, v23.4s
-
-       // x8[0-3] += s2[0]
-       // x9[0-3] += s2[1]
-       // x10[0-3] += s2[2]
-       // x11[0-3] += s2[3]
-       add             v8.4s, v8.4s, v24.4s
-       add             v9.4s, v9.4s, v25.4s
-       add             v10.4s, v10.4s, v26.4s
-       add             v11.4s, v11.4s, v27.4s
-
-       // x12[0-3] += s3[0]
-       // x13[0-3] += s3[1]
-       // x14[0-3] += s3[2]
-       // x15[0-3] += s3[3]
-       add             v12.4s, v12.4s, v28.4s
-       add             v13.4s, v13.4s, v29.4s
-       add             v14.4s, v14.4s, v30.4s
-       add             v15.4s, v15.4s, v31.4s
-
-       // interleave 32-bit words in state n, n+1
-       zip1            v16.4s, v0.4s, v1.4s
-       zip2            v17.4s, v0.4s, v1.4s
-       zip1            v18.4s, v2.4s, v3.4s
-       zip2            v19.4s, v2.4s, v3.4s
-       zip1            v20.4s, v4.4s, v5.4s
-       zip2            v21.4s, v4.4s, v5.4s
-       zip1            v22.4s, v6.4s, v7.4s
-       zip2            v23.4s, v6.4s, v7.4s
-       zip1            v24.4s, v8.4s, v9.4s
-       zip2            v25.4s, v8.4s, v9.4s
-       zip1            v26.4s, v10.4s, v11.4s
-       zip2            v27.4s, v10.4s, v11.4s
-       zip1            v28.4s, v12.4s, v13.4s
-       zip2            v29.4s, v12.4s, v13.4s
-       zip1            v30.4s, v14.4s, v15.4s
-       zip2            v31.4s, v14.4s, v15.4s
-
-       // interleave 64-bit words in state n, n+2
-       zip1            v0.2d, v16.2d, v18.2d
-       zip2            v4.2d, v16.2d, v18.2d
-       zip1            v8.2d, v17.2d, v19.2d
-       zip2            v12.2d, v17.2d, v19.2d
-       ld1             {v16.16b-v19.16b}, [x2], #64
-
-       zip1            v1.2d, v20.2d, v22.2d
-       zip2            v5.2d, v20.2d, v22.2d
-       zip1            v9.2d, v21.2d, v23.2d
-       zip2            v13.2d, v21.2d, v23.2d
-       ld1             {v20.16b-v23.16b}, [x2], #64
-
-       zip1            v2.2d, v24.2d, v26.2d
-       zip2            v6.2d, v24.2d, v26.2d
-       zip1            v10.2d, v25.2d, v27.2d
-       zip2            v14.2d, v25.2d, v27.2d
-       ld1             {v24.16b-v27.16b}, [x2], #64
-
-       zip1            v3.2d, v28.2d, v30.2d
-       zip2            v7.2d, v28.2d, v30.2d
-       zip1            v11.2d, v29.2d, v31.2d
-       zip2            v15.2d, v29.2d, v31.2d
-       ld1             {v28.16b-v31.16b}, [x2]
-
-       // xor with corresponding input, write to output
-       eor             v16.16b, v16.16b, v0.16b
-       eor             v17.16b, v17.16b, v1.16b
-       eor             v18.16b, v18.16b, v2.16b
-       eor             v19.16b, v19.16b, v3.16b
-       eor             v20.16b, v20.16b, v4.16b
-       eor             v21.16b, v21.16b, v5.16b
-       st1             {v16.16b-v19.16b}, [x1], #64
-       eor             v22.16b, v22.16b, v6.16b
-       eor             v23.16b, v23.16b, v7.16b
-       eor             v24.16b, v24.16b, v8.16b
-       eor             v25.16b, v25.16b, v9.16b
-       st1             {v20.16b-v23.16b}, [x1], #64
-       eor             v26.16b, v26.16b, v10.16b
-       eor             v27.16b, v27.16b, v11.16b
-       eor             v28.16b, v28.16b, v12.16b
-       st1             {v24.16b-v27.16b}, [x1], #64
-       eor             v29.16b, v29.16b, v13.16b
-       eor             v30.16b, v30.16b, v14.16b
-       eor             v31.16b, v31.16b, v15.16b
-       st1             {v28.16b-v31.16b}, [x1]
-
-       ret
-ENDPROC(chacha20_4block_xor_neon)
-
-CTRINC:        .word           0, 1, 2, 3
-ROT8:  .word           0x02010003, 0x06050407, 0x0a09080b, 0x0e0d0c0f
arch/arm64/crypto/chacha20-neon-glue.c deleted +0 -133
deleted file mode 100644
index 727579c93ded..000000000000
@@ -1,133 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, arm64 NEON functions
- *
- * Copyright (C) 2016 - 2017 Linaro, Ltd. <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Based on:
- * ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <crypto/algapi.h>
-#include <crypto/chacha20.h>
-#include <crypto/internal/skcipher.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-#include <asm/hwcap.h>
-#include <asm/neon.h>
-#include <asm/simd.h>
-
-asmlinkage void chacha20_block_xor_neon(u32 *state, u8 *dst, const u8 *src);
-asmlinkage void chacha20_4block_xor_neon(u32 *state, u8 *dst, const u8 *src);
-
-static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src,
-                           unsigned int bytes)
-{
-       u8 buf[CHACHA20_BLOCK_SIZE];
-
-       while (bytes >= CHACHA20_BLOCK_SIZE * 4) {
-               kernel_neon_begin();
-               chacha20_4block_xor_neon(state, dst, src);
-               kernel_neon_end();
-               bytes -= CHACHA20_BLOCK_SIZE * 4;
-               src += CHACHA20_BLOCK_SIZE * 4;
-               dst += CHACHA20_BLOCK_SIZE * 4;
-               state[12] += 4;
-       }
-
-       if (!bytes)
-               return;
-
-       kernel_neon_begin();
-       while (bytes >= CHACHA20_BLOCK_SIZE) {
-               chacha20_block_xor_neon(state, dst, src);
-               bytes -= CHACHA20_BLOCK_SIZE;
-               src += CHACHA20_BLOCK_SIZE;
-               dst += CHACHA20_BLOCK_SIZE;
-               state[12]++;
-       }
-       if (bytes) {
-               memcpy(buf, src, bytes);
-               chacha20_block_xor_neon(state, buf, buf);
-               memcpy(dst, buf, bytes);
-       }
-       kernel_neon_end();
-}
-
-static int chacha20_neon(struct skcipher_request *req)
-{
-       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
-       struct chacha20_ctx *ctx = crypto_skcipher_ctx(tfm);
-       struct skcipher_walk walk;
-       u32 state[16];
-       int err;
-
-       if (!may_use_simd() || req->cryptlen <= CHACHA20_BLOCK_SIZE)
-               return crypto_chacha20_crypt(req);
-
-       err = skcipher_walk_virt(&walk, req, false);
-
-       crypto_chacha20_init(state, ctx, walk.iv);
-
-       while (walk.nbytes > 0) {
-               unsigned int nbytes = walk.nbytes;
-
-               if (nbytes < walk.total)
-                       nbytes = round_down(nbytes, walk.stride);
-
-               chacha20_doneon(state, walk.dst.virt.addr, walk.src.virt.addr,
-                               nbytes);
-               err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-       }
-
-       return err;
-}
-
-static struct skcipher_alg alg = {
-       .base.cra_name          = "chacha20",
-       .base.cra_driver_name   = "chacha20-neon",
-       .base.cra_priority      = 300,
-       .base.cra_blocksize     = 1,
-       .base.cra_ctxsize       = sizeof(struct chacha20_ctx),
-       .base.cra_module        = THIS_MODULE,
-
-       .min_keysize            = CHACHA20_KEY_SIZE,
-       .max_keysize            = CHACHA20_KEY_SIZE,
-       .ivsize                 = CHACHA20_IV_SIZE,
-       .chunksize              = CHACHA20_BLOCK_SIZE,
-       .walksize               = 4 * CHACHA20_BLOCK_SIZE,
-       .setkey                 = crypto_chacha20_setkey,
-       .encrypt                = chacha20_neon,
-       .decrypt                = chacha20_neon,
-};
-
-static int __init chacha20_simd_mod_init(void)
-{
-       if (!(elf_hwcap & HWCAP_ASIMD))
-               return -ENODEV;
-
-       return crypto_register_skcipher(&alg);
-}
-
-static void __exit chacha20_simd_mod_fini(void)
-{
-       crypto_unregister_skcipher(&alg);
-}
-
-module_init(chacha20_simd_mod_init);
-module_exit(chacha20_simd_mod_fini);
-
-MODULE_AUTHOR("Ard Biesheuvel <[email protected]>");
-MODULE_LICENSE("GPL v2");
-MODULE_ALIAS_CRYPTO("chacha20");
arch/x86/crypto/Makefile +0 -3
index cf830219846b..419212c31246 100644
@@ -23,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o
 obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o
-obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o
 obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
@@ -76,7 +75,6 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o
 blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o
 twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o
 twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o
-chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o
 serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o
 
 aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -99,7 +97,6 @@ endif
 
 ifeq ($(avx2_supported),yes)
        camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o
-       chacha20-x86_64-y += chacha20-avx2-x86_64.o
        serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o
 
        morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
arch/x86/crypto/chacha20-avx2-x86_64.S deleted +0 -448
deleted file mode 100644
index f3cd26f48332..000000000000
@@ -1,448 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, x64 AVX2 functions
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/linkage.h>
-
-.section       .rodata.cst32.ROT8, "aM", @progbits, 32
-.align 32
-ROT8:  .octa 0x0e0d0c0f0a09080b0605040702010003
-       .octa 0x0e0d0c0f0a09080b0605040702010003
-
-.section       .rodata.cst32.ROT16, "aM", @progbits, 32
-.align 32
-ROT16: .octa 0x0d0c0f0e09080b0a0504070601000302
-       .octa 0x0d0c0f0e09080b0a0504070601000302
-
-.section       .rodata.cst32.CTRINC, "aM", @progbits, 32
-.align 32
-CTRINC:        .octa 0x00000003000000020000000100000000
-       .octa 0x00000007000000060000000500000004
-
-.text
-
-ENTRY(chacha20_8block_xor_avx2)
-       # %rdi: Input state matrix, s
-       # %rsi: 8 data blocks output, o
-       # %rdx: 8 data blocks input, i
-
-       # This function encrypts eight consecutive ChaCha20 blocks by loading
-       # the state matrix in AVX registers eight times. As we need some
-       # scratch registers, we save the first four registers on the stack. The
-       # algorithm performs each operation on the corresponding word of each
-       # state matrix, hence requires no word shuffling. For final XORing step
-       # we transpose the matrix by interleaving 32-, 64- and then 128-bit
-       # words, which allows us to do XOR in AVX registers. 8/16-bit word
-       # rotation is done with the slightly better performing byte shuffling,
-       # 7/12-bit word rotation uses traditional shift+OR.
-
-       vzeroupper
-       # 4 * 32 byte stack, 32-byte aligned
-       lea             8(%rsp),%r10
-       and             $~31, %rsp
-       sub             $0x80, %rsp
-
-       # x0..15[0-7] = s[0..15]
-       vpbroadcastd    0x00(%rdi),%ymm0
-       vpbroadcastd    0x04(%rdi),%ymm1
-       vpbroadcastd    0x08(%rdi),%ymm2
-       vpbroadcastd    0x0c(%rdi),%ymm3
-       vpbroadcastd    0x10(%rdi),%ymm4
-       vpbroadcastd    0x14(%rdi),%ymm5
-       vpbroadcastd    0x18(%rdi),%ymm6
-       vpbroadcastd    0x1c(%rdi),%ymm7
-       vpbroadcastd    0x20(%rdi),%ymm8
-       vpbroadcastd    0x24(%rdi),%ymm9
-       vpbroadcastd    0x28(%rdi),%ymm10
-       vpbroadcastd    0x2c(%rdi),%ymm11
-       vpbroadcastd    0x30(%rdi),%ymm12
-       vpbroadcastd    0x34(%rdi),%ymm13
-       vpbroadcastd    0x38(%rdi),%ymm14
-       vpbroadcastd    0x3c(%rdi),%ymm15
-       # x0..3 on stack
-       vmovdqa         %ymm0,0x00(%rsp)
-       vmovdqa         %ymm1,0x20(%rsp)
-       vmovdqa         %ymm2,0x40(%rsp)
-       vmovdqa         %ymm3,0x60(%rsp)
-
-       vmovdqa         CTRINC(%rip),%ymm1
-       vmovdqa         ROT8(%rip),%ymm2
-       vmovdqa         ROT16(%rip),%ymm3
-
-       # x12 += counter values 0-3
-       vpaddd          %ymm1,%ymm12,%ymm12
-
-       mov             $10,%ecx
-
-.Ldoubleround8:
-       # x0 += x4, x12 = rotl32(x12 ^ x0, 16)
-       vpaddd          0x00(%rsp),%ymm4,%ymm0
-       vmovdqa         %ymm0,0x00(%rsp)
-       vpxor           %ymm0,%ymm12,%ymm12
-       vpshufb         %ymm3,%ymm12,%ymm12
-       # x1 += x5, x13 = rotl32(x13 ^ x1, 16)
-       vpaddd          0x20(%rsp),%ymm5,%ymm0
-       vmovdqa         %ymm0,0x20(%rsp)
-       vpxor           %ymm0,%ymm13,%ymm13
-       vpshufb         %ymm3,%ymm13,%ymm13
-       # x2 += x6, x14 = rotl32(x14 ^ x2, 16)
-       vpaddd          0x40(%rsp),%ymm6,%ymm0
-       vmovdqa         %ymm0,0x40(%rsp)
-       vpxor           %ymm0,%ymm14,%ymm14
-       vpshufb         %ymm3,%ymm14,%ymm14
-       # x3 += x7, x15 = rotl32(x15 ^ x3, 16)
-       vpaddd          0x60(%rsp),%ymm7,%ymm0
-       vmovdqa         %ymm0,0x60(%rsp)
-       vpxor           %ymm0,%ymm15,%ymm15
-       vpshufb         %ymm3,%ymm15,%ymm15
-
-       # x8 += x12, x4 = rotl32(x4 ^ x8, 12)
-       vpaddd          %ymm12,%ymm8,%ymm8
-       vpxor           %ymm8,%ymm4,%ymm4
-       vpslld          $12,%ymm4,%ymm0
-       vpsrld          $20,%ymm4,%ymm4
-       vpor            %ymm0,%ymm4,%ymm4
-       # x9 += x13, x5 = rotl32(x5 ^ x9, 12)
-       vpaddd          %ymm13,%ymm9,%ymm9
-       vpxor           %ymm9,%ymm5,%ymm5
-       vpslld          $12,%ymm5,%ymm0
-       vpsrld          $20,%ymm5,%ymm5
-       vpor            %ymm0,%ymm5,%ymm5
-       # x10 += x14, x6 = rotl32(x6 ^ x10, 12)
-       vpaddd          %ymm14,%ymm10,%ymm10
-       vpxor           %ymm10,%ymm6,%ymm6
-       vpslld          $12,%ymm6,%ymm0
-       vpsrld          $20,%ymm6,%ymm6
-       vpor            %ymm0,%ymm6,%ymm6
-       # x11 += x15, x7 = rotl32(x7 ^ x11, 12)
-       vpaddd          %ymm15,%ymm11,%ymm11
-       vpxor           %ymm11,%ymm7,%ymm7
-       vpslld          $12,%ymm7,%ymm0
-       vpsrld          $20,%ymm7,%ymm7
-       vpor            %ymm0,%ymm7,%ymm7
-
-       # x0 += x4, x12 = rotl32(x12 ^ x0, 8)
-       vpaddd          0x00(%rsp),%ymm4,%ymm0
-       vmovdqa         %ymm0,0x00(%rsp)
-       vpxor           %ymm0,%ymm12,%ymm12
-       vpshufb         %ymm2,%ymm12,%ymm12
-       # x1 += x5, x13 = rotl32(x13 ^ x1, 8)
-       vpaddd          0x20(%rsp),%ymm5,%ymm0
-       vmovdqa         %ymm0,0x20(%rsp)
-       vpxor           %ymm0,%ymm13,%ymm13
-       vpshufb         %ymm2,%ymm13,%ymm13
-       # x2 += x6, x14 = rotl32(x14 ^ x2, 8)
-       vpaddd          0x40(%rsp),%ymm6,%ymm0
-       vmovdqa         %ymm0,0x40(%rsp)
-       vpxor           %ymm0,%ymm14,%ymm14
-       vpshufb         %ymm2,%ymm14,%ymm14
-       # x3 += x7, x15 = rotl32(x15 ^ x3, 8)
-       vpaddd          0x60(%rsp),%ymm7,%ymm0
-       vmovdqa         %ymm0,0x60(%rsp)
-       vpxor           %ymm0,%ymm15,%ymm15
-       vpshufb         %ymm2,%ymm15,%ymm15
-
-       # x8 += x12, x4 = rotl32(x4 ^ x8, 7)
-       vpaddd          %ymm12,%ymm8,%ymm8
-       vpxor           %ymm8,%ymm4,%ymm4
-       vpslld          $7,%ymm4,%ymm0
-       vpsrld          $25,%ymm4,%ymm4
-       vpor            %ymm0,%ymm4,%ymm4
-       # x9 += x13, x5 = rotl32(x5 ^ x9, 7)
-       vpaddd          %ymm13,%ymm9,%ymm9
-       vpxor           %ymm9,%ymm5,%ymm5
-       vpslld          $7,%ymm5,%ymm0
-       vpsrld          $25,%ymm5,%ymm5
-       vpor            %ymm0,%ymm5,%ymm5
-       # x10 += x14, x6 = rotl32(x6 ^ x10, 7)
-       vpaddd          %ymm14,%ymm10,%ymm10
-       vpxor           %ymm10,%ymm6,%ymm6
-       vpslld          $7,%ymm6,%ymm0
-       vpsrld          $25,%ymm6,%ymm6
-       vpor            %ymm0,%ymm6,%ymm6
-       # x11 += x15, x7 = rotl32(x7 ^ x11, 7)
-       vpaddd          %ymm15,%ymm11,%ymm11
-       vpxor           %ymm11,%ymm7,%ymm7
-       vpslld          $7,%ymm7,%ymm0
-       vpsrld          $25,%ymm7,%ymm7
-       vpor            %ymm0,%ymm7,%ymm7
-
-       # x0 += x5, x15 = rotl32(x15 ^ x0, 16)
-       vpaddd          0x00(%rsp),%ymm5,%ymm0
-       vmovdqa         %ymm0,0x00(%rsp)
-       vpxor           %ymm0,%ymm15,%ymm15
-       vpshufb         %ymm3,%ymm15,%ymm15
-       # x1 += x6, x12 = rotl32(x12 ^ x1, 16)%ymm0
-       vpaddd          0x20(%rsp),%ymm6,%ymm0
-       vmovdqa         %ymm0,0x20(%rsp)
-       vpxor           %ymm0,%ymm12,%ymm12
-       vpshufb         %ymm3,%ymm12,%ymm12
-       # x2 += x7, x13 = rotl32(x13 ^ x2, 16)
-       vpaddd          0x40(%rsp),%ymm7,%ymm0
-       vmovdqa         %ymm0,0x40(%rsp)
-       vpxor           %ymm0,%ymm13,%ymm13
-       vpshufb         %ymm3,%ymm13,%ymm13
-       # x3 += x4, x14 = rotl32(x14 ^ x3, 16)
-       vpaddd          0x60(%rsp),%ymm4,%ymm0
-       vmovdqa         %ymm0,0x60(%rsp)
-       vpxor           %ymm0,%ymm14,%ymm14
-       vpshufb         %ymm3,%ymm14,%ymm14
-
-       # x10 += x15, x5 = rotl32(x5 ^ x10, 12)
-       vpaddd          %ymm15,%ymm10,%ymm10
-       vpxor           %ymm10,%ymm5,%ymm5
-       vpslld          $12,%ymm5,%ymm0
-       vpsrld          $20,%ymm5,%ymm5
-       vpor            %ymm0,%ymm5,%ymm5
-       # x11 += x12, x6 = rotl32(x6 ^ x11, 12)
-       vpaddd          %ymm12,%ymm11,%ymm11
-       vpxor           %ymm11,%ymm6,%ymm6
-       vpslld          $12,%ymm6,%ymm0
-       vpsrld          $20,%ymm6,%ymm6
-       vpor            %ymm0,%ymm6,%ymm6
-       # x8 += x13, x7 = rotl32(x7 ^ x8, 12)
-       vpaddd          %ymm13,%ymm8,%ymm8
-       vpxor           %ymm8,%ymm7,%ymm7
-       vpslld          $12,%ymm7,%ymm0
-       vpsrld          $20,%ymm7,%ymm7
-       vpor            %ymm0,%ymm7,%ymm7
-       # x9 += x14, x4 = rotl32(x4 ^ x9, 12)
-       vpaddd          %ymm14,%ymm9,%ymm9
-       vpxor           %ymm9,%ymm4,%ymm4
-       vpslld          $12,%ymm4,%ymm0
-       vpsrld          $20,%ymm4,%ymm4
-       vpor            %ymm0,%ymm4,%ymm4
-
-       # x0 += x5, x15 = rotl32(x15 ^ x0, 8)
-       vpaddd          0x00(%rsp),%ymm5,%ymm0
-       vmovdqa         %ymm0,0x00(%rsp)
-       vpxor           %ymm0,%ymm15,%ymm15
-       vpshufb         %ymm2,%ymm15,%ymm15
-       # x1 += x6, x12 = rotl32(x12 ^ x1, 8)
-       vpaddd          0x20(%rsp),%ymm6,%ymm0
-       vmovdqa         %ymm0,0x20(%rsp)
-       vpxor           %ymm0,%ymm12,%ymm12
-       vpshufb         %ymm2,%ymm12,%ymm12
-       # x2 += x7, x13 = rotl32(x13 ^ x2, 8)
-       vpaddd          0x40(%rsp),%ymm7,%ymm0
-       vmovdqa         %ymm0,0x40(%rsp)
-       vpxor           %ymm0,%ymm13,%ymm13
-       vpshufb         %ymm2,%ymm13,%ymm13
-       # x3 += x4, x14 = rotl32(x14 ^ x3, 8)
-       vpaddd          0x60(%rsp),%ymm4,%ymm0
-       vmovdqa         %ymm0,0x60(%rsp)
-       vpxor           %ymm0,%ymm14,%ymm14
-       vpshufb         %ymm2,%ymm14,%ymm14
-
-       # x10 += x15, x5 = rotl32(x5 ^ x10, 7)
-       vpaddd          %ymm15,%ymm10,%ymm10
-       vpxor           %ymm10,%ymm5,%ymm5
-       vpslld          $7,%ymm5,%ymm0
-       vpsrld          $25,%ymm5,%ymm5
-       vpor            %ymm0,%ymm5,%ymm5
-       # x11 += x12, x6 = rotl32(x6 ^ x11, 7)
-       vpaddd          %ymm12,%ymm11,%ymm11
-       vpxor           %ymm11,%ymm6,%ymm6
-       vpslld          $7,%ymm6,%ymm0
-       vpsrld          $25,%ymm6,%ymm6
-       vpor            %ymm0,%ymm6,%ymm6
-       # x8 += x13, x7 = rotl32(x7 ^ x8, 7)
-       vpaddd          %ymm13,%ymm8,%ymm8
-       vpxor           %ymm8,%ymm7,%ymm7
-       vpslld          $7,%ymm7,%ymm0
-       vpsrld          $25,%ymm7,%ymm7
-       vpor            %ymm0,%ymm7,%ymm7
-       # x9 += x14, x4 = rotl32(x4 ^ x9, 7)
-       vpaddd          %ymm14,%ymm9,%ymm9
-       vpxor           %ymm9,%ymm4,%ymm4
-       vpslld          $7,%ymm4,%ymm0
-       vpsrld          $25,%ymm4,%ymm4
-       vpor            %ymm0,%ymm4,%ymm4
-
-       dec             %ecx
-       jnz             .Ldoubleround8
-
-       # x0..15[0-3] += s[0..15]
-       vpbroadcastd    0x00(%rdi),%ymm0
-       vpaddd          0x00(%rsp),%ymm0,%ymm0
-       vmovdqa         %ymm0,0x00(%rsp)
-       vpbroadcastd    0x04(%rdi),%ymm0
-       vpaddd          0x20(%rsp),%ymm0,%ymm0
-       vmovdqa         %ymm0,0x20(%rsp)
-       vpbroadcastd    0x08(%rdi),%ymm0
-       vpaddd          0x40(%rsp),%ymm0,%ymm0
-       vmovdqa         %ymm0,0x40(%rsp)
-       vpbroadcastd    0x0c(%rdi),%ymm0
-       vpaddd          0x60(%rsp),%ymm0,%ymm0
-       vmovdqa         %ymm0,0x60(%rsp)
-       vpbroadcastd    0x10(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm4,%ymm4
-       vpbroadcastd    0x14(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm5,%ymm5
-       vpbroadcastd    0x18(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm6,%ymm6
-       vpbroadcastd    0x1c(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm7,%ymm7
-       vpbroadcastd    0x20(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm8,%ymm8
-       vpbroadcastd    0x24(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm9,%ymm9
-       vpbroadcastd    0x28(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm10,%ymm10
-       vpbroadcastd    0x2c(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm11,%ymm11
-       vpbroadcastd    0x30(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm12,%ymm12
-       vpbroadcastd    0x34(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm13,%ymm13
-       vpbroadcastd    0x38(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm14,%ymm14
-       vpbroadcastd    0x3c(%rdi),%ymm0
-       vpaddd          %ymm0,%ymm15,%ymm15
-
-       # x12 += counter values 0-3
-       vpaddd          %ymm1,%ymm12,%ymm12
-
-       # interleave 32-bit words in state n, n+1
-       vmovdqa         0x00(%rsp),%ymm0
-       vmovdqa         0x20(%rsp),%ymm1
-       vpunpckldq      %ymm1,%ymm0,%ymm2
-       vpunpckhdq      %ymm1,%ymm0,%ymm1
-       vmovdqa         %ymm2,0x00(%rsp)
-       vmovdqa         %ymm1,0x20(%rsp)
-       vmovdqa         0x40(%rsp),%ymm0
-       vmovdqa         0x60(%rsp),%ymm1
-       vpunpckldq      %ymm1,%ymm0,%ymm2
-       vpunpckhdq      %ymm1,%ymm0,%ymm1
-       vmovdqa         %ymm2,0x40(%rsp)
-       vmovdqa         %ymm1,0x60(%rsp)
-       vmovdqa         %ymm4,%ymm0
-       vpunpckldq      %ymm5,%ymm0,%ymm4
-       vpunpckhdq      %ymm5,%ymm0,%ymm5
-       vmovdqa         %ymm6,%ymm0
-       vpunpckldq      %ymm7,%ymm0,%ymm6
-       vpunpckhdq      %ymm7,%ymm0,%ymm7
-       vmovdqa         %ymm8,%ymm0
-       vpunpckldq      %ymm9,%ymm0,%ymm8
-       vpunpckhdq      %ymm9,%ymm0,%ymm9
-       vmovdqa         %ymm10,%ymm0
-       vpunpckldq      %ymm11,%ymm0,%ymm10
-       vpunpckhdq      %ymm11,%ymm0,%ymm11
-       vmovdqa         %ymm12,%ymm0
-       vpunpckldq      %ymm13,%ymm0,%ymm12
-       vpunpckhdq      %ymm13,%ymm0,%ymm13
-       vmovdqa         %ymm14,%ymm0
-       vpunpckldq      %ymm15,%ymm0,%ymm14
-       vpunpckhdq      %ymm15,%ymm0,%ymm15
-
-       # interleave 64-bit words in state n, n+2
-       vmovdqa         0x00(%rsp),%ymm0
-       vmovdqa         0x40(%rsp),%ymm2
-       vpunpcklqdq     %ymm2,%ymm0,%ymm1
-       vpunpckhqdq     %ymm2,%ymm0,%ymm2
-       vmovdqa         %ymm1,0x00(%rsp)
-       vmovdqa         %ymm2,0x40(%rsp)
-       vmovdqa         0x20(%rsp),%ymm0
-       vmovdqa         0x60(%rsp),%ymm2
-       vpunpcklqdq     %ymm2,%ymm0,%ymm1
-       vpunpckhqdq     %ymm2,%ymm0,%ymm2
-       vmovdqa         %ymm1,0x20(%rsp)
-       vmovdqa         %ymm2,0x60(%rsp)
-       vmovdqa         %ymm4,%ymm0
-       vpunpcklqdq     %ymm6,%ymm0,%ymm4
-       vpunpckhqdq     %ymm6,%ymm0,%ymm6
-       vmovdqa         %ymm5,%ymm0
-       vpunpcklqdq     %ymm7,%ymm0,%ymm5
-       vpunpckhqdq     %ymm7,%ymm0,%ymm7
-       vmovdqa         %ymm8,%ymm0
-       vpunpcklqdq     %ymm10,%ymm0,%ymm8
-       vpunpckhqdq     %ymm10,%ymm0,%ymm10
-       vmovdqa         %ymm9,%ymm0
-       vpunpcklqdq     %ymm11,%ymm0,%ymm9
-       vpunpckhqdq     %ymm11,%ymm0,%ymm11
-       vmovdqa         %ymm12,%ymm0
-       vpunpcklqdq     %ymm14,%ymm0,%ymm12
-       vpunpckhqdq     %ymm14,%ymm0,%ymm14
-       vmovdqa         %ymm13,%ymm0
-       vpunpcklqdq     %ymm15,%ymm0,%ymm13
-       vpunpckhqdq     %ymm15,%ymm0,%ymm15
-
-       # interleave 128-bit words in state n, n+4
-       vmovdqa         0x00(%rsp),%ymm0
-       vperm2i128      $0x20,%ymm4,%ymm0,%ymm1
-       vperm2i128      $0x31,%ymm4,%ymm0,%ymm4
-       vmovdqa         %ymm1,0x00(%rsp)
-       vmovdqa         0x20(%rsp),%ymm0
-       vperm2i128      $0x20,%ymm5,%ymm0,%ymm1
-       vperm2i128      $0x31,%ymm5,%ymm0,%ymm5
-       vmovdqa         %ymm1,0x20(%rsp)
-       vmovdqa         0x40(%rsp),%ymm0
-       vperm2i128      $0x20,%ymm6,%ymm0,%ymm1
-       vperm2i128      $0x31,%ymm6,%ymm0,%ymm6
-       vmovdqa         %ymm1,0x40(%rsp)
-       vmovdqa         0x60(%rsp),%ymm0
-       vperm2i128      $0x20,%ymm7,%ymm0,%ymm1
-       vperm2i128      $0x31,%ymm7,%ymm0,%ymm7
-       vmovdqa         %ymm1,0x60(%rsp)
-       vperm2i128      $0x20,%ymm12,%ymm8,%ymm0
-       vperm2i128      $0x31,%ymm12,%ymm8,%ymm12
-       vmovdqa         %ymm0,%ymm8
-       vperm2i128      $0x20,%ymm13,%ymm9,%ymm0
-       vperm2i128      $0x31,%ymm13,%ymm9,%ymm13
-       vmovdqa         %ymm0,%ymm9
-       vperm2i128      $0x20,%ymm14,%ymm10,%ymm0
-       vperm2i128      $0x31,%ymm14,%ymm10,%ymm14
-       vmovdqa         %ymm0,%ymm10
-       vperm2i128      $0x20,%ymm15,%ymm11,%ymm0
-       vperm2i128      $0x31,%ymm15,%ymm11,%ymm15
-       vmovdqa         %ymm0,%ymm11
-
-       # xor with corresponding input, write to output
-       vmovdqa         0x00(%rsp),%ymm0
-       vpxor           0x0000(%rdx),%ymm0,%ymm0
-       vmovdqu         %ymm0,0x0000(%rsi)
-       vmovdqa         0x20(%rsp),%ymm0
-       vpxor           0x0080(%rdx),%ymm0,%ymm0
-       vmovdqu         %ymm0,0x0080(%rsi)
-       vmovdqa         0x40(%rsp),%ymm0
-       vpxor           0x0040(%rdx),%ymm0,%ymm0
-       vmovdqu         %ymm0,0x0040(%rsi)
-       vmovdqa         0x60(%rsp),%ymm0
-       vpxor           0x00c0(%rdx),%ymm0,%ymm0
-       vmovdqu         %ymm0,0x00c0(%rsi)
-       vpxor           0x0100(%rdx),%ymm4,%ymm4
-       vmovdqu         %ymm4,0x0100(%rsi)
-       vpxor           0x0180(%rdx),%ymm5,%ymm5
-       vmovdqu         %ymm5,0x00180(%rsi)
-       vpxor           0x0140(%rdx),%ymm6,%ymm6
-       vmovdqu         %ymm6,0x0140(%rsi)
-       vpxor           0x01c0(%rdx),%ymm7,%ymm7
-       vmovdqu         %ymm7,0x01c0(%rsi)
-       vpxor           0x0020(%rdx),%ymm8,%ymm8
-       vmovdqu         %ymm8,0x0020(%rsi)
-       vpxor           0x00a0(%rdx),%ymm9,%ymm9
-       vmovdqu         %ymm9,0x00a0(%rsi)
-       vpxor           0x0060(%rdx),%ymm10,%ymm10
-       vmovdqu         %ymm10,0x0060(%rsi)
-       vpxor           0x00e0(%rdx),%ymm11,%ymm11
-       vmovdqu         %ymm11,0x00e0(%rsi)
-       vpxor           0x0120(%rdx),%ymm12,%ymm12
-       vmovdqu         %ymm12,0x0120(%rsi)
-       vpxor           0x01a0(%rdx),%ymm13,%ymm13
-       vmovdqu         %ymm13,0x01a0(%rsi)
-       vpxor           0x0160(%rdx),%ymm14,%ymm14
-       vmovdqu         %ymm14,0x0160(%rsi)
-       vpxor           0x01e0(%rdx),%ymm15,%ymm15
-       vmovdqu         %ymm15,0x01e0(%rsi)
-
-       vzeroupper
-       lea             -8(%r10),%rsp
-       ret
-ENDPROC(chacha20_8block_xor_avx2)
arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0 -630
deleted file mode 100644
index 512a2b500fd1..000000000000
@@ -1,630 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, x64 SSSE3 functions
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/linkage.h>
-
-.section       .rodata.cst16.ROT8, "aM", @progbits, 16
-.align 16
-ROT8:  .octa 0x0e0d0c0f0a09080b0605040702010003
-.section       .rodata.cst16.ROT16, "aM", @progbits, 16
-.align 16
-ROT16: .octa 0x0d0c0f0e09080b0a0504070601000302
-.section       .rodata.cst16.CTRINC, "aM", @progbits, 16
-.align 16
-CTRINC:        .octa 0x00000003000000020000000100000000
-
-.text
-
-ENTRY(chacha20_block_xor_ssse3)
-       # %rdi: Input state matrix, s
-       # %rsi: 1 data block output, o
-       # %rdx: 1 data block input, i
-
-       # This function encrypts one ChaCha20 block by loading the state matrix
-       # in four SSE registers. It performs matrix operation on four words in
-       # parallel, but requireds shuffling to rearrange the words after each
-       # round. 8/16-bit word rotation is done with the slightly better
-       # performing SSSE3 byte shuffling, 7/12-bit word rotation uses
-       # traditional shift+OR.
-
-       # x0..3 = s0..3
-       movdqa          0x00(%rdi),%xmm0
-       movdqa          0x10(%rdi),%xmm1
-       movdqa          0x20(%rdi),%xmm2
-       movdqa          0x30(%rdi),%xmm3
-       movdqa          %xmm0,%xmm8
-       movdqa          %xmm1,%xmm9
-       movdqa          %xmm2,%xmm10
-       movdqa          %xmm3,%xmm11
-
-       movdqa          ROT8(%rip),%xmm4
-       movdqa          ROT16(%rip),%xmm5
-
-       mov     $10,%ecx
-
-.Ldoubleround:
-
-       # x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       paddd           %xmm1,%xmm0
-       pxor            %xmm0,%xmm3
-       pshufb          %xmm5,%xmm3
-
-       # x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       paddd           %xmm3,%xmm2
-       pxor            %xmm2,%xmm1
-       movdqa          %xmm1,%xmm6
-       pslld           $12,%xmm6
-       psrld           $20,%xmm1
-       por             %xmm6,%xmm1
-
-       # x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       paddd           %xmm1,%xmm0
-       pxor            %xmm0,%xmm3
-       pshufb          %xmm4,%xmm3
-
-       # x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       paddd           %xmm3,%xmm2
-       pxor            %xmm2,%xmm1
-       movdqa          %xmm1,%xmm7
-       pslld           $7,%xmm7
-       psrld           $25,%xmm1
-       por             %xmm7,%xmm1
-
-       # x1 = shuffle32(x1, MASK(0, 3, 2, 1))
-       pshufd          $0x39,%xmm1,%xmm1
-       # x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       pshufd          $0x4e,%xmm2,%xmm2
-       # x3 = shuffle32(x3, MASK(2, 1, 0, 3))
-       pshufd          $0x93,%xmm3,%xmm3
-
-       # x0 += x1, x3 = rotl32(x3 ^ x0, 16)
-       paddd           %xmm1,%xmm0
-       pxor            %xmm0,%xmm3
-       pshufb          %xmm5,%xmm3
-
-       # x2 += x3, x1 = rotl32(x1 ^ x2, 12)
-       paddd           %xmm3,%xmm2
-       pxor            %xmm2,%xmm1
-       movdqa          %xmm1,%xmm6
-       pslld           $12,%xmm6
-       psrld           $20,%xmm1
-       por             %xmm6,%xmm1
-
-       # x0 += x1, x3 = rotl32(x3 ^ x0, 8)
-       paddd           %xmm1,%xmm0
-       pxor            %xmm0,%xmm3
-       pshufb          %xmm4,%xmm3
-
-       # x2 += x3, x1 = rotl32(x1 ^ x2, 7)
-       paddd           %xmm3,%xmm2
-       pxor            %xmm2,%xmm1
-       movdqa          %xmm1,%xmm7
-       pslld           $7,%xmm7
-       psrld           $25,%xmm1
-       por             %xmm7,%xmm1
-
-       # x1 = shuffle32(x1, MASK(2, 1, 0, 3))
-       pshufd          $0x93,%xmm1,%xmm1
-       # x2 = shuffle32(x2, MASK(1, 0, 3, 2))
-       pshufd          $0x4e,%xmm2,%xmm2
-       # x3 = shuffle32(x3, MASK(0, 3, 2, 1))
-       pshufd          $0x39,%xmm3,%xmm3
-
-       dec             %ecx
-       jnz             .Ldoubleround
-
-       # o0 = i0 ^ (x0 + s0)
-       movdqu          0x00(%rdx),%xmm4
-       paddd           %xmm8,%xmm0
-       pxor            %xmm4,%xmm0
-       movdqu          %xmm0,0x00(%rsi)
-       # o1 = i1 ^ (x1 + s1)
-       movdqu          0x10(%rdx),%xmm5
-       paddd           %xmm9,%xmm1
-       pxor            %xmm5,%xmm1
-       movdqu          %xmm1,0x10(%rsi)
-       # o2 = i2 ^ (x2 + s2)
-       movdqu          0x20(%rdx),%xmm6
-       paddd           %xmm10,%xmm2
-       pxor            %xmm6,%xmm2
-       movdqu          %xmm2,0x20(%rsi)
-       # o3 = i3 ^ (x3 + s3)
-       movdqu          0x30(%rdx),%xmm7
-       paddd           %xmm11,%xmm3
-       pxor            %xmm7,%xmm3
-       movdqu          %xmm3,0x30(%rsi)
-
-       ret
-ENDPROC(chacha20_block_xor_ssse3)
-
-ENTRY(chacha20_4block_xor_ssse3)
-       # %rdi: Input state matrix, s
-       # %rsi: 4 data blocks output, o
-       # %rdx: 4 data blocks input, i
-
-       # This function encrypts four consecutive ChaCha20 blocks by loading the
-       # the state matrix in SSE registers four times. As we need some scratch
-       # registers, we save the first four registers on the stack. The
-       # algorithm performs each operation on the corresponding word of each
-       # state matrix, hence requires no word shuffling. For final XORing step
-       # we transpose the matrix by interleaving 32- and then 64-bit words,
-       # which allows us to do XOR in SSE registers. 8/16-bit word rotation is
-       # done with the slightly better performing SSSE3 byte shuffling,
-       # 7/12-bit word rotation uses traditional shift+OR.
-
-       lea             8(%rsp),%r10
-       sub             $0x80,%rsp
-       and             $~63,%rsp
-
-       # x0..15[0-3] = s0..3[0..3]
-       movq            0x00(%rdi),%xmm1
-       pshufd          $0x00,%xmm1,%xmm0
-       pshufd          $0x55,%xmm1,%xmm1
-       movq            0x08(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       movq            0x10(%rdi),%xmm5
-       pshufd          $0x00,%xmm5,%xmm4
-       pshufd          $0x55,%xmm5,%xmm5
-       movq            0x18(%rdi),%xmm7
-       pshufd          $0x00,%xmm7,%xmm6
-       pshufd          $0x55,%xmm7,%xmm7
-       movq            0x20(%rdi),%xmm9
-       pshufd          $0x00,%xmm9,%xmm8
-       pshufd          $0x55,%xmm9,%xmm9
-       movq            0x28(%rdi),%xmm11
-       pshufd          $0x00,%xmm11,%xmm10
-       pshufd          $0x55,%xmm11,%xmm11
-       movq            0x30(%rdi),%xmm13
-       pshufd          $0x00,%xmm13,%xmm12
-       pshufd          $0x55,%xmm13,%xmm13
-       movq            0x38(%rdi),%xmm15
-       pshufd          $0x00,%xmm15,%xmm14
-       pshufd          $0x55,%xmm15,%xmm15
-       # x0..3 on stack
-       movdqa          %xmm0,0x00(%rsp)
-       movdqa          %xmm1,0x10(%rsp)
-       movdqa          %xmm2,0x20(%rsp)
-       movdqa          %xmm3,0x30(%rsp)
-
-       movdqa          CTRINC(%rip),%xmm1
-       movdqa          ROT8(%rip),%xmm2
-       movdqa          ROT16(%rip),%xmm3
-
-       # x12 += counter values 0-3
-       paddd           %xmm1,%xmm12
-
-       mov             $10,%ecx
-
-.Ldoubleround4:
-       # x0 += x4, x12 = rotl32(x12 ^ x0, 16)
-       movdqa          0x00(%rsp),%xmm0
-       paddd           %xmm4,%xmm0
-       movdqa          %xmm0,0x00(%rsp)
-       pxor            %xmm0,%xmm12
-       pshufb          %xmm3,%xmm12
-       # x1 += x5, x13 = rotl32(x13 ^ x1, 16)
-       movdqa          0x10(%rsp),%xmm0
-       paddd           %xmm5,%xmm0
-       movdqa          %xmm0,0x10(%rsp)
-       pxor            %xmm0,%xmm13
-       pshufb          %xmm3,%xmm13
-       # x2 += x6, x14 = rotl32(x14 ^ x2, 16)
-       movdqa          0x20(%rsp),%xmm0
-       paddd           %xmm6,%xmm0
-       movdqa          %xmm0,0x20(%rsp)
-       pxor            %xmm0,%xmm14
-       pshufb          %xmm3,%xmm14
-       # x3 += x7, x15 = rotl32(x15 ^ x3, 16)
-       movdqa          0x30(%rsp),%xmm0
-       paddd           %xmm7,%xmm0
-       movdqa          %xmm0,0x30(%rsp)
-       pxor            %xmm0,%xmm15
-       pshufb          %xmm3,%xmm15
-
-       # x8 += x12, x4 = rotl32(x4 ^ x8, 12)
-       paddd           %xmm12,%xmm8
-       pxor            %xmm8,%xmm4
-       movdqa          %xmm4,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm4
-       por             %xmm0,%xmm4
-       # x9 += x13, x5 = rotl32(x5 ^ x9, 12)
-       paddd           %xmm13,%xmm9
-       pxor            %xmm9,%xmm5
-       movdqa          %xmm5,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm5
-       por             %xmm0,%xmm5
-       # x10 += x14, x6 = rotl32(x6 ^ x10, 12)
-       paddd           %xmm14,%xmm10
-       pxor            %xmm10,%xmm6
-       movdqa          %xmm6,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm6
-       por             %xmm0,%xmm6
-       # x11 += x15, x7 = rotl32(x7 ^ x11, 12)
-       paddd           %xmm15,%xmm11
-       pxor            %xmm11,%xmm7
-       movdqa          %xmm7,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm7
-       por             %xmm0,%xmm7
-
-       # x0 += x4, x12 = rotl32(x12 ^ x0, 8)
-       movdqa          0x00(%rsp),%xmm0
-       paddd           %xmm4,%xmm0
-       movdqa          %xmm0,0x00(%rsp)
-       pxor            %xmm0,%xmm12
-       pshufb          %xmm2,%xmm12
-       # x1 += x5, x13 = rotl32(x13 ^ x1, 8)
-       movdqa          0x10(%rsp),%xmm0
-       paddd           %xmm5,%xmm0
-       movdqa          %xmm0,0x10(%rsp)
-       pxor            %xmm0,%xmm13
-       pshufb          %xmm2,%xmm13
-       # x2 += x6, x14 = rotl32(x14 ^ x2, 8)
-       movdqa          0x20(%rsp),%xmm0
-       paddd           %xmm6,%xmm0
-       movdqa          %xmm0,0x20(%rsp)
-       pxor            %xmm0,%xmm14
-       pshufb          %xmm2,%xmm14
-       # x3 += x7, x15 = rotl32(x15 ^ x3, 8)
-       movdqa          0x30(%rsp),%xmm0
-       paddd           %xmm7,%xmm0
-       movdqa          %xmm0,0x30(%rsp)
-       pxor            %xmm0,%xmm15
-       pshufb          %xmm2,%xmm15
-
-       # x8 += x12, x4 = rotl32(x4 ^ x8, 7)
-       paddd           %xmm12,%xmm8
-       pxor            %xmm8,%xmm4
-       movdqa          %xmm4,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm4
-       por             %xmm0,%xmm4
-       # x9 += x13, x5 = rotl32(x5 ^ x9, 7)
-       paddd           %xmm13,%xmm9
-       pxor            %xmm9,%xmm5
-       movdqa          %xmm5,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm5
-       por             %xmm0,%xmm5
-       # x10 += x14, x6 = rotl32(x6 ^ x10, 7)
-       paddd           %xmm14,%xmm10
-       pxor            %xmm10,%xmm6
-       movdqa          %xmm6,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm6
-       por             %xmm0,%xmm6
-       # x11 += x15, x7 = rotl32(x7 ^ x11, 7)
-       paddd           %xmm15,%xmm11
-       pxor            %xmm11,%xmm7
-       movdqa          %xmm7,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm7
-       por             %xmm0,%xmm7
-
-       # x0 += x5, x15 = rotl32(x15 ^ x0, 16)
-       movdqa          0x00(%rsp),%xmm0
-       paddd           %xmm5,%xmm0
-       movdqa          %xmm0,0x00(%rsp)
-       pxor            %xmm0,%xmm15
-       pshufb          %xmm3,%xmm15
-       # x1 += x6, x12 = rotl32(x12 ^ x1, 16)
-       movdqa          0x10(%rsp),%xmm0
-       paddd           %xmm6,%xmm0
-       movdqa          %xmm0,0x10(%rsp)
-       pxor            %xmm0,%xmm12
-       pshufb          %xmm3,%xmm12
-       # x2 += x7, x13 = rotl32(x13 ^ x2, 16)
-       movdqa          0x20(%rsp),%xmm0
-       paddd           %xmm7,%xmm0
-       movdqa          %xmm0,0x20(%rsp)
-       pxor            %xmm0,%xmm13
-       pshufb          %xmm3,%xmm13
-       # x3 += x4, x14 = rotl32(x14 ^ x3, 16)
-       movdqa          0x30(%rsp),%xmm0
-       paddd           %xmm4,%xmm0
-       movdqa          %xmm0,0x30(%rsp)
-       pxor            %xmm0,%xmm14
-       pshufb          %xmm3,%xmm14
-
-       # x10 += x15, x5 = rotl32(x5 ^ x10, 12)
-       paddd           %xmm15,%xmm10
-       pxor            %xmm10,%xmm5
-       movdqa          %xmm5,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm5
-       por             %xmm0,%xmm5
-       # x11 += x12, x6 = rotl32(x6 ^ x11, 12)
-       paddd           %xmm12,%xmm11
-       pxor            %xmm11,%xmm6
-       movdqa          %xmm6,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm6
-       por             %xmm0,%xmm6
-       # x8 += x13, x7 = rotl32(x7 ^ x8, 12)
-       paddd           %xmm13,%xmm8
-       pxor            %xmm8,%xmm7
-       movdqa          %xmm7,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm7
-       por             %xmm0,%xmm7
-       # x9 += x14, x4 = rotl32(x4 ^ x9, 12)
-       paddd           %xmm14,%xmm9
-       pxor            %xmm9,%xmm4
-       movdqa          %xmm4,%xmm0
-       pslld           $12,%xmm0
-       psrld           $20,%xmm4
-       por             %xmm0,%xmm4
-
-       # x0 += x5, x15 = rotl32(x15 ^ x0, 8)
-       movdqa          0x00(%rsp),%xmm0
-       paddd           %xmm5,%xmm0
-       movdqa          %xmm0,0x00(%rsp)
-       pxor            %xmm0,%xmm15
-       pshufb          %xmm2,%xmm15
-       # x1 += x6, x12 = rotl32(x12 ^ x1, 8)
-       movdqa          0x10(%rsp),%xmm0
-       paddd           %xmm6,%xmm0
-       movdqa          %xmm0,0x10(%rsp)
-       pxor            %xmm0,%xmm12
-       pshufb          %xmm2,%xmm12
-       # x2 += x7, x13 = rotl32(x13 ^ x2, 8)
-       movdqa          0x20(%rsp),%xmm0
-       paddd           %xmm7,%xmm0
-       movdqa          %xmm0,0x20(%rsp)
-       pxor            %xmm0,%xmm13
-       pshufb          %xmm2,%xmm13
-       # x3 += x4, x14 = rotl32(x14 ^ x3, 8)
-       movdqa          0x30(%rsp),%xmm0
-       paddd           %xmm4,%xmm0
-       movdqa          %xmm0,0x30(%rsp)
-       pxor            %xmm0,%xmm14
-       pshufb          %xmm2,%xmm14
-
-       # x10 += x15, x5 = rotl32(x5 ^ x10, 7)
-       paddd           %xmm15,%xmm10
-       pxor            %xmm10,%xmm5
-       movdqa          %xmm5,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm5
-       por             %xmm0,%xmm5
-       # x11 += x12, x6 = rotl32(x6 ^ x11, 7)
-       paddd           %xmm12,%xmm11
-       pxor            %xmm11,%xmm6
-       movdqa          %xmm6,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm6
-       por             %xmm0,%xmm6
-       # x8 += x13, x7 = rotl32(x7 ^ x8, 7)
-       paddd           %xmm13,%xmm8
-       pxor            %xmm8,%xmm7
-       movdqa          %xmm7,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm7
-       por             %xmm0,%xmm7
-       # x9 += x14, x4 = rotl32(x4 ^ x9, 7)
-       paddd           %xmm14,%xmm9
-       pxor            %xmm9,%xmm4
-       movdqa          %xmm4,%xmm0
-       pslld           $7,%xmm0
-       psrld           $25,%xmm4
-       por             %xmm0,%xmm4
-
-       dec             %ecx
-       jnz             .Ldoubleround4
-
-       # x0[0-3] += s0[0]
-       # x1[0-3] += s0[1]
-       movq            0x00(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           0x00(%rsp),%xmm2
-       movdqa          %xmm2,0x00(%rsp)
-       paddd           0x10(%rsp),%xmm3
-       movdqa          %xmm3,0x10(%rsp)
-       # x2[0-3] += s0[2]
-       # x3[0-3] += s0[3]
-       movq            0x08(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           0x20(%rsp),%xmm2
-       movdqa          %xmm2,0x20(%rsp)
-       paddd           0x30(%rsp),%xmm3
-       movdqa          %xmm3,0x30(%rsp)
-
-       # x4[0-3] += s1[0]
-       # x5[0-3] += s1[1]
-       movq            0x10(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm4
-       paddd           %xmm3,%xmm5
-       # x6[0-3] += s1[2]
-       # x7[0-3] += s1[3]
-       movq            0x18(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm6
-       paddd           %xmm3,%xmm7
-
-       # x8[0-3] += s2[0]
-       # x9[0-3] += s2[1]
-       movq            0x20(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm8
-       paddd           %xmm3,%xmm9
-       # x10[0-3] += s2[2]
-       # x11[0-3] += s2[3]
-       movq            0x28(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm10
-       paddd           %xmm3,%xmm11
-
-       # x12[0-3] += s3[0]
-       # x13[0-3] += s3[1]
-       movq            0x30(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm12
-       paddd           %xmm3,%xmm13
-       # x14[0-3] += s3[2]
-       # x15[0-3] += s3[3]
-       movq            0x38(%rdi),%xmm3
-       pshufd          $0x00,%xmm3,%xmm2
-       pshufd          $0x55,%xmm3,%xmm3
-       paddd           %xmm2,%xmm14
-       paddd           %xmm3,%xmm15
-
-       # x12 += counter values 0-3
-       paddd           %xmm1,%xmm12
-
-       # interleave 32-bit words in state n, n+1
-       movdqa          0x00(%rsp),%xmm0
-       movdqa          0x10(%rsp),%xmm1
-       movdqa          %xmm0,%xmm2
-       punpckldq       %xmm1,%xmm2
-       punpckhdq       %xmm1,%xmm0
-       movdqa          %xmm2,0x00(%rsp)
-       movdqa          %xmm0,0x10(%rsp)
-       movdqa          0x20(%rsp),%xmm0
-       movdqa          0x30(%rsp),%xmm1
-       movdqa          %xmm0,%xmm2
-       punpckldq       %xmm1,%xmm2
-       punpckhdq       %xmm1,%xmm0
-       movdqa          %xmm2,0x20(%rsp)
-       movdqa          %xmm0,0x30(%rsp)
-       movdqa          %xmm4,%xmm0
-       punpckldq       %xmm5,%xmm4
-       punpckhdq       %xmm5,%xmm0
-       movdqa          %xmm0,%xmm5
-       movdqa          %xmm6,%xmm0
-       punpckldq       %xmm7,%xmm6
-       punpckhdq       %xmm7,%xmm0
-       movdqa          %xmm0,%xmm7
-       movdqa          %xmm8,%xmm0
-       punpckldq       %xmm9,%xmm8
-       punpckhdq       %xmm9,%xmm0
-       movdqa          %xmm0,%xmm9
-       movdqa          %xmm10,%xmm0
-       punpckldq       %xmm11,%xmm10
-       punpckhdq       %xmm11,%xmm0
-       movdqa          %xmm0,%xmm11
-       movdqa          %xmm12,%xmm0
-       punpckldq       %xmm13,%xmm12
-       punpckhdq       %xmm13,%xmm0
-       movdqa          %xmm0,%xmm13
-       movdqa          %xmm14,%xmm0
-       punpckldq       %xmm15,%xmm14
-       punpckhdq       %xmm15,%xmm0
-       movdqa          %xmm0,%xmm15
-
-       # interleave 64-bit words in state n, n+2
-       movdqa          0x00(%rsp),%xmm0
-       movdqa          0x20(%rsp),%xmm1
-       movdqa          %xmm0,%xmm2
-       punpcklqdq      %xmm1,%xmm2
-       punpckhqdq      %xmm1,%xmm0
-       movdqa          %xmm2,0x00(%rsp)
-       movdqa          %xmm0,0x20(%rsp)
-       movdqa          0x10(%rsp),%xmm0
-       movdqa          0x30(%rsp),%xmm1
-       movdqa          %xmm0,%xmm2
-       punpcklqdq      %xmm1,%xmm2
-       punpckhqdq      %xmm1,%xmm0
-       movdqa          %xmm2,0x10(%rsp)
-       movdqa          %xmm0,0x30(%rsp)
-       movdqa          %xmm4,%xmm0
-       punpcklqdq      %xmm6,%xmm4
-       punpckhqdq      %xmm6,%xmm0
-       movdqa          %xmm0,%xmm6
-       movdqa          %xmm5,%xmm0
-       punpcklqdq      %xmm7,%xmm5
-       punpckhqdq      %xmm7,%xmm0
-       movdqa          %xmm0,%xmm7
-       movdqa          %xmm8,%xmm0
-       punpcklqdq      %xmm10,%xmm8
-       punpckhqdq      %xmm10,%xmm0
-       movdqa          %xmm0,%xmm10
-       movdqa          %xmm9,%xmm0
-       punpcklqdq      %xmm11,%xmm9
-       punpckhqdq      %xmm11,%xmm0
-       movdqa          %xmm0,%xmm11
-       movdqa          %xmm12,%xmm0
-       punpcklqdq      %xmm14,%xmm12
-       punpckhqdq      %xmm14,%xmm0
-       movdqa          %xmm0,%xmm14
-       movdqa          %xmm13,%xmm0
-       punpcklqdq      %xmm15,%xmm13
-       punpckhqdq      %xmm15,%xmm0
-       movdqa          %xmm0,%xmm15
-
-       # xor with corresponding input, write to output
-       movdqa          0x00(%rsp),%xmm0
-       movdqu          0x00(%rdx),%xmm1
-       pxor            %xmm1,%xmm0
-       movdqu          %xmm0,0x00(%rsi)
-       movdqa          0x10(%rsp),%xmm0
-       movdqu          0x80(%rdx),%xmm1
-       pxor            %xmm1,%xmm0
-       movdqu          %xmm0,0x80(%rsi)
-       movdqa          0x20(%rsp),%xmm0
-       movdqu          0x40(%rdx),%xmm1
-       pxor            %xmm1,%xmm0
-       movdqu          %xmm0,0x40(%rsi)
-       movdqa          0x30(%rsp),%xmm0
-       movdqu          0xc0(%rdx),%xmm1
-       pxor            %xmm1,%xmm0
-       movdqu          %xmm0,0xc0(%rsi)
-       movdqu          0x10(%rdx),%xmm1
-       pxor            %xmm1,%xmm4
-       movdqu          %xmm4,0x10(%rsi)
-       movdqu          0x90(%rdx),%xmm1
-       pxor            %xmm1,%xmm5
-       movdqu          %xmm5,0x90(%rsi)
-       movdqu          0x50(%rdx),%xmm1
-       pxor            %xmm1,%xmm6
-       movdqu          %xmm6,0x50(%rsi)
-       movdqu          0xd0(%rdx),%xmm1
-       pxor            %xmm1,%xmm7
-       movdqu          %xmm7,0xd0(%rsi)
-       movdqu          0x20(%rdx),%xmm1
-       pxor            %xmm1,%xmm8
-       movdqu          %xmm8,0x20(%rsi)
-       movdqu          0xa0(%rdx),%xmm1
-       pxor            %xmm1,%xmm9
-       movdqu          %xmm9,0xa0(%rsi)
-       movdqu          0x60(%rdx),%xmm1
-       pxor            %xmm1,%xmm10
-       movdqu          %xmm10,0x60(%rsi)
-       movdqu          0xe0(%rdx),%xmm1
-       pxor            %xmm1,%xmm11
-       movdqu          %xmm11,0xe0(%rsi)
-       movdqu          0x30(%rdx),%xmm1
-       pxor            %xmm1,%xmm12
-       movdqu          %xmm12,0x30(%rsi)
-       movdqu          0xb0(%rdx),%xmm1
-       pxor            %xmm1,%xmm13
-       movdqu          %xmm13,0xb0(%rsi)
-       movdqu          0x70(%rdx),%xmm1
-       pxor            %xmm1,%xmm14
-       movdqu          %xmm14,0x70(%rsi)
-       movdqu          0xf0(%rdx),%xmm1
-       pxor            %xmm1,%xmm15
-       movdqu          %xmm15,0xf0(%rsi)
-
-       lea             -8(%r10),%rsp
-       ret
-ENDPROC(chacha20_4block_xor_ssse3)
arch/x86/crypto/chacha20_glue.c deleted +0 -146
deleted file mode 100644
index dce7c5d39c2f..000000000000
@@ -1,146 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539, SIMD glue code
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <crypto/algapi.h>
-#include <crypto/chacha20.h>
-#include <crypto/internal/skcipher.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <asm/fpu/api.h>
-#include <asm/simd.h>
-
-#define CHACHA20_STATE_ALIGN 16
-
-asmlinkage void chacha20_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src);
-asmlinkage void chacha20_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src);
-#ifdef CONFIG_AS_AVX2
-asmlinkage void chacha20_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src);
-static bool chacha20_use_avx2;
-#endif
-
-static void chacha20_dosimd(u32 *state, u8 *dst, const u8 *src,
-                           unsigned int bytes)
-{
-       u8 buf[CHACHA20_BLOCK_SIZE];
-
-#ifdef CONFIG_AS_AVX2
-       if (chacha20_use_avx2) {
-               while (bytes >= CHACHA20_BLOCK_SIZE * 8) {
-                       chacha20_8block_xor_avx2(state, dst, src);
-                       bytes -= CHACHA20_BLOCK_SIZE * 8;
-                       src += CHACHA20_BLOCK_SIZE * 8;
-                       dst += CHACHA20_BLOCK_SIZE * 8;
-                       state[12] += 8;
-               }
-       }
-#endif
-       while (bytes >= CHACHA20_BLOCK_SIZE * 4) {
-               chacha20_4block_xor_ssse3(state, dst, src);
-               bytes -= CHACHA20_BLOCK_SIZE * 4;
-               src += CHACHA20_BLOCK_SIZE * 4;
-               dst += CHACHA20_BLOCK_SIZE * 4;
-               state[12] += 4;
-       }
-       while (bytes >= CHACHA20_BLOCK_SIZE) {
-               chacha20_block_xor_ssse3(state, dst, src);
-               bytes -= CHACHA20_BLOCK_SIZE;
-               src += CHACHA20_BLOCK_SIZE;
-               dst += CHACHA20_BLOCK_SIZE;
-               state[12]++;
-       }
-       if (bytes) {
-               memcpy(buf, src, bytes);
-               chacha20_block_xor_ssse3(state, buf, buf);
-               memcpy(dst, buf, bytes);
-       }
-}
-
-static int chacha20_simd(struct skcipher_request *req)
-{
-       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
-       struct chacha20_ctx *ctx = crypto_skcipher_ctx(tfm);
-       u32 *state, state_buf[16 + 2] __aligned(8);
-       struct skcipher_walk walk;
-       int err;
-
-       BUILD_BUG_ON(CHACHA20_STATE_ALIGN != 16);
-       state = PTR_ALIGN(state_buf + 0, CHACHA20_STATE_ALIGN);
-
-       if (req->cryptlen <= CHACHA20_BLOCK_SIZE || !may_use_simd())
-               return crypto_chacha20_crypt(req);
-
-       err = skcipher_walk_virt(&walk, req, true);
-
-       crypto_chacha20_init(state, ctx, walk.iv);
-
-       kernel_fpu_begin();
-
-       while (walk.nbytes >= CHACHA20_BLOCK_SIZE) {
-               chacha20_dosimd(state, walk.dst.virt.addr, walk.src.virt.addr,
-                               rounddown(walk.nbytes, CHACHA20_BLOCK_SIZE));
-               err = skcipher_walk_done(&walk,
-                                        walk.nbytes % CHACHA20_BLOCK_SIZE);
-       }
-
-       if (walk.nbytes) {
-               chacha20_dosimd(state, walk.dst.virt.addr, walk.src.virt.addr,
-                               walk.nbytes);
-               err = skcipher_walk_done(&walk, 0);
-       }
-
-       kernel_fpu_end();
-
-       return err;
-}
-
-static struct skcipher_alg alg = {
-       .base.cra_name          = "chacha20",
-       .base.cra_driver_name   = "chacha20-simd",
-       .base.cra_priority      = 300,
-       .base.cra_blocksize     = 1,
-       .base.cra_ctxsize       = sizeof(struct chacha20_ctx),
-       .base.cra_module        = THIS_MODULE,
-
-       .min_keysize            = CHACHA20_KEY_SIZE,
-       .max_keysize            = CHACHA20_KEY_SIZE,
-       .ivsize                 = CHACHA20_IV_SIZE,
-       .chunksize              = CHACHA20_BLOCK_SIZE,
-       .setkey                 = crypto_chacha20_setkey,
-       .encrypt                = chacha20_simd,
-       .decrypt                = chacha20_simd,
-};
-
-static int __init chacha20_simd_mod_init(void)
-{
-       if (!boot_cpu_has(X86_FEATURE_SSSE3))
-               return -ENODEV;
-
-#ifdef CONFIG_AS_AVX2
-       chacha20_use_avx2 = boot_cpu_has(X86_FEATURE_AVX) &&
-                           boot_cpu_has(X86_FEATURE_AVX2) &&
-                           cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL);
-#endif
-       return crypto_register_skcipher(&alg);
-}
-
-static void __exit chacha20_simd_mod_fini(void)
-{
-       crypto_unregister_skcipher(&alg);
-}
-
-module_init(chacha20_simd_mod_init);
-module_exit(chacha20_simd_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Martin Willi <[email protected]>");
-MODULE_DESCRIPTION("chacha20 cipher algorithm, SIMD accelerated");
-MODULE_ALIAS_CRYPTO("chacha20");
-MODULE_ALIAS_CRYPTO("chacha20-simd");
crypto/Kconfig +0 -16
index 47859a0f8052..93cd4d199447 100644
@@ -1433,22 +1433,6 @@ config CRYPTO_CHACHA20
 
          ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
          Bernstein and further specified in RFC7539 for use in IETF protocols.
-         This is the portable C implementation of ChaCha20.
-
-         See also:
-         <http://cr.yp.to/chacha/chacha-20080128.pdf>
-
-config CRYPTO_CHACHA20_X86_64
-       tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)"
-       depends on X86 && 64BIT
-       select CRYPTO_BLKCIPHER
-       select CRYPTO_CHACHA20
-       help
-         ChaCha20 cipher algorithm, RFC7539.
-
-         ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
-         Bernstein and further specified in RFC7539 for use in IETF protocols.
-         This is the x86_64 assembler implementation using SIMD instructions.
 
          See also:
          <http://cr.yp.to/chacha/chacha-20080128.pdf>
crypto/Makefile +1 -1
index 5e60348d02e2..587103b87890 100644
@@ -117,7 +117,7 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o
 obj-$(CONFIG_CRYPTO_SEED) += seed.o
 obj-$(CONFIG_CRYPTO_SPECK) += speck.o
 obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o
-obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o
+obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_zinc.o
 obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
crypto/chacha20_generic.c deleted +0 -136
deleted file mode 100644
index e451c3cb6a56..000000000000
@@ -1,136 +0,0 @@
-/*
- * ChaCha20 256-bit cipher algorithm, RFC7539
- *
- * Copyright (C) 2015 Martin Willi
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <asm/unaligned.h>
-#include <crypto/algapi.h>
-#include <crypto/chacha20.h>
-#include <crypto/internal/skcipher.h>
-#include <linux/module.h>
-
-static void chacha20_docrypt(u32 *state, u8 *dst, const u8 *src,
-                            unsigned int bytes)
-{
-       u32 stream[CHACHA20_BLOCK_WORDS];
-
-       if (dst != src)
-               memcpy(dst, src, bytes);
-
-       while (bytes >= CHACHA20_BLOCK_SIZE) {
-               chacha20_block(state, stream);
-               crypto_xor(dst, (const u8 *)stream, CHACHA20_BLOCK_SIZE);
-               bytes -= CHACHA20_BLOCK_SIZE;
-               dst += CHACHA20_BLOCK_SIZE;
-       }
-       if (bytes) {
-               chacha20_block(state, stream);
-               crypto_xor(dst, (const u8 *)stream, bytes);
-       }
-}
-
-void crypto_chacha20_init(u32 *state, struct chacha20_ctx *ctx, u8 *iv)
-{
-       state[0]  = 0x61707865; /* "expa" */
-       state[1]  = 0x3320646e; /* "nd 3" */
-       state[2]  = 0x79622d32; /* "2-by" */
-       state[3]  = 0x6b206574; /* "te k" */
-       state[4]  = ctx->key[0];
-       state[5]  = ctx->key[1];
-       state[6]  = ctx->key[2];
-       state[7]  = ctx->key[3];
-       state[8]  = ctx->key[4];
-       state[9]  = ctx->key[5];
-       state[10] = ctx->key[6];
-       state[11] = ctx->key[7];
-       state[12] = get_unaligned_le32(iv +  0);
-       state[13] = get_unaligned_le32(iv +  4);
-       state[14] = get_unaligned_le32(iv +  8);
-       state[15] = get_unaligned_le32(iv + 12);
-}
-EXPORT_SYMBOL_GPL(crypto_chacha20_init);
-
-int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key,
-                          unsigned int keysize)
-{
-       struct chacha20_ctx *ctx = crypto_skcipher_ctx(tfm);
-       int i;
-
-       if (keysize != CHACHA20_KEY_SIZE)
-               return -EINVAL;
-
-       for (i = 0; i < ARRAY_SIZE(ctx->key); i++)
-               ctx->key[i] = get_unaligned_le32(key + i * sizeof(u32));
-
-       return 0;
-}
-EXPORT_SYMBOL_GPL(crypto_chacha20_setkey);
-
-int crypto_chacha20_crypt(struct skcipher_request *req)
-{
-       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
-       struct chacha20_ctx *ctx = crypto_skcipher_ctx(tfm);
-       struct skcipher_walk walk;
-       u32 state[16];
-       int err;
-
-       err = skcipher_walk_virt(&walk, req, true);
-
-       crypto_chacha20_init(state, ctx, walk.iv);
-
-       while (walk.nbytes > 0) {
-               unsigned int nbytes = walk.nbytes;
-
-               if (nbytes < walk.total)
-                       nbytes = round_down(nbytes, walk.stride);
-
-               chacha20_docrypt(state, walk.dst.virt.addr, walk.src.virt.addr,
-                                nbytes);
-               err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-       }
-
-       return err;
-}
-EXPORT_SYMBOL_GPL(crypto_chacha20_crypt);
-
-static struct skcipher_alg alg = {
-       .base.cra_name          = "chacha20",
-       .base.cra_driver_name   = "chacha20-generic",
-       .base.cra_priority      = 100,
-       .base.cra_blocksize     = 1,
-       .base.cra_ctxsize       = sizeof(struct chacha20_ctx),
-       .base.cra_module        = THIS_MODULE,
-
-       .min_keysize            = CHACHA20_KEY_SIZE,
-       .max_keysize            = CHACHA20_KEY_SIZE,
-       .ivsize                 = CHACHA20_IV_SIZE,
-       .chunksize              = CHACHA20_BLOCK_SIZE,
-       .setkey                 = crypto_chacha20_setkey,
-       .encrypt                = crypto_chacha20_crypt,
-       .decrypt                = crypto_chacha20_crypt,
-};
-
-static int __init chacha20_generic_mod_init(void)
-{
-       return crypto_register_skcipher(&alg);
-}
-
-static void __exit chacha20_generic_mod_fini(void)
-{
-       crypto_unregister_skcipher(&alg);
-}
-
-module_init(chacha20_generic_mod_init);
-module_exit(chacha20_generic_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Martin Willi <[email protected]>");
-MODULE_DESCRIPTION("chacha20 cipher algorithm");
-MODULE_ALIAS_CRYPTO("chacha20");
-MODULE_ALIAS_CRYPTO("chacha20-generic");
crypto/chacha20_zinc.c new file +100 -0
new file mode 100644
index 000000000000..5df88fdee066
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/unaligned.h>
+#include <crypto/algapi.h>
+#include <crypto/internal/skcipher.h>
+#include <zinc/chacha20.h>
+#include <linux/module.h>
+
+struct chacha20_key_ctx {
+       u32 key[8];
+};
+
+static int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key,
+                                 unsigned int keysize)
+{
+       struct chacha20_key_ctx *key_ctx = crypto_skcipher_ctx(tfm);
+       int i;
+
+       if (keysize != CHACHA20_KEY_SIZE)
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(key_ctx->key); ++i)
+               key_ctx->key[i] = get_unaligned_le32(key + i * sizeof(u32));
+
+       return 0;
+}
+
+static int crypto_chacha20_crypt(struct skcipher_request *req)
+{
+       struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+       struct chacha20_key_ctx *key_ctx = crypto_skcipher_ctx(tfm);
+       struct chacha20_ctx ctx;
+       struct skcipher_walk walk;
+       simd_context_t simd_context;
+       int err, i;
+
+       err = skcipher_walk_virt(&walk, req, true);
+       if (unlikely(err))
+               return err;
+
+       memcpy(ctx.key, key_ctx->key, sizeof(ctx.key));
+       for (i = 0; i < ARRAY_SIZE(ctx.counter); ++i)
+               ctx.counter[i] = get_unaligned_le32(walk.iv + i * sizeof(u32));
+
+       simd_context = simd_get();
+       while (walk.nbytes > 0) {
+               unsigned int nbytes = walk.nbytes;
+
+               if (nbytes < walk.total)
+                       nbytes = round_down(nbytes, walk.stride);
+
+               chacha20(&ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes,
+                        simd_context);
+
+               err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+               simd_context = simd_relax(simd_context);
+       }
+       simd_put(simd_context);
+
+       return err;
+}
+
+static struct skcipher_alg alg = {
+       .base.cra_name          = "chacha20",
+       .base.cra_driver_name   = "chacha20-software",
+       .base.cra_priority      = 100,
+       .base.cra_blocksize     = 1,
+       .base.cra_ctxsize       = sizeof(struct chacha20_key_ctx),
+       .base.cra_module        = THIS_MODULE,
+
+       .min_keysize            = CHACHA20_KEY_SIZE,
+       .max_keysize            = CHACHA20_KEY_SIZE,
+       .ivsize                 = CHACHA20_IV_SIZE,
+       .chunksize              = CHACHA20_BLOCK_SIZE,
+       .setkey                 = crypto_chacha20_setkey,
+       .encrypt                = crypto_chacha20_crypt,
+       .decrypt                = crypto_chacha20_crypt,
+};
+
+static int __init chacha20_mod_init(void)
+{
+       return crypto_register_skcipher(&alg);
+}
+
+static void __exit chacha20_mod_exit(void)
+{
+       crypto_unregister_skcipher(&alg);
+}
+
+module_init(chacha20_mod_init);
+module_exit(chacha20_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Jason A. Donenfeld <[email protected]>");
+MODULE_DESCRIPTION("ChaCha20 stream cipher");
+MODULE_ALIAS_CRYPTO("chacha20");
+MODULE_ALIAS_CRYPTO("chacha20-software");
crypto/chacha20poly1305.c +1 -1
index bf523797bef3..b26adb9ed898 100644
@@ -13,7 +13,7 @@
 #include <crypto/internal/hash.h>
 #include <crypto/internal/skcipher.h>
 #include <crypto/scatterwalk.h>
-#include <crypto/chacha20.h>
+#include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
 #include <linux/err.h>
 #include <linux/init.h>
include/crypto/chacha20.h +0 -13
index b83d66073db0..3b92f58f3891 100644
@@ -6,23 +6,11 @@
 #ifndef _CRYPTO_CHACHA20_H
 #define _CRYPTO_CHACHA20_H
 
-#include <crypto/skcipher.h>
-#include <linux/types.h>
-#include <linux/crypto.h>
-
 #define CHACHA20_IV_SIZE       16
 #define CHACHA20_KEY_SIZE      32
 #define CHACHA20_BLOCK_SIZE    64
 #define CHACHA20_BLOCK_WORDS   (CHACHA20_BLOCK_SIZE / sizeof(u32))
 
-struct chacha20_ctx {
-       u32 key[8];
-};
-
 void chacha20_block(u32 *state, u32 *stream);
-void crypto_chacha20_init(u32 *state, struct chacha20_ctx *ctx, u8 *iv);
-int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key,
-                          unsigned int keysize);
-int crypto_chacha20_crypt(struct skcipher_request *req);
 
 #endif
--

2.19.0

author maintainer
#25 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

Quoted 1 line · Ard Biesheuvel

so could we please bring that discussion to a close before we drop the ARM code?

My understanding is that either these will find their way up to AndyP and then back down here, or Eric or you will augment the .S in this patch at a later date with an improvement commit that includes some benchmarks.

Jason

A message that is not in the archive (sent off-list or not yet received): 201809160711.HzjdJedZ%[email protected]

author maintainer
#27 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

Hi Mr. Ro Bot,

Quoted 2 lines · kbuild test robot

crypto/chacha20_zinc.o: In function `crypto_chacha20_crypt':

Quoted 1 line
Quoted 1 line

crypto/chacha20_zinc.c:55: undefined reference to `chacha20'

Looks like my Kconfig change didn't get squashed in as intended. Fixed for v5.

Thanks,
Jason

#28 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

In reply to Jason A. Donenfeld

Hi Jason,

Quoted 5 lines

Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it.

 delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S

I did some trivial benchmarking with tcrypt for the ChaCha20Poly1305 AEAD as used by IPsec. This is on a box with AVX2, which is probably the configuration mostly used these days. With Zinc I get:

Quoted 9 lines

testing speed of rfc7539esp(chacha20,poly1305) (rfc7539esp(chacha20-software,poly1305-software)) decryption test 0 (288 bit key, 16 byte blocks): 743510 operations in 1 seconds (11896160 bytes)
test 1 (288 bit key, 64 byte blocks): 743190 operations in 1 seconds (47564160 bytes)
test 2 (288 bit key, 256 byte blocks): 701461 operations in 1 seconds (179574016 bytes) test 3 (288 bit key, 512 byte blocks): 681567 operations in 1 seconds (348962304 bytes) test 4 (288 bit key, 1024 byte blocks): 572854 operations in 1 seconds (586602496 bytes) test 5 (288 bit key, 2048 byte blocks): 434477 operations in 1 seconds (889808896 bytes) test 6 (288 bit key, 4096 byte blocks): 293553 operations in 1 seconds (1202393088 bytes) test 7 (288 bit key, 8192 byte blocks): 173351 operations in 1 seconds (1420091392 bytes)

Using the existing implementation, this was:

Quoted 9 lines

testing speed of rfc7539esp(chacha20,poly1305) (rfc7539esp(chacha20-simd,poly1305-simd)) decryption test 0 (288 bit key, 16 byte blocks): 1064524 operations in 1 seconds (17032384 bytes) test 1 (288 bit key, 64 byte blocks): 1016046 operations in 1 seconds (65026944 bytes) test 2 (288 bit key, 256 byte blocks): 829566 operations in 1 seconds (212368896 bytes) test 3 (288 bit key, 512 byte blocks): 778912 operations in 1 seconds (398802944 bytes) test 4 (288 bit key, 1024 byte blocks): 622331 operations in 1 seconds (637266944 bytes) test 5 (288 bit key, 2048 byte blocks): 441790 operations in 1 seconds (904785920 bytes) test 6 (288 bit key, 4096 byte blocks): 280616 operations in 1 seconds (1149403136 bytes) test 7 (288 bit key, 8192 byte blocks): 158800 operations in 1 seconds (1300889600 bytes)

I've also experimented with the SIMD context save/restore amortization from patch one on the existing implementation:

Quoted 9 lines

testing speed of rfc7539esp(chacha20,poly1305) (rfc7539esp(chacha20-simd,poly1305-simd)) decryption test 0 (288 bit key, 16 byte blocks): 1088215 operations in 1 seconds (17411440 bytes) test 1 (288 bit key, 64 byte blocks): 1001788 operations in 1 seconds (64114432 bytes) test 2 (288 bit key, 256 byte blocks): 870193 operations in 1 seconds (222769408 bytes) test 3 (288 bit key, 512 byte blocks): 822149 operations in 1 seconds (420940288 bytes) test 4 (288 bit key, 1024 byte blocks): 647447 operations in 1 seconds (662985728 bytes) test 5 (288 bit key, 2048 byte blocks): 454734 operations in 1 seconds (931295232 bytes) test 6 (288 bit key, 4096 byte blocks): 286995 operations in 1 seconds (1175531520 bytes) test 7 (288 bit key, 8192 byte blocks): 162028 operations in 1 seconds (1327333376 bytes)

For large blocks your implementation is faster; for typical IPsec MTUs this degrades performance by ~10% and more.

Martin

author maintainer
#29 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

Hey Martin,

Thanks for running these and pointing this out. I've replicated the results with tcrypt and fixed some issues, and the next patch series should be a lot closer to what you'd expect, instead of the regression you noticed. Most of the slowdown happened as a result of over-eager XSAVEs, which I've now rectified. I'm still working on a few other facets of it, but I believe v5 will be more satisfactory when posted.

Regards,
Jason

#30 lore raw

Re: [PATCH net-next v4 18/20] crypto: port ChaCha20 to Zinc

In reply to Jason A. Donenfeld
I love your patch! Yet something to improve:

Hi Jason,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: i386-randconfig-s3-09171149 (attached as .config)
compiler: gcc-6 (Debian 6.4.0-9) 6.4.0 20171026 reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

>> ERROR: "chacha20" [crypto/chacha20_zinc.ko] undefined!

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#31 lore raw

PATCH v4 19/20 security/keys: rewrite big_key crypto to use Zinc

In reply to Jason A. Donenfeld

A while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function. This makes the file considerably more simple; the diffstat alone should justify this commit. It also should be faster, since it no longer requires a mutex around the "aead api object" (nor allocations), allowing us to encrypt multiple items in parallel. We also benefit from being able to pass any type of pointer to Zinc, so we can get rid of the ridiculously complex custom page allocator that big_key really doesn't need.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
Cc David Howells <[email protected]>
---
 security/keys/Kconfig   |   4 +-
 security/keys/big_key.c | 230 +++++-----------------------------------
 2 files changed, 28 insertions(+), 206 deletions(-)
security/keys/Kconfig +1 -3
index 6462e6654ccf..66ff26298fb3 100644
@@ -45,9 +45,7 @@ config BIG_KEYS
 	bool "Large payload keys"
 	depends on KEYS
 	depends on TMPFS
-	select CRYPTO
-	select CRYPTO_AES
-	select CRYPTO_GCM
+	select ZINC_CHACHA20POLY1305
 	help
 	  This option provides support for holding large keys within the kernel
 	  (for example Kerberos ticket caches).  The data may be stored out to
security/keys/big_key.c +27 -203

347 lines not on this page · raw message

Signature · 1 line
2.19.0
#32 lore raw

Re: [PATCH net-next v4 19/20] security/keys: rewrite big_key crypto to use Zinc

I love your patch! Yet something to improve:

Hi Jason,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: arm64-defconfig
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
GCC_VERSION=7.2.0 make.cross ARCH=arm64 defconfig
GCC_VERSION=7.2.0 make.cross ARCH=arm64

All errors (new ones prefixed by >>):

>> drivers/acpi/Kconfig:9:error: recursive dependency detected!
   drivers/acpi/Kconfig:9: symbol ACPI depends on ARCH_SUPPORTS_ACPI
   drivers/acpi/Kconfig:6: symbol ARCH_SUPPORTS_ACPI is selected by EFI
   arch/arm64/Kconfig:1253: symbol EFI depends on KERNEL_MODE_NEON
   arch/arm64/Kconfig:262: symbol KERNEL_MODE_NEON is implied by ZINC_ARCH_ARM
   lib/zinc/Kconfig:42: symbol ZINC_ARCH_ARM depends on ZINC
>> lib/zinc/Kconfig:1: symbol ZINC is selected by ZINC_CHACHA20
>> lib/zinc/Kconfig:4: symbol ZINC_CHACHA20 is selected by ZINC_CHACHA20POLY1305
>> lib/zinc/Kconfig:13: symbol ZINC_CHACHA20POLY1305 is selected by BIG_KEYS
>> security/keys/Kconfig:44: symbol BIG_KEYS depends on KEYS
>> security/keys/Kconfig:5: symbol KEYS is selected by FS_ENCRYPTION
>> fs/crypto/Kconfig:1: symbol FS_ENCRYPTION is selected by UBIFS_FS_ENCRYPTION
>> fs/ubifs/Kconfig:65: symbol UBIFS_FS_ENCRYPTION depends on MISC_FILESYSTEMS
>> fs/Kconfig:218: symbol MISC_FILESYSTEMS is selected by ACPI_APEI
>> drivers/acpi/apei/Kconfig:8: symbol ACPI_APEI depends on ACPI

For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"

vim +4 lib/zinc/Kconfig

32bbe22e Jason A. Donenfeld 2018-09-14 @1 config ZINC
32bbe22e Jason A. Donenfeld 2018-09-14 2 tristate
32bbe22e Jason A. Donenfeld 2018-09-14 3

35f45248 Jason A. Donenfeld 2018-09-14  @4  config ZINC_CHACHA20
35f45248 Jason A. Donenfeld 2018-09-14   5  	bool
35f45248 Jason A. Donenfeld 2018-09-14   6  	select ZINC
35f45248 Jason A. Donenfeld 2018-09-14   7  	select CRYPTO_ALGAPI
35f45248 Jason A. Donenfeld 2018-09-14   8  
0a36c146 Jason A. Donenfeld 2018-09-14   9  config ZINC_POLY1305
0a36c146 Jason A. Donenfeld 2018-09-14  10  	bool
0a36c146 Jason A. Donenfeld 2018-09-14  11  	select ZINC
0a36c146 Jason A. Donenfeld 2018-09-14  12  
1b5dbb86 Jason A. Donenfeld 2018-09-14 @13  config ZINC_CHACHA20POLY1305
1b5dbb86 Jason A. Donenfeld 2018-09-14  14  	bool
1b5dbb86 Jason A. Donenfeld 2018-09-14  15  	select ZINC
1b5dbb86 Jason A. Donenfeld 2018-09-14  16  	select ZINC_CHACHA20
1b5dbb86 Jason A. Donenfeld 2018-09-14  17  	select ZINC_POLY1305
1b5dbb86 Jason A. Donenfeld 2018-09-14  18  	select CRYPTO_BLKCIPHER
1b5dbb86 Jason A. Donenfeld 2018-09-14  19  
a740374c Jason A. Donenfeld 2018-09-14  20  config ZINC_BLAKE2S
a740374c Jason A. Donenfeld 2018-09-14  21  	bool
a740374c Jason A. Donenfeld 2018-09-14  22  	select ZINC
a740374c Jason A. Donenfeld 2018-09-14  23  
cec5aa7c Jason A. Donenfeld 2018-09-14  24  config ZINC_CURVE25519
cec5aa7c Jason A. Donenfeld 2018-09-14  25  	bool
cec5aa7c Jason A. Donenfeld 2018-09-14  26  	select ZINC
cec5aa7c Jason A. Donenfeld 2018-09-14  27  	select CONFIG_CRYPTO
cec5aa7c Jason A. Donenfeld 2018-09-14  28  
32bbe22e Jason A. Donenfeld 2018-09-14  29  config ZINC_DEBUG
32bbe22e Jason A. Donenfeld 2018-09-14  30  	bool "Zinc cryptography library debugging and self-tests"
32bbe22e Jason A. Donenfeld 2018-09-14  31  	depends on ZINC
32bbe22e Jason A. Donenfeld 2018-09-14  32  	help
32bbe22e Jason A. Donenfeld 2018-09-14  33  	  This builds a series of self-tests for the Zinc crypto library, which
32bbe22e Jason A. Donenfeld 2018-09-14  34  	  help diagnose any cryptographic algorithm implementation issues that
32bbe22e Jason A. Donenfeld 2018-09-14  35  	  might be at the root cause of potential bugs. It also adds various
32bbe22e Jason A. Donenfeld 2018-09-14  36  	  debugging traps.
32bbe22e Jason A. Donenfeld 2018-09-14  37  
32bbe22e Jason A. Donenfeld 2018-09-14  38  	  Unless you're developing and testing cryptographic routines, or are
32bbe22e Jason A. Donenfeld 2018-09-14  39  	  especially paranoid about correctness on your hardware, you may say
32bbe22e Jason A. Donenfeld 2018-09-14  40  	  N here.
32bbe22e Jason A. Donenfeld 2018-09-14  41  
32bbe22e Jason A. Donenfeld 2018-09-14 @42  config ZINC_ARCH_ARM
32bbe22e Jason A. Donenfeld 2018-09-14  43  	def_bool y
32bbe22e Jason A. Donenfeld 2018-09-14  44  	depends on ARM
32bbe22e Jason A. Donenfeld 2018-09-14  45  	depends on ZINC
32bbe22e Jason A. Donenfeld 2018-09-14  46  	imply VFP
32bbe22e Jason A. Donenfeld 2018-09-14  47  	imply VFPv3 if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14  48  	imply NEON if CPU_V7
32bbe22e Jason A. Donenfeld 2018-09-14  49  	imply KERNEL_MODE_NEON if CPU_V7

32bbe22e Jason A. Donenfeld 2018-09-14 50

:::::: The code at line 4 was first introduced by commit
:::::: 35f45248597b5a2c80f0f4a680344c22c86efe7d zinc: ChaCha20 generic C implementation

:::::: TO: Jason A. Donenfeld <[email protected]> :::::: CC: 0day robot <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#33 lore raw

Re: [PATCH net-next v4 19/20] security/keys: rewrite big_key crypto to use Zinc

Hey again Ro,

Quoted 1 line

32bbe22e Jason A. Donenfeld 2018-09-14 49 imply KERNEL_MODE_NEON if CPU_V7

This shouldn't have ever been there anyway. Fixed now for v5.

Thanks,
Jason

author maintainer
#34 lore raw

PATCH v4 20/20 net: WireGuard secure network tunnel

In reply to Jason A. Donenfeld

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace have already implemented the API.

This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.

The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:

* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.

* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.

* allowedips.[ch], hashtables.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the specific hashtables we use.

* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.

* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.

* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.

* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.

* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.

* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.

* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.

* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.

* main.c, version.h: Initialization and deinitialization of the module.

* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.

* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.

This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.

We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.

---
 MAINTAINERS                                  |   8 +
 drivers/net/Kconfig                          |  30 +
 drivers/net/Makefile                         |   1 +
 drivers/net/wireguard/Makefile               |  18 +
 drivers/net/wireguard/allowedips.c           | 404 ++++++++++
 drivers/net/wireguard/allowedips.h           |  55 ++
 drivers/net/wireguard/cookie.c               | 234 ++++++
 drivers/net/wireguard/cookie.h               |  59 ++
 drivers/net/wireguard/device.c               | 438 +++++++++++
 drivers/net/wireguard/device.h               |  65 ++
 drivers/net/wireguard/hashtables.c           | 209 +++++
 drivers/net/wireguard/hashtables.h           |  63 ++
 drivers/net/wireguard/main.c                 |  65 ++
 drivers/net/wireguard/messages.h             | 128 +++
 drivers/net/wireguard/netlink.c              | 605 ++++++++++++++
 drivers/net/wireguard/netlink.h              |  12 +
 drivers/net/wireguard/noise.c                | 784 +++++++++++++++++++
 drivers/net/wireguard/noise.h                | 129 +++
 drivers/net/wireguard/peer.c                 | 191 +++++
 drivers/net/wireguard/peer.h                 |  87 ++
 drivers/net/wireguard/queueing.c             |  52 ++
 drivers/net/wireguard/queueing.h             | 193 +++++
 drivers/net/wireguard/ratelimiter.c          | 220 ++++++
 drivers/net/wireguard/ratelimiter.h          |  19 +
 drivers/net/wireguard/receive.c              | 597 ++++++++++++++
 drivers/net/wireguard/selftest/allowedips.h  | 656 ++++++++++++++++
 drivers/net/wireguard/selftest/counter.h     | 103 +++
 drivers/net/wireguard/selftest/ratelimiter.h | 174 ++++
 drivers/net/wireguard/send.c                 | 420 ++++++++++
 drivers/net/wireguard/socket.c               | 435 ++++++++++
 drivers/net/wireguard/socket.h               |  44 ++
 drivers/net/wireguard/timers.c               | 256 ++++++
 drivers/net/wireguard/timers.h               |  30 +
 drivers/net/wireguard/version.h              |   1 +
 include/uapi/linux/wireguard.h               | 190 +++++
 tools/testing/selftests/wireguard/netns.sh   | 499 ++++++++++++
 36 files changed, 7474 insertions(+)
 create mode 100644 drivers/net/wireguard/Makefile
 create mode 100644 drivers/net/wireguard/allowedips.c
 create mode 100644 drivers/net/wireguard/allowedips.h
 create mode 100644 drivers/net/wireguard/cookie.c
 create mode 100644 drivers/net/wireguard/cookie.h
 create mode 100644 drivers/net/wireguard/device.c
 create mode 100644 drivers/net/wireguard/device.h
 create mode 100644 drivers/net/wireguard/hashtables.c
 create mode 100644 drivers/net/wireguard/hashtables.h
 create mode 100644 drivers/net/wireguard/main.c
 create mode 100644 drivers/net/wireguard/messages.h
 create mode 100644 drivers/net/wireguard/netlink.c
 create mode 100644 drivers/net/wireguard/netlink.h
 create mode 100644 drivers/net/wireguard/noise.c
 create mode 100644 drivers/net/wireguard/noise.h
 create mode 100644 drivers/net/wireguard/peer.c
 create mode 100644 drivers/net/wireguard/peer.h
 create mode 100644 drivers/net/wireguard/queueing.c
 create mode 100644 drivers/net/wireguard/queueing.h
 create mode 100644 drivers/net/wireguard/ratelimiter.c
 create mode 100644 drivers/net/wireguard/ratelimiter.h
 create mode 100644 drivers/net/wireguard/receive.c
 create mode 100644 drivers/net/wireguard/selftest/allowedips.h
 create mode 100644 drivers/net/wireguard/selftest/counter.h
 create mode 100644 drivers/net/wireguard/selftest/ratelimiter.h
 create mode 100644 drivers/net/wireguard/send.c
 create mode 100644 drivers/net/wireguard/socket.c
 create mode 100644 drivers/net/wireguard/socket.h
 create mode 100644 drivers/net/wireguard/timers.c
 create mode 100644 drivers/net/wireguard/timers.h
 create mode 100644 drivers/net/wireguard/version.h
 create mode 100644 include/uapi/linux/wireguard.h
 create mode 100755 tools/testing/selftests/wireguard/netns.sh
MAINTAINERS +8 -0
index d2092e52320d..2043437adf0b 100644
@@ -15813,6 +15813,14 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/gpio/gpio-ws16c48.c
 
+WIREGUARD SECURE NETWORK TUNNEL
+M:	Jason A. Donenfeld <[email protected]>
+S:	Maintained
+F:	drivers/net/wireguard/
+F:	tools/testing/selftests/wireguard/
+L:	[email protected]
+L:	[email protected]
+
 WISTRON LAPTOP BUTTON DRIVER
 M:	Miloslav Trmac <[email protected]>
 S:	Maintained
drivers/net/Kconfig +30 -0
index d03775100f7d..aa631fe3b395 100644
@@ -70,6 +70,36 @@ config DUMMY
 	  To compile this driver as a module, choose M here: the module
 	  will be called dummy.
 
+config WIREGUARD
+	tristate "WireGuard secure network tunnel"
+	depends on NET && INET
+	select NET_UDP_TUNNEL
+	select DST_CACHE
+	select ZINC_CHACHA20POLY1305
+	select ZINC_BLAKE2S
+	select ZINC_CURVE25519
+	default m
+	help
+	  WireGuard is a secure, fast, and easy to use replacement for IPSec
+	  that uses modern cryptography and clever networking tricks. It's
+	  designed to be fairly general purpose and abstract enough to fit most
+	  use cases, while at the same time remaining extremely simple to
+	  configure. See www.wireguard.com for more info.
+
+	  It's safe to say Y or M here, as the driver is very lightweight and
+	  is only in use when an administrator chooses to add an interface.
+
+config WIREGUARD_DEBUG
+	bool "Debugging checks and verbose messages"
+	depends on WIREGUARD
+	help
+	  This will write log messages for handshake and other events
+	  that occur for a WireGuard interface. It will also perform some
+	  extra validation checks and unit tests at various points. This is
+	  only useful for debugging.
+
+	  Say N here unless you know what you're doing.
+
 config EQUALIZER
 	tristate "EQL (serial line load balancing) support"
 	---help---
drivers/net/Makefile +1 -0
index 21cde7e78621..f0acd11a143d 100644
@@ -24,6 +24,7 @@ obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/
 obj-$(CONFIG_TUN) += tun.o
 obj-$(CONFIG_TAP) += tap.o
+obj-$(CONFIG_WIREGUARD) += wireguard/
 obj-$(CONFIG_VETH) += veth.o
 obj-$(CONFIG_VIRTIO_NET) += virtio_net.o
 obj-$(CONFIG_VXLAN) += vxlan.o
drivers/net/wireguard/Makefile new file +18 -0
new file mode 100644
index 000000000000..d8856255bc9d
@@ -0,0 +1,18 @@
+ccflags-y := -O3
+ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
+ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG
+wireguard-y := main.o
+wireguard-y += noise.o
+wireguard-y += device.o
+wireguard-y += peer.o
+wireguard-y += timers.o
+wireguard-y += queueing.o
+wireguard-y += send.o
+wireguard-y += receive.o
+wireguard-y += socket.o
+wireguard-y += hashtables.o
+wireguard-y += allowedips.o
+wireguard-y += ratelimiter.o
+wireguard-y += cookie.o
+wireguard-y += netlink.o
+obj-$(CONFIG_WIREGUARD) := wireguard.o
drivers/net/wireguard/allowedips.c new file +404 -0

404 lines not on this page · raw message

drivers/net/wireguard/allowedips.h new file +55 -0

55 lines not on this page · raw message

drivers/net/wireguard/cookie.c new file +234 -0

234 lines not on this page · raw message

drivers/net/wireguard/cookie.h new file +59 -0

59 lines not on this page · raw message

drivers/net/wireguard/device.c new file +438 -0

438 lines not on this page · raw message

drivers/net/wireguard/device.h new file +65 -0

65 lines not on this page · raw message

drivers/net/wireguard/hashtables.c new file +209 -0

209 lines not on this page · raw message

drivers/net/wireguard/hashtables.h new file +63 -0

63 lines not on this page · raw message

drivers/net/wireguard/main.c new file +65 -0

65 lines not on this page · raw message

drivers/net/wireguard/messages.h new file +128 -0

128 lines not on this page · raw message

drivers/net/wireguard/netlink.c new file +605 -0

605 lines not on this page · raw message

drivers/net/wireguard/netlink.h new file +12 -0
new file mode 100644
index 000000000000..c1cd9b019bd1
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _WG_NETLINK_H
+#define _WG_NETLINK_H
+
+int genetlink_init(void);
+void genetlink_uninit(void);
+
+#endif /* _WG_NETLINK_H */
drivers/net/wireguard/noise.c new file +784 -0

588 lines not on this page · raw message

drivers/net/wireguard/noise.h new file +129 -0

129 lines not on this page · raw message

drivers/net/wireguard/peer.c new file +191 -0

191 lines not on this page · raw message

drivers/net/wireguard/peer.h new file +87 -0

87 lines not on this page · raw message

drivers/net/wireguard/queueing.c new file +52 -0

52 lines not on this page · raw message

drivers/net/wireguard/queueing.h new file +193 -0

193 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.c new file +220 -0

220 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.h new file +19 -0
new file mode 100644
index 000000000000..8931c0615374

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/receive.c new file +597 -0

597 lines not on this page · raw message

drivers/net/wireguard/selftest/allowedips.h new file +656 -0

656 lines not on this page · raw message

drivers/net/wireguard/selftest/counter.h new file +103 -0

103 lines not on this page · raw message

drivers/net/wireguard/selftest/ratelimiter.h new file +174 -0

174 lines not on this page · raw message

drivers/net/wireguard/send.c new file +420 -0

420 lines not on this page · raw message

drivers/net/wireguard/socket.c new file +435 -0

435 lines not on this page · raw message

drivers/net/wireguard/socket.h new file +44 -0

44 lines not on this page · raw message

drivers/net/wireguard/timers.c new file +256 -0

256 lines not on this page · raw message

drivers/net/wireguard/timers.h new file +30 -0
new file mode 100644
index 000000000000..483529c9d873

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/version.h new file +1 -0
new file mode 100644
index 000000000000..fba9c7ab9423

Lines of this file are not shown. View the raw message.

include/uapi/linux/wireguard.h new file +190 -0

190 lines not on this page · raw message

tools/testing/selftests/wireguard/netns.sh new file +499 -0

499 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

#35 lore raw

Re: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel

I love your patch! Perhaps something to improve:

Hi Jason,

I love your patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: powerpc-canyonlands_defconfig (attached as .config)
compiler: powerpc-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=powerpc

All warnings (new ones prefixed by >>):

drivers/net/wireguard/send.c: In function 'packet_encrypt_worker':

>> drivers/net/wireguard/send.c:320:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]
    }
    ^
--
   drivers/net/wireguard/receive.c: In function 'packet_decrypt_worker':
>> drivers/net/wireguard/receive.c:520:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]
    }
    ^

vim +320 drivers/net/wireguard/send.c

294
295 void packet_encrypt_worker(struct work_struct *work)
296 {
297 struct crypt_queue *queue =
298 container_of(work, struct multicore_worker, work)->ptr;
299 struct sk_buff *first, *skb, *next;
300 simd_context_t simd_context = simd_get();
301
302 while ((first = ptr_ring_consume_bh(&queue->ring)) != NULL) {
303 enum packet_state state = PACKET_STATE_CRYPTED;
304
305 skb_walk_null_queue_safe (first, skb, next) {
306 if (likely(skb_encrypt(skb, PACKET_CB(first)->keypair,
307 simd_context)))
308 skb_reset(skb);
309 else {
310 state = PACKET_STATE_DEAD;
311 break;
312 }
313 }
314 queue_enqueue_per_peer(&PACKET_PEER(first)->tx_queue, first,
315 state);
316
317 simd_context = simd_relax(simd_context);
318 }
319 simd_put(simd_context);

 > 320	}

321

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#36 lore raw

Re: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel

Hi Mr. Ro Bot,

Quoted 3 lines · kbuild test robot

drivers/net/wireguard/send.c: In function 'packet_encrypt_worker':

Quoted 1 line
Quoted 1 line

drivers/net/wireguard/send.c:320:1: warning: the frame size of 1136 bytes is larger than 1024 bytes [-Wframe-larger-than=]

    }

Thanks, fixed for v5.

Jason

#37 lore raw

Re: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel

In reply to Jason A. Donenfeld
I love your patch! Perhaps something to improve:

Hi Jason,

I love your patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url: https://github.com/0day-ci/linux/commits/Jason-A-Donenfeld/WireGuard-Secure-Network-Tunnel/20180916-043623
config: x86_64-randconfig-b0-09160521 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All warnings (new ones prefixed by >>):

In file included from lib/zinc/curve25519/curve25519.c:45:0:
lib/zinc/curve25519/curve25519-hacl64.h: In function 'curve25519_generic':

>> lib/zinc/curve25519/curve25519-hacl64.h:785:1: warning: the frame size of 13216 bytes is larger than 8192 bytes [-Wframe-larger-than=]
    }
    ^

vim +785 lib/zinc/curve25519/curve25519-hacl64.h

cec5aa7c Jason A. Donenfeld 2018-09-14  755  
cec5aa7c Jason A. Donenfeld 2018-09-14  756  static void curve25519_generic(u8 mypublic[CURVE25519_POINT_SIZE],
cec5aa7c Jason A. Donenfeld 2018-09-14  757  			       const u8 secret[CURVE25519_POINT_SIZE],
cec5aa7c Jason A. Donenfeld 2018-09-14  758  			       const u8 basepoint[CURVE25519_POINT_SIZE])
cec5aa7c Jason A. Donenfeld 2018-09-14  759  {
cec5aa7c Jason A. Donenfeld 2018-09-14  760  	u64 buf0[10] __aligned(32) = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14  761  	u64 *x0 = buf0;
cec5aa7c Jason A. Donenfeld 2018-09-14  762  	u64 *z = buf0 + 5;
cec5aa7c Jason A. Donenfeld 2018-09-14  763  	u64 *q;
cec5aa7c Jason A. Donenfeld 2018-09-14  764  	format_fexpand(x0, basepoint);
cec5aa7c Jason A. Donenfeld 2018-09-14  765  	z[0] = 1;
cec5aa7c Jason A. Donenfeld 2018-09-14  766  	q = buf0;
cec5aa7c Jason A. Donenfeld 2018-09-14  767  	{
cec5aa7c Jason A. Donenfeld 2018-09-14  768  		u8 e[32] __aligned(32) = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14  769  		u8 *scalar;
cec5aa7c Jason A. Donenfeld 2018-09-14  770  		memcpy(e, secret, 32);
cec5aa7c Jason A. Donenfeld 2018-09-14  771  		normalize_secret(e);
cec5aa7c Jason A. Donenfeld 2018-09-14  772  		scalar = e;
cec5aa7c Jason A. Donenfeld 2018-09-14  773  		{
cec5aa7c Jason A. Donenfeld 2018-09-14  774  			u64 buf[15] = { 0 };
cec5aa7c Jason A. Donenfeld 2018-09-14  775  			u64 *nq = buf;
cec5aa7c Jason A. Donenfeld 2018-09-14  776  			u64 *x = nq;
cec5aa7c Jason A. Donenfeld 2018-09-14  777  			x[0] = 1;
cec5aa7c Jason A. Donenfeld 2018-09-14  778  			ladder_cmult(nq, scalar, q);
cec5aa7c Jason A. Donenfeld 2018-09-14  779  			format_scalar_of_point(mypublic, nq);
cec5aa7c Jason A. Donenfeld 2018-09-14  780  			memzero_explicit(buf, sizeof(buf));
cec5aa7c Jason A. Donenfeld 2018-09-14  781  		}
cec5aa7c Jason A. Donenfeld 2018-09-14  782  		memzero_explicit(e, sizeof(e));
cec5aa7c Jason A. Donenfeld 2018-09-14  783  	}
cec5aa7c Jason A. Donenfeld 2018-09-14  784  	memzero_explicit(buf0, sizeof(buf0));

cec5aa7c Jason A. Donenfeld 2018-09-14 @785 }

:::::: The code at line 785 was first introduced by commit
:::::: cec5aa7cd396d909f8555d2df80d3e1a0f94688b zinc: Curve25519 generic C implementations and selftest

:::::: TO: Jason A. Donenfeld <[email protected]>
:::::: CC: 0day robot <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#38 lore raw

Re: [PATCH net-next v4 20/20] net: WireGuard secure network tunnel

Hi Mr. Ro Bot,

Looks like this is related to stack frames with KASAN. I've fixed this for v5.

Thanks,
Jason