Skip to contentSkip to conversation

WireGuard: Secure Network Tunnel

PATCH Superseded Jason A. Donenfeld 9 participants 65 messages Sep 18, 2018 – Sep 25, 2018

Subsystems (30)

v5 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

v5 cover WireGuard: Secure Network Tunnel

v5 has the most comprehensive set of changes yet, and I think should finally address all of the relevant issues brought up on the mailing list. In particular, this feedback has come from:

- Andy Lutomirski
- Eric Biggers
- Ard Biesheuvel
- Kevin Easton
- Andrew Lunn
- Martin Willi

Changes v4->v5:
---------------
- Use fewer inlines, except when measured as necessary.
- Reduce size of scattergather array to fit within stack on small systems.
- Account for larger stack frames with KASAN.
- The x86_64 implementations are selected according to input length.
- Avoid using simd for small blocks on x86_64.
- The simd_get/put API is now pass by reference, so that the user can lazily use the context based on whether or not it's needed. See the description again in the first commit for this.
- Add cycle counts for different sizes for x86_64 commit messages.
- Relax simd during chapoly sg loop.
- Replace -include with #if defined(...)
- Saner and simpler Kconfig.
- Split into separate modules instead of one monolithic zinc.
- The combination of these three last items means that there no longer are any conditionals in our Makefile.
- Martin showed a performance regression using tcrypt in v4. This has been triaged and fixed, and now the Zinc code runs faster than the previous code.
- While I initially wasn't going to do this for the initial patchset, it was just so simple to do: now there's a nosimd module parameter that can be used to disable simd instructions for debugging and testing, or on weird systems.

-----------------------------------------------------------

This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:

* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

-----------------------------------------------------------

WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.

Ample information, including documentation, installation instructions, and project details, is available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:

* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/

The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:

* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf

The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:

* https://www.wireguard.com/formal-verification/

This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.

It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.

Enjoy,
Jason

author maintainer
#1 lore raw

PATCH v5 1/20 asm: simd context helper API

Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related FPU/SIMD functions over a number of calls, because FPU restoration is quite expensive. This adds a simple header for carrying out this pattern:

    simd_context_t simd_context;

    simd_get(&simd_context);
    while ((item = get_item_from_queue()) != NULL) {
        encrypt_item(item, simd_context);
        simd_relax(&simd_context);
    }
    simd_put(&simd_context);

The relaxation step ensures that we don't trample over preemption, and the get/put API should be a familiar paradigm in the kernel.

On the other end, code that actually wants to use SIMD instructions can accept this as a parameter and check it via:

   void encrypt_item(struct item *item, simd_context_t *simd_context)
   {
       if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
           wild_simd_code(item);
       else
           boring_scalar_code(item);
   }

The actual XSAVE happens during simd_use (and only on the first time), so that if the context is never actually used, no performance penalty is hit.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Greg KH <[email protected]>
---
 arch/alpha/include/asm/Kbuild      |  5 ++-
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/simd.h        | 63 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/simd.h      | 51 +++++++++++++++++++++---
 arch/c6x/include/asm/Kbuild        |  3 +-
 arch/h8300/include/asm/Kbuild      |  3 +-
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/nds32/include/asm/Kbuild      |  7 ++--
 arch/nios2/include/asm/Kbuild      |  1 +
 arch/openrisc/include/asm/Kbuild   |  7 ++--
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  3 +-
 arch/riscv/include/asm/Kbuild      |  3 +-
 arch/s390/include/asm/Kbuild       |  3 +-
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/um/include/asm/Kbuild         |  3 +-
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/simd.h        | 44 ++++++++++++++++++++-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/simd.h         | 20 ++++++++++
 include/linux/simd.h               | 28 +++++++++++++
 26 files changed, 234 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm/include/asm/simd.h
 create mode 100644 include/linux/simd.h
arch/alpha/include/asm/Kbuild +3 -2
index 0580cb8c84b2..07b2c1025d34 100644
@@ -2,14 +2,15 @@
 
 
 generic-y += compat.h
+generic-y += current.h
 generic-y += exec.h
 generic-y += export.h
 generic-y += fb.h
 generic-y += irq_work.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h
arch/arc/include/asm/Kbuild +1 -0
index feed50ce89fa..a7f4255f1649 100644
@@ -22,6 +22,7 @@ generic-y += parport.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/arm/include/asm/simd.h new file +63 -0

63 lines not on this page · raw message

arch/arm64/include/asm/simd.h +45 -6

71 lines not on this page · raw message

arch/c6x/include/asm/Kbuild +2 -1
index 33a2c94fed0d..22f3d8333c74 100644
@@ -5,8 +5,8 @@ generic-y += compat.h
 generic-y += current.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -30,6 +30,7 @@ generic-y += pgalloc.h
 generic-y += preempt.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += tlbflush.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/h8300/include/asm/Kbuild +2 -1
index a5d0b2991f47..f5c2f12d593e 100644
@@ -8,8 +8,8 @@ generic-y += current.h
 generic-y += delay.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += extable.h
@@ -39,6 +39,7 @@ generic-y += preempt.h
 generic-y += scatterlist.h
 generic-y += sections.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += spinlock.h
 generic-y += timex.h
arch/hexagon/include/asm/Kbuild +1 -0
index dd2fd9c0d292..217d4695fd8a 100644
@@ -29,6 +29,7 @@ generic-y += rwsem.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/ia64/include/asm/Kbuild +1 -0
index 557bbc8ba9f5..41c5ebdf79e5 100644
@@ -4,6 +4,7 @@ generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += vtime.h
 generic-y += word-at-a-time.h
arch/m68k/include/asm/Kbuild +1 -0
index a4b8d3331a9e..73898dd1a4d0 100644
@@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/microblaze/include/asm/Kbuild +1 -0
index 569ba9e670c1..7a877eea99d3 100644
@@ -25,6 +25,7 @@ generic-y += parport.h
 generic-y += percpu.h
 generic-y += preempt.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += syscalls.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/mips/include/asm/Kbuild +1 -0
index 58351e48421e..e8868e0fb2c3 100644
@@ -16,6 +16,7 @@ generic-y += qrwlock.h
 generic-y += qspinlock.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += user.h
arch/nds32/include/asm/Kbuild +4 -3
index dbc4e5422550..603c1d020620 100644
@@ -7,14 +7,14 @@ generic-y += bug.h
 generic-y += bugs.h
 generic-y += checksum.h
 generic-y += clkdev.h
-generic-y += cmpxchg.h
 generic-y += cmpxchg-local.h
+generic-y += cmpxchg.h
 generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,14 +46,15 @@ generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
 generic-y += shmbuf.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += stat.h
 generic-y += switch_to.h
 generic-y += timex.h
 generic-y += topology.h
 generic-y += trace_clock.h
-generic-y += xor.h
 generic-y += unaligned.h
 generic-y += user.h
 generic-y += vga.h
 generic-y += word-at-a-time.h
+generic-y += xor.h
arch/nios2/include/asm/Kbuild +1 -0
index 8fde4fa2c34f..571a9d9ad107 100644
@@ -33,6 +33,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += spinlock.h
 generic-y += topology.h
 generic-y += trace_clock.h
arch/openrisc/include/asm/Kbuild +4 -3
index eb87cd8327c8..5e9f2f4c4d39 100644
@@ -28,12 +28,13 @@ generic-y += module.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
-generic-y += qspinlock_types.h
-generic-y += qspinlock.h
-generic-y += qrwlock_types.h
 generic-y += qrwlock.h
+generic-y += qrwlock_types.h
+generic-y += qspinlock.h
+generic-y += qspinlock_types.h
 generic-y += sections.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += string.h
 generic-y += switch_to.h
 generic-y += topology.h
arch/parisc/include/asm/Kbuild +1 -0
index 2013d639e735..97970b4d05ab 100644
@@ -17,6 +17,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += seccomp.h
 generic-y += segment.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += user.h
arch/powerpc/include/asm/Kbuild +2 -1
index 3196d227e351..64290f48e733 100644
@@ -4,7 +4,8 @@ generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
+generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += vtime.h
-generic-y += msi.h
arch/riscv/include/asm/Kbuild +2 -1
index efdbe311e936..6669b7374c0a 100644
@@ -5,9 +5,9 @@ generic-y += compat.h
 generic-y += cputime.h
 generic-y += device.h
 generic-y += div64.h
-generic-y += dma.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
+generic-y += dma.h
 generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += exec.h
@@ -46,6 +46,7 @@ generic-y += setup.h
 generic-y += shmbuf.h
 generic-y += shmparam.h
 generic-y += signal.h
+generic-y += simd.h
 generic-y += socket.h
 generic-y += sockios.h
 generic-y += stat.h
arch/s390/include/asm/Kbuild +2 -1
index e3239772887a..7a26dc6ce815 100644
@@ -7,9 +7,9 @@ generated-y += unistd_nr.h
 generic-y += asm-offsets.h
 generic-y += cacheflush.h
 generic-y += device.h
+generic-y += div64.h
 generic-y += dma-contiguous.h
 generic-y += dma-mapping.h
-generic-y += div64.h
 generic-y += emergency-restart.h
 generic-y += export.h
 generic-y += fb.h
@@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += preempt.h
 generic-y += rwsem.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += unaligned.h
 generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +1 -0
index 6a5609a55965..8e64ff35a933 100644
@@ -16,6 +16,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += trace_clock.h
 generic-y += xor.h
arch/sparc/include/asm/Kbuild +1 -0
index 410b263ef5c8..72b9e08fb350 100644
@@ -17,5 +17,6 @@ generic-y += msi.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
arch/um/include/asm/Kbuild +2 -1
index b10dde6cb793..d37288b08dd2 100644
@@ -16,15 +16,16 @@ generic-y += io.h
 generic-y += irq_regs.h
 generic-y += irq_work.h
 generic-y += kdebug.h
+generic-y += kprobes.h
 generic-y += mcs_spinlock.h
 generic-y += mm-arch-hooks.h
 generic-y += param.h
 generic-y += pci.h
 generic-y += percpu.h
 generic-y += preempt.h
+generic-y += simd.h
 generic-y += switch_to.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
 generic-y += xor.h
-generic-y += kprobes.h
arch/unicore32/include/asm/Kbuild +1 -0
index bfc7abe77905..98a908720bbd 100644
@@ -27,6 +27,7 @@ generic-y += preempt.h
 generic-y += sections.h
 generic-y += segment.h
 generic-y += serial.h
+generic-y += simd.h
 generic-y += sizes.h
 generic-y += syscalls.h
 generic-y += topology.h
arch/x86/include/asm/simd.h +43 -1

50 lines not on this page · raw message

arch/xtensa/include/asm/Kbuild +1 -0
index 82c756431b49..7950f359649d 100644
@@ -24,6 +24,7 @@ generic-y += percpu.h
 generic-y += preempt.h
 generic-y += rwsem.h
 generic-y += sections.h
+generic-y += simd.h
 generic-y += topology.h
 generic-y += trace_clock.h
 generic-y += word-at-a-time.h
include/asm-generic/simd.h +20 -0
index d0343d58a74a..b3dd61ac010e 100644
@@ -1,5 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 
+#include <linux/simd.h>
+#ifndef _ASM_SIMD_H
+#define _ASM_SIMD_H
+
 #include <linux/hardirq.h>
 
 /*
@@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void)
 {
 	return !in_interrupt();
 }
+
+static inline void simd_get(simd_context_t *ctx)
+{
+	*ctx = HAVE_NO_SIMD;
+}
+
+static inline void simd_put(simd_context_t *ctx)
+{
+}
+
+static __must_check inline bool simd_use(simd_context_t *ctx)
+{
+	return false;
+}
+
+#endif /* _ASM_SIMD_H */
include/linux/simd.h new file +28 -0
new file mode 100644
index 000000000000..33bba21012ff
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _SIMD_H
+#define _SIMD_H
+
+typedef enum {
+	HAVE_NO_SIMD = 1 << 0,
+	HAVE_FULL_SIMD = 1 << 1,
+	HAVE_SIMD_IN_USE = 1 << 31
+} simd_context_t;
+
+#include <linux/sched.h>
+#include <asm/simd.h>
+
+static inline void simd_relax(simd_context_t *ctx)
+{
+#ifdef CONFIG_PREEMPT
+	if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
+		simd_put(ctx);
+		simd_get(ctx);
+	}
+#endif
+}
+
+#endif /* _SIMD_H */
Signature · 1 line
2.19.0
author maintainer
#2 lore raw

PATCH v5 2/20 zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld

Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.

Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:

* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.

* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.

* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.

This last observation has seen itself play out several times over and over again within the kernel:

* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.

* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.

* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.

Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.

It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.

All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].

Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.

SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.

More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.

Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.

Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.

Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.

As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.

[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 MAINTAINERS       |  8 ++++++++
 lib/Kconfig       |  2 ++
 lib/Makefile      |  2 ++
 lib/zinc/Kconfig  | 32 ++++++++++++++++++++++++++++++++
 lib/zinc/Makefile |  4 ++++
 5 files changed, 48 insertions(+)
 create mode 100644 lib/zinc/Kconfig
 create mode 100644 lib/zinc/Makefile
MAINTAINERS +8 -0
index 4327777dce57..5967c737f3ce 100644
@@ -16170,6 +16170,14 @@ Q:	https://patchwork.linuxtv.org/project/linux-media/list/
 S:	Maintained
 F:	drivers/media/dvb-frontends/zd1301_demod*
 
+ZINC CRYPTOGRAPHY LIBRARY
+M:	Jason A. Donenfeld <[email protected]>
+M:	Samuel Neves <[email protected]>
+S:	Maintained
+F:	lib/zinc/
+F:	include/zinc/
+L:	[email protected]
+
 ZPOOL COMPRESSED PAGE STORAGE API
 M:	Dan Streetman <[email protected]>
 L:	[email protected]
lib/Kconfig +2 -0
index a3928d4438b5..3e6848269c66 100644
@@ -485,6 +485,8 @@ config GLOB_SELFTEST
 	  module load) by a small amount, so you're welcome to play with
 	  it, but you probably don't need it.
 
+source "lib/zinc/Kconfig"
+
 #
 # Netlink attribute parsing support is select'ed if needed
 #
lib/Makefile +2 -0
index ca3f7ebb900d..571d28d3f143 100644
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o
 
 obj-$(CONFIG_ASN1) += asn1_decoder.o
 
+obj-y += zinc/
+
 obj-$(CONFIG_FONT_SUPPORT) += fonts/
 
 obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +32 -0
new file mode 100644
index 000000000000..4e2e59126a67
@@ -0,0 +1,32 @@
+config ZINC_DEBUG
+	bool "Zinc cryptography library debugging and self-tests"
+	help
+	  This builds a series of self-tests for the Zinc crypto library, which
+	  help diagnose any cryptographic algorithm implementation issues that
+	  might be at the root cause of potential bugs. It also adds various
+	  debugging traps.
+
+	  Unless you're developing and testing cryptographic routines, or are
+	  especially paranoid about correctness on your hardware, you may say
+	  N here.
+
+config ZINC_ARCH_ARM
+	def_bool y
+	depends on ARM && !64BIT
+
+config ZINC_ARCH_ARM64
+	def_bool y
+	depends on ARM64 && 64BIT
+
+config ZINC_ARCH_X86_64
+	def_bool y
+	depends on X86_64 && 64BIT
+	depends on !UML
+
+config ZINC_ARCH_MIPS
+	def_bool y
+	depends on MIPS && CPU_MIPS32_R2 && !64BIT
+
+config ZINC_ARCH_MIPS64
+	def_bool y
+	depends on MIPS && 64BIT
lib/zinc/Makefile new file +4 -0
new file mode 100644
index 000000000000..83dfd63988c0
@@ -0,0 +1,4 @@
+ccflags-y := -O3
+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
Signature · 1 line
2.19.0
#3 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

(+ Arnd, Eric)

On 18 September 2018 at 09:16, Jason A. Donenfeld <[email protected]> wrote:
...

lib/zinc/Makefile

Quoted 6 lines
lib/zinc/Makefile new file +0 -0
new file mode 100644
index 000000000000..83dfd63988c0
@@ -0,0 +1,4 @@

Apologies for not spotting these before:

lib/zinc/Makefile line 1

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y := -O3

-O3 optimization has been problematic in the past, at least on x86 but I think on other architectures as well. Please stick with -O2.

lib/zinc/Makefile line 2

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)

There is no way we can support code in the kernel with that kind of stack space requirements. I will let Arnd comment on what we typically allow, since he deals with such issues on a regular basis.

Quoted 5 lines
lib/zinc/Makefile new file +2 -1
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
--

2.19.0

#4 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Quoted 20 lines · Ard Biesheuvel

(+ Arnd, Eric)

On 18 September 2018 at 09:16, Jason A. Donenfeld <[email protected]> wrote:
...

lib/zinc/Makefile

Quoted 6 lines
lib/zinc/Makefile new file +0 -0
new file mode 100644
index 000000000000..83dfd63988c0
@@ -0,0 +1,4 @@

Apologies for not spotting these before:

lib/zinc/Makefile line 1

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y := -O3
Quoted 8 lines

-O3 optimization has been problematic in the past, at least on x86 but I think on other architectures as well. Please stick with -O2.

lib/zinc/Makefile line 2

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)

There is no way we can support code in the kernel with that kind of stack space requirements. I will let Arnd comment on what we typically allow, since he deals with such issues on a regular basis.

To make matters worse, KASAN is incompatible with VMAP_STACK right now, and KASAN is not so good at detecting stack overflow.

Quoted 6 lines
Quoted 6 lines
lib/zinc/Makefile new file +2 -1
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
--

2.19.0

maintainer
#5 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Ard Biesheuvel
Quoted 17 lines · Ard Biesheuvel

(+ Arnd, Eric)

On 18 September 2018 at 09:16, Jason A. Donenfeld <[email protected]> wrote:
...

lib/zinc/Makefile

Quoted 6 lines
lib/zinc/Makefile new file +0 -0
new file mode 100644
index 000000000000..83dfd63988c0
@@ -0,0 +1,4 @@

Apologies for not spotting these before:

lib/zinc/Makefile line 1

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y := -O3
Quoted 8 lines

-O3 optimization has been problematic in the past, at least on x86 but I think on other architectures as well. Please stick with -O2.

lib/zinc/Makefile line 2

Quoted 1 line
lib/zinc/Makefile new file +1 -0
+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)

There is no way we can support code in the kernel with that kind of stack space requirements. I will let Arnd comment on what we typically allow, since he deals with such issues on a regular basis.

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Arnd

author maintainer
#6 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hey Arnd,

Quoted 9 lines · Arnd Bergmann

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

Quoted 5 lines

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Thanks for the explanation. Fortunately in my case, the issues were trivially fixable to get it under 1024/1280. (By the way, why does 64-bit have a slightly larger stack frame? To account for 32 pointers taking double the space or something?) That will be rectified in v6.

There is one exception though: sometimes KASAN bloats the frame on 64-bit compiles. How would you feel about me adding 'ccflags-$(CONFIG_KASAN) += -Wframe-larger-than=16384' in my makefile? I'm not remotely close to reaching that limit (it's just a tiny bit over 1280), but it does seem like telling gcc to quiet down about stack frames when KASAN is being used might make sense. Alternatively, I see the defaults for FRAME_WARN are:

config FRAME_WARN

       int "Warn for stack frames larger than (needs gcc 4.4)"
       range 0 8192
       default 3072 if KASAN_EXTRA
       default 2048 if GCC_PLUGIN_LATENT_ENTROPY
       default 1280 if (!64BIT && PARISC)
       default 1024 if (!64BIT && !PARISC)
       default 2048 if 64BIT

What about changing that KASAN_EXTRA into just KASAN? This seems cleanest; I'll send a patch for it.

On the other hand, this KASAN behavior is only observable on 64-bit systems when I force it to be 1280, whereas the default is still 2048, so probably this isn't a problem *now* for this patchset. But it is something to think about for down the road when you lower the default frame.

Regards,
Jason

#7 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Quoted 18 lines · Jason A. Donenfeld

Hey Arnd,

Quoted 9 lines · Arnd Bergmann

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

Quoted 5 lines

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Quoted 4 lines

Thanks for the explanation. Fortunately in my case, the issues were trivially fixable to get it under 1024/1280. (By the way, why does 64-bit have a slightly larger stack frame? To account for 32 pointers taking double the space or something?) That will be rectified in v6.

Hi Jason

Do you any stack usage information?

A VPN can be at the bottom of some deep stack calls. Swap on NFS over the VPN? If you have one frame of 1K, you might be O.K. But if you have a few of these, i can see there might be issues of overflowing the stack.

Andrew

author maintainer
#8 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hi Andrew,

Quoted 6 lines · Andrew Lunn

Do you any stack usage information?

A VPN can be at the bottom of some deep stack calls. Swap on NFS over the VPN? If you have one frame of 1K, you might be O.K. But if you have a few of these, i can see there might be issues of overflowing the stack.

It's pretty small, and most of the work is punted onto workqueues pretty early, so the depth isn't as much of an issue as it would be if I did everything from ndo_start_xmit. I've done some stack tracking using the debug option for that, and it's generally far from the deepest thing running.

Jason

#9 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Andrew Lunn
Quoted 27 lines · Andrew Lunn
Quoted 20 lines · Jason A. Donenfeld

Hey Arnd,

Quoted 10 lines · Arnd Bergmann

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

Quoted 5 lines

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Quoted 4 lines

Thanks for the explanation. Fortunately in my case, the issues were trivially fixable to get it under 1024/1280. (By the way, why does 64-bit have a slightly larger stack frame? To account for 32 pointers taking double the space or something?) That will be rectified in v6.

Quoted 10 lines

Hi Jason

Do you any stack usage information?

A VPN can be at the bottom of some deep stack calls. Swap on NFS over the VPN? If you have one frame of 1K, you might be O.K. But if you have a few of these, i can see there might be issues of overflowing the stack.

At the risk on suggesting something awful: on x86_64, since we turn preemption off for simd, it wouldn’t be *completely* insane to do the crypto on the irq stack. It would look like:

kernel_fpu_call(func, arg);

And this helper would disable preemption, enable FPU, switch to the irq stack, call func(arg), disable FPU, enable preemption, and return. And we can have large IRQ stacks.

I refuse to touch this with a ten-foot pole until the lazy FPU restore patches land.

All that being said, why are these frames so large? It sounds like something may be spilling that ought not to.

author maintainer
#10 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hi Andy,

Quoted 7 lines · Andy Lutomirski

At the risk on suggesting something awful: on x86_64, since we turn preemption off for simd, it wouldn’t be *completely* insane to do the crypto on the irq stack. It would look like:

kernel_fpu_call(func, arg);

And this helper would disable preemption, enable FPU, switch to the irq stack, call func(arg), disable FPU, enable preemption, and return. And we can have large IRQ stacks.

I refuse to touch this with a ten-foot pole until the lazy FPU restore patches land.

Haha. That's fun, and maybe we'll do that at some point, but I have some other reasons too for being on a workqueue now.

Quoted 2 lines

All that being said, why are these frames so large? It sounds like something may be spilling that ought not to.

They're not. Well, they're not anymore. I had a silly thing before like "u8 buffer[1 << 12]" in some debugging code, which is what prompted the ccflag-y addition. I cleaned up the mistakes like that and frames are now reasonable everywhere. Non-issue.

Jason

#11 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Quoted 5 lines · Jason A. Donenfeld

Hi Andy,

Quoted 2 lines · Andy Lutomirski

At the risk on suggesting something awful: on x86_64, since we turn preemption off for simd, it wouldn’t be *completely* insane to do the crypto on the irq stack. It would look like:

Quoted 9 lines
Quoted 5 lines

kernel_fpu_call(func, arg);

And this helper would disable preemption, enable FPU, switch to the irq stack, call func(arg), disable FPU, enable preemption, and return. And we can have large IRQ stacks.

I refuse to touch this with a ten-foot pole until the lazy FPU restore patches land.

Haha. That's fun, and maybe we'll do that at some point, but I have some other reasons too for being on a workqueue now.

Kernel mode crypto is callable from any context, and SIMD can be used in softirq context on arm64 (and on x86, even from hardirq context IIRC if the interrupt is taken from userland), in which case we'd already be on the irq stack.

Quoted 8 lines
Quoted 2 lines

All that being said, why are these frames so large? It sounds like something may be spilling that ought not to.

They're not. Well, they're not anymore. I had a silly thing before like "u8 buffer[1 << 12]" in some debugging code, which is what prompted the ccflag-y addition. I cleaned up the mistakes like that and frames are now reasonable everywhere. Non-issue.

Did any of the fuzzing/testing you mention in the cover letter occur on the kernel versions of these algorithms?

author maintainer
#12 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hi Ard,

Quoted 2 lines · Ard Biesheuvel

Did any of the fuzzing/testing you mention in the cover letter occur on the kernel versions of these algorithms?

Yes, of course.

Jason

#13 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Ard Biesheuvel
Quoted 16 lines · Ard Biesheuvel
Quoted 13 lines · Jason A. Donenfeld

Hi Andy,

Quoted 8 lines · Andy Lutomirski

At the risk on suggesting something awful: on x86_64, since we turn preemption off for simd, it wouldn’t be *completely* insane to do the crypto on the irq stack. It would look like:

kernel_fpu_call(func, arg);

And this helper would disable preemption, enable FPU, switch to the irq stack, call func(arg), disable FPU, enable preemption, and return. And we can have large IRQ stacks.

I refuse to touch this with a ten-foot pole until the lazy FPU restore patches land.

Quoted 8 lines
Quoted 3 lines

Haha. That's fun, and maybe we'll do that at some point, but I have some other reasons too for being on a workqueue now.

Kernel mode crypto is callable from any context, and SIMD can be used in softirq context on arm64 (and on x86, even from hardirq context IIRC if the interrupt is taken from userland), in which case we'd already be on the irq stack.

The x86_64 irq stack handles nesting already.

maintainer
#14 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld
Quoted 13 lines · Jason A. Donenfeld

Hey Arnd,

Quoted 9 lines · Arnd Bergmann

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

Quoted 8 lines
Quoted 5 lines

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Thanks for the explanation. Fortunately in my case, the issues were trivially fixable to get it under 1024/1280

A lot of these bugs are not trivial, but we still need a full analysis of what failed and what the possible mititgations are. Can you describe more specifically what causes it?

Quoted 3 lines

(By the way, why does
64-bit have a slightly larger stack frame? To account for 32 pointers taking double the space or something?) That will be rectified in v6.

Correct. We have some functions in the kernel that take 128 pointers on the stack. This fits fine within the 1024 byte limit on 32 bit, but goes beyond the same limit on 64 bit. Everything that took more than 1280 bytes (without KASAN) in my analysis was a real problem and reducing the stack size fixed a bug or improved the code in other ways. I never got around to reposting all those bug fixes I made back then, and there are probably a couple new ones we'd need.

Quoted 3 lines

There is one exception though: sometimes KASAN bloats the frame on 64-bit compiles. How would you feel about me adding 'ccflags-$(CONFIG_KASAN) += -Wframe-larger-than=16384' in my makefile?

I ran into several ciphers in the kernel that required a large stack allocation, with or without KASAN. Typically these turned out to be optimization bugs in gcc where gcc tried to be smart about optimizing part of the cipher that was intentionally designed to not be optimizable, but in the process it ended up spilling registers to the stack. Adding KASAN into the mix then added an extra redzone around each of those.

My best approach here was to figure out which optimization step in gcc caused the register spilling in the first place, and adjust the set of optimizations done to a specific function in that cipher.

Note that some crypto ciphers are rather performance sensitive and we have users that care about every percent of performance one can get out. The bugs that cause the stack size explode beyond a few hundred bytes that one might expect are usually the same bugs that make the ciphers much slower than they should be even without KASAN.

For a cryptographic library it might therefore even make sense to have an even smaller limit (e.g. 400 bytes) in the Makefile and then audit each case that does not fit into that after determining what exactly caused the problem, and making sure we find workaround that produces optmized object code. When you end up with a function that gcc-8 still misoptimizes in all optmization levels and you find no other workaround, please set the higher limit specifically for that function with a gcc version check in it, and add a comment pointing to the gcc bugzilla report you have. gcc developers tend to be good about fixing these issues in new compilers, but the version check helps to ensure we find it if a future version regresses again, and of course we have to make sure we apply the hack on all unpatched compilers.

Quoted 14 lines

I'm not remotely close to reaching that limit (it's just a tiny bit over 1280), but it does seem like telling gcc to quiet down about stack frames when KASAN is being used might make sense. Alternatively, I see the defaults for FRAME_WARN are:

config FRAME_WARN

       int "Warn for stack frames larger than (needs gcc 4.4)"
       range 0 8192
       default 3072 if KASAN_EXTRA
       default 2048 if GCC_PLUGIN_LATENT_ENTROPY
       default 1280 if (!64BIT && PARISC)
       default 1024 if (!64BIT && !PARISC)
       default 2048 if 64BIT
Quoted 2 lines

What about changing that KASAN_EXTRA into just KASAN? This seems cleanest; I'll send a patch for it.

I saw you already posted a patch for that, sorry for not being able to reply earlier. I actually spent several months of my work to introduce the KASAN_EXTRA option specifically in order to have a separate stack size limit for that, so I definitely object to that one. We need the default (allmodconfig) stack size check to be as small as we can in order to catch kernel and gcc bugs that bloat the stack more than it is currently at.

Quoted 5 lines

On the other hand, this KASAN behavior is only observable on 64-bit systems when I force it to be 1280, whereas the default is still 2048, so probably this isn't a problem *now* for this patchset. But it is something to think about for down the road when you lower the default frame.

Right. Some background about that limit: 2048 has been what we used traditionally on 64-bit before KASAN. When I first saw the problems with KASAN, I had a more detailed look at it and found that we can significantly reduce it without KASAN, but obviously KASAN needs a little extra space. I got all my patches upstream to get the KASAN limit below 2048 and remove that special case for it, and I sent a lot of the other patches to reduce the limit further to 1280/1535 without/with KASAN, but not all those patches got accepted, so for the moment that traditional limit is still in place.

clang with KASAN still has known bugs that make it go beyond 2048 bytes, as I said in the other thread, and this is something that first needs to be addressed in clang before we can take specific measures in the kernel to work around any remaining unrelated bugs with clang building our kernel with KASAN.

Arnd

maintainer
#15 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Quoted 21 lines · Arnd Bergmann
Quoted 19 lines · Jason A. Donenfeld

Hey Arnd,

Quoted 14 lines · Arnd Bergmann

Right, if you hit a stack requirement like this, it's usually the compiler doing something bad, not just using too much stack but also generating rather slow object code in the process. It's better to fix the bug by optimizing the code to not spill registers to the stack.

In the long run, I'd like to reduce the stack frame size further, so best assume that anything over 1024 bytes (on 32-bit) or 1280 bytes (on 64-bit) is a bug in the code, and stay below that.

For prototyping, you can just mark the broken functions individually by setting the warning limit for a specific function that is known to be misoptimized by the compiler (with a comment about which compiler and architectures are affected), but not override the limit for the entire file.

Quoted 6 lines
Quoted 2 lines

Thanks for the explanation. Fortunately in my case, the issues were trivially fixable to get it under 1024/1280

A lot of these bugs are not trivial, but we still need a full analysis of what failed and what the possible mititgations are. Can you describe more specifically what causes it?

I think I misread your earlier sentence and thought you had said the exact opposite.

For confirmation, I've downloaded your git tree and built it with my collection of compilers (gcc-4.6 through 8.1) and tried building it in various configurations. Nothing alarming stood out, the only thing that I think would might warrant some investigation is this one:

lib/zinc/curve25519/curve25519-hacl64.h: In function 'curve25519_generic':
lib/zinc/curve25519/curve25519-hacl64.h:785:1: warning: the frame size of 1536 bytes is larger than 500 bytes [-Wframe-larger-than=]

Without KASAN, this takes 832 bytes, which is still more than it should use from a look at the source code.

I first suspected some misoptimization around the get/put_unaligned_le64() calls, but playing around with it some more led me to this patch:

(quoted diff) +0 -0

b/lib/zinc/curve25519/curve25519-hacl64.h
index c7b2924a68c2..1f6eb5708a0e 100644

lib/zinc/curve25519/curve25519-hacl64.h +0 -0
@@ -182,8 +182,7 @@ static __always_inline void

fmul_mul_shift_reduce_(u128 *output, u64 *input,

static __always_inline void fmul_fmul(u64 *output, u64 *input, u64 *input21) {

-       u64 tmp[5];
-       memcpy(tmp, input, 5 * sizeof(*input));
+       u64 tmp[5] = { input[0], input[1], input[2], input[3], input[4] };
        {
                u128 b4;
                u128 b0;

That change gets it down to

lib/zinc/curve25519/curve25519-hacl64.h: In function 'curve25519_generic':
lib/zinc/curve25519/curve25519-hacl64.h:788:1: warning: the frame size of 640 bytes is larger than 500 bytes [-Wframe-larger-than=]

with KASAN, or 496 bytes without it. This indicates that there might be something wrong with either gcc-8 or with our fortified memset() function that requires more investigation. Maybe you can see something there that I missed.

Arnd

author maintainer
#16 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hey Arnd,

Quoted 12 lines · Arnd Bergmann

I think I misread your earlier sentence and thought you had said the exact opposite.

For confirmation, I've downloaded your git tree and built it with my collection of compilers (gcc-4.6 through 8.1) and tried building it in various configurations. Nothing alarming stood out, the only thing that I think would might warrant some investigation is this one:

lib/zinc/curve25519/curve25519-hacl64.h: In function 'curve25519_generic':
lib/zinc/curve25519/curve25519-hacl64.h:785:1: warning: the frame size of 1536 bytes is larger than 500 bytes [-Wframe-larger-than=]

Quoted 5 lines

Without KASAN, this takes 832 bytes, which is still more than it should use from a look at the source code.

I first suspected some misoptimization around the get/put_unaligned_le64() calls, but playing around with it some more led me to this patch:

Excellent detective work. Thanks for spotting that before I had a time to respond here. I'll also send that same fix to the HACL* team at INRIA, so they can add it as a heuristic.

Jason

author maintainer
#17 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Ard Biesheuvel

Hi Ard,

Quoted 6 lines · Ard Biesheuvel

Apologies for not spotting these before:

Quoted 1 line

+ccflags-y := -O3

-O3 optimization has been problematic in the past, at least on x86 but I think on other architectures as well. Please stick with -O2.

Fixed up for v6.

Quoted 6 lines
Quoted 2 lines

+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)

There is no way we can support code in the kernel with that kind of stack space requirements. I will let Arnd comment on what we typically allow, since he deals with such issues on a regular basis.

Also fixed up for v6 by just removing that line entirely.

Jason

#18 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

In reply to Jason A. Donenfeld

Hi Jason,

I have another couple of questions which are unrelated to the below, so I will just respond again to the top message.

Quoted 8 lines · Jason A. Donenfeld

Zinc stands for "Zinc Is Neat Crypto" or "Zinc as IN Crypto" or maybe just "Zx2c4's INsane Cryptolib." It's also short, easy to type, and plays nicely with the recent trend of naming crypto libraries after elements. The guiding principle is "don't overdo it". It's less of a library and more of a directory tree for organizing well-curated direct implementations of cryptography primitives.

Kees is currently dealing with VLA uses in crypto API skcipher invocations [0] that don't benefit from its async capabilities nor from the runtime resolution of cipher name strings, given that they always select the same one.

drivers/net/ppp/ppp_mppe.c: "ecb(arc4)" drivers/usb/wusbcore/crypto.c: "cbc(aes)" net/ceph/crypto.c: "cbc(aes)" net/mac802154/llsec.c: "ctr(aes)" net/rxrpc/rxkad.c: "pcbc(fcrypt)" net/rxrpc/rxkad.c: "pcbc(fcrypt)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cbc(des)" net/sunrpc/auth_gss/gss_krb5_mech.c: "ecb(arc4)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cbc(des3_ede)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cts(cbc(aes))" net/sunrpc/auth_gss/gss_krb5_mech.c: "cts(cbc(aes))" net/wireless/lib80211_crypt_tkip.c: "ecb(arc4)" net/wireless/lib80211_crypt_wep.c: "ecb(arc4)"

To me, these are prime candidates for moving into your library [at some point]. I guess AES should be non-controversial, but moving the others is actually more important in my view, since we will be able to stop exposing them via the crypto API in that case. Any thoughts?

Quoted 3 lines

[From 00/20] ..., WireGuard patches will go through
DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.

Given the cross dependencies, I'd prefer it if we take these through Herbert's tree [which feeds into DaveM's tree as well].

Also, you haven't yet responded to my question about WireGuard's limitation to synchronous encryption, or whether and how you expect to support asynchronous accelerators for ChaCha20/Poly1305 in the future. This shouldn't impede adoption of this series, but this is something that is going to come up sooner than you think, and so I would like to understand whether this means your library will grow asynchronous interfaces as well, or whether it will be moved to the crypto API. (Also, I'd like to know whether the RFC7539 construction of ChaCha20 and Poly1305 is compatible with WireGuard's)

Thanks,
Ard.

[0] https://marc.info/?l=linux-kernel&m=153732317132090&w=2

Quoted 245 lines

Zinc is a new cryptography API that is much more minimal and lower-level than the current one. It intends to complement it and provide a basis upon which the current crypto API might build, as the provider of software implementations of cryptographic primitives. It is motivated by three primary observations in crypto API design:

* Highly composable "cipher modes" and related abstractions from 90s cryptographers did not turn out to be as terrific an idea as hoped, leading to a host of API misuse problems.

* Most programmers are afraid of crypto code, and so prefer to integrate it into libraries in a highly abstracted manner, so as to shield themselves from implementation details. Cryptographers, on the other hand, prefer simple direct implementations, which they're able to verify for high assurance and optimize in accordance with their expertise.

* Overly abstracted and flexible cryptography APIs lead to a host of dangerous problems and performance issues. The kernel is in the business usually not of coming up with new uses of crypto, but rather implementing various constructions, which means it essentially needs a library of primitives, not a highly abstracted enterprise-ready pluggable system, with a few particular exceptions.

This last observation has seen itself play out several times over and over again within the kernel:

* The perennial move of actual primitives away from crypto/ and into lib/, so that users can actually call these functions directly with no overhead and without lots of allocations, function pointers, string specifier parsing, and general clunkiness. For example:
sha256, chacha20, siphash, sha1, and so forth live in lib/ rather than in crypto/. Zinc intends to stop the cluttering of lib/ and introduce these direct primitives into their proper place, lib/zinc/.

* An abundance of misuse bugs with the present crypto API that have been very unpleasant to clean up.

* A hesitance to even use cryptography, because of the overhead and headaches involved in accessing the routines.

Zinc goes in a rather different direction. Rather than providing a thoroughly designed and abstracted API, Zinc gives you simple functions, which implement some primitive, or some particular and specific construction of primitives. It is not dynamic in the least, though one could imagine implementing a complex dynamic dispatch mechanism (such as the current crypto API) on top of these basic functions. After all, dynamic dispatch is usually needed for applications with cipher agility, such as IPsec, dm-crypt, AF_ALG, and so forth, and the existing crypto API will continue to play that role. However, Zinc will provide a non- haphazard way of directly utilizing crypto routines in applications that do have neither the need nor desire for abstraction and dynamic dispatch.

It also organizes the implementations in a simple, straight-forward, and direct manner, making it enjoyable and intuitive to work on. Rather than moving optimized assembly implementations into arch/, it keeps them all together in lib/zinc/, making it simple and obvious to compare and contrast what's happening. This is, notably, exactly what the lib/raid6/ tree does, and that seems to work out rather well. It's also the pattern of most successful crypto libraries. The architecture- specific glue-code is made a part of each translation unit, rather than being in a separate one, so that generic and architecture-optimized code are combined at compile-time, and incompatibility branches compiled out by the optimizer.

All implementations have been extensively tested and fuzzed, and are selected for their quality, trustworthiness, and performance. Wherever possible and performant, formally verified implementations are used, such as those from HACL* [1] and Fiat-Crypto [2]. The routines also take special care to zero out secrets using memzero_explicit (and future work is planned to have gcc do this more reliably and performantly with compiler plugins). The performance of the selected implementations is state-of-the-art and unrivaled on a broad array of hardware, though of course we will continue to fine tune these to the hardware demands needed by kernel contributors. Each implementation also comes with extensive self-tests and crafted test vectors, pulled from various places such as Wycheproof [9].

Regularity of function signatures is important, so that users can easily "guess" the name of the function they want. Though, individual primitives are oftentimes not trivially interchangeable, having been designed for different things and requiring different parameters and semantics, and so the function signatures they provide will directly reflect the realities of the primitives' usages, rather than hiding it behind (inevitably leaky) abstractions. Also, in contrast to the current crypto API, Zinc functions can work on stack buffers, and can be called with different keys, without requiring allocations or locking.

SIMD is used automatically when available, though some routines may benefit from either having their SIMD disabled for particular invocations, or to have the SIMD initialization calls amortized over several invocations of the function, and so Zinc utilizes function signatures enabling that in conjunction with the recently introduced simd_context_t.

More generally, Zinc provides function signatures that allow just what is required by the various callers. This isn't to say that users of the functions will be permitted to pollute the function semantics with weird particular needs, but we are trying very hard not to overdo it, and that means looking carefully at what's actually necessary, and doing just that, and not much more than that. Remember: practicality and cleanliness rather than over-zealous infrastructure.

Zinc provides also an opening for the best implementers in academia to contribute their time and effort to the kernel, by being sufficiently simple and inviting. In discussing this commit with some of the best and brightest over the last few years, there are many who are eager to devote rare talent and energy to this effort.

Following the merging of this, I expect for the primitives that currently exist in lib/ to work their way into lib/zinc/, after intense scrutiny of each implementation, potentially replacing them with either formally-verified implementations, or better studied and faster state-of-the-art implementations.

Also following the merging of this, I expect for the old crypto API implementations to be ported over to use Zinc for their software-based implementations.

As Zinc is simply library code, its config options are un-menued, with the exception of CONFIG_ZINC_DEBUG, which enables various selftests and BUG_ONs.

[1] https://github.com/project-everest/hacl-star
[2] https://github.com/mit-plv/fiat-crypto
[3] https://cr.yp.to/ecdh.html
[4] https://cr.yp.to/chacha.html
[5] https://cr.yp.to/snuffle/xsalsa-20081128.pdf
[6] https://cr.yp.to/mac.html
[7] https://blake2.net/
[8] https://tools.ietf.org/html/rfc8439
[9] https://github.com/google/wycheproof

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 MAINTAINERS       |  8 ++++++++
 lib/Kconfig       |  2 ++
 lib/Makefile      |  2 ++
 lib/zinc/Kconfig  | 32 ++++++++++++++++++++++++++++++++
 lib/zinc/Makefile |  4 ++++
 5 files changed, 48 insertions(+)
 create mode 100644 lib/zinc/Kconfig
 create mode 100644 lib/zinc/Makefile
MAINTAINERS +8 -0
index 4327777dce57..5967c737f3ce 100644
@@ -16170,6 +16170,14 @@ Q:     https://patchwork.linuxtv.org/project/linux-media/list/
 S:     Maintained
 F:     drivers/media/dvb-frontends/zd1301_demod*
 
+ZINC CRYPTOGRAPHY LIBRARY
+M:     Jason A. Donenfeld <[email protected]>
+M:     Samuel Neves <[email protected]>
+S:     Maintained
+F:     lib/zinc/
+F:     include/zinc/
+L:     [email protected]
+
 ZPOOL COMPRESSED PAGE STORAGE API
 M:     Dan Streetman <[email protected]>
 L:     [email protected]
lib/Kconfig +2 -0
index a3928d4438b5..3e6848269c66 100644
@@ -485,6 +485,8 @@ config GLOB_SELFTEST
          module load) by a small amount, so you're welcome to play with
          it, but you probably don't need it.
 
+source "lib/zinc/Kconfig"
+
 #
 # Netlink attribute parsing support is select'ed if needed
 #
lib/Makefile +2 -0
index ca3f7ebb900d..571d28d3f143 100644
@@ -214,6 +214,8 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o
 
 obj-$(CONFIG_ASN1) += asn1_decoder.o
 
+obj-y += zinc/
+
 obj-$(CONFIG_FONT_SUPPORT) += fonts/
 
 obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o
lib/zinc/Kconfig new file +32 -0
new file mode 100644
index 000000000000..4e2e59126a67
@@ -0,0 +1,32 @@
+config ZINC_DEBUG
+       bool "Zinc cryptography library debugging and self-tests"
+       help
+         This builds a series of self-tests for the Zinc crypto library, which
+         help diagnose any cryptographic algorithm implementation issues that
+         might be at the root cause of potential bugs. It also adds various
+         debugging traps.
+
+         Unless you're developing and testing cryptographic routines, or are
+         especially paranoid about correctness on your hardware, you may say
+         N here.
+
+config ZINC_ARCH_ARM
+       def_bool y
+       depends on ARM && !64BIT
+
+config ZINC_ARCH_ARM64
+       def_bool y
+       depends on ARM64 && 64BIT
+
+config ZINC_ARCH_X86_64
+       def_bool y
+       depends on X86_64 && 64BIT
+       depends on !UML
+
+config ZINC_ARCH_MIPS
+       def_bool y
+       depends on MIPS && CPU_MIPS32_R2 && !64BIT
+
+config ZINC_ARCH_MIPS64
+       def_bool y
+       depends on MIPS && 64BIT
lib/zinc/Makefile new file +4 -1
new file mode 100644
index 000000000000..83dfd63988c0
@@ -0,0 +1,4 @@
+ccflags-y := -O3
+ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)
+ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
+ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
--

2.19.0

author maintainer
#19 lore raw

Re: [PATCH net-next v5 02/20] zinc: introduce minimal cryptography library

Hey Ard,

Quoted 19 lines · Ard Biesheuvel

Kees is currently dealing with VLA uses in crypto API skcipher invocations [0] that don't benefit from its async capabilities nor from the runtime resolution of cipher name strings, given that they always select the same one.

drivers/net/ppp/ppp_mppe.c: "ecb(arc4)" drivers/usb/wusbcore/crypto.c: "cbc(aes)" net/ceph/crypto.c: "cbc(aes)" net/mac802154/llsec.c: "ctr(aes)" net/rxrpc/rxkad.c: "pcbc(fcrypt)" net/rxrpc/rxkad.c: "pcbc(fcrypt)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cbc(des)" net/sunrpc/auth_gss/gss_krb5_mech.c: "ecb(arc4)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cbc(des3_ede)" net/sunrpc/auth_gss/gss_krb5_mech.c: "cts(cbc(aes))" net/sunrpc/auth_gss/gss_krb5_mech.c: "cts(cbc(aes))" net/wireless/lib80211_crypt_tkip.c: "ecb(arc4)" net/wireless/lib80211_crypt_wep.c: "ecb(arc4)"

Quoted 4 lines

To me, these are prime candidates for moving into your library [at some point]. I guess AES should be non-controversial, but moving the others is actually more important in my view, since we will be able to stop exposing them via the crypto API in that case. Any thoughts?

In order of priority, I'll probably tackle lib/ first and then the cases like you mentioned after. Indeed AES is an obvious candidate. For the others, we'll evaluate them on a case-by-case basis. For example, Ted T'so's "halfmd4" algorithm was moved from lib/ directly into that portion of the ext4 driver, since it's some "half"-baked random crypto that should only be used in that one place and then never again. On the other hand, it seems likely RC4 and DES are used multiple places, and so we'll have to carefully evaluate these. We can also discuss this in November and see where thoughts are at that time.

Quoted 7 lines

Also, you haven't yet responded to my question about WireGuard's limitation to synchronous encryption, or whether and how you expect to support asynchronous accelerators for ChaCha20/Poly1305 in the future. This shouldn't impede adoption of this series, but this is something that is going to come up sooner than you think, and so I would like to understand whether this means your library will grow asynchronous interfaces as well, or whether it will be moved to the crypto API.

I have no concrete plans to introduce an asynchronous interface to Zinc at this time, but that could change at some later date. At the moment however, I prefer for it to be just a simple collection of software ciphers, just as the description reads. Regarding hardware acceleration in WireGuard: I've actually been talking to some people interested in producing these types of ASICs lately, and hopefully something cool will come out of it. It's not obvious, however, that this _must_ imply an asynchronous interface, even though that may very well seem like the intuitive thing. This is, as well, a discussion for the future indeed.

Quoted 2 lines

(Also, I'd like to know whether the RFC7539 construction of ChaCha20 and Poly1305 is compatible with WireGuard's)

WireGuard uses 64-bit nonces, but since they're both little-endian, and because of the maximum size of a series of IP fragments (namely, less than 2^32), they're "compatible".

Jason

author maintainer
#20 lore raw

PATCH v5 3/20 zinc: ChaCha20 generic C implementation and selftest

In reply to Jason A. Donenfeld

This implements the ChaCha20 permutation as a single C statement, by way of the comma operator, which the compiler is able to simplify terrifically.

Information: https://cr.yp.to/chacha.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20.h      |   52 ++
 lib/zinc/Kconfig             |    4 +
 lib/zinc/Makefile            |    3 +
 lib/zinc/chacha20/chacha20.c |  193 ++++++
 lib/zinc/selftest/chacha20.h | 1182 ++++++++++++++++++++++++++++++++++
 5 files changed, 1434 insertions(+)
 create mode 100644 include/zinc/chacha20.h
 create mode 100644 lib/zinc/chacha20/chacha20.c
 create mode 100644 lib/zinc/selftest/chacha20.h
include/zinc/chacha20.h new file +52 -0
new file mode 100644
index 000000000000..afad69409f5f
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20_H
+#define _ZINC_CHACHA20_H
+
+#include <asm/unaligned.h>
+#include <linux/simd.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+enum {
+	CHACHA20_IV_SIZE = 16,
+	CHACHA20_KEY_SIZE = 32,
+	CHACHA20_BLOCK_SIZE = 64,
+	CHACHA20_BLOCK_WORDS = CHACHA20_BLOCK_SIZE / sizeof(u32),
+	HCHACHA20_KEY_SIZE = 32,
+	HCHACHA20_NONCE_SIZE = 16
+};
+
+struct chacha20_ctx {
+	u32 key[8];
+	u32 counter[4];
+} __aligned(32);
+
+static inline void chacha20_init(struct chacha20_ctx *state,
+				 const u8 key[CHACHA20_KEY_SIZE],
+				 const u64 nonce)
+{
+	state->key[0] = get_unaligned_le32(key + 0);
+	state->key[1] = get_unaligned_le32(key + 4);
+	state->key[2] = get_unaligned_le32(key + 8);
+	state->key[3] = get_unaligned_le32(key + 12);
+	state->key[4] = get_unaligned_le32(key + 16);
+	state->key[5] = get_unaligned_le32(key + 20);
+	state->key[6] = get_unaligned_le32(key + 24);
+	state->key[7] = get_unaligned_le32(key + 28);
+	state->counter[0] = state->counter[1] = 0;
+	state->counter[2] = nonce & U32_MAX;
+	state->counter[3] = nonce >> 32;
+}
+void chacha20(struct chacha20_ctx *state, u8 *dst, const u8 *src, u32 len,
+	      simd_context_t *simd_context);
+
+/* Derived key should be 32-bit aligned */
+void hchacha20(u8 derived_key[CHACHA20_KEY_SIZE],
+	       const u8 nonce[HCHACHA20_NONCE_SIZE],
+	       const u8 key[HCHACHA20_KEY_SIZE], simd_context_t *simd_context);
+
+#endif /* _ZINC_CHACHA20_H */
lib/zinc/Kconfig +4 -0
index 4e2e59126a67..1ca1ae1e9ea9 100644
@@ -1,3 +1,7 @@
+config ZINC_CHACHA20
+	tristate
+	select CRYPTO_ALGAPI
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index 83dfd63988c0..83b320f1ace2 100644
@@ -2,3 +2,6 @@ ccflags-y := -O3
 ccflags-y += -Wframe-larger-than=$(if (CONFIG_KASAN),16384,8192)
 ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
+
+zinc_chacha20-y := chacha20/chacha20.o
+obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20.c new file +193 -0

193 lines not on this page · raw message

lib/zinc/selftest/chacha20.h new file +1182 -0

1,182 lines not on this page · raw message

Signature · 1 line
2.19.0
#21 lore raw

Re: [PATCH net-next v5 03/20] zinc: ChaCha20 generic C implementation and selftest

Quoted 69 lines · Jason A. Donenfeld
lib/zinc/chacha20/chacha20.c new file +63 -0
new file mode 100644
index 000000000000..3f00e1edd4c8
@@ -0,0 +1,193 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ *
+ * Implementation of the ChaCha20 stream cipher.
+ *
+ * Information: https://cr.yp.to/chacha.html
+ */
+
+#include <zinc/chacha20.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <crypto/algapi.h>
+
+#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION
+void __init chacha20_fpu_init(void)
+{
+}
+static inline bool chacha20_arch(u8 *out, const u8 *in, const size_t len,
+				 const u32 key[8], const u32 counter[4],
+				 simd_context_t *simd_context)
+{
+	return false;
+}
+static inline bool hchacha20_arch(u8 *derived_key, const u8 *nonce,
+				  const u8 *key, simd_context_t *simd_context)
+{
+	return false;
+}
+#endif
+
+#define EXPAND_32_BYTE_K 0x61707865U, 0x3320646eU, 0x79622d32U, 0x6b206574U
+
+#define QUARTER_ROUND(x, a, b, c, d) ( \
+	x[a] += x[b], \
+	x[d] = rol32((x[d] ^ x[a]), 16), \
+	x[c] += x[d], \
+	x[b] = rol32((x[b] ^ x[c]), 12), \
+	x[a] += x[b], \
+	x[d] = rol32((x[d] ^ x[a]), 8), \
+	x[c] += x[d], \
+	x[b] = rol32((x[b] ^ x[c]), 7) \
+)
+
+#define C(i, j) (i * 4 + j)
+
+#define DOUBLE_ROUND(x) ( \
+	/* Column Round */ \
+	QUARTER_ROUND(x, C(0, 0), C(1, 0), C(2, 0), C(3, 0)), \
+	QUARTER_ROUND(x, C(0, 1), C(1, 1), C(2, 1), C(3, 1)), \
+	QUARTER_ROUND(x, C(0, 2), C(1, 2), C(2, 2), C(3, 2)), \
+	QUARTER_ROUND(x, C(0, 3), C(1, 3), C(2, 3), C(3, 3)), \
+	/* Diagonal Round */ \
+	QUARTER_ROUND(x, C(0, 0), C(1, 1), C(2, 2), C(3, 3)), \
+	QUARTER_ROUND(x, C(0, 1), C(1, 2), C(2, 3), C(3, 0)), \
+	QUARTER_ROUND(x, C(0, 2), C(1, 3), C(2, 0), C(3, 1)), \
+	QUARTER_ROUND(x, C(0, 3), C(1, 0), C(2, 1), C(3, 2)) \
+)
+
+#define TWENTY_ROUNDS(x) ( \
+	DOUBLE_ROUND(x), \

lib/zinc/chacha20/chacha20.c line 73

Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10 -0
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x), \
+	DOUBLE_ROUND(x) \
+)

Does this consistently perform as well as an implementation that organizes the operations such that the quarterrounds for all columns/diagonals are interleaved? As-is, there are tight dependencies in QUARTER_ROUND() (as well as in the existing chacha20_block() in lib/chacha20.c, for that matter), so we're heavily depending on the compiler to do the needed interleaving so as to not get potentially disastrous performance. Making it explicit could be a good idea.

Quoted 32 lines
lib/zinc/chacha20/chacha20.c new file +32 -0
+
+static void chacha20_block_generic(__le32 *stream, u32 *state)
+{
+	u32 x[CHACHA20_BLOCK_WORDS];
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(x); ++i)
+		x[i] = state[i];
+
+	TWENTY_ROUNDS(x);
+
+	for (i = 0; i < ARRAY_SIZE(x); ++i)
+		stream[i] = cpu_to_le32(x[i] + state[i]);
+
+	++state[12];
+}
+
+static void chacha20_generic(u8 *out, const u8 *in, u32 len, const u32 key[8],
+			     const u32 counter[4])
+{
+	__le32 buf[CHACHA20_BLOCK_WORDS];
+	u32 x[] = {
+		EXPAND_32_BYTE_K,
+		key[0], key[1], key[2], key[3],
+		key[4], key[5], key[6], key[7],
+		counter[0], counter[1], counter[2], counter[3]
+	};
+
+	if (out != in)
+		memmove(out, in, len);
+
+	while (len >= CHACHA20_BLOCK_SIZE) {

lib/zinc/chacha20/chacha20.c line 115

Quoted 10 lines
lib/zinc/chacha20/chacha20.c new file +10 -0
+		chacha20_block_generic(buf, x);
+		crypto_xor(out, (u8 *)buf, CHACHA20_BLOCK_SIZE);
+		len -= CHACHA20_BLOCK_SIZE;
+		out += CHACHA20_BLOCK_SIZE;
+	}
+	if (len) {
+		chacha20_block_generic(buf, x);
+		crypto_xor(out, (u8 *)buf, len);
+	}
+}

If crypto_xor_cpy() is used instead of crypto_xor(), and 'in' is incremented along with 'out', then the memmove() is not needed.

- Eric

author maintainer
#22 lore raw

Re: [PATCH net-next v5 03/20] zinc: ChaCha20 generic C implementation and selftest

Quoted 6 lines · Eric Biggers

Does this consistently perform as well as an implementation that organizes the operations such that the quarterrounds for all columns/diagonals are interleaved? As-is, there are tight dependencies in QUARTER_ROUND() (as well as in the existing chacha20_block() in lib/chacha20.c, for that matter), so we're heavily depending on the compiler to do the needed interleaving so as to not get potentially disastrous performance. Making it explicit could be a good idea.

It does perform as well, and the compiler outputs good code, even on older compilers. Notably that's all a single statement (via the comma operator).

Quoted 22 lines
Quoted 22 lines
lib/zinc/chacha20/chacha20.c new file +22 -0
+}
+
+static void chacha20_generic(u8 *out, const u8 *in, u32 len, const u32 key[8],
+                          const u32 counter[4])
+{
+     __le32 buf[CHACHA20_BLOCK_WORDS];
+     u32 x[] = {
+             EXPAND_32_BYTE_K,
+             key[0], key[1], key[2], key[3],
+             key[4], key[5], key[6], key[7],
+             counter[0], counter[1], counter[2], counter[3]
+     };
+
+     if (out != in)
+             memmove(out, in, len);
+
+     while (len >= CHACHA20_BLOCK_SIZE) {
+             chacha20_block_generic(buf, x);
+             crypto_xor(out, (u8 *)buf, CHACHA20_BLOCK_SIZE);
+             len -= CHACHA20_BLOCK_SIZE;
+             out += CHACHA20_BLOCK_SIZE;
+     }
Quoted 8 lines

lib/zinc/chacha20/chacha20.c line 115

Quoted 5 lines
lib/zinc/chacha20/chacha20.c new file +5 -0
+     if (len) {
+             chacha20_block_generic(buf, x);
+             crypto_xor(out, (u8 *)buf, len);
+     }
+}

If crypto_xor_cpy() is used instead of crypto_xor(), and 'in' is incremented along with 'out', then the memmove() is not needed.

Nice idea, thanks. Implemented.

Jason

author maintainer
#23 lore raw

PATCH v5 4/20 zinc: ChaCha20 x86_64 implementation

In reply to Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- Eliminate translating certain instructions, such as pshufb, palignr, vprotd, etc, to .byte directives. This is meant for compatibility with ancient toolchains, but presumably it is unnecessary here, since the build system already does checks on what GNU as can assemble.

- When aligning the stack, the original code was saving %rsp to %r9.

    To keep objtool happy, we use instead the DRAP idiom to save %rsp
    to %r10:

      leaq    8(%rsp),%r10
      ... code here ...
      leaq    -8(%r10),%rsp

- The original code assumes the stack comes aligned to 16 bytes. This is not necessarily the case, and to avoid crashes,
andq $-alignment, %rsp was added in the prolog of a few functions.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:

size old new
---- ---- ----
0 62 52
16 414 376
32 410 400
48 414 422
64 362 356
80 714 666
96 714 700
112 712 718
128 692 646
144 1042 674
160 1042 694
176 1042 726
192 1018 650
208 1366 686
224 1366 696
240 1366 722
256 640 656
272 988 1246
288 988 1276
304 992 1296
320 972 1222
336 1318 1256
352 1318 1276
368 1316 1294
384 1294 1218
400 1642 1258
416 1642 1282
432 1642 1302
448 1628 1224
464 1970 1258
480 1970 1280
496 1970 1300
512 656 676
528 1010 1290
544 1010 1306
560 1010 1332
576 986 1254
592 1340 1284
608 1334 1310
624 1340 1334
640 1314 1254
656 1664 1282
672 1674 1306
688 1662 1336
704 1638 1250
720 1992 1292
736 1994 1308
752 1988 1334
768 1252 1254
784 1596 1290
800 1596 1314
816 1596 1330
832 1576 1256
848 1922 1286
864 1922 1314
880 1926 1338
896 1898 1258
912 2248 1288
928 2248 1320
944 2248 1338
960 2226 1268
976 2574 1288
992 2576 1312
1008 2574 1340

Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:

size old new
---- ---- ----
0 64 54
16 386 372
32 388 396
48 388 420
64 366 350
80 708 666
96 708 692
112 706 736
128 692 648
144 1036 682
160 1036 708
176 1036 730
192 1016 658
208 1360 684
224 1362 708
240 1360 732
256 644 500
272 990 526
288 988 556
304 988 576
320 972 500
336 1314 532
352 1316 558
368 1318 578
384 1308 506
400 1644 532
416 1644 556
432 1644 594
448 1624 508
464 1970 534
480 1970 556
496 1968 582
512 660 624
528 1016 682
544 1016 702
560 1018 728
576 998 654
592 1344 680
608 1344 708
624 1344 730
640 1326 654
656 1670 686
672 1670 708
688 1670 732
704 1652 658
720 1998 682
736 1998 710
752 1996 734
768 1256 662
784 1606 688
800 1606 714
816 1606 736
832 1584 660
848 1948 688
864 1950 714
880 1948 736
896 1912 688
912 2258 718
928 2258 744
944 2256 768
960 2238 692
976 2584 718
992 2584 744
1008 2584 770

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                        |    1 +
 lib/zinc/chacha20/chacha20-x86_64-glue.h |  100 +
 lib/zinc/chacha20/chacha20-x86_64.S      | 2632 ++++++++++++++++++++++
 lib/zinc/chacha20/chacha20.c             |    4 +-
 4 files changed, 2736 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-x86_64.S
lib/zinc/Makefile +1 -0
index 83b320f1ace2..e929acc909c6 100644
@@ -4,4 +4,5 @@ ccflags-y += -D'pr_fmt(fmt)="zinc: " fmt'
 ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-x86_64-glue.h new file +100 -0

100 lines not on this page · raw message

lib/zinc/chacha20/chacha20-x86_64.S new file +2632 -0

2,632 lines not on this page · raw message

lib/zinc/chacha20/chacha20.c +3 -1
index 3f00e1edd4c8..3ba2d592e7b0 100644
@@ -14,7 +14,9 @@
 #include <linux/init.h>
 #include <crypto/algapi.h>
 
-#ifndef HAVE_CHACHA20_ARCH_IMPLEMENTATION
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "chacha20-x86_64-glue.h"
+#else
 void __init chacha20_fpu_init(void)
 {
 }
Signature · 1 line
2.19.0
#24 lore raw

Re: [PATCH net-next v5 04/20] zinc: ChaCha20 x86_64 implementation

Jason,

Quoted 5 lines · Jason A. Donenfeld

This provides SSSE3, AVX-2, AVX-512F, and AVX-512VL implementations for ChaCha20. The AVX-512F implementation is disabled on Skylake, due to throttling, and the VL ymm implementation is used instead. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

...

Quoted 2 lines

I'm a bit confused by this SOB chain. So above you write that it's from Andy Polakovs implementation and Samuel did the changes. But here it seems you are the main author. If Samuel just did some modifications then you want to use the Co-developed-by tag along with his SOB.

Also I'd recommend to add a Originally-by or Based-on-code-from: Andy Polyakov tag. Both are not formal tags but widely in use for attributions.

lib/zinc/chacha20/chacha20-x86_64-glue.h line 1

Quoted 3 lines

+++ b/lib/zinc/chacha20/chacha20-x86_64-glue.h

lib/zinc/chacha20/chacha20-x86_64-glue.h new file +1 -0
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: MIT

Please put that into a separate one liner comment. Also this should be 'GPL-2.0[+] or MIT' I think.

lib/zinc/chacha20/chacha20-x86_64.S line 1

Quoted 3 lines
lib/zinc/chacha20/chacha20-x86_64.S new file +2 -0
+++ b/lib/zinc/chacha20/chacha20-x86_64.S
@@ -0,0 +1,2632 @@
+/* SPDX-License-Identifier: BSD-3-Clause

See above. That applies to all the others as well.

Thanks,

tglx

author maintainer
#25 lore raw

Re: [PATCH net-next v5 04/20] zinc: ChaCha20 x86_64 implementation

Hi Thomas,

Quoted 4 lines · Thomas Gleixner

I'm a bit confused by this SOB chain. So above you write that it's from Andy Polakovs implementation and Samuel did the changes. But here it seems you are the main author. If Samuel just did some modifications then you want to use the Co-developed-by tag along with his SOB.

Thanks, I'll use that tag.

Quoted 3 lines

Also I'd recommend to add a Originally-by or Based-on-code-from: Andy Polyakov tag. Both are not formal tags but widely in use for attributions.

Great idea.

Quoted 7 lines

lib/zinc/chacha20/chacha20-x86_64-glue.h line 1

Quoted 4 lines

+++ b/lib/zinc/chacha20/chacha20-x86_64-glue.h

lib/zinc/chacha20/chacha20-x86_64-glue.h new file +1 -0
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: MIT

Please put that into a separate one liner comment. Also this should be 'GPL-2.0[+] or MIT' I think.

I had that originally, but changed it to just MIT, since MIT is a subset of GPL-2.0. And looking at tree repo, it appears this is what others do too.

Jason

#26 lore raw

Re: [PATCH net-next v5 04/20] zinc: ChaCha20 x86_64 implementation

Jason,

Quoted 9 lines · Jason A. Donenfeld
Quoted 7 lines · Thomas Gleixner

lib/zinc/chacha20/chacha20-x86_64-glue.h line 1

Quoted 4 lines

+++ b/lib/zinc/chacha20/chacha20-x86_64-glue.h

lib/zinc/chacha20/chacha20-x86_64-glue.h new file +1 -0
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: MIT

Please put that into a separate one liner comment. Also this should be 'GPL-2.0[+] or MIT' I think.

Quoted 3 lines

I had that originally, but changed it to just MIT, since MIT is a subset of GPL-2.0. And looking at tree repo, it appears this is what others do too.

Subset? Not really. Both MIT and BSD3-Clause are GPL2.0 compatible licenses. And if your intention is to have those files MIT/BSD only, yes then the single license identifier is the right thing. If you want it dual licensed then it should be expressed there clearly.

Thanks,

tglx

author maintainer
#27 lore raw

Re: [PATCH net-next v5 04/20] zinc: ChaCha20 x86_64 implementation

Hi Thomas,

Quoted 4 lines · Thomas Gleixner

Subset? Not really. Both MIT and BSD3-Clause are GPL2.0 compatible licenses. And if your intention is to have those files MIT/BSD only, yes then the single license identifier is the right thing. If you want it dual licensed then it should be expressed there clearly.

I always thought "GPL2 compatible" was the same as "subset" because of the restrictions clause of GPL2. But IANAL, so for the avoidance of doubt I'll take your advice and put both.

Jason

author maintainer
#28 lore raw

PATCH v5 5/20 zinc: ChaCha20 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed (prefixed with .L) to fit kernel conventions.
- Constants have been rearranged so that they are closer to the code that is using them. [ARM only]
- The neon code can jump to the scalar code when it makes sense to do so.
- The neon_512 function as a separate function has been removed, leaving the decision up to the main neon entry point. [ARM64 only]

After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.

ARM:

-ChaCha20_ctr32:
-.LChaCha20_ctr32:
+ENTRY(chacha20_arm)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
- sub r14,pc,#16 @ ChaCha20_ctr32
- adr r14,.LChaCha20_ctr32
cmp r2,#0 @ len==0?
itt eq
addeq sp,sp,#4*3
- beq .Lno_data
- cmp r2,#192 @ test len
- bls .Lshort
- ldr r4,[r14,#-32]
- ldr r4,[r14,r4]
- ldr r4,[r4]
- tst r4,#ARMV7_NEON
- bne .LChaCha20_neon
+ beq .Lno_data_arm
.Lshort:
ldmia r12,{r4-r7} @ load counter and nonce
sub sp,sp,#4*(16) @ off-load area
- sub r14,r14,#64 @ .Lsigma
+ sub r14,pc,#100 @ .Lsigma
+ adr r14,.Lsigma @ .Lsigma
stmdb sp!,{r4-r7} @ copy counter and nonce
ldmia r3,{r4-r11} @ load key
ldmia r14,{r0-r3} @ load sigma
@@ -617,14 +615,25 @@

.Ldone:
add sp,sp,#4*(32+3)
-.Lno_data:
+.Lno_data_arm:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_arm)

-ChaCha20_neon:
+ENTRY(chacha20_neon)
ldr r12,[sp,#0] @ pull pointer to counter and nonce
stmdb sp!,{r0-r2,r4-r11,lr}
-.LChaCha20_neon:
- adr r14,.Lsigma
+ cmp r2,#0 @ len==0?
+ itt eq
+ addeq sp,sp,#4*3
+ beq .Lno_data_neon
+ cmp r2,#192 @ test len
+ bls .Lshort
+.Lchacha20_neon_begin:
+ adr r14,.Lsigma2
vstmdb sp!,{d8-d15} @ ABI spec says so
stmdb sp!,{r0-r3}

@@ -1265,4 +1274,6 @@
add sp,sp,#4*(32+4)
vldmia sp,{d8-d15}
add sp,sp,#4*(16+3)
+.Lno_data_neon:
ldmia sp!,{r4-r11,pc}
+ENDPROC(chacha20_neon)

ARM64:

-ChaCha20_ctr32:
+ENTRY(chacha20_arm)
cbz x2,.Labort
- adr x5,.LOPENSSL_armcap_P
- cmp x2,#192
- b.lo .Lshort
- ldrsw x6,[x5]
- ldr x6,[x5]
- ldr w17,[x6,x5]
- tst w17,#ARMV7_NEON
- b.ne ChaCha20_neon
-
.Lshort:
stp x29,x30,[sp,#-96]!
add x29,sp,#0
@@ -279,8 +274,13 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
+ENDPROC(chacha20_arm)
+
+ENTRY(chacha20_neon)
+ cbz x2,.Labort_neon
+ cmp x2,#192
+ b.lo .Lshort

-ChaCha20_neon:
stp x29,x30,[sp,#-96]!
add x29,sp,#0

@@ -763,16 +763,6 @@
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
ret
-ChaCha20_512_neon:
- stp x29,x30,[sp,#-96]!
- add x29,sp,#0
-
- adr x5,.Lsigma
- stp x19,x20,[sp,#16]
- stp x21,x22,[sp,#32]
- stp x23,x24,[sp,#48]
- stp x25,x26,[sp,#64]
- stp x27,x28,[sp,#80]

.L512_or_more_neon:
sub sp,sp,#128+64
@@ -1920,4 +1910,6 @@
ldp x25,x26,[x29,#64]
ldp x27,x28,[x29,#80]
ldp x29,x30,[sp],#96
+.Labort_neon:
ret
+ENDPROC(chacha20_neon)

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                     |    2 +
 lib/zinc/chacha20/chacha20-arm-glue.h |   47 +
 lib/zinc/chacha20/chacha20-arm.S      | 1473 +++++++++++++++++++
 lib/zinc/chacha20/chacha20-arm64.S    | 1942 +++++++++++++++++++++++++
 lib/zinc/chacha20/chacha20.c          |    2 +
 5 files changed, 3466 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-arm.S
 create mode 100644 lib/zinc/chacha20/chacha20-arm64.S
lib/zinc/Makefile +2 -0
index e929acc909c6..36b61145666d 100644
@@ -5,4 +5,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
 
 zinc_chacha20-y := chacha20/chacha20.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-arm-glue.h new file +47 -0

47 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm.S new file +1473 -0

1,473 lines not on this page · raw message

lib/zinc/chacha20/chacha20-arm64.S new file +1942 -0

1,474 lines not on this page · raw message

lib/zinc/chacha20/chacha20.c +2 -0
index 3ba2d592e7b0..385e7350adc3 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#29 lore raw

PATCH v5 6/20 zinc: ChaCha20 MIPS32r2 implementation

In reply to Jason A. Donenfeld

This MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Ralf Baechle <[email protected]>
Cc Paul Burton <[email protected]>
Cc James Hogan <[email protected]>
---
 lib/zinc/Makefile                      |   1 +
 lib/zinc/chacha20/chacha20-mips-glue.h |  24 ++
 lib/zinc/chacha20/chacha20-mips.S      | 474 +++++++++++++++++++++++++
 lib/zinc/chacha20/chacha20.c           |   2 +
 4 files changed, 501 insertions(+)
 create mode 100644 lib/zinc/chacha20/chacha20-mips-glue.h
 create mode 100644 lib/zinc/chacha20/chacha20-mips.S
lib/zinc/Makefile +1 -0
index 36b61145666d..5c7b452d6a19 100644
@@ -7,4 +7,5 @@ zinc_chacha20-y := chacha20/chacha20.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
+zinc_chacha20-$(CONFIG_ZINC_ARCH_MIPS) += chacha20/chacha20-mips.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
lib/zinc/chacha20/chacha20-mips-glue.h new file +24 -0
new file mode 100644
index 000000000000..518deb5e6c34
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+asmlinkage void chacha20_mips(u8 *out, const u8 *in, const size_t len,
+			      const u32 key[8], const u32 counter[4]);
+static void __init chacha20_fpu_init(void)
+{
+}
+
+static inline bool chacha20_arch(u8 *dst, const u8 *src, const size_t len,
+				 const u32 key[8], const u32 counter[4],
+				 simd_context_t *simd_context)
+{
+	chacha20_mips(dst, src, len, key, counter);
+	return true;
+}
+
+static inline bool hchacha20_arch(u8 *derived_key, const u8 *nonce,
+				  const u8 *key, simd_context_t *simd_context)
+{
+	return false;
+}
lib/zinc/chacha20/chacha20-mips.S new file +474 -0

474 lines not on this page · raw message

lib/zinc/chacha20/chacha20.c +2 -0
index 385e7350adc3..0cf490ca23be 100644
@@ -18,6 +18,8 @@
 #include "chacha20-x86_64-glue.h"
 #elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
 #include "chacha20-arm-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_MIPS)
+#include "chacha20-mips-glue.h"
 #else
 void __init chacha20_fpu_init(void)
 {
Signature · 1 line
2.19.0
#30 lore raw

Re: [PATCH net-next v5 06/20] zinc: ChaCha20 MIPS32r2 implementation

Hi Jason,

Quoted 7 lines · Jason A. Donenfeld

+.Lchacha20_mips_xor_aligned_4_b:

lib/zinc/chacha20/chacha20-mips.S new file +6 -0
+	/* STORE_ALIGNED( 4,  0, $sp, 0+CONSTANT_OFS_SP) */
+	lw	T0, 0+CONSTANT_OFS_SP($sp)
+	lw	T1, 0(IN)
+	addu	X0, T0
+	CPU_TO_LE32(X0)
+	xor	X0, T1

lib/zinc/chacha20/chacha20-mips.S line 338

Quoted 10 lines
lib/zinc/chacha20/chacha20-mips.S new file +10 -0
+	.set noreorder
+	bne	OUT, PTR_LAST_ROUND, .Loop_chacha20_rounds
+	sw	X0, 0(OUT)
+	.set reorder
+
+	.set noreorder
+	bne	$at, BYTES, .Lchacha20_mips_xor_bytes
+	/* Empty delayslot, Increase NONCE_0, return NONCE_0 value */
+	addiu	NONCE_0, 1
+	.set noreorder

Should this be .set reorder?

Even better - could we not just place the addiu before the bne & drop the .set noreorder, allowing the assembler to fill the delay slot with the addiu? Likewise in many other places throughout the patch.

That would be more future proof - particularly if we ever want to adjust this for use with the nanoMIPS ISA which has no delay slots. It may also allow the assembler the choice to use compact branches (ie. branches without visible delay slots) when targeting MIPS32r6. I know neither of these will currently build this code, but I think avoiding all the noreorder blocks would be a nice cleanup just for the sake of readability anyway.

Thanks,
Paul

author maintainer
#31 lore raw

Re: [PATCH net-next v5 06/20] zinc: ChaCha20 MIPS32r2 implementation

Hi Paul,

Thanks a bunch for the review.

Quoted 1 line · Paul Burton

Should this be .set reorder?

Nice catch. Fixed here:
https://git.zx2c4.com/WireGuard/commit/?id=23d97fc333cf85dd07445a9d21a28cbef47c553c
But then...

Quoted 11 lines

Even better - could we not just place the addiu before the bne & drop the .set noreorder, allowing the assembler to fill the delay slot with the addiu? Likewise in many other places throughout the patch.

That would be more future proof - particularly if we ever want to adjust this for use with the nanoMIPS ISA which has no delay slots. It may also allow the assembler the choice to use compact branches (ie. branches without visible delay slots) when targeting MIPS32r6. I know neither of these will currently build this code, but I think avoiding all the noreorder blocks would be a nice cleanup just for the sake of readability anyway.

Great idea. Rene has committed that here:
https://git.zx2c4.com/WireGuard/commit/?id=5c153a59ac3aa58a3ff17c69fee63d599e5f2758

These will be in the v6 patchset whenever that's posted, and it's already been merged into the dev tree:
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

Regards,
Jason

author maintainer
#32 lore raw

PATCH v5 7/20 zinc: Poly1305 generic C implementations and selftest

In reply to Jason A. Donenfeld

These two C implementations -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel and for usage with accelerated primitives.

Information: https://cr.yp.to/mac.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/poly1305.h              |  31 +
 lib/zinc/Kconfig                     |   3 +
 lib/zinc/Makefile                    |   3 +
 lib/zinc/poly1305/poly1305-donna32.h | 205 +++++++
 lib/zinc/poly1305/poly1305-donna64.h | 182 ++++++
 lib/zinc/poly1305/poly1305.c         | 155 +++++
 lib/zinc/selftest/poly1305.h         | 875 +++++++++++++++++++++++++++
 7 files changed, 1454 insertions(+)
 create mode 100644 include/zinc/poly1305.h
 create mode 100644 lib/zinc/poly1305/poly1305-donna32.h
 create mode 100644 lib/zinc/poly1305/poly1305-donna64.h
 create mode 100644 lib/zinc/poly1305/poly1305.c
 create mode 100644 lib/zinc/selftest/poly1305.h
include/zinc/poly1305.h new file +31 -0
new file mode 100644
index 000000000000..f6ea9dc26c95
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_POLY1305_H
+#define _ZINC_POLY1305_H
+
+#include <linux/simd.h>
+#include <linux/types.h>
+
+enum poly1305_lengths {
+	POLY1305_BLOCK_SIZE = 16,
+	POLY1305_KEY_SIZE = 32,
+	POLY1305_MAC_SIZE = 16
+};
+
+struct poly1305_ctx {
+	u8 opaque[24 * sizeof(u64)];
+	u32 nonce[4];
+	u8 data[POLY1305_BLOCK_SIZE];
+	size_t num;
+} __aligned(8);
+
+void poly1305_init(struct poly1305_ctx *ctx, const u8 key[POLY1305_KEY_SIZE]);
+void poly1305_update(struct poly1305_ctx *ctx, const u8 *input, size_t len,
+		     simd_context_t *simd_context);
+void poly1305_final(struct poly1305_ctx *ctx, u8 mac[POLY1305_MAC_SIZE],
+		    simd_context_t *simd_context);
+
+#endif /* _ZINC_POLY1305_H */
lib/zinc/Kconfig +3 -0
index 1ca1ae1e9ea9..f9ef8f7e3c25 100644
@@ -2,6 +2,9 @@ config ZINC_CHACHA20
 	tristate
 	select CRYPTO_ALGAPI
 
+config ZINC_POLY1305
+	tristate
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index 5c7b452d6a19..ce9707d79ea8 100644
@@ -9,3 +9,6 @@ zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
 zinc_chacha20-$(CONFIG_ZINC_ARCH_MIPS) += chacha20/chacha20-mips.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
+
+zinc_poly1305-y := poly1305/poly1305.o
+obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
lib/zinc/poly1305/poly1305-donna32.h new file +205 -0

205 lines not on this page · raw message

lib/zinc/poly1305/poly1305-donna64.h new file +182 -0

182 lines not on this page · raw message

lib/zinc/poly1305/poly1305.c new file +155 -0

155 lines not on this page · raw message

lib/zinc/selftest/poly1305.h new file +875 -0

875 lines not on this page · raw message

Signature · 1 line
2.19.0
#33 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 77 lines · Jason A. Donenfeld
lib/zinc/poly1305/poly1305.c new file +71 -0
new file mode 100644
index 000000000000..dbab82f33aa7
@@ -0,0 +1,155 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ *
+ * Implementation of the Poly1305 message authenticator.
+ *
+ * Information: https://cr.yp.to/mac.html
+ */
+
+#include <zinc/poly1305.h>
+
+#include <asm/unaligned.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <linux/module.h>
+#include <linux/init.h>
+
+#ifndef HAVE_POLY1305_ARCH_IMPLEMENTATION
+static inline bool poly1305_init_arch(void *ctx,
+				      const u8 key[POLY1305_KEY_SIZE])
+{
+	return false;
+}
+static inline bool poly1305_blocks_arch(void *ctx, const u8 *input,
+					const size_t len, const u32 padbit,
+					simd_context_t *simd_context)
+{
+	return false;
+}
+static inline bool poly1305_emit_arch(void *ctx, u8 mac[POLY1305_MAC_SIZE],
+				      const u32 nonce[4],
+				      simd_context_t *simd_context)
+{
+	return false;
+}
+void __init poly1305_fpu_init(void)
+{
+}
+#endif
+
+#if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__)
+#include "poly1305-donna64.h"
+#else
+#include "poly1305-donna32.h"
+#endif
+
+void poly1305_init(struct poly1305_ctx *ctx, const u8 key[POLY1305_KEY_SIZE])
+{
+	ctx->nonce[0] = get_unaligned_le32(&key[16]);
+	ctx->nonce[1] = get_unaligned_le32(&key[20]);
+	ctx->nonce[2] = get_unaligned_le32(&key[24]);
+	ctx->nonce[3] = get_unaligned_le32(&key[28]);
+
+	if (!poly1305_init_arch(ctx->opaque, key))
+		poly1305_init_generic(ctx->opaque, key);
+
+	ctx->num = 0;
+}
+EXPORT_SYMBOL(poly1305_init);
+
+static inline void poly1305_blocks(void *ctx, const u8 *input, const size_t len,
+				   const u32 padbit,
+				   simd_context_t *simd_context)
+{
+	if (!poly1305_blocks_arch(ctx, input, len, padbit, simd_context))
+		poly1305_blocks_generic(ctx, input, len, padbit);
+}
+
+static inline void poly1305_emit(void *ctx, u8 mac[POLY1305_KEY_SIZE],
+				 const u32 nonce[4],
+				 simd_context_t *simd_context)

lib/zinc/poly1305/poly1305.c line 81

Quoted 10 lines
lib/zinc/poly1305/poly1305.c new file +10 -0
+{
+	if (!poly1305_emit_arch(ctx, mac, nonce, simd_context))
+		poly1305_emit_generic(ctx, mac, nonce);
+}
+
+void poly1305_update(struct poly1305_ctx *ctx, const u8 *input, size_t len,
+		     simd_context_t *simd_context)
+{
+	const size_t num = ctx->num % POLY1305_BLOCK_SIZE;
+	size_t rem;

0 <= ctx->num < POLY1305_BLOCK_SIZE, so no need to mod by POLY1305_BLOCK_SIZE.

Quoted 24 lines
lib/zinc/poly1305/poly1305.c new file +24 -0
+
+	if (num) {
+		rem = POLY1305_BLOCK_SIZE - num;
+		if (len < rem) {
+			memcpy(ctx->data + num, input, len);
+			ctx->num = num + len;
+			return;
+		}
+		memcpy(ctx->data + num, input, rem);
+		poly1305_blocks(ctx->opaque, ctx->data, POLY1305_BLOCK_SIZE, 1,
+				simd_context);
+		input += rem;
+		len -= rem;
+	}
+
+	rem = len % POLY1305_BLOCK_SIZE;
+	len -= rem;
+
+	if (len >= POLY1305_BLOCK_SIZE) {
+		poly1305_blocks(ctx->opaque, input, len, 1, simd_context);
+		input += len;
+	}
+
+	if (rem)

lib/zinc/poly1305/poly1305.c line 115

Quoted 10 lines
lib/zinc/poly1305/poly1305.c new file +10 -0
+		memcpy(ctx->data, input, rem);
+
+	ctx->num = rem;
+}
+EXPORT_SYMBOL(poly1305_update);
+
+void poly1305_final(struct poly1305_ctx *ctx, u8 mac[POLY1305_MAC_SIZE],
+		    simd_context_t *simd_context)
+{
+	size_t num = ctx->num % POLY1305_BLOCK_SIZE;

Same here.

Quoted 20 lines
lib/zinc/poly1305/poly1305.c new file +19 -0
+++ b/lib/zinc/selftest/poly1305.h
@@ -0,0 +1,875 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifdef DEBUG
+struct poly1305_testvec {
+	u8 input[600];
+	u8 output[POLY1305_MAC_SIZE];
+	u8 key[POLY1305_KEY_SIZE];
+	size_t ilen;
+};
+
+static const struct poly1305_testvec poly1305_testvecs[] __initconst = {
+{ /* RFC7539 */
+	.input	= { 0x43, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x67, 0x72,
+		    0x61, 0x70, 0x68, 0x69, 0x63, 0x20, 0x46, 0x6f,
+		    0x72, 0x75, 0x6d, 0x20, 0x52, 0x65, 0x73, 0x65,

lib/zinc/poly1305/poly1305.c line 28

Quoted 10 lines
lib/zinc/poly1305/poly1305.c new file +10 -0
+		    0x61, 0x72, 0x63, 0x68, 0x20, 0x47, 0x72, 0x6f,
+		    0x75, 0x70 },
+	.ilen	= 34,
+	.output	= { 0xa8, 0x06, 0x1d, 0xc1, 0x30, 0x51, 0x36, 0xc6,
+		    0xc2, 0x2b, 0x8b, 0xaf, 0x0c, 0x01, 0x27, 0xa9 },
+	.key	= { 0x85, 0xd6, 0xbe, 0x78, 0x57, 0x55, 0x6d, 0x33,
+		    0x7f, 0x44, 0x52, 0xfe, 0x42, 0xd5, 0x06, 0xa8,
+		    0x01, 0x03, 0x80, 0x8a, 0xfb, 0x0d, 0xb2, 0xfd,
+		    0x4a, 0xbf, 0xf6, 0xaf, 0x41, 0x49, 0xf5, 0x1b },
+}, { /* "The Poly1305-AES message-authentication code" */

Hardcoding the 'input' array to 600 bytes forces the full amount of space to be reserved in the kernel image for every test vector. Also, if anyone adds a longer test vector they will need to remember to increase the value.

It should be a const pointer instead, like the test vectors in crypto/testmgr.h.

- Eric

author maintainer
#34 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 5 lines · Eric Biggers

Hardcoding the 'input' array to 600 bytes forces the full amount of space to be reserved in the kernel image for every test vector. Also, if anyone adds a longer test vector they will need to remember to increase the value.

It should be a const pointer instead, like the test vectors in crypto/testmgr.h.

I know. The agony. This has been really annoying me. I originally did it the right way, but removed it last week, when I noticed that gcc failed to put it in the initconst section:

https://git.zx2c4.com/WireGuard/commit/?id=f4698d20f13946afc6ce99e98685ba3f9adc4474

Even changing the (u8[]){ ... } into a (const u8[]){ ... } or even into a const string literal does not do the trick. It makes it into the constant data section with const, but it does not make it into the initconst section. What a bummer.

I went asking about this on the gcc mailing list, to see if there was just some aspect of C that I had overlooked:
https://gcc.gnu.org/ml/gcc/2018-09/msg00043.html So far, it looks like we're SOL. I could probably make some macros to do this in a .S file but... that's pretty unacceptably gross. Or I could define lots and lots of variables in __initconst, and then connect them all together at the end, but that's pretty gross too. Or I could have all this data in one variable and record offsets into it, but that's even more atrocious.

So I think it comes down to these two non-ugly options:
- We use fixed sized buffers, waste a lot of space, and be happy that it's cleared from memory immediately after init/insertion anyway, so it's not actually wasting ram.
- We use const string literals / constant compound literals, save space on disk, but not benefit from having it cleared from memory after init/insertion.

Of these, which would you prefer? I can see the argument both ways, but in the end opted for the first. Or perhaps you have a better third option?

Jason

#35 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 16 lines · Jason A. Donenfeld
Quoted 7 lines · Eric Biggers

Hardcoding the 'input' array to 600 bytes forces the full amount of space to be reserved in the kernel image for every test vector. Also, if anyone adds a longer test vector they will need to remember to increase the value.

It should be a const pointer instead, like the test vectors in crypto/testmgr.h.

I know. The agony. This has been really annoying me. I originally did it the right way, but removed it last week, when I noticed that gcc failed to put it in the initconst section:

https://git.zx2c4.com/WireGuard/commit/?id=f4698d20f13946afc6ce99e98685ba3f9adc4474

Quoted 8 lines

Even changing the (u8[]){ ... } into a (const u8[]){ ... } or even into a const string literal does not do the trick. It makes it into the constant data section with const, but it does not make it into the initconst section. What a bummer.

I went asking about this on the gcc mailing list, to see if there was just some aspect of C that I had overlooked:
https://gcc.gnu.org/ml/gcc/2018-09/msg00043.html

Can you not uglify the code a bit by using normal (non-compound) liberals as described in the response to that email?

author maintainer
#36 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 1 line · Andy Lutomirski

Can you not uglify the code a bit by using normal (non-compound) liberals as described in the response to that email?

I can define a bunch of variables, and then string them all together in an array at the end. This is a bit ugly and less maintainable, but would be the best of both worlds.

author maintainer
#37 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 7 lines · Jason A. Donenfeld
Quoted 1 line · Andy Lutomirski

Can you not uglify the code a bit by using normal (non-compound) liberals as described in the response to that email?

I can define a bunch of variables, and then string them all together in an array at the end. This is a bit ugly and less maintainable, but would be the best of both worlds.

Alright, I wrote a little program to do this for me in a way that actually doesn't look half bad. Thanks for suggesting.

Jason

author maintainer
#38 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

In reply to Eric Biggers
Quoted 4 lines
Quoted 1 line

+ const size_t num = ctx->num % POLY1305_BLOCK_SIZE;

0 <= ctx->num < POLY1305_BLOCK_SIZE, so no need to mod by POLY1305_BLOCK_SIZE.

Quoted 1 line

+ size_t num = ctx->num % POLY1305_BLOCK_SIZE;

Same here.

I know, but I was having a hard time convincing gcc-8 of that invariant, and it was warning me. Perhaps this is something they fixed, though, between 8.1 and 8.2 though. I'll check back and adjust accordingly.

author maintainer
#39 lore raw

Re: [PATCH net-next v5 07/20] zinc: Poly1305 generic C implementations and selftest

Quoted 10 lines · Jason A. Donenfeld
Quoted 5 lines
Quoted 2 lines

+ const size_t num = ctx->num % POLY1305_BLOCK_SIZE;

0 <= ctx->num < POLY1305_BLOCK_SIZE, so no need to mod by POLY1305_BLOCK_SIZE.

Quoted 1 line

+ size_t num = ctx->num % POLY1305_BLOCK_SIZE;

Same here.

I know, but I was having a hard time convincing gcc-8 of that invariant, and it was warning me. Perhaps this is something they fixed, though, between 8.1 and 8.2 though. I'll check back and adjust accordingly.

This was changed here:
https://git.zx2c4.com/WireGuard/commit/?id=37f114a73ba37219b00a66f0a51219a696599745

I can't reproduce with 8.2 anymore, so perhaps I should remove it now. Unless you'd like to avoid a warning on old compilers. Since there's no difference in speed, probably we should avoid the 8.1 warning and leave it be?

author maintainer
#40 lore raw

PATCH v5 8/20 zinc: Poly1305 x86_64 implementation

In reply to Jason A. Donenfeld

This provides AVX, AVX-2, and AVX-512F implementations for Poly1305. The AVX-512F implementation is disabled on Skylake, due to throttling. These come from Andy Polyakov's implementation, with the following modifications from Samuel Neves:

- Some cosmetic changes, like renaming labels to .Lname, constants, and other Linux conventions.

- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.

- poly1305_blocks_avx512 jumped to the middle of the poly1305_blocks_avx2 for the final blocks. To appease objtool, the relevant tail avx2 code was duplicated for the avx512 function.

- The original uses %rbp as a scratch register. However, the kernel expects %rbp to be a valid frame pointer at any given time in order to do proper unwinding. Thus we need to alter the code in order to preserve it. The most straightforward manner in which this was accomplished was by replacing $d3, formerly %r10, by %rdi, and replacing %rbp by %r10. Because %rdi, a pointer to the context structure, does not change and is not used by poly1305_iteration, it is safe to use it here, and the overhead of saving and restoring it should be minimal.

- The original hardcodes returns as .byte 0xf3,0xc3, aka "rep ret". We replace this by "ret". "rep ret" was meant to help with AMD K8 chips, cf. http://repzret.org/p/repzret. It makes no sense to continue to use this kludge for code that won't even run on ancient AMD chips.

Cycle counts on a Core i7 6700HQ using the AVX-2 codepath:

size old new
---- ---- ----
0 70 68
16 92 90
32 134 104
48 172 120
64 218 136
80 254 158
96 298 174
112 342 192
128 388 212
144 428 228
160 466 246
176 510 264
192 550 282
208 594 302
224 628 316
240 676 334
256 716 354
272 764 374
288 802 352
304 420 366
320 428 360
336 484 378
352 426 384
368 478 400
384 488 394
400 542 408
416 486 416
432 534 430
448 544 422
464 600 438
480 540 448
496 594 464
512 602 456
528 656 476
544 600 480
560 650 494
576 664 490
592 714 508
608 656 514
624 708 532
640 716 524
656 770 536
672 716 548
688 770 562
704 774 552
720 826 568
736 768 574
752 822 592
768 830 584
784 884 602
800 828 610
816 884 628
832 888 618
848 942 632
864 884 644
880 936 660
896 948 652
912 1000 664
928 942 676
944 994 690
960 1002 680
976 1054 694
992 1002 706
1008 1052 720

Cycle counts on a Xeon Gold 5120 using the AVX-512 codepath:

size old new
---- ---- ----
0 74 70
16 96 92
32 136 106
48 184 124
64 218 138
80 260 160
96 300 176
112 342 194
128 384 212
144 420 226
160 464 248
176 504 264
192 544 282
208 582 300
224 624 318
240 662 338
256 708 358
272 748 372
288 788 358
304 422 370
320 432 364
336 486 380
352 434 390
368 480 408
384 490 398
400 542 412
416 492 426
432 538 436
448 546 432
464 600 448
480 548 456
496 594 476
512 606 470
528 656 480
544 606 498
560 652 512
576 662 508
592 716 522
608 664 538
624 710 552
640 720 516
656 772 526
672 722 544
688 768 556
704 778 556
720 832 568
736 780 584
752 826 600
768 836 560
784 888 572
800 838 588
816 884 604
832 894 598
848 946 612
864 896 628
880 942 644
896 952 608
912 1004 616
928 954 634
944 1000 646
960 1008 646
976 1062 658
992 1012 674
1008 1058 690

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                        |    1 +
 lib/zinc/poly1305/poly1305-x86_64-glue.h |  125 +
 lib/zinc/poly1305/poly1305-x86_64.S      | 2792 ++++++++++++++++++++++
 lib/zinc/poly1305/poly1305.c             |    4 +-
 4 files changed, 2921 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/poly1305/poly1305-x86_64-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-x86_64.S
lib/zinc/Makefile +1 -0
index ce9707d79ea8..f61d5ff4e386 100644
@@ -11,4 +11,5 @@ zinc_chacha20-$(CONFIG_ZINC_ARCH_MIPS) += chacha20/chacha20-mips.o
 obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
 
 zinc_poly1305-y := poly1305/poly1305.o
+zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o
 obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
lib/zinc/poly1305/poly1305-x86_64-glue.h new file +125 -0

125 lines not on this page · raw message

lib/zinc/poly1305/poly1305-x86_64.S new file +2792 -0

2,792 lines not on this page · raw message

lib/zinc/poly1305/poly1305.c +3 -1
index dbab82f33aa7..8bf2b95ca615 100644
@@ -15,7 +15,9 @@
 #include <linux/module.h>
 #include <linux/init.h>
 
-#ifndef HAVE_POLY1305_ARCH_IMPLEMENTATION
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "poly1305-x86_64-glue.h"
+#else
 static inline bool poly1305_init_arch(void *ctx,
 				      const u8 key[POLY1305_KEY_SIZE])
 {
Signature · 1 line
2.19.0
author maintainer
#41 lore raw

PATCH v5 9/20 zinc: Poly1305 ARM and ARM64 implementations

In reply to Jason A. Donenfeld

These NEON and non-NEON implementations come from Andy Polyakov's implementation. They are exactly the same as Andy Polyakov's original, with the following exceptions:

- Entries and exits use the proper kernel convention macro.
- CPU feature checking is done in C by the glue code, so that has been removed from the assembly.
- The function names have been renamed to fit kernel conventions.
- Labels have been renamed to fit kernel conventions.
- The neon code can jump to the scalar code when it makes sense to do so.

After '/^#/d;/^\..*[^:]$/d', the code has the following diff in actual instructions from the original.

ARM:

-poly1305_init:
-.Lpoly1305_init:
+ENTRY(poly1305_init_arm)
stmdb sp!,{r4-r11}

eor r3,r3,r3
@@ -18,8 +25,6 @@
moveq r0,#0
beq .Lno_key

- adr r11,.Lpoly1305_init
- ldr r12,.LOPENSSL_armcap
ldrb r4,[r1,#0]
mov r10,#0x0fffffff
ldrb r5,[r1,#1]
@@ -34,8 +39,6 @@
ldrb r7,[r1,#6]
and r4,r4,r10

- ldr r12,[r11,r12] @ OPENSSL_armcap_P
- ldr r12,[r12]
ldrb r8,[r1,#7]
orr r5,r5,r6,lsl#8
ldrb r6,[r1,#8]
@@ -45,22 +48,6 @@
ldrb r8,[r1,#10]
and r5,r5,r3

- tst r12,#ARMV7_NEON @ check for NEON
- adr r9,poly1305_blocks_neon
- adr r11,poly1305_blocks
- it ne
- movne r11,r9
- adr r12,poly1305_emit
- adr r10,poly1305_emit_neon
- it ne
- movne r12,r10
- itete eq
- addeq r12,r11,#(poly1305_emit-.Lpoly1305_init)
- addne r12,r11,#(poly1305_emit_neon-.Lpoly1305_init)
- addeq r11,r11,#(poly1305_blocks-.Lpoly1305_init)
- addne r11,r11,#(poly1305_blocks_neon-.Lpoly1305_init)
- orr r12,r12,#1 @ thumb-ify address
- orr r11,r11,#1
ldrb r9,[r1,#11]
orr r6,r6,r7,lsl#8
ldrb r7,[r1,#12]
@@ -79,17 +66,16 @@
str r6,[r0,#8]
and r7,r7,r3
str r7,[r0,#12]
- stmia r2,{r11,r12} @ fill functions table
- mov r0,#1
- mov r0,#0
.Lno_key:
ldmia sp!,{r4-r11}
bx lr @ bx lr
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_blocks:
-.Lpoly1305_blocks:
+ENDPROC(poly1305_init_arm)
+
+ENTRY(poly1305_blocks_arm)
+.Lpoly1305_blocks_arm:
stmdb sp!,{r3-r11,lr}

ands r2,r2,#-16
@@ -231,10 +217,11 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
-poly1305_emit:
+ENDPROC(poly1305_blocks_arm)
+
+ENTRY(poly1305_emit_arm)
stmdb sp!,{r4-r11}
.Lpoly1305_emit_enter:
-
ldmia r0,{r3-r7}
adds r8,r3,#5 @ compare to modulus
adcs r9,r4,#0
@@ -305,8 +292,12 @@
tst lr,#1
moveq pc,lr @ be binary compatible with V4, yet
.word 0xe12fff1e @ interoperable with Thumb ISA:-)
+ENDPROC(poly1305_emit_arm)
+
+

-poly1305_init_neon:
+ENTRY(poly1305_init_neon)
+.Lpoly1305_init_neon:
ldr r4,[r0,#20] @ load key base 2^32
ldr r5,[r0,#24]
ldr r6,[r0,#28]
@@ -515,8 +506,9 @@
vst1.32 {d8[1]},[r7]

bx lr @ bx lr
+ENDPROC(poly1305_init_neon)

-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr ip,[r0,#36] @ is_base2_26
ands r2,r2,#-16
beq .Lno_data_neon
@@ -524,7 +516,7 @@
cmp r2,#64
bhs .Lenter_neon
tst ip,ip @ is_base2_26?
- beq .Lpoly1305_blocks
+ beq .Lpoly1305_blocks_arm

.Lenter_neon:
stmdb sp!,{r4-r7}
@@ -534,7 +526,7 @@
bne .Lbase2_26_neon

stmdb sp!,{r1-r3,lr}
- bl poly1305_init_neon
+ bl .Lpoly1305_init_neon

ldr r4,[r0,#0] @ load hash value base 2^32
ldr r5,[r0,#4]
@@ -989,8 +981,9 @@
ldmia sp!,{r4-r7}
.Lno_data_neon:
bx lr @ bx lr
+ENDPROC(poly1305_blocks_neon)

-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr ip,[r0,#36] @ is_base2_26

stmdb sp!,{r4-r11}
@@ -1055,6 +1048,6 @@

ldmia sp!,{r4-r11}
bx lr @ bx lr
+ENDPROC(poly1305_emit_neon)

ARM64:

-poly1305_init:
+ENTRY(poly1305_init_arm)
cmp x1,xzr
stp xzr,xzr,[x0] // zero hash value
stp xzr,xzr,[x0,#16] // [along with is_base2_26]
@@ -11,14 +15,9 @@
csel x0,xzr,x0,eq
b.eq .Lno_key

- ldrsw x11,.LOPENSSL_armcap_P
- ldr x11,.LOPENSSL_armcap_P
- adr x10,.LOPENSSL_armcap_P
-
ldp x7,x8,[x1] // load key
mov x9,#0xfffffffc0fffffff
movk x9,#0x0fff,lsl#48
- ldr w17,[x10,x11]
rev x7,x7 // flip bytes
rev x8,x8
and x7,x7,x9 // &=0ffffffc0fffffff
@@ -26,24 +25,11 @@
and x8,x8,x9 // &=0ffffffc0ffffffc
stp x7,x8,[x0,#32] // save key value

- tst w17,#ARMV7_NEON
-
- adr x12,poly1305_blocks
- adr x7,poly1305_blocks_neon
- adr x13,poly1305_emit
- adr x8,poly1305_emit_neon
-
- csel x12,x12,x7,eq
- csel x13,x13,x8,eq
-
- stp w12,w13,[x2]
- stp x12,x13,[x2]
-
- mov x0,#1
.Lno_key:
ret
+ENDPROC(poly1305_init_arm)

-poly1305_blocks:
+ENTRY(poly1305_blocks_arm)
ands x2,x2,#-16
b.eq .Lno_data

@@ -100,8 +86,9 @@

.Lno_data:
ret
+ENDPROC(poly1305_blocks_arm)

-poly1305_emit:
+ENTRY(poly1305_emit_arm)
ldp x4,x5,[x0] // load hash base 2^64
ldr x6,[x0,#16]
ldp x10,x11,[x2] // load nonce
@@ -124,7 +111,9 @@
stp x4,x5,[x1] // write result

ret
-poly1305_mult:
+ENDPROC(poly1305_emit_arm)
+
+__poly1305_mult:
mul x12,x4,x7 // h0*r0
umulh x13,x4,x7

@@ -158,7 +147,7 @@

ret

-poly1305_splat:
+__poly1305_splat:
and x12,x4,#0x03ffffff // base 2^64 -> base 2^26
ubfx x13,x4,#26,#26
extr x14,x5,x4,#52
@@ -182,11 +171,11 @@

ret

-poly1305_blocks_neon:
+ENTRY(poly1305_blocks_neon)
ldr x17,[x0,#24]
cmp x2,#128
b.hs .Lblocks_neon
- cbz x17,poly1305_blocks
+ cbz x17,poly1305_blocks_arm

.Lblocks_neon:
stp x29,x30,[sp,#-80]!
@@ -232,7 +221,7 @@
adcs x5,x5,x13
adc x6,x6,x3

- bl poly1305_mult
+ bl __poly1305_mult
ldr x30,[sp,#8]

cbz x3,.Lstore_base2_64_neon
@@ -274,7 +263,7 @@
adcs x5,x5,x13
adc x6,x6,x3

- bl poly1305_mult
+ bl __poly1305_mult

.Linit_neon:
and x10,x4,#0x03ffffff // base 2^64 -> base 2^26
@@ -301,19 +290,19 @@
mov x5,x8
mov x6,xzr
add x0,x0,#48+12
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^2
+ bl __poly1305_mult // r^2
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^3
+ bl __poly1305_mult // r^3
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat

- bl poly1305_mult // r^4
+ bl __poly1305_mult // r^4
sub x0,x0,#4
- bl poly1305_splat
+ bl __poly1305_splat
ldr x30,[sp,#8]

add x16,x1,#32
@@ -743,10 +732,11 @@
.Lno_data_neon:
ldr x29,[sp],#80
ret
+ENDPROC(poly1305_blocks_neon)

-poly1305_emit_neon:
+ENTRY(poly1305_emit_neon)
ldr x17,[x0,#24]
- cbz x17,poly1305_emit
+ cbz x17,poly1305_emit_arm

ldp w10,w11,[x0] // load hash value base 2^26
ldp w12,w13,[x0,#8]
@@ -788,6 +778,6 @@
stp x4,x5,[x1] // write result

ret
+ENDPROC(poly1305_emit_neon)

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                     |    2 +
 lib/zinc/poly1305/poly1305-arm-glue.h |   65 ++
 lib/zinc/poly1305/poly1305-arm.S      | 1117 +++++++++++++++++++++++++
 lib/zinc/poly1305/poly1305-arm64.S    |  822 ++++++++++++++++++
 lib/zinc/poly1305/poly1305.c          |    2 +
 5 files changed, 2008 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-arm-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-arm.S
 create mode 100644 lib/zinc/poly1305/poly1305-arm64.S
lib/zinc/Makefile +2 -0
index f61d5ff4e386..1970d912b4df 100644
@@ -12,4 +12,6 @@ obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
 
 zinc_poly1305-y := poly1305/poly1305.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o
+zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM) += poly1305/poly1305-arm.o
+zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM64) += poly1305/poly1305-arm64.o
 obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
lib/zinc/poly1305/poly1305-arm-glue.h new file +65 -0

65 lines not on this page · raw message

lib/zinc/poly1305/poly1305-arm.S new file +1117 -0

1,117 lines not on this page · raw message

lib/zinc/poly1305/poly1305-arm64.S new file +822 -0

822 lines not on this page · raw message

lib/zinc/poly1305/poly1305.c +2 -0
index 8bf2b95ca615..75032146fd50 100644
@@ -17,6 +17,8 @@
 
 #if defined(CONFIG_ZINC_ARCH_X86_64)
 #include "poly1305-x86_64-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
+#include "poly1305-arm-glue.h"
 #else
 static inline bool poly1305_init_arch(void *ctx,
 				      const u8 key[POLY1305_KEY_SIZE])
Signature · 1 line
2.19.0
#42 lore raw

Re: [PATCH net-next v5 09/20] zinc: Poly1305 ARM and ARM64 implementations

Quoted 47 lines · Jason A. Donenfeld
lib/zinc/poly1305/poly1305-arm-glue.h new file +41 -0
new file mode 100644
index 000000000000..dd3fa5a38c62
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/hwcap.h>
+#include <asm/neon.h>
+
+asmlinkage void poly1305_init_arm(void *ctx, const u8 key[16]);
+asmlinkage void poly1305_blocks_arm(void *ctx, const u8 *inp, const size_t len,
+				    const u32 padbit);
+asmlinkage void poly1305_emit_arm(void *ctx, u8 mac[16], const u32 nonce[4]);
+#if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&                                     \
+	(defined(CONFIG_64BIT) || __LINUX_ARM_ARCH__ >= 7)
+#define ARM_USE_NEON
+asmlinkage void poly1305_blocks_neon(void *ctx, const u8 *inp, const size_t len,
+				     const u32 padbit);
+asmlinkage void poly1305_emit_neon(void *ctx, u8 mac[16], const u32 nonce[4]);
+#endif
+
+static bool poly1305_use_neon __ro_after_init;
+
+static void __init poly1305_fpu_init(void)
+{
+#if defined(CONFIG_ARM64)
+	poly1305_use_neon = elf_hwcap & HWCAP_ASIMD;
+#elif defined(CONFIG_ARM)
+	poly1305_use_neon = elf_hwcap & HWCAP_NEON;
+#endif
+}
+
+static inline bool poly1305_init_arch(void *ctx,
+				      const u8 key[POLY1305_KEY_SIZE])
+{
+	poly1305_init_arm(ctx, key);
+	return true;
+}
+
+static inline bool poly1305_blocks_arch(void *ctx, const u8 *inp,
+					const size_t len, const u32 padbit,
+					simd_context_t *simd_context)

lib/zinc/poly1305/poly1305-arm-glue.h line 51

Quoted 10 lines
lib/zinc/poly1305/poly1305-arm-glue.h new file +10 -0
+{
+#if defined(ARM_USE_NEON)
+	if (poly1305_use_neon && simd_use(simd_context)) {
+		poly1305_blocks_neon(ctx, inp, len, padbit);
+		return true;
+	}
+#endif
+	poly1305_blocks_arm(ctx, inp, len, padbit);
+	return true;
+}

This will compute the wrong digest if called with simd_context=HAVE_FULL_SIMD and then later with simd_context=HAVE_NO_SIMD, since poly1305_blocks_neon() converts the accumulator from base 32 to base 26, whereas poly1305_blocks_arm() assumes it is still in base 32. Is that intentional? I'm sure this is a rare case, but my understanding is that the existing crypto API doesn't preclude calling successive steps in different contexts. And I'm concerned that it could be relevant in some cases, e.g. especially if people are importing a hash state that was exported earlier. Handling it by silently computing the wrong digest is not a great idea...

- Eric

author maintainer
#43 lore raw

Re: [PATCH net-next v5 09/20] zinc: Poly1305 ARM and ARM64 implementations

Hi Eric,

Quoted 9 lines · Eric Biggers

This will compute the wrong digest if called with simd_context=HAVE_FULL_SIMD and then later with simd_context=HAVE_NO_SIMD, since poly1305_blocks_neon() converts the accumulator from base 32 to base 26, whereas poly1305_blocks_arm() assumes it is still in base 32. Is that intentional? I'm sure this is a rare case, but my understanding is that the existing crypto API doesn't preclude calling successive steps in different contexts. And I'm concerned that it could be relevant in some cases, e.g. especially if people are importing a hash state that was exported earlier. Handling it by silently computing the wrong digest is not a great idea...

Indeed you're right; Samuel and I were just discussing that recently. I'd rather handle this correctly even if the contexts change, so I'll see if I can fix this up properly for that unlikely case in the next revision.

Jason

author maintainer
#44 lore raw

PATCH v5 10/20 zinc: Poly1305 MIPS32r2 and MIPS64 implementations

In reply to Jason A. Donenfeld

This MIPS32r2 implementation comes from René van Dorst and me and results in a nice speedup on the usual OpenWRT targets. The MIPS64 implementation comes from Andy Polyakov results in a nice speedup on commodity Octeon hardware, and has been modified slightly from the original:

- The function names have been renamed to fit kernel conventions.
- A comment has been added.

No changes have been made to the actual instructions.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Andy Polyakov <[email protected]>
Cc Ralf Baechle <[email protected]>
Cc Paul Burton <[email protected]>
Cc James Hogan <[email protected]>
---
 lib/zinc/Makefile                      |   2 +
 lib/zinc/poly1305/poly1305-mips-glue.h |  35 +++
 lib/zinc/poly1305/poly1305-mips.S      | 417 +++++++++++++++++++++++++
 lib/zinc/poly1305/poly1305-mips64.S    | 359 +++++++++++++++++++++
 lib/zinc/poly1305/poly1305.c           |   2 +
 5 files changed, 815 insertions(+)
 create mode 100644 lib/zinc/poly1305/poly1305-mips-glue.h
 create mode 100644 lib/zinc/poly1305/poly1305-mips.S
 create mode 100644 lib/zinc/poly1305/poly1305-mips64.S
lib/zinc/Makefile +2 -0
index 1970d912b4df..db51764a9c05 100644
@@ -14,4 +14,6 @@ zinc_poly1305-y := poly1305/poly1305.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_X86_64) += poly1305/poly1305-x86_64.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM) += poly1305/poly1305-arm.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM64) += poly1305/poly1305-arm64.o
+zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS) += poly1305/poly1305-mips.o
+zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS64) += poly1305/poly1305-mips64.o
 obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
lib/zinc/poly1305/poly1305-mips-glue.h new file +35 -0
new file mode 100644
index 000000000000..2932bb987fd9
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+asmlinkage void poly1305_init_mips(void *ctx, const u8 key[16]);
+asmlinkage void poly1305_blocks_mips(void *ctx, const u8 *inp, const size_t len,
+				     const u32 padbit);
+asmlinkage void poly1305_emit_mips(void *ctx, u8 mac[16], const u32 nonce[4]);
+static void __init poly1305_fpu_init(void)
+{
+}
+
+static inline bool poly1305_init_arch(void *ctx,
+				      const u8 key[POLY1305_KEY_SIZE])
+{
+	poly1305_init_mips(ctx, key);
+	return true;
+}
+
+static inline bool poly1305_blocks_arch(void *ctx, const u8 *inp,
+					const size_t len, const u32 padbit,
+					simd_context_t *simd_context)
+{
+	poly1305_blocks_mips(ctx, inp, len, padbit);
+	return true;
+}
+
+static inline bool poly1305_emit_arch(void *ctx, u8 mac[POLY1305_MAC_SIZE],
+				      const u32 nonce[4],
+				      simd_context_t *simd_context)
+{
+	poly1305_emit_mips(ctx, mac, nonce);
+	return true;
+}
lib/zinc/poly1305/poly1305-mips.S new file +417 -0

417 lines not on this page · raw message

lib/zinc/poly1305/poly1305-mips64.S new file +359 -0

359 lines not on this page · raw message

lib/zinc/poly1305/poly1305.c +2 -0
index 75032146fd50..ac53a1e36d5b 100644
@@ -19,6 +19,8 @@
 #include "poly1305-x86_64-glue.h"
 #elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
 #include "poly1305-arm-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_MIPS) || defined(CONFIG_ZINC_ARCH_MIPS64)
+#include "poly1305-mips-glue.h"
 #else
 static inline bool poly1305_init_arch(void *ctx,
 				      const u8 key[POLY1305_KEY_SIZE])
Signature · 1 line
2.19.0
author maintainer
#45 lore raw

PATCH v5 11/20 zinc: ChaCha20Poly1305 construction and selftest

In reply to Jason A. Donenfeld

This is an implementation of the ChaCha20Poly1305 AEAD, with an easy API for encrypting either contiguous buffers or scatter gather lists (such as those created from skb_to_sgvec).

Information: https://tools.ietf.org/html/rfc8439

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/chacha20poly1305.h      |   50 +
 lib/zinc/Kconfig                     |    6 +
 lib/zinc/Makefile                    |    3 +
 lib/zinc/chacha20poly1305.c          |  359 ++
 lib/zinc/selftest/chacha20poly1305.h | 7854 ++++++++++++++++++++++++++
 5 files changed, 8272 insertions(+)
 create mode 100644 include/zinc/chacha20poly1305.h
 create mode 100644 lib/zinc/chacha20poly1305.c
 create mode 100644 lib/zinc/selftest/chacha20poly1305.h
include/zinc/chacha20poly1305.h new file +50 -0
new file mode 100644
index 000000000000..339980deb4df
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CHACHA20POLY1305_H
+#define _ZINC_CHACHA20POLY1305_H
+
+#include <linux/simd.h>
+#include <linux/types.h>
+
+struct scatterlist;
+
+enum chacha20poly1305_lengths {
+	XCHACHA20POLY1305_NONCELEN = 24,
+	CHACHA20POLY1305_KEYLEN = 32,
+	CHACHA20POLY1305_AUTHTAGLEN = 16
+};
+
+void chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
+			      const u8 *ad, const size_t ad_len,
+			      const u64 nonce,
+			      const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check chacha20poly1305_encrypt_sg(
+	struct scatterlist *dst, struct scatterlist *src, const size_t src_len,
+	const u8 *ad, const size_t ad_len, const u64 nonce,
+	const u8 key[CHACHA20POLY1305_KEYLEN], simd_context_t *simd_context);
+
+bool __must_check
+chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
+			 const u8 *ad, const size_t ad_len, const u64 nonce,
+			 const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check chacha20poly1305_decrypt_sg(
+	struct scatterlist *dst, struct scatterlist *src, const size_t src_len,
+	const u8 *ad, const size_t ad_len, const u64 nonce,
+	const u8 key[CHACHA20POLY1305_KEYLEN], simd_context_t *simd_context);
+
+void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
+			       const u8 *ad, const size_t ad_len,
+			       const u8 nonce[XCHACHA20POLY1305_NONCELEN],
+			       const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+bool __must_check xchacha20poly1305_decrypt(
+	u8 *dst, const u8 *src, const size_t src_len, const u8 *ad,
+	const size_t ad_len, const u8 nonce[XCHACHA20POLY1305_NONCELEN],
+	const u8 key[CHACHA20POLY1305_KEYLEN]);
+
+#endif /* _ZINC_CHACHA20POLY1305_H */
lib/zinc/Kconfig +6 -0
index f9ef8f7e3c25..dec4633cdfb4 100644
@@ -5,6 +5,12 @@ config ZINC_CHACHA20
 config ZINC_POLY1305
 	tristate
 
+config ZINC_CHACHA20POLY1305
+	tristate
+	select ZINC_CHACHA20
+	select ZINC_POLY1305
+	select CRYPTO_BLKCIPHER
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index db51764a9c05..9da1463cdfcb 100644
@@ -17,3 +17,6 @@ zinc_poly1305-$(CONFIG_ZINC_ARCH_ARM64) += poly1305/poly1305-arm64.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS) += poly1305/poly1305-mips.o
 zinc_poly1305-$(CONFIG_ZINC_ARCH_MIPS64) += poly1305/poly1305-mips64.o
 obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
+
+zinc_chacha20poly1305-y := chacha20poly1305.o
+obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o
lib/zinc/chacha20poly1305.c new file +359 -0

359 lines not on this page · raw message

lib/zinc/selftest/chacha20poly1305.h new file +7854 -0

2,573 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#46 lore raw

PATCH v5 12/20 zinc: BLAKE2s generic C implementation and selftest

In reply to Jason A. Donenfeld

The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.

Information: https://blake2.net/

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/blake2s.h      |   95 ++
 lib/zinc/Kconfig            |    3 +
 lib/zinc/Makefile           |    3 +
 lib/zinc/blake2s/blake2s.c  |  301 +++++
 lib/zinc/selftest/blake2s.h | 2095 +++++++++++++++++++++++++++++++++++
 5 files changed, 2497 insertions(+)
 create mode 100644 include/zinc/blake2s.h
 create mode 100644 lib/zinc/blake2s/blake2s.c
 create mode 100644 lib/zinc/selftest/blake2s.h
include/zinc/blake2s.h new file +95 -0
new file mode 100644
index 000000000000..951281596274
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_BLAKE2S_H
+#define _ZINC_BLAKE2S_H
+
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <crypto/algapi.h>
+
+enum blake2s_lengths {
+	BLAKE2S_BLOCKBYTES = 64,
+	BLAKE2S_OUTBYTES = 32,
+	BLAKE2S_KEYBYTES = 32
+};
+
+struct blake2s_state {
+	u32 h[8];
+	u32 t[2];
+	u32 f[2];
+	u8 buf[BLAKE2S_BLOCKBYTES];
+	size_t buflen;
+	u8 last_node;
+};
+
+void blake2s_init(struct blake2s_state *state, const size_t outlen);
+void blake2s_init_key(struct blake2s_state *state, const size_t outlen,
+		      const void *key, const size_t keylen);
+void blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen);
+void __blake2s_final(struct blake2s_state *state);
+static inline void blake2s_final(struct blake2s_state *state, u8 *out,
+				 const size_t outlen)
+{
+	int i;
+
+#ifdef DEBUG
+	BUG_ON(!out || !outlen || outlen > BLAKE2S_OUTBYTES);
+#endif
+	__blake2s_final(state);
+
+	if (__builtin_constant_p(outlen) && !(outlen % sizeof(u32))) {
+		if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+		    IS_ALIGNED((unsigned long)out, __alignof__(u32))) {
+			__le32 *outwords = (__le32 *)out;
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				outwords[i] = cpu_to_le32(state->h[i]);
+		} else {
+			__le32 buffer[BLAKE2S_OUTBYTES];
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				buffer[i] = cpu_to_le32(state->h[i]);
+			memcpy(out, buffer, outlen);
+			memzero_explicit(buffer, sizeof(buffer));
+		}
+	} else {
+		u8 buffer[BLAKE2S_OUTBYTES] __aligned(__alignof__(u32));
+		__le32 *outwords = (__le32 *)buffer;
+
+		for (i = 0; i < 8; ++i)
+			outwords[i] = cpu_to_le32(state->h[i]);
+		memcpy(out, buffer, outlen);
+		memzero_explicit(buffer, sizeof(buffer));
+	}
+
+	memzero_explicit(state, sizeof(*state));
+}
+
+static inline void blake2s(u8 *out, const u8 *in, const u8 *key,
+			   const size_t outlen, const size_t inlen,
+			   const size_t keylen)
+{
+	struct blake2s_state state;
+
+#ifdef DEBUG
+	BUG_ON((!in && inlen > 0) || !out || !outlen ||
+	       outlen > BLAKE2S_OUTBYTES || keylen > BLAKE2S_KEYBYTES ||
+	       (!key && keylen));
+#endif
+
+	if (keylen)
+		blake2s_init_key(&state, outlen, key, keylen);
+	else
+		blake2s_init(&state, outlen);
+
+	blake2s_update(&state, in, inlen);
+	blake2s_final(&state, out, outlen);
+}
+
+void blake2s_hmac(u8 *out, const u8 *in, const u8 *key, const size_t outlen,
+		  const size_t inlen, const size_t keylen);
+
+#endif /* _ZINC_BLAKE2S_H */
lib/zinc/Kconfig +3 -0
index dec4633cdfb4..7bf4bc88f81f 100644
@@ -11,6 +11,9 @@ config ZINC_CHACHA20POLY1305
 	select ZINC_POLY1305
 	select CRYPTO_BLKCIPHER
 
+config ZINC_BLAKE2S
+	tristate
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index 9da1463cdfcb..0e0020f58d27 100644
@@ -20,3 +20,6 @@ obj-$(CONFIG_ZINC_POLY1305) += zinc_poly1305.o
 
 zinc_chacha20poly1305-y := chacha20poly1305.o
 obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o
+
+zinc_blake2s-y := blake2s/blake2s.o
+obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o
lib/zinc/blake2s/blake2s.c new file +301 -0

301 lines not on this page · raw message

lib/zinc/selftest/blake2s.h new file +2095 -0

2,095 lines not on this page · raw message

Signature · 1 line
2.19.0
#47 lore raw

Re: [PATCH net-next v5 12/20] zinc: BLAKE2s generic C implementation and selftest

Quoted 70 lines · Jason A. Donenfeld

The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file.

Information: https://blake2.net/

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
---
 include/zinc/blake2s.h      |   95 ++
 lib/zinc/Kconfig            |    3 +
 lib/zinc/Makefile           |    3 +
 lib/zinc/blake2s/blake2s.c  |  301 +++++
 lib/zinc/selftest/blake2s.h | 2095 +++++++++++++++++++++++++++++++++++
 5 files changed, 2497 insertions(+)
 create mode 100644 include/zinc/blake2s.h
 create mode 100644 lib/zinc/blake2s/blake2s.c
 create mode 100644 lib/zinc/selftest/blake2s.h
include/zinc/blake2s.h new file +41 -0
new file mode 100644
index 000000000000..951281596274
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_BLAKE2S_H
+#define _ZINC_BLAKE2S_H
+
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <crypto/algapi.h>
+
+enum blake2s_lengths {
+	BLAKE2S_BLOCKBYTES = 64,
+	BLAKE2S_OUTBYTES = 32,
+	BLAKE2S_KEYBYTES = 32
+};
+
+struct blake2s_state {
+	u32 h[8];
+	u32 t[2];
+	u32 f[2];
+	u8 buf[BLAKE2S_BLOCKBYTES];
+	size_t buflen;
+	u8 last_node;
+};
+
+void blake2s_init(struct blake2s_state *state, const size_t outlen);
+void blake2s_init_key(struct blake2s_state *state, const size_t outlen,
+		      const void *key, const size_t keylen);
+void blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen);
+void __blake2s_final(struct blake2s_state *state);
+static inline void blake2s_final(struct blake2s_state *state, u8 *out,
+				 const size_t outlen)
+{
+	int i;
+
+#ifdef DEBUG
+	BUG_ON(!out || !outlen || outlen > BLAKE2S_OUTBYTES);
+#endif
+	__blake2s_final(state);

include/zinc/blake2s.h line 51

Quoted 10 lines
include/zinc/blake2s.h new file +10 -0
+
+	if (__builtin_constant_p(outlen) && !(outlen % sizeof(u32))) {
+		if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+		    IS_ALIGNED((unsigned long)out, __alignof__(u32))) {
+			__le32 *outwords = (__le32 *)out;
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				outwords[i] = cpu_to_le32(state->h[i]);
+		} else {
+			__le32 buffer[BLAKE2S_OUTBYTES];

This buffer is 4 times too long.

Quoted 8 lines
include/zinc/blake2s.h new file +8 -0
+
+			for (i = 0; i < outlen / sizeof(u32); ++i)
+				buffer[i] = cpu_to_le32(state->h[i]);
+			memcpy(out, buffer, outlen);
+			memzero_explicit(buffer, sizeof(buffer));
+		}
+	} else {
+		u8 buffer[BLAKE2S_OUTBYTES] __aligned(__alignof__(u32));

include/zinc/blake2s.h line 69

Quoted 10 lines
include/zinc/blake2s.h new file +10 -0
+		__le32 *outwords = (__le32 *)buffer;
+
+		for (i = 0; i < 8; ++i)
+			outwords[i] = cpu_to_le32(state->h[i]);
+		memcpy(out, buffer, outlen);
+		memzero_explicit(buffer, sizeof(buffer));
+	}
+
+	memzero_explicit(state, sizeof(*state));
+}

Or how about something much simpler:

static inline void blake2s_final(struct blake2s_state *state, u8 *out,
const size_t outlen)
{

#ifdef DEBUG
	BUG_ON(!out || !outlen || outlen > BLAKE2S_OUTBYTES);
#endif
	__blake2s_final(state);

	cpu_to_le32_array(state->h, ARRAY_SIZE(state->h));
	memcpy(out, state->h, outlen);

	memzero_explicit(state, sizeof(*state));
}
author maintainer
#48 lore raw

Re: [PATCH net-next v5 12/20] zinc: BLAKE2s generic C implementation and selftest

Hey Eric,

Quoted 1 line · Eric Biggers

This buffer is 4 times too long.

Nice catch.

Quoted 5 lines

Or how about something much simpler:

static inline void blake2s_final(struct blake2s_state *state, u8 *out,
const size_t outlen)
{

Quoted 10 lines
#ifdef DEBUG
        BUG_ON(!out || !outlen || outlen > BLAKE2S_OUTBYTES);
#endif
        __blake2s_final(state);

        cpu_to_le32_array(state->h, ARRAY_SIZE(state->h));
        memcpy(out, state->h, outlen);

        memzero_explicit(state, sizeof(*state));
}

Oh, that's excellent, thanks. Much better than prior. I'll do exactly that.

Jason

author maintainer
#49 lore raw

PATCH v5 13/20 zinc: BLAKE2s x86_64 implementation

In reply to Jason A. Donenfeld

These implementations from Samuel Neves support AVX and AVX-512VL. Originally this used AVX-512F, but Skylake thermal throttling made AVX-512VL more attractive and possible to do with negligable difference.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/Makefile                      |   1 +
 lib/zinc/blake2s/blake2s-x86_64-glue.h |  59 +++
 lib/zinc/blake2s/blake2s-x86_64.S      | 685 +++++++++++++++++++++++++
 lib/zinc/blake2s/blake2s.c             |   4 +-
 4 files changed, 748 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/blake2s/blake2s-x86_64-glue.h
 create mode 100644 lib/zinc/blake2s/blake2s-x86_64.S
lib/zinc/Makefile +1 -0
index 0e0020f58d27..47607486c39d 100644
@@ -22,4 +22,5 @@ zinc_chacha20poly1305-y := chacha20poly1305.o
 obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o
 
 zinc_blake2s-y := blake2s/blake2s.o
+zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o
 obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o
lib/zinc/blake2s/blake2s-x86_64-glue.h new file +59 -0

59 lines not on this page · raw message

lib/zinc/blake2s/blake2s-x86_64.S new file +685 -0

685 lines not on this page · raw message

lib/zinc/blake2s/blake2s.c +3 -1
index 0b62313267fb..c54f017fccfe 100644
@@ -113,7 +113,9 @@ void blake2s_init_key(struct blake2s_state *state, const size_t outlen,
 }
 EXPORT_SYMBOL(blake2s_init_key);
 
-#ifndef HAVE_BLAKE2S_ARCH_IMPLEMENTATION
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "blake2s-x86_64-glue.h"
+#else
 static void __init blake2s_fpu_init(void)
 {
 }
Signature · 1 line
2.19.0
author maintainer
#50 lore raw

PATCH v5 14/20 zinc: Curve25519 generic C implementations and selftest

In reply to Jason A. Donenfeld

This contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic.

The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star
The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto

Information: https://cr.yp.to/ecdh.html

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Karthikeyan Bhargavan <[email protected]>
---
 include/zinc/curve25519.h               |   22 +
 lib/zinc/Kconfig                        |    4 +
 lib/zinc/Makefile                       |    3 +
 lib/zinc/curve25519/curve25519-fiat32.h |  862 +++++++++++++++
 lib/zinc/curve25519/curve25519-hacl64.h |  785 ++++++++++++++
 lib/zinc/curve25519/curve25519.c        |  109 ++
 lib/zinc/selftest/curve25519.h          | 1321 +++++++++++++++++++++++
 7 files changed, 3106 insertions(+)
 create mode 100644 include/zinc/curve25519.h
 create mode 100644 lib/zinc/curve25519/curve25519-fiat32.h
 create mode 100644 lib/zinc/curve25519/curve25519-hacl64.h
 create mode 100644 lib/zinc/curve25519/curve25519.c
 create mode 100644 lib/zinc/selftest/curve25519.h
include/zinc/curve25519.h new file +22 -0
new file mode 100644
index 000000000000..5da800ff0b0d
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _ZINC_CURVE25519_H
+#define _ZINC_CURVE25519_H
+
+#include <linux/types.h>
+
+enum curve25519_lengths {
+	CURVE25519_POINT_SIZE = 32
+};
+
+bool __must_check curve25519(u8 mypublic[CURVE25519_POINT_SIZE],
+			     const u8 secret[CURVE25519_POINT_SIZE],
+			     const u8 basepoint[CURVE25519_POINT_SIZE]);
+void curve25519_generate_secret(u8 secret[CURVE25519_POINT_SIZE]);
+bool __must_check curve25519_generate_public(
+	u8 pub[CURVE25519_POINT_SIZE], const u8 secret[CURVE25519_POINT_SIZE]);
+
+#endif /* _ZINC_CURVE25519_H */
lib/zinc/Kconfig +4 -0
index 7bf4bc88f81f..0d8a94dd73a8 100644
@@ -14,6 +14,10 @@ config ZINC_CHACHA20POLY1305
 config ZINC_BLAKE2S
 	tristate
 
+config ZINC_CURVE25519
+	tristate
+	select CONFIG_CRYPTO
+
 config ZINC_DEBUG
 	bool "Zinc cryptography library debugging and self-tests"
 	help
lib/zinc/Makefile +3 -0
index 47607486c39d..aba1740876d8 100644
@@ -24,3 +24,6 @@ obj-$(CONFIG_ZINC_CHACHA20POLY1305) += zinc_chacha20poly1305.o
 zinc_blake2s-y := blake2s/blake2s.o
 zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o
 obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o
+
+zinc_curve25519-y := curve25519/curve25519.o
+obj-$(CONFIG_ZINC_CURVE25519) += zinc_curve25519.o
lib/zinc/curve25519/curve25519-fiat32.h new file +862 -0

862 lines not on this page · raw message

lib/zinc/curve25519/curve25519-hacl64.h new file +785 -0

785 lines not on this page · raw message

lib/zinc/curve25519/curve25519.c new file +109 -0

109 lines not on this page · raw message

lib/zinc/selftest/curve25519.h new file +1321 -0

1,206 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

author maintainer
#51 lore raw

PATCH v5 15/20 zinc: Curve25519 x86_64 implementation

In reply to Jason A. Donenfeld

This implementation is the fastest available x86_64 implementation, and unlike Sandy2x, it doesn't requie use of the floating point registers at all. Instead it makes use of BMI2 and ADX, available on recent microarchitectures. The implementation was written by Armando Faz-Hernández with contributions (upstream) from Samuel Neves and me, in addition to further changes in the kernel implementation from us.

Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Armando Faz-Hernández <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 lib/zinc/curve25519/curve25519-x86_64-glue.h |   46 +
 lib/zinc/curve25519/curve25519-x86_64.h      | 2333 ++++++++++++++++++
 lib/zinc/curve25519/curve25519.c             |    4 +-
 3 files changed, 2382 insertions(+), 1 deletion(-)
 create mode 100644 lib/zinc/curve25519/curve25519-x86_64-glue.h
 create mode 100644 lib/zinc/curve25519/curve25519-x86_64.h
lib/zinc/curve25519/curve25519-x86_64-glue.h new file +46 -0
new file mode 100644
index 000000000000..1b9b71922a82
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#include <asm/cpufeature.h>
+#include <asm/processor.h>
+
+#include "curve25519-x86_64.h"
+
+static bool curve25519_use_bmi2 __ro_after_init;
+static bool curve25519_use_adx __ro_after_init;
+
+static void __init curve25519_fpu_init(void)
+{
+	curve25519_use_bmi2 = boot_cpu_has(X86_FEATURE_BMI2);
+	curve25519_use_adx = boot_cpu_has(X86_FEATURE_BMI2) &&
+			     boot_cpu_has(X86_FEATURE_ADX);
+}
+
+static inline bool curve25519_arch(u8 mypublic[CURVE25519_POINT_SIZE],
+				   const u8 secret[CURVE25519_POINT_SIZE],
+				   const u8 basepoint[CURVE25519_POINT_SIZE])
+{
+	if (curve25519_use_adx) {
+		curve25519_adx(mypublic, secret, basepoint);
+		return true;
+	} else if (curve25519_use_bmi2) {
+		curve25519_bmi2(mypublic, secret, basepoint);
+		return true;
+	}
+	return false;
+}
+
+static inline bool curve25519_base_arch(u8 pub[CURVE25519_POINT_SIZE],
+					const u8 secret[CURVE25519_POINT_SIZE])
+{
+	if (curve25519_use_adx) {
+		curve25519_adx_base(pub, secret);
+		return true;
+	} else if (curve25519_use_bmi2) {
+		curve25519_bmi2_base(pub, secret);
+		return true;
+	}
+	return false;
+}
lib/zinc/curve25519/curve25519-x86_64.h new file +2333 -0

2,333 lines not on this page · raw message

lib/zinc/curve25519/curve25519.c +3 -1
index 61dff8578a77..d3d4445d1ac2 100644
@@ -19,7 +19,9 @@
 #include <linux/init.h>
 #include <crypto/algapi.h>
 
-#ifndef HAVE_CURVE25519_ARCH_IMPLEMENTATION
+#if defined(CONFIG_ZINC_ARCH_X86_64)
+#include "curve25519-x86_64-glue.h"
+#else
 void __init curve25519_fpu_init(void)
 {
 }
Signature · 1 line
2.19.0
author maintainer
#52 lore raw

PATCH v5 16/20 zinc: Curve25519 ARM implementation

In reply to Jason A. Donenfeld

This comes from Dan Bernstein and Peter Schwabe's public domain NEON code, and has been modified to be friendly for kernel space, as well as removing some qhasm strangeness to be more idiomatic.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Russell King <[email protected]>
Cc linux-arm-kernel at lists.infradead.org
---
 lib/zinc/Makefile                         |    1 +
 lib/zinc/curve25519/curve25519-arm-glue.h |   43 +
 lib/zinc/curve25519/curve25519-arm.S      | 2095 +++++++++++++++++++++
 lib/zinc/curve25519/curve25519.c          |    2 +
 4 files changed, 2141 insertions(+)
 create mode 100644 lib/zinc/curve25519/curve25519-arm-glue.h
 create mode 100644 lib/zinc/curve25519/curve25519-arm.S
lib/zinc/Makefile +1 -0
index aba1740876d8..29658d39c3e8 100644
@@ -26,4 +26,5 @@ zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o
 obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o
 
 zinc_curve25519-y := curve25519/curve25519.o
+zinc_curve25519-$(CONFIG_ZINC_ARCH_ARM) += curve25519/curve25519-arm.o
 obj-$(CONFIG_ZINC_CURVE25519) += zinc_curve25519.o
lib/zinc/curve25519/curve25519-arm-glue.h new file +43 -0

43 lines not on this page · raw message

lib/zinc/curve25519/curve25519-arm.S new file +2095 -0

2,095 lines not on this page · raw message

lib/zinc/curve25519/curve25519.c +2 -0
index d3d4445d1ac2..38a624bba913 100644
@@ -21,6 +21,8 @@
 
 #if defined(CONFIG_ZINC_ARCH_X86_64)
 #include "curve25519-x86_64-glue.h"
+#elif defined(CONFIG_ZINC_ARCH_ARM)
+#include "curve25519-arm-glue.h"
 #else
 void __init curve25519_fpu_init(void)
 {
Signature · 1 line
2.19.0
author maintainer
#53 lore raw

PATCH v5 17/20 crypto: port Poly1305 to Zinc

In reply to Jason A. Donenfeld

Now that Poly1305 is in Zinc, we can have the crypto API code simply call into it. We have to do a little bit of book keeping here, because the crypto API receives the key in the first few calls to update.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
---
 arch/x86/crypto/Makefile               |   3 -
 arch/x86/crypto/poly1305-avx2-x86_64.S | 388 ----------------
 arch/x86/crypto/poly1305-sse2-x86_64.S | 584 -------------------------
 arch/x86/crypto/poly1305_glue.c        | 205 ---------
 crypto/Kconfig                         |  15 +-
 crypto/Makefile                        |   2 +-
 crypto/chacha20poly1305.c              |  12 +-
 crypto/poly1305_generic.c              | 304 -------------
 crypto/poly1305_zinc.c                 |  98 +++++
 include/crypto/poly1305.h              |  40 --
 10 files changed, 107 insertions(+), 1544 deletions(-)
 delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305_glue.c
 delete mode 100644 crypto/poly1305_generic.c
 create mode 100644 crypto/poly1305_zinc.c
 delete mode 100644 include/crypto/poly1305.h
arch/x86/crypto/Makefile +0 -3
index a450ad573dcb..cf830219846b 100644
@@ -34,7 +34,6 @@ obj-$(CONFIG_CRYPTO_CRC32_PCLMUL) += crc32-pclmul.o
 obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o
 obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
 obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o
-obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o
 obj-$(CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2) += aegis128l-aesni.o
@@ -110,10 +109,8 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
-poly1305-x86_64-y := poly1305-sse2-x86_64.o poly1305_glue.o
 ifeq ($(avx2_supported),yes)
 sha1-ssse3-y += sha1_avx2_x86_64_asm.o
-poly1305-x86_64-y += poly1305-avx2-x86_64.o
 endif
 ifeq ($(sha1_ni_supported),yes)
 sha1-ssse3-y += sha1_ni_asm.o
arch/x86/crypto/poly1305-avx2-x86_64.S deleted +0 -388

388 lines not on this page · raw message

arch/x86/crypto/poly1305-sse2-x86_64.S deleted +0 -584

584 lines not on this page · raw message

arch/x86/crypto/poly1305_glue.c deleted +0 -205

205 lines not on this page · raw message

crypto/Kconfig +2 -13
index f3e40ac56d93..47859a0f8052 100644
@@ -656,24 +656,13 @@ config CRYPTO_GHASH
 config CRYPTO_POLY1305
 	tristate "Poly1305 authenticator algorithm"
 	select CRYPTO_HASH
+	select ZINC_POLY1305
 	help
 	  Poly1305 authenticator algorithm, RFC7539.
 
 	  Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein.
 	  It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use
-	  in IETF protocols. This is the portable C implementation of Poly1305.
-
-config CRYPTO_POLY1305_X86_64
-	tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)"
-	depends on X86 && 64BIT
-	select CRYPTO_POLY1305
-	help
-	  Poly1305 authenticator algorithm, RFC7539.
-
-	  Poly1305 is an authenticator algorithm designed by Daniel J. Bernstein.
-	  It is used for the ChaCha20-Poly1305 AEAD, specified in RFC7539 for use
-	  in IETF protocols. This is the x86_64 assembler implementation using SIMD
-	  instructions.
+	  in IETF protocols.
 
 config CRYPTO_MD4
 	tristate "MD4 digest algorithm"
crypto/Makefile +1 -1
index 6d1d40eeb964..5e60348d02e2 100644
@@ -118,7 +118,7 @@ obj-$(CONFIG_CRYPTO_SEED) += seed.o
 obj-$(CONFIG_CRYPTO_SPECK) += speck.o
 obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o
 obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o
-obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_generic.o
+obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
 obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o
crypto/chacha20poly1305.c +6 -6
index 600afa99941f..bf523797bef3 100644
@@ -14,7 +14,7 @@
 #include <crypto/internal/skcipher.h>
 #include <crypto/scatterwalk.h>
 #include <crypto/chacha20.h>
-#include <crypto/poly1305.h>
+#include <zinc/poly1305.h>
 #include <linux/err.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
@@ -62,7 +62,7 @@ struct chachapoly_req_ctx {
 	/* the key we generate for Poly1305 using Chacha20 */
 	u8 key[POLY1305_KEY_SIZE];
 	/* calculated Poly1305 tag */
-	u8 tag[POLY1305_DIGEST_SIZE];
+	u8 tag[POLY1305_MAC_SIZE];
 	/* length of data to en/decrypt, without ICV */
 	unsigned int cryptlen;
 	/* Actual AD, excluding IV */
@@ -471,7 +471,7 @@ static int chachapoly_decrypt(struct aead_request *req)
 {
 	struct chachapoly_req_ctx *rctx = aead_request_ctx(req);
 
-	rctx->cryptlen = req->cryptlen - POLY1305_DIGEST_SIZE;
+	rctx->cryptlen = req->cryptlen - POLY1305_MAC_SIZE;
 
 	/* decrypt call chain:
 	 * - poly_genkey/done()
@@ -513,7 +513,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 static int chachapoly_setauthsize(struct crypto_aead *tfm,
 				  unsigned int authsize)
 {
-	if (authsize != POLY1305_DIGEST_SIZE)
+	if (authsize != POLY1305_MAC_SIZE)
 		return -EINVAL;
 
 	return 0;
@@ -613,7 +613,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
 	poly_hash = __crypto_hash_alg_common(poly);
 
 	err = -EINVAL;
-	if (poly_hash->digestsize != POLY1305_DIGEST_SIZE)
+	if (poly_hash->digestsize != POLY1305_MAC_SIZE)
 		goto out_put_poly;
 
 	err = -ENOMEM;
@@ -666,7 +666,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
 				     ctx->saltlen;
 	inst->alg.ivsize = ivsize;
 	inst->alg.chunksize = crypto_skcipher_alg_chunksize(chacha);
-	inst->alg.maxauthsize = POLY1305_DIGEST_SIZE;
+	inst->alg.maxauthsize = POLY1305_MAC_SIZE;
 	inst->alg.init = chachapoly_init;
 	inst->alg.exit = chachapoly_exit;
 	inst->alg.encrypt = chachapoly_encrypt;
crypto/poly1305_generic.c deleted +0 -304

304 lines not on this page · raw message

crypto/poly1305_zinc.c new file +98 -0

98 lines not on this page · raw message

include/crypto/poly1305.h deleted +0 -40
deleted file mode 100644
index f718a19da82f..000000000000
@@ -1,40 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Common values for the Poly1305 algorithm
- */
-
-#ifndef _CRYPTO_POLY1305_H
-#define _CRYPTO_POLY1305_H
-
-#include <linux/types.h>
-#include <linux/crypto.h>
-
-#define POLY1305_BLOCK_SIZE	16
-#define POLY1305_KEY_SIZE	32
-#define POLY1305_DIGEST_SIZE	16
-
-struct poly1305_desc_ctx {
-	/* key */
-	u32 r[5];
-	/* finalize key */
-	u32 s[4];
-	/* accumulator */
-	u32 h[5];
-	/* partial buffer */
-	u8 buf[POLY1305_BLOCK_SIZE];
-	/* bytes used in partial buffer */
-	unsigned int buflen;
-	/* r key has been set */
-	bool rset;
-	/* s key has been set */
-	bool sset;
-};
-
-int crypto_poly1305_init(struct shash_desc *desc);
-unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx,
-					const u8 *src, unsigned int srclen);
-int crypto_poly1305_update(struct shash_desc *desc,
-			   const u8 *src, unsigned int srclen);
-int crypto_poly1305_final(struct shash_desc *desc, u8 *dst);
-
-#endif
Signature · 1 line
2.19.0
author maintainer
#54 lore raw

PATCH v5 18/20 crypto: port ChaCha20 to Zinc

In reply to Jason A. Donenfeld

Now that ChaCha20 is in Zinc, we can have the crypto API code simply call into it. The crypto API expects to have a stored key per instance and independent nonces, so we follow suite and store the key and initialize the nonce independently.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
---
 arch/arm/configs/exynos_defconfig       |   1 -
 arch/arm/configs/multi_v7_defconfig     |   1 -
 arch/arm/configs/omap2plus_defconfig    |   1 -
 arch/arm/crypto/Kconfig                 |   6 -
 arch/arm/crypto/Makefile                |   2 -
 arch/arm/crypto/chacha20-neon-core.S    | 521 --------------------
 arch/arm/crypto/chacha20-neon-glue.c    | 127 -----
 arch/arm64/configs/defconfig            |   1 -
 arch/arm64/crypto/Kconfig               |   6 -
 arch/arm64/crypto/Makefile              |   3 -
 arch/arm64/crypto/chacha20-neon-core.S  | 450 -----------------
 arch/arm64/crypto/chacha20-neon-glue.c  | 133 -----
 arch/x86/crypto/Makefile                |   3 -
 arch/x86/crypto/chacha20-avx2-x86_64.S  | 448 -----------------
 arch/x86/crypto/chacha20-ssse3-x86_64.S | 630 ------------------------
 arch/x86/crypto/chacha20_glue.c         | 146 ------
 crypto/Kconfig                          |  17 +-
 crypto/Makefile                         |   2 +-
 crypto/chacha20_generic.c               | 136 -----
 crypto/chacha20_zinc.c                  | 100 ++++
 crypto/chacha20poly1305.c               |   2 +-
 include/crypto/chacha20.h               |  12 -
 22 files changed, 103 insertions(+), 2645 deletions(-)
 delete mode 100644 arch/arm/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/arm64/crypto/chacha20-neon-core.S
 delete mode 100644 arch/arm64/crypto/chacha20-neon-glue.c
 delete mode 100644 arch/x86/crypto/chacha20-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20-ssse3-x86_64.S
 delete mode 100644 arch/x86/crypto/chacha20_glue.c
 delete mode 100644 crypto/chacha20_generic.c
 create mode 100644 crypto/chacha20_zinc.c
arch/arm/configs/exynos_defconfig +0 -1
index 27ea6dfcf2f2..95929b5e7b10 100644
@@ -350,7 +350,6 @@ CONFIG_CRYPTO_SHA1_ARM_NEON=m
 CONFIG_CRYPTO_SHA256_ARM=m
 CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_FONTS=y
 CONFIG_FONT_7x14=y
arch/arm/configs/multi_v7_defconfig +0 -1
index fc33444e94f0..63be07724db3 100644
@@ -1000,4 +1000,3 @@ CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_AES_ARM_CE=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
 CONFIG_CRYPTO_CRC32_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
arch/arm/configs/omap2plus_defconfig +0 -1
index 6491419b1dad..f585a8ecc336 100644
@@ -547,7 +547,6 @@ CONFIG_CRYPTO_SHA512_ARM=m
 CONFIG_CRYPTO_AES_ARM=m
 CONFIG_CRYPTO_AES_ARM_BS=m
 CONFIG_CRYPTO_GHASH_ARM_CE=m
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRC_CCITT=y
 CONFIG_CRC_T10DIF=y
 CONFIG_CRC_ITU_T=y
arch/arm/crypto/Kconfig +0 -6
index 925d1364727a..fb80fd89f0e7 100644
@@ -115,12 +115,6 @@ config CRYPTO_CRC32_ARM_CE
 	depends on KERNEL_MODE_NEON && CRC32
 	select CRYPTO_HASH
 
-config CRYPTO_CHACHA20_NEON
-	tristate "NEON accelerated ChaCha20 symmetric cipher"
-	depends on KERNEL_MODE_NEON
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-
 config CRYPTO_SPECK_NEON
 	tristate "NEON accelerated Speck cipher algorithms"
 	depends on KERNEL_MODE_NEON
arch/arm/crypto/Makefile +0 -2
index 8de542c48ade..bbfa98447063 100644
@@ -9,7 +9,6 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o
 obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o
 obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 
 ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
@@ -53,7 +52,6 @@ aes-arm-ce-y	:= aes-ce-core.o aes-ce-glue.o
 ghash-arm-ce-y	:= ghash-ce-core.o ghash-ce-glue.o
 crct10dif-arm-ce-y	:= crct10dif-ce-core.o crct10dif-ce-glue.o
 crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
 ifdef REGENERATE_ARM_CRYPTO
arch/arm/crypto/chacha20-neon-core.S deleted +0 -521

521 lines not on this page · raw message

arch/arm/crypto/chacha20-neon-glue.c deleted +0 -127

127 lines not on this page · raw message

arch/arm64/configs/defconfig +0 -1
index db8d364f8476..6cc3c8a0ad88 100644
@@ -709,5 +709,4 @@ CONFIG_CRYPTO_CRCT10DIF_ARM64_CE=m
 CONFIG_CRYPTO_CRC32_ARM64_CE=m
 CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
 CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
-CONFIG_CRYPTO_CHACHA20_NEON=m
 CONFIG_CRYPTO_AES_ARM64_BS=m
arch/arm64/crypto/Kconfig +0 -6
index e3fdb0fd6f70..9db6d775a880 100644
@@ -105,12 +105,6 @@ config CRYPTO_AES_ARM64_NEON_BLK
 	select CRYPTO_AES
 	select CRYPTO_SIMD
 
-config CRYPTO_CHACHA20_NEON
-	tristate "NEON accelerated ChaCha20 symmetric cipher"
-	depends on KERNEL_MODE_NEON
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-
 config CRYPTO_AES_ARM64_BS
 	tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm"
 	depends on KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +0 -3
index bcafd016618e..507c4bfb86e3 100644
@@ -53,9 +53,6 @@ sha256-arm64-y := sha256-glue.o sha256-core.o
 obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o
 sha512-arm64-y := sha512-glue.o sha512-core.o
 
-obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
-chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
-
 obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
 speck-neon-y := speck-neon-core.o speck-neon-glue.o
 
arch/arm64/crypto/chacha20-neon-core.S deleted +0 -450

450 lines not on this page · raw message

arch/arm64/crypto/chacha20-neon-glue.c deleted +0 -133

133 lines not on this page · raw message

arch/x86/crypto/Makefile +0 -3
index cf830219846b..419212c31246 100644
@@ -23,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o
 obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o
 obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o
-obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o
 obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
@@ -76,7 +75,6 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o
 blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o
 twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o
 twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o
-chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o
 serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o
 
 aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -99,7 +97,6 @@ endif
 
 ifeq ($(avx2_supported),yes)
 	camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o
-	chacha20-x86_64-y += chacha20-avx2-x86_64.o
 	serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o
 
 	morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
arch/x86/crypto/chacha20-avx2-x86_64.S deleted +0 -448

448 lines not on this page · raw message

arch/x86/crypto/chacha20-ssse3-x86_64.S deleted +0 -630

630 lines not on this page · raw message

arch/x86/crypto/chacha20_glue.c deleted +0 -146

146 lines not on this page · raw message

crypto/Kconfig +1 -16
index 47859a0f8052..42dc48aa9b81 100644
@@ -1428,27 +1428,12 @@ config CRYPTO_SALSA20
 config CRYPTO_CHACHA20
 	tristate "ChaCha20 cipher algorithm"
 	select CRYPTO_BLKCIPHER
+	select ZINC_CHACHA20
 	help
 	  ChaCha20 cipher algorithm, RFC7539.
 
 	  ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
 	  Bernstein and further specified in RFC7539 for use in IETF protocols.
-	  This is the portable C implementation of ChaCha20.
-
-	  See also:
-	  <http://cr.yp.to/chacha/chacha-20080128.pdf>
-
-config CRYPTO_CHACHA20_X86_64
-	tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)"
-	depends on X86 && 64BIT
-	select CRYPTO_BLKCIPHER
-	select CRYPTO_CHACHA20
-	help
-	  ChaCha20 cipher algorithm, RFC7539.
-
-	  ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J.
-	  Bernstein and further specified in RFC7539 for use in IETF protocols.
-	  This is the x86_64 assembler implementation using SIMD instructions.
 
 	  See also:
 	  <http://cr.yp.to/chacha/chacha-20080128.pdf>
crypto/Makefile +1 -1
index 5e60348d02e2..587103b87890 100644
@@ -117,7 +117,7 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o
 obj-$(CONFIG_CRYPTO_SEED) += seed.o
 obj-$(CONFIG_CRYPTO_SPECK) += speck.o
 obj-$(CONFIG_CRYPTO_SALSA20) += salsa20_generic.o
-obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_generic.o
+obj-$(CONFIG_CRYPTO_CHACHA20) += chacha20_zinc.o
 obj-$(CONFIG_CRYPTO_POLY1305) += poly1305_zinc.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
crypto/chacha20_generic.c deleted +0 -136

136 lines not on this page · raw message

crypto/chacha20_zinc.c new file +100 -0

100 lines not on this page · raw message

crypto/chacha20poly1305.c +1 -1
index bf523797bef3..b26adb9ed898 100644
@@ -13,7 +13,7 @@
 #include <crypto/internal/hash.h>
 #include <crypto/internal/skcipher.h>
 #include <crypto/scatterwalk.h>
-#include <crypto/chacha20.h>
+#include <zinc/chacha20.h>
 #include <zinc/poly1305.h>
 #include <linux/err.h>
 #include <linux/init.h>
include/crypto/chacha20.h +0 -12
index b83d66073db0..3b92f58f3891 100644
@@ -6,23 +6,11 @@
 #ifndef _CRYPTO_CHACHA20_H
 #define _CRYPTO_CHACHA20_H
 
-#include <crypto/skcipher.h>
-#include <linux/types.h>
-#include <linux/crypto.h>
-
 #define CHACHA20_IV_SIZE	16
 #define CHACHA20_KEY_SIZE	32
 #define CHACHA20_BLOCK_SIZE	64
 #define CHACHA20_BLOCK_WORDS	(CHACHA20_BLOCK_SIZE / sizeof(u32))
 
-struct chacha20_ctx {
-	u32 key[8];
-};
-
 void chacha20_block(u32 *state, u32 *stream);
-void crypto_chacha20_init(u32 *state, struct chacha20_ctx *ctx, u8 *iv);
-int crypto_chacha20_setkey(struct crypto_skcipher *tfm, const u8 *key,
-			   unsigned int keysize);
-int crypto_chacha20_crypt(struct skcipher_request *req);
 
 #endif
Signature · 1 line
2.19.0
author maintainer
#55 lore raw

PATCH v5 19/20 security/keys: rewrite big_key crypto to use Zinc

In reply to Jason A. Donenfeld

A while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function. This makes the file considerably more simple; the diffstat alone should justify this commit. It also should be faster, since it no longer requires a mutex around the "aead api object" (nor allocations), allowing us to encrypt multiple items in parallel. We also benefit from being able to pass any type of pointer to Zinc, so we can get rid of the ridiculously complex custom page allocator that big_key really doesn't need.

Cc Samuel Neves <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Greg KH <[email protected]>
Cc Jean-Philippe Aumasson <[email protected]>
Cc Eric Biggers <[email protected]>
Cc David Howells <[email protected]>
---
 security/keys/Kconfig   |   4 +-
 security/keys/big_key.c | 230 +++++-----------------------------------
 2 files changed, 28 insertions(+), 206 deletions(-)
security/keys/Kconfig +1 -3
index 6462e6654ccf..66ff26298fb3 100644
@@ -45,9 +45,7 @@ config BIG_KEYS
 	bool "Large payload keys"
 	depends on KEYS
 	depends on TMPFS
-	select CRYPTO
-	select CRYPTO_AES
-	select CRYPTO_GCM
+	select ZINC_CHACHA20POLY1305
 	help
 	  This option provides support for holding large keys within the kernel
 	  (for example Kerberos ticket caches).  The data may be stored out to
security/keys/big_key.c +27 -203

347 lines not on this page · raw message

Signature · 1 line
2.19.0
maintainer
#56 lore raw

Re: [PATCH net-next v5 19/20] security/keys: rewrite big_key crypto to use Zinc

Quoted 4 lines · Jason A. Donenfeld

A while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function.

warthog>git grep chacha20poly1305_decrypt net-next/master warthog1>

Where do I find this?

David

author maintainer
#57 lore raw

Re: [PATCH net-next v5 19/20] security/keys: rewrite big_key crypto to use Zinc

Hi David,

Quoted 7 lines · David Howells
Quoted 4 lines · Jason A. Donenfeld

A while back, I noticed that the crypto and crypto API usage in big_keys were entirely broken in multiple ways, so I rewrote it. Now, I'm rewriting it again, but this time using Zinc's ChaCha20Poly1305 function.

Quoted 6 lines

warthog>git grep chacha20poly1305_decrypt net-next/master warthog1>

Where do I find this?

David

Previously in the patchset:
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard
https://lore.kernel.org/lkml/[email protected]/

Regards,
Jason

author maintainer
#58 lore raw

PATCH v5 20/20 net: WireGuard secure network tunnel

In reply to Jason A. Donenfeld

WireGuard is a layer 3 secure networking tunnel made specifically for the kernel, that aims to be much simpler and easier to audit than IPsec. Extensive documentation and description of the protocol and considerations, along with formal proofs of the cryptography, are available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

This commit implements WireGuard as a simple network device driver, accessible in the usual RTNL way used by virtual network drivers. It makes use of the udp_tunnel APIs, GRO, GSO, NAPI, and the usual set of networking subsystem APIs. It has a somewhat novel multicore queueing system designed for maximum throughput and minimal latency of encryption operations, but it is implemented modestly using workqueues and NAPI. Configuration is done via generic Netlink, and following a review from the Netlink maintainer a year ago, several high profile userspace have already implemented the API.

This commit also comes with several different tests, both in-kernel tests and out-of-kernel tests based on network namespaces, taking profit of the fact that sockets used by WireGuard intentionally stay in the namespace the WireGuard interface was originally created, exactly like the semantics of userspace tun devices. See wireguard.com/netns/ for pictures and examples.

The source code is fairly short, but rather than combining everything into a single file, WireGuard is developed as cleanly separable files, making auditing and comprehension easier. Things are laid out as follows:

* noise.[ch], cookie.[ch], messages.h: These implement the bulk of the cryptographic aspects of the protocol, and are mostly data-only in nature, taking in buffers of bytes and spitting out buffers of bytes. They also handle reference counting for their various shared pieces of data, like keys and key lists.

* ratelimiter.[ch]: Used as an integral part of cookie.[ch] for ratelimiting certain types of cryptographic operations in accordance with particular WireGuard semantics.

* allowedips.[ch], hashtables.[ch]: The main lookup structures of WireGuard, the former being trie-like with particular semantics, an integral part of the design of the protocol, and the latter just being nice helper functions around the specific hashtables we use.

* device.[ch]: Implementation of functions for the netdevice and for rtnl, responsible for maintaining the life of a given interface and wiring it up to the rest of WireGuard.

* peer.[ch]: Each interface has a list of peers, with helper functions available here for creation, destruction, and reference counting.

* socket.[ch]: Implementation of functions related to udp_socket and the general set of kernel socket APIs, for sending and receiving ciphertext UDP packets, and taking care of WireGuard-specific sticky socket routing semantics for the automatic roaming.

* netlink.[ch]: Userspace API entry point for configuring WireGuard peers and devices. The API has been implemented by several userspace tools and network management utility, and the WireGuard project distributes the basic wg(8) tool.

* queueing.[ch]: Shared function on the rx and tx path for handling the various queues used in the multicore algorithms.

* send.c: Handles encrypting outgoing packets in parallel on multiple cores, before sending them in order on a single core, via workqueues and ring buffers. Also handles sending handshake and cookie messages as part of the protocol, in parallel.

* receive.c: Handles decrypting incoming packets in parallel on multiple cores, before passing them off in order to be ingested via the rest of the networking subsystem with GRO via the typical NAPI poll function. Also handles receiving handshake and cookie messages as part of the protocol, in parallel.

* timers.[ch]: Uses the timer wheel to implement protocol particular event timeouts, and gives a set of very simple event-driven entry point functions for callers.

* main.c, version.h: Initialization and deinitialization of the module.

* selftest/*.h: Runtime unit tests for some of the most security sensitive functions.

* tools/testing/selftests/wireguard/netns.sh: Aforementioned testing script using network namespaces.

This commit aims to be as self-contained as possible, implementing WireGuard as a standalone module not needing much special handling or coordination from the network subsystem. I expect for future optimizations to the network stack to positively improve WireGuard, and vice-versa, but for the time being, this exists as intentionally standalone.

We introduce a menu option for CONFIG_WIREGUARD, as well as providing a verbose debug log and self-tests via CONFIG_WIREGUARD_DEBUG.

---
 MAINTAINERS                                  |   8 +
 drivers/net/Kconfig                          |  30 +
 drivers/net/Makefile                         |   1 +
 drivers/net/wireguard/Makefile               |  18 +
 drivers/net/wireguard/allowedips.c           | 404 ++++++++++
 drivers/net/wireguard/allowedips.h           |  55 ++
 drivers/net/wireguard/cookie.c               | 234 ++++++
 drivers/net/wireguard/cookie.h               |  59 ++
 drivers/net/wireguard/device.c               | 438 +++++++++++
 drivers/net/wireguard/device.h               |  65 ++
 drivers/net/wireguard/hashtables.c           | 209 +++++
 drivers/net/wireguard/hashtables.h           |  63 ++
 drivers/net/wireguard/main.c                 |  65 ++
 drivers/net/wireguard/messages.h             | 128 +++
 drivers/net/wireguard/netlink.c              | 605 ++++++++++++++
 drivers/net/wireguard/netlink.h              |  12 +
 drivers/net/wireguard/noise.c                | 784 +++++++++++++++++++
 drivers/net/wireguard/noise.h                | 129 +++
 drivers/net/wireguard/peer.c                 | 191 +++++
 drivers/net/wireguard/peer.h                 |  87 ++
 drivers/net/wireguard/queueing.c             |  52 ++
 drivers/net/wireguard/queueing.h             | 193 +++++
 drivers/net/wireguard/ratelimiter.c          | 220 ++++++
 drivers/net/wireguard/ratelimiter.h          |  19 +
 drivers/net/wireguard/receive.c              | 595 ++++++++++++++
 drivers/net/wireguard/selftest/allowedips.h  | 656 ++++++++++++++++
 drivers/net/wireguard/selftest/counter.h     | 103 +++
 drivers/net/wireguard/selftest/ratelimiter.h | 174 ++++
 drivers/net/wireguard/send.c                 | 420 ++++++++++
 drivers/net/wireguard/socket.c               | 432 ++++++++++
 drivers/net/wireguard/socket.h               |  44 ++
 drivers/net/wireguard/timers.c               | 256 ++++++
 drivers/net/wireguard/timers.h               |  30 +
 drivers/net/wireguard/version.h              |   1 +
 include/uapi/linux/wireguard.h               | 190 +++++
 tools/testing/selftests/wireguard/netns.sh   | 499 ++++++++++++
 36 files changed, 7469 insertions(+)
 create mode 100644 drivers/net/wireguard/Makefile
 create mode 100644 drivers/net/wireguard/allowedips.c
 create mode 100644 drivers/net/wireguard/allowedips.h
 create mode 100644 drivers/net/wireguard/cookie.c
 create mode 100644 drivers/net/wireguard/cookie.h
 create mode 100644 drivers/net/wireguard/device.c
 create mode 100644 drivers/net/wireguard/device.h
 create mode 100644 drivers/net/wireguard/hashtables.c
 create mode 100644 drivers/net/wireguard/hashtables.h
 create mode 100644 drivers/net/wireguard/main.c
 create mode 100644 drivers/net/wireguard/messages.h
 create mode 100644 drivers/net/wireguard/netlink.c
 create mode 100644 drivers/net/wireguard/netlink.h
 create mode 100644 drivers/net/wireguard/noise.c
 create mode 100644 drivers/net/wireguard/noise.h
 create mode 100644 drivers/net/wireguard/peer.c
 create mode 100644 drivers/net/wireguard/peer.h
 create mode 100644 drivers/net/wireguard/queueing.c
 create mode 100644 drivers/net/wireguard/queueing.h
 create mode 100644 drivers/net/wireguard/ratelimiter.c
 create mode 100644 drivers/net/wireguard/ratelimiter.h
 create mode 100644 drivers/net/wireguard/receive.c
 create mode 100644 drivers/net/wireguard/selftest/allowedips.h
 create mode 100644 drivers/net/wireguard/selftest/counter.h
 create mode 100644 drivers/net/wireguard/selftest/ratelimiter.h
 create mode 100644 drivers/net/wireguard/send.c
 create mode 100644 drivers/net/wireguard/socket.c
 create mode 100644 drivers/net/wireguard/socket.h
 create mode 100644 drivers/net/wireguard/timers.c
 create mode 100644 drivers/net/wireguard/timers.h
 create mode 100644 drivers/net/wireguard/version.h
 create mode 100644 include/uapi/linux/wireguard.h
 create mode 100755 tools/testing/selftests/wireguard/netns.sh
MAINTAINERS +8 -0
index 5967c737f3ce..32db7ebad86e 100644
@@ -15823,6 +15823,14 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/gpio/gpio-ws16c48.c
 
+WIREGUARD SECURE NETWORK TUNNEL
+M:	Jason A. Donenfeld <[email protected]>
+S:	Maintained
+F:	drivers/net/wireguard/
+F:	tools/testing/selftests/wireguard/
+L:	[email protected]
+L:	[email protected]
+
 WISTRON LAPTOP BUTTON DRIVER
 M:	Miloslav Trmac <[email protected]>
 S:	Maintained
drivers/net/Kconfig +30 -0
index d03775100f7d..aa631fe3b395 100644
@@ -70,6 +70,36 @@ config DUMMY
 	  To compile this driver as a module, choose M here: the module
 	  will be called dummy.
 
+config WIREGUARD
+	tristate "WireGuard secure network tunnel"
+	depends on NET && INET
+	select NET_UDP_TUNNEL
+	select DST_CACHE
+	select ZINC_CHACHA20POLY1305
+	select ZINC_BLAKE2S
+	select ZINC_CURVE25519
+	default m
+	help
+	  WireGuard is a secure, fast, and easy to use replacement for IPSec
+	  that uses modern cryptography and clever networking tricks. It's
+	  designed to be fairly general purpose and abstract enough to fit most
+	  use cases, while at the same time remaining extremely simple to
+	  configure. See www.wireguard.com for more info.
+
+	  It's safe to say Y or M here, as the driver is very lightweight and
+	  is only in use when an administrator chooses to add an interface.
+
+config WIREGUARD_DEBUG
+	bool "Debugging checks and verbose messages"
+	depends on WIREGUARD
+	help
+	  This will write log messages for handshake and other events
+	  that occur for a WireGuard interface. It will also perform some
+	  extra validation checks and unit tests at various points. This is
+	  only useful for debugging.
+
+	  Say N here unless you know what you're doing.
+
 config EQUALIZER
 	tristate "EQL (serial line load balancing) support"
 	---help---
drivers/net/Makefile +1 -0
index 21cde7e78621..f0acd11a143d 100644
@@ -24,6 +24,7 @@ obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/
 obj-$(CONFIG_TUN) += tun.o
 obj-$(CONFIG_TAP) += tap.o
+obj-$(CONFIG_WIREGUARD) += wireguard/
 obj-$(CONFIG_VETH) += veth.o
 obj-$(CONFIG_VIRTIO_NET) += virtio_net.o
 obj-$(CONFIG_VXLAN) += vxlan.o
drivers/net/wireguard/Makefile new file +18 -0
new file mode 100644
index 000000000000..d8856255bc9d
@@ -0,0 +1,18 @@
+ccflags-y := -O3
+ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
+ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG
+wireguard-y := main.o
+wireguard-y += noise.o
+wireguard-y += device.o
+wireguard-y += peer.o
+wireguard-y += timers.o
+wireguard-y += queueing.o
+wireguard-y += send.o
+wireguard-y += receive.o
+wireguard-y += socket.o
+wireguard-y += hashtables.o
+wireguard-y += allowedips.o
+wireguard-y += ratelimiter.o
+wireguard-y += cookie.o
+wireguard-y += netlink.o
+obj-$(CONFIG_WIREGUARD) := wireguard.o
drivers/net/wireguard/allowedips.c new file +404 -0

404 lines not on this page · raw message

drivers/net/wireguard/allowedips.h new file +55 -0

55 lines not on this page · raw message

drivers/net/wireguard/cookie.c new file +234 -0

234 lines not on this page · raw message

drivers/net/wireguard/cookie.h new file +59 -0

59 lines not on this page · raw message

drivers/net/wireguard/device.c new file +438 -0

438 lines not on this page · raw message

drivers/net/wireguard/device.h new file +65 -0

65 lines not on this page · raw message

drivers/net/wireguard/hashtables.c new file +209 -0

209 lines not on this page · raw message

drivers/net/wireguard/hashtables.h new file +63 -0

63 lines not on this page · raw message

drivers/net/wireguard/main.c new file +65 -0

65 lines not on this page · raw message

drivers/net/wireguard/messages.h new file +128 -0

128 lines not on this page · raw message

drivers/net/wireguard/netlink.c new file +605 -0

605 lines not on this page · raw message

drivers/net/wireguard/netlink.h new file +12 -0
new file mode 100644
index 000000000000..c1cd9b019bd1
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Copyright (C) 2015-2018 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+ */
+
+#ifndef _WG_NETLINK_H
+#define _WG_NETLINK_H
+
+int genetlink_init(void);
+void genetlink_uninit(void);
+
+#endif /* _WG_NETLINK_H */
drivers/net/wireguard/noise.c new file +784 -0

588 lines not on this page · raw message

drivers/net/wireguard/noise.h new file +129 -0

129 lines not on this page · raw message

drivers/net/wireguard/peer.c new file +191 -0

191 lines not on this page · raw message

drivers/net/wireguard/peer.h new file +87 -0

87 lines not on this page · raw message

drivers/net/wireguard/queueing.c new file +52 -0

52 lines not on this page · raw message

drivers/net/wireguard/queueing.h new file +193 -0

193 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.c new file +220 -0

220 lines not on this page · raw message

drivers/net/wireguard/ratelimiter.h new file +19 -0
new file mode 100644
index 000000000000..8931c0615374

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/receive.c new file +595 -0

595 lines not on this page · raw message

drivers/net/wireguard/selftest/allowedips.h new file +656 -0

656 lines not on this page · raw message

drivers/net/wireguard/selftest/counter.h new file +103 -0

103 lines not on this page · raw message

drivers/net/wireguard/selftest/ratelimiter.h new file +174 -0

174 lines not on this page · raw message

drivers/net/wireguard/send.c new file +420 -0

420 lines not on this page · raw message

drivers/net/wireguard/socket.c new file +432 -0

432 lines not on this page · raw message

drivers/net/wireguard/socket.h new file +44 -0

44 lines not on this page · raw message

drivers/net/wireguard/timers.c new file +256 -0

256 lines not on this page · raw message

drivers/net/wireguard/timers.h new file +30 -0
new file mode 100644
index 000000000000..483529c9d873

Lines of this file are not shown. View the raw message.

drivers/net/wireguard/version.h new file +1 -0
new file mode 100644
index 000000000000..42d40e0939c4

Lines of this file are not shown. View the raw message.

include/uapi/linux/wireguard.h new file +190 -0

190 lines not on this page · raw message

tools/testing/selftests/wireguard/netns.sh new file +499 -0

499 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.19.0

Part of the diff is not shown here. View the raw message.

#59 lore raw

Re: [PATCH net-next v5 20/20] net: WireGuard secure network tunnel

Quoted 8 lines
drivers/net/wireguard/allowedips.c new file +8 -0
+#define push_rcu(stack, p, len) ({                                             \
+		if (rcu_access_pointer(p)) {                                   \
+			BUG_ON(len >= 128);                                    \
+			stack[len++] = rcu_dereference_raw(p);                 \
+		}                                                              \
+		true;                                                          \
+	})
+static void root_free_rcu(struct rcu_head *rcu)

drivers/net/wireguard/allowedips.c line 68

Quoted 10 lines
drivers/net/wireguard/allowedips.c new file +10 -0
+{
+	struct allowedips_node *node, *stack[128] = {
+		container_of(rcu, struct allowedips_node, rcu) };
+	unsigned int len = 1;
+
+	while (len > 0 && (node = stack[--len]) &&
+	       push_rcu(stack, node->bit[0], len) &&
+	       push_rcu(stack, node->bit[1], len))
+		kfree(node);
+}

Hi Jason

I see this BUG_ON() is still here. It really needs to be removed. It does not look like you need to crash the kernel here. Can you add in a test of len >= 128, do a WARN and then return. I think you then leak some memory, but i would much prefer that to a crashed machine.

Andrew

author maintainer
#60 lore raw

Re: [PATCH net-next v5 20/20] net: WireGuard secure network tunnel

Hi Andrew,

Quoted 4 lines · Andrew Lunn

I see this BUG_ON() is still here. It really needs to be removed. It does not look like you need to crash the kernel here. Can you add in a test of len >= 128, do a WARN and then return. I think you then leak some memory, but i would much prefer that to a crashed machine.

Sure, I'll change it to that.

Regards,
Jason

#61 lore raw

Re: [PATCH net-next v5 20/20] net: WireGuard secure network tunnel

Quoted 9 lines · Jason A. Donenfeld

Hi Andrew,

Quoted 4 lines · Andrew Lunn

I see this BUG_ON() is still here. It really needs to be removed. It does not look like you need to crash the kernel here. Can you add in a test of len >= 128, do a WARN and then return. I think you then leak some memory, but i would much prefer that to a crashed machine.

Sure, I'll change it to that.

Great, thanks. I noticed there is at least one more BUG() statements. It would be good to remove them all. BUG() should only be used when something bad has already happened and we want to minimise the damage by killing the machine immediately.

Andrew

#62 lore raw
In reply to Jason A. Donenfeld
Quoted 28 lines · Jason A. Donenfeld

v5 has the most comprehensive set of changes yet, and I think should finally address all of the relevant issues brought up on the mailing list. In particular, this feedback has come from:

- Andy Lutomirski
- Eric Biggers
- Ard Biesheuvel
- Kevin Easton
- Andrew Lunn
- Martin Willi

Changes v4->v5:
---------------
- Use fewer inlines, except when measured as necessary.
- Reduce size of scattergather array to fit within stack on small systems.
- Account for larger stack frames with KASAN.
- The x86_64 implementations are selected according to input length.
- Avoid using simd for small blocks on x86_64.
- The simd_get/put API is now pass by reference, so that the user can lazily use the context based on whether or not it's needed. See the description again in the first commit for this.
- Add cycle counts for different sizes for x86_64 commit messages.
- Relax simd during chapoly sg loop.
- Replace -include with #if defined(...)
- Saner and simpler Kconfig.
- Split into separate modules instead of one monolithic zinc.
- The combination of these three last items means that there no

Quoted 9 lines

longer are any conditionals in our Makefile.
- Martin showed a performance regression using tcrypt in v4. This has been triaged and fixed, and now the Zinc code runs faster than the previous code.
- While I initially wasn't going to do this for the initial patchset, it was just so simple to do: now there's a nosimd module parameter that can be used to disable simd instructions for debugging and testing, or on weird systems.

I was going to respond in the other thread but it is probably better to move the discussion here.

My concern about the monolithic nature of each algo module is not only about SIMD, and it has nothing to do with weird systems. It has to do with micro-architectural differences which are more common on ARM than on other architectures *, I suppose. But generalizing from that, it has to do with policy which is currently owned by userland and not by the kernel. This will also be important for choosing between the time variant but less safe table based scalar AES and the much slower time invariant version (which is substantially slower, especially on decryption) once we move AES into this library.

So a command line option for the kernel is not the solution here. If we can't have separate modules, could we at least have per-module options that put the policy decisions back into userland?

* as an example, the SHA256 NEON code I collaborated on with Andy Polyakov 2 years ago is significantly faster on some cores and not on others

Quoted 75 lines

-----------------------------------------------------------

This patchset is available on git.kernel.org in this branch, where it may be pulled directly for inclusion into net-next:

* https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/linux.git/log/?h=jd/wireguard

-----------------------------------------------------------

WireGuard is a secure network tunnel written especially for Linux, which has faced around three years of serious development, deployment, and scrutiny. It delivers excellent performance and is extremely easy to use and configure. It has been designed with the primary goal of being both easy to audit by virtue of being small and highly secure from a cryptography and systems security perspective. WireGuard is used by some massive companies pushing enormous amounts of traffic, and likely already today you've consumed bytes that at some point transited through a WireGuard tunnel. Even as an out-of-tree module, WireGuard has been integrated into various userspace tools, Linux distributions, mobile phones, and data centers. There are ports in several languages to several operating systems, and even commercial hardware and services sold integrating WireGuard. It is time, therefore, for WireGuard to be properly integrated into Linux.

Ample information, including documentation, installation instructions, and project details, is available at:

* https://www.wireguard.com/
* https://www.wireguard.com/papers/wireguard.pdf

As it is currently an out-of-tree module, it lives in its own git repo and has its own mailing list, and every commit for the module is tested against every stable kernel since 3.10 on a variety of architectures using an extensive test suite:

* https://git.zx2c4.com/WireGuard
https://git.kernel.org/pub/scm/linux/kernel/git/zx2c4/WireGuard.git/
* https://lists.zx2c4.com/mailman/listinfo/wireguard
* https://www.wireguard.com/build-status/

The project has been broadly discussed at conferences, and was presented to the Netdev developers in Seoul last November, where a paper was released detailing some interesting aspects of the project. Dave asked me after the talk if I would consider sending in a v1 "sooner rather than later", hence this patchset. A decision is still waiting from the Linux Plumbers Conference, but an update on these topics may be presented in Vancouver in a few months. Prior presentations:

* https://www.wireguard.com/presentations/
* https://www.wireguard.com/papers/wireguard-netdev22.pdf

The cryptography in the protocol itself has been formally verified by several independent academic teams with positive results, and I know of two additional efforts on their way to further corroborate those findings. The version 1 protocol is "complete", and so the purpose of this review is to assess the implementation of the protocol. However, it still may be of interest to know that the thing you're reviewing uses a protocol with various nice security properties:

* https://www.wireguard.com/formal-verification/

This patchset is divided into four segments. The first introduces a very simple helper for working with the FPU state for the purposes of amortizing SIMD operations. The second segment is a small collection of cryptographic primitives, split up into several commits by primitive and by hardware. The third shows usage of Zinc within the existing crypto API and as a replacement to the existing crypto API. The last is WireGuard itself, presented as an unintrusive and self-contained virtual network driver.

It is intended that this entire patch series enter the kernel through DaveM's net-next tree. Subsequently, WireGuard patches will go through DaveM's net-next tree, while Zinc patches will go through Greg KH's tree.

Enjoy,
Jason

author maintainer
#63 lore raw

Hi Ard,

Quoted 20 lines · Ard Biesheuvel
Quoted 5 lines · Jason A. Donenfeld

- While I initially wasn't going to do this for the initial patchset, it was just so simple to do: now there's a nosimd module parameter that can be used to disable simd instructions for debugging and testing, or on weird systems.

I was going to respond in the other thread but it is probably better to move the discussion here.

My concern about the monolithic nature of each algo module is not only about SIMD, and it has nothing to do with weird systems. It has to do with micro-architectural differences which are more common on ARM than on other architectures *, I suppose. But generalizing from that, it has to do with policy which is currently owned by userland and not by the kernel. This will also be important for choosing between the time variant but less safe table based scalar AES and the much slower time invariant version (which is substantially slower, especially on decryption) once we move AES into this library.

Quoted 7 lines

So a command line option for the kernel is not the solution here. If we can't have separate modules, could we at least have per-module options that put the policy decisions back into userland?

* as an example, the SHA256 NEON code I collaborated on with Andy Polyakov 2 years ago is significantly faster on some cores and not on others

Interesting concern. There are micro-architectural quirks on x86 too that the current code actually already considers. Notably, we use an AVX-512VL path for Skylake-X but an AVX-512F path for Knights Landing and Coffee Lake and others, due to thermal throttling when touching the zmm registers on Skylake-X. So, in the code, we have it automatically select the right thing based on the micro-architecture.

Is the same thing not possible with ARM? Do you not have access to this information already, such that the module can just always do the right thing and not require any user intervention?

If so, that would be ideal. If not (and I'm curious to learn why not exactly), then indeed we could add some runtime nobs in /sys/module/ {algo}/parameters/{nob}, or the like. This would be super easy to do, should we ever encounter a situation where we're unable to auto-detect the correct thing.

Regards,
Jason

#64 lore raw
Quoted 38 lines · Jason A. Donenfeld

Hi Ard,

Quoted 20 lines · Ard Biesheuvel
Quoted 5 lines · Jason A. Donenfeld

- While I initially wasn't going to do this for the initial patchset, it was just so simple to do: now there's a nosimd module parameter that can be used to disable simd instructions for debugging and testing, or on weird systems.

I was going to respond in the other thread but it is probably better to move the discussion here.

My concern about the monolithic nature of each algo module is not only about SIMD, and it has nothing to do with weird systems. It has to do with micro-architectural differences which are more common on ARM than on other architectures *, I suppose. But generalizing from that, it has to do with policy which is currently owned by userland and not by the kernel. This will also be important for choosing between the time variant but less safe table based scalar AES and the much slower time invariant version (which is substantially slower, especially on decryption) once we move AES into this library.

Quoted 7 lines

So a command line option for the kernel is not the solution here. If we can't have separate modules, could we at least have per-module options that put the policy decisions back into userland?

* as an example, the SHA256 NEON code I collaborated on with Andy Polyakov 2 years ago is significantly faster on some cores and not on others

Interesting concern. There are micro-architectural quirks on x86 too that the current code actually already considers. Notably, we use an AVX-512VL path for Skylake-X but an AVX-512F path for Knights Landing and Coffee Lake and others, due to thermal throttling when touching the zmm registers on Skylake-X. So, in the code, we have it automatically select the right thing based on the micro-architecture.

Quoted 4 lines

Is the same thing not possible with ARM? Do you not have access to this information already, such that the module can just always do the right thing and not require any user intervention?

That depends on what the right thing is. 'Fastest' does not necessarily mean 'optimal', and I guess the thermal throttling on Skylake-X may still result in the most power efficient implementation, which may be the preferred one in some contexts.

The point is that this is a policy decision, and those belong in userland not in the kernel.

Quoted 8 lines

If so, that would be ideal. If not (and I'm curious to learn why not exactly), then indeed we could add some runtime nobs in /sys/module/ {algo}/parameters/{nob}, or the like. This would be super easy to do, should we ever encounter a situation where we're unable to auto-detect the correct thing.

Regards,
Jason