crypto: arm/ghash - implement fused AES/GHASH implementation of GCM

On 32-bit ARM, AES in GCM mode takes full advantage of the ARMv8 Crypto Extensions when available, resulting in a performance of 6-7 cycles per byte for typical IPsec frames on cores such as Cortex-A53, using the generic GCM template encapsulating the accelerated AES-CTR and GHASH implementations.

At such high rates, any time spent copying data or doing other poorly optimized work in the generic layer hurts disproportionately, and we can get a significant performance improvement by combining the optimized AES-CTR and GHASH implementations into a single one.

On Cortex-A53, this results in a performance improvement of around 70%, or 4.2 cycles per byte for AES-256-GCM-128 with RFC4106 encapsulation. The fastest mode on this core is bare AES-128-GCM using 8k blocks, which manages 2.66 cycles per byte.

Signed-off-by Ard Biesheuvel <[email protected]>

---
Note: this patch depends on the softirq context patches for kernel mode NEON I sent last week. More specifically, this implements a sync AEAD that does not implement a !simd fallback, as AEADs are not callable in IRQ context anyway.

 arch/arm/crypto/Kconfig         |   2 +
 arch/arm/crypto/ghash-ce-core.S | 381 +++++++++++++++++++-
 arch/arm/crypto/ghash-ce-glue.c | 350 +++++++++++++++++-
 3 files changed, 718 insertions(+), 15 deletions(-)

Quoted 23 lines · Ard Biesheuvel

On 32-bit ARM, AES in GCM mode takes full advantage of the ARMv8 Crypto Extensions when available, resulting in a performance of 6-7 cycles per byte for typical IPsec frames on cores such as Cortex-A53, using the generic GCM template encapsulating the accelerated AES-CTR and GHASH implementations.

At such high rates, any time spent copying data or doing other poorly optimized work in the generic layer hurts disproportionately, and we can get a significant performance improvement by combining the optimized AES-CTR and GHASH implementations into a single one.

On Cortex-A53, this results in a performance improvement of around 70%, or 4.2 cycles per byte for AES-256-GCM-128 with RFC4106 encapsulation. The fastest mode on this core is bare AES-128-GCM using 8k blocks, which manages 2.66 cycles per byte.

Signed-off-by Ard Biesheuvel <[email protected]>

---
Note: this patch depends on the softirq context patches for kernel mode NEON I sent last week. More specifically, this implements a sync AEAD that does not implement a !simd fallback, as AEADs are not callable in IRQ context anyway.

Quoted 4 lines

 arch/arm/crypto/Kconfig         |   2 +
 arch/arm/crypto/ghash-ce-core.S | 381 +++++++++++++++++++-
 arch/arm/crypto/ghash-ce-glue.c | 350 +++++++++++++++++-
 3 files changed, 718 insertions(+), 15 deletions(-)

Patch applied. Thanks.