lib/crypto: AES-GCM x86 arch hooks: any plan?

Hi Eric,

while working on optimising ovpn (OpenVPN data channel offload) we
prototyped what an x86 implementation of aes_gcm_{en,de}crypt_update_arch() would give us: ovpn calls the existing VAES-AVX512 GCM code directly on
linear records (AAD update, en/decrypt update and final in one FPU
section), instead of going through the crypto_aead API.

On a Ryzen 9 9950X (Zen 5), ovpn over 100G, MTU 1500, the receiver (the bottleneck) saves ~120-127 cycles per record at every record size, i.e. the request/scatterlist/skcipher_walk overhead:

Gbit/s, AES-GCM
crypto_aead direct

  1 stream, 1000 B      21.34      22.86   +7.1%
  1 stream, 1500 B      29.73      31.67   +6.5%
  1 stream, 2500 B      40.78      43.13   +5.8%
  4 streams, 1500 B     28.79      30.48   +5.9%

With RX prefetching in
the NIC driver the receiver gets cheaper and the gain grows to +10.8% (41.23 -> 45.70 Gbit/s, 1 stream). On the sending side (single packets and UDP GSO segments) the direct path saves ~40-100 cycles per packet; throughput there is set by the receiver.

Since the commit adding GCM to the library says the arch-optimized code will be migrated into it, do you already have work in progress for x86?

If not, we'd be happy to help there, or to test yours on Zen 5 with ovpn as a user.

One design question I have though: the VAES code needs the hash key powers H^16..H^1 (and H^8..H^1 for VAES-AVX2 / AES-NI), while the x86 ghash_key only holds the PCLMUL state.

Would you rather extend struct ghash_key on x86, or have an arch-specific part in struct aes_gcm_key that aes_gcm_preparekey() fills when the arch code is usable?

Thanks a lot!

Antonio Quartulli

CEO and Co-Founder
Mandelbit Srl
https://www.mandelbit.com

I'm planning it. I've just been trying to get other things out of the way first. If you're interested you can try to do it; just keep in mind it will be quite a bit of work to do properly.

The tentative plan is to implement the GHASH and AES-GCM arch hooks that lib/crypto/aes.c already has in place. And yes, that would mean that the x86 AES-GCM code would have to be reorganized to operate on struct ghash_key (expanded to hold up to 16 key powers) and struct aes_enckey rather than a combined struct.

That by itself may result in performance regressions on short messages, though, as each AES-GCM operation would be broken up into multiple calls to the arch code. There likely will also need to be new hooks for one-shot AES-GCM operations to handle them efficiently.

- Eric

Hi Eric,

Thanks! Then I'll leave it to you, and I'm happy to test the series on Zen 5 with ovpn when it's ready.

[...]

One data point I wanted to share on this part from the ovpn prototype.

I did split its single FPU section into three (AAD update, en/decrypt update and final, each in its own kernel_fpu_begin()/kernel_fpu_end(), same asm otherwise) and measured the receiver, which is the bottleneck (Ryzen 9 9950X, 100G, 1 stream):

                 one section   split      receiver cycles/record
  920 B records    34.22       33.86      1120 -> 1131 (+11)
  1420 B           45.59       44.95      1306 -> 1317 (+10)
  2420 B           55.43       54.79      1836 -> 1850 (+14)
                  (Gbit/s)

So the split costs at most ~10-14 cycles per record.
The sender showed no difference.
The receiver mostly runs in ksoftirqd, so there is no user FPU state to save in kernel_fpu_begin().

Therefore I think most of the ~120 cycles per record the direct path saves come from avoiding the crypto_aead request, scatterlist and walk, not from the single FPU section: for ovpn, update-level hooks would already get nearly all of it, as long as the C code between the calls stays light (that part I couldn't measure).

All in all, a one-shot hook would still be welcome (although it doesn't look super critical here).

Thanks a lot!

Antonio Quartulli

CEO and Co-Founder
Mandelbit Srl
https://www.mandelbit.com