Skip to content

migration: workaround GNUTLS live migration crashes

Daniel P. Berrangé qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 12 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4crypto: implement workaround for GNUTLS thread safety problemscrypto/tlssession.c include/crypto/tlssession.h meson.build +2 more+116 -32 replies
2/4io: add support for activating TLS thread safety workaroundinclude/io/channel.h io/channel-tls.c+6 -01 reply
3/4migration: activate TLS thread safety workaroundmigration/tls.c+9 -01 reply
4/4crypto: add tracing & warning about GNUTLS countermeasurescrypto/tlssession.c crypto/trace-events+12 -02 replies

Reviews trailers per patch

All 4 patches have a Reviewed-by or Acked-by.

PatchDaniel P. BerrangéFabiano Rosas
1/4R-b
2/4R-b
3/4R-b
4/4R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/4] migration: workaround GNUTLS live migration crashes

TL:DR: GNUTLS is liable to crash QEMU when live migration is run with TLS enabled and a return path channel is present, if approx 64 GB of data is transferred. This is easily triggered in a 16 GB VM with 4 CPUs, by running 'stress-ng --vm 4 --vm-bytes 80%' to prevent convergance until 64 GB of RAM has been copied. Then triggering post-copy switchover, or removing the stress workload to allow completion, will crash it.

The only live migration scenario that should avoid this danger is multifd, since the high volume data transfers are handled in dedicated TCP connections which are unidirectional. The main bi-directionl TCP connection is only for co-ordination purposes

This patch implements a workaround that will prevent future QEMU versions from triggering the crash.

The only way to avoid the crash with *existing* running QEMU processes is to change the TLS cipher priority string to avoid use of AES with TLS 1.3. This can be done with the 'priority' field in the 'tls-creds-x509' object.eg

-object tls-creds-x509,id=tls0,priority=NORMAL:-AES-256-GCM:-AES-128-GCM:-AES-128-CCM

which should force the use of CHACHA20-POLY1305 which does not require TLS re-keying after 16 million sent records (64 GB of migration data).

https://gitlab.com/qemu-project/qemu/-/issues/1937

On RHEL/Fedora distros you can also use the system wide crypto priorities to override this from the migration *target* host by creating /etc/crypto-policies/local.d/gnutls-qemu.config containing

QEMU=NONE:+ECDHE-RSA:+ECDHE-ECDSA:+RSA:+DHE-RSA:+GROUP-X25519:+GROUP-X448:+GROUP-SECP256R1:+GROUP-SECP384R1:+GROUP-SECP521R1:+GROUP-FF

and running 'update-crypto-policies'. I recommend the QEMU level 'tls-creds-x509' workaround though, which new libvirt patches can soon do:

https://lists.libvirt.org/archives/list/[email protected]/thread/LX5KMIUFZSP5DPUXKJDFYBZI5TIE3E5N/

Daniel P. Berrangé (4):
crypto: implement workaround for GNUTLS thread safety problems
io: add support for activating TLS thread safety workaround
migration: activate TLS thread safety workaround
crypto: add tracing & warning about GNUTLS countermeasures

 crypto/tlssession.c           | 99 +++++++++++++++++++++++++++++++++--
 crypto/trace-events           |  2 +
 include/crypto/tlssession.h   | 14 +++++
 include/io/channel.h          |  1 +
 io/channel-tls.c              |  5 ++
 meson.build                   |  9 ++++
 meson_options.txt             |  2 +
 migration/tls.c               |  9 ++++
 scripts/meson-buildoptions.sh |  5 ++
 9 files changed, 143 insertions(+), 3 deletions(-)
Signature · 1 line
2.50.1