libvirk KVM save (migrate) terribly slow
Hi,
I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant.
I guess this is not what should be expected, is there something particular I should check?
Thanks a lot in advance
with best regards
nik
Signature · 13 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 8 lines · Nikola Ciprich
Hi,
I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant.
I guess this is not what should be expected, is there something particular I should check?
This matches behaviour that I see when saving VMs. QEMU takes a really very long time to save itself using migrate exec:. It is not CPU limited even in gzip mode - there is near zero CPU usage while this is going on. QEMU just seems to be very slow at sending the data. I've not had time to investigate whether this is a flaw in QEMU's exec: migration handling, or whether it is being hurt by too small pipe buffers, or something else entirely. I must say I'm not entirely convinced that it is a good idea for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but that could be a red-herring.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 20 lines · Daniel P. Berrange
Quoted 8 lines · Nikola Ciprich
Hi,
I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant.
I guess this is not what should be expected, is there something particular I should check?This matches behaviour that I see when saving VMs. QEMU takes a really very long time to save itself using migrate exec:. It is not CPU limited even in gzip mode - there is near zero CPU usage while this is going on. QEMU just seems to be very slow at sending the data. I've not had time to investigate whether this is a flaw in QEMU's exec: migration handling, or whether it is being hurt by too small pipe buffers, or something else entirely. I must say I'm not entirely convinced that it is a good idea for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but that could be a red-herring.
Quoted 1 line
Daniel
I've reported this issue back when the regression was introduced in qemu.
Anthony had the same idea (not mixing popen with non-blocking I/O), but no solution was found at the time. I haven't tried recently (I'm using TCP migration only) but the problem must still be here.
The thread on qemu-devel:
http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg01557.html
http://lists.gnu.org/archive/html/qemu-devel/2009-09/msg00020.html
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/