QEMU P2P migration speed
Hello,
I`ve got strange results during benchmarking migration speed for different kinds of loads on source/target host: when source host is 'empty', migration takes approx. 30 percent longer than the same for host already occupied by one VM with CPU overcommit ratio=1.
[src host, three equal vms, each with ability to eat all cores once] [tgt host, one VM, with same appetite and limitations]
All VMs was put into cgroups with same cpu ceiling and cpu shares values.
Migration time is almost independent of VM RSS(varies by ten percent at maximum), for situation when VM is active on target host, time is about 85 seconds to migrate 8G between hosts, and when it is turned off, migration time *increasing* to 120s. For curious ones, frequency management is completely inactive on both nodes, neither CStates mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
Anyone have a suggestions on how to possibly explain/fix this?
Paolo Bonzini
maintainerIl 04/02/2014 18:06, Andrey Korolyov ha scritto:
Quoted 6 lines
Migration time is almost independent of VM RSS(varies by ten percent at maximum), for situation when VM is active on target host, time is about 85 seconds to migrate 8G between hosts, and when it is turned off, migration time *increasing* to 120s. For curious ones, frequency management is completely inactive on both nodes, neither CStates mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
What version of QEMU?
Paolo
Quoted 11 lines · Paolo Bonzini
Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
Quoted 6 lines
Migration time is almost independent of VM RSS(varies by ten percent at maximum), for situation when VM is active on target host, time is about 85 seconds to migrate 8G between hosts, and when it is turned off, migration time *increasing* to 120s. For curious ones, frequency management is completely inactive on both nodes, neither CStates mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
What version of QEMU?
Paolo
Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?
Paolo Bonzini
maintainerIl 05/02/2014 11:46, Andrey Korolyov ha scritto:
Quoted 9 lines · Paolo Bonzini
Quoted 8 lines
Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
Quoted 6 lines
Migration time is almost independent of VM RSS(varies by ten percent at maximum), for situation when VM is active on target host, time is about 85 seconds to migrate 8G between hosts, and when it is turned off, migration time *increasing* to 120s. For curious ones, frequency management is completely inactive on both nodes, neither CStates mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
Quoted 5 lines
Quoted 3 lines
What version of QEMU?
Paolo
Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?
Yeah, you can checkout the release notes on wiki.qemu.org to find out which versions had good improvements. You can also try compiling straight from git, there are more speedups there.
Paolo
Quoted 14 lines · Paolo Bonzini
Il 05/02/2014 11:46, Andrey Korolyov ha scritto:
Quoted 13 lines · Paolo Bonzini
Quoted 11 lines
Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
Quoted 6 lines
Migration time is almost independent of VM RSS(varies by ten percent at maximum), for situation when VM is active on target host, time is about 85 seconds to migrate 8G between hosts, and when it is turned off, migration time *increasing* to 120s. For curious ones, frequency management is completely inactive on both nodes, neither CStates mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
What version of QEMU?
Paolo
Quoted 8 lines
Quoted 1 line
Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?
Yeah, you can checkout the release notes on wiki.qemu.org to find out which versions had good improvements. You can also try compiling straight from git, there are more speedups there.
Paolo
Took and build 1.6.2 and faced a problem - after a couple of bounce iterations of migration (1->2->1->2) VM is not able to migrate anymore back in a probabilistic manner with an error 'internal error unexpected migration status in setup'. Error may disappear over a time, or may not disappear at all and it may took a lot of tries in a row to succeed. There are no obvious hints with default logging level in libvirt/qemu logs and seemingly libvirt is not a cause because "accumulated error state" preserves over service restarts. Also every VM is affected, not ones which are experiencing multiple migration actions. Error happens on 3rd-5th second of the migration procedure, if it may help.
What is more interesting, the original counter-intuitive behavior is not disappeared but increased relative span: 25 vs 70 seconds for fully commited 8G VM. I am suspecting some mechanism falling back to the idle and dropping overall performance therefore but can not image one beyond standard freq/cstates which are definitely turned off.
Paolo Bonzini
maintainerIl 06/02/2014 14:40, Andrey Korolyov ha scritto:
Quoted 10 lines
Took and build 1.6.2 and faced a problem - after a couple of bounce iterations of migration (1->2->1->2) VM is not able to migrate anymore back in a probabilistic manner with an error 'internal error unexpected migration status in setup'. Error may disappear over a time, or may not disappear at all and it may took a lot of tries in a row to succeed. There are no obvious hints with default logging level in libvirt/qemu logs and seemingly libvirt is not a cause because "accumulated error state" preserves over service restarts. Also every VM is affected, not ones which are experiencing multiple migration actions. Error happens on 3rd-5th second of the migration procedure, if it may help.
You need to update libvirt too.
Paolo
Quoted 12 lines · Paolo Bonzini
Il 06/02/2014 14:40, Andrey Korolyov ha scritto:
Quoted 10 lines
Took and build 1.6.2 and faced a problem - after a couple of bounce iterations of migration (1->2->1->2) VM is not able to migrate anymore back in a probabilistic manner with an error 'internal error unexpected migration status in setup'. Error may disappear over a time, or may not disappear at all and it may took a lot of tries in a row to succeed. There are no obvious hints with default logging level in libvirt/qemu logs and seemingly libvirt is not a cause because "accumulated error state" preserves over service restarts. Also every VM is affected, not ones which are experiencing multiple migration actions. Error happens on 3rd-5th second of the migration procedure, if it may help.
Quoted 3 lines
You need to update libvirt too.
Paolo
Ok, I will do, but looks like libvirt version(1.0.2) in not relevant - it meets criteria set by debian packagers and again, 'broken state' is not relevant to the libvirt state history, it more likely to be qemu/kvm problem.
Paolo Bonzini
maintainerIl 07/02/2014 14:07, Andrey Korolyov ha scritto:
Quoted 4 lines
Ok, I will do, but looks like libvirt version(1.0.2) in not relevant - it meets criteria set by debian packagers and again, 'broken state' is not relevant to the libvirt state history, it more likely to be qemu/kvm problem.
It is relevant, qemu introduced a new migration status before "active" ("setup") and libvirt doesn't recognize it.
Paolo
Paolo Bonzini
maintainerIl 07/02/2014 14:07, Andrey Korolyov ha scritto:
Quoted 2 lines
Ok, I will do, but looks like libvirt version(1.0.2) in not relevant - it meets criteria set by debian packagers
Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict with libvirt <1.2.0.
Quoted 3 lines
and again, 'broken state' is
not relevant to the libvirt state history, it more likely to be qemu/kvm problem.
It is relevant, qemu introduced a new migration status before "active" ("setup") and libvirt doesn't recognize it. That's why you need at least 1.2.0.
Paolo
Quoted 6 lines · Paolo Bonzini
Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
Quoted 2 lines
Ok, I will do, but looks like libvirt version(1.0.2) in not relevant - it meets criteria set by debian packagers
Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict with libvirt <1.2.0.
I've no idea when qemu breaks libvirt. I found out - just by a chance - that qemu 1.3+ breaks libvirt <1.0, and I stated this in the deps. But the thing that 1.6 requires libvirt 1.2 is news for me.
I'll add this requiriment to the debian package.
At any rate, there's no libvirt 1.0 in debian. Current stable has 0.9, and current testing has 1.2.1, and this version is also available in backports for stable. 1.2 was the first version past 0.9 which were backproted to stable. There's no other versions of libvirt in debian.
So whomever installed that mess did that on their own, it is definitely not supported on debian ;)
Thanks,
/mjt
Quoted 10 lines
Quoted 3 lines
and again, 'broken state' is
not relevant to the libvirt state history, it more likely to be qemu/kvm problem.It is relevant, qemu introduced a new migration status before "active" ("setup") and libvirt doesn't recognize it. That's why you need at least 1.2.0.
Paolo
Quoted 11 lines · Paolo Bonzini
Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
Quoted 2 lines
Ok, I will do, but looks like libvirt version(1.0.2) in not relevant - it meets criteria set by debian packagers
Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict with libvirt <1.2.0.
Quoted 3 lines
and again, 'broken state' is
not relevant to the libvirt state history, it more likely to be qemu/kvm problem.
Quoted 6 lines
It is relevant, qemu introduced a new migration status before "active" ("setup") and libvirt doesn't recognize it. That's why you need at least 1.2.0.
Paolo
Thanks, both issues - with reverted CPU dependency and with migration itself went away.