Skip to contentSkip to conversation

Live migration without shared storage

PATCH In review [email protected] 3 participants 6 messages Sep 7, 2009 – Sep 8, 2009

lore
Share & tools
lore
#0 lore raw

cover Live migration without shared storage

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
kvm-87

#1 lore raw
Quoted 11 lines · On 7 sept. 2009, at 11:24, [email protected] wrote:

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of
allocated
data (user responsibility to verify that the same backend file
reside
on src and destination).

Quoted 10 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
kvm-87

I'm trying blk migration (not incremental) between two machines connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire).
While the load on the sending end is low (vmstat says ~2000 blocks in/sec, and top says ~1% in io wait), on the receiving end I see almost 40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports ~14000 blocks out/sec.

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches.
The guest is also running Debian Lenny and is idle io wise. I tried with both idle and full cpu utilization, it doesn't change anything.

Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
#2 lore raw
Quoted 11 lines · Pierre Riteau

I'm trying blk migration (not incremental) between two machines connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire). While the load on the sending end is low (vmstat says ~2000 blocks in/ sec, and top says ~1% in io wait), on the receiving end I see almost 40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports ~14000 blocks out/sec.

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches. The guest is also running Debian Lenny and is idle io wise. I tried with both idle and full cpu utilization, it doesn't change anything.

If you happen to be using ext3, then that might explain it. ext3 has long had issues with performance while slowly writing a large file (this affects mythtv a lot for example). 2.6.30 should improve things a lot, although ext3 will never be great. ext4 is better, and hopefully btrfs will be great (when done). xfs is also supposed to handle such tasks very well.

Signature · 1 line
Len Sorensen
#3 lore raw

[email protected] (Lennart Sorensen) wrote on 08/09/2009 16:26:31:

Quoted 6 lines · Pierre Riteau
Quoted 5 lines

I'm trying blk migration (not incremental) between two machines connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire). While the load on the sending end is low (vmstat says ~2000 blocks in/ sec, and top says ~1% in io wait), on the receiving end I see almost

40%

Quoted 1 line
Quoted 1 line

CPU in io wait, kjournald takes 20% of the CPU and vmstat reports

~14000

Quoted 4 lines
Quoted 4 lines

blocks out/sec.

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches. The guest is also running Debian Lenny and is idle io wise. I tried

with

Quoted 11 lines
Quoted 1 line

both idle and full cpu utilization, it doesn't change anything.

If you happen to be using ext3, then that might explain it. ext3 has long had issues with performance while slowly writing a large file (this affects mythtv a lot for example). 2.6.30 should improve things a lot, although ext3 will never be great. ext4 is better, and hopefully btrfs will be great (when done). xfs is also supposed to handle such tasks very well.

--
Len Sorensen

I found performance issues in the patch itself. I will first submit a new patch
and maybe then will be able to talk about the fs performance.

Thanks,
- Liran

#4 lore raw
In reply to Lennart Sorensen
Quoted 13 lines · Lennart Sorensen
Quoted 11 lines · Pierre Riteau

I'm trying blk migration (not incremental) between two machines connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire). While the load on the sending end is low (vmstat says ~2000 blocks in/ sec, and top says ~1% in io wait), on the receiving end I see almost 40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports ~14000 blocks out/sec.

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches. The guest is also running Debian Lenny and is idle io wise. I tried with both idle and full cpu utilization, it doesn't change anything.

Quoted 9 lines

If you happen to be using ext3, then that might explain it. ext3 has long had issues with performance while slowly writing a large file (this affects mythtv a lot for example). 2.6.30 should improve things a lot, although ext3 will never be great. ext4 is better, and hopefully btrfs will be great (when done). xfs is also supposed to handle such tasks very well.

Signature · 1 line
Len Sorensen

Yes, I'm using ext3. When I tested last week, I created the disk image on the destination machine using qemu-img create -f raw, which creates a sparse file.
I experimented again today, but this time I allocated all the blocks in the disk image prior to the migration.
This resulted in increased throughput (between 6 and 8 MB/s on the wire), reduced CPU usage on the destination node (now qemu is the top CPU user, not kjournald), and the IO interrupt numbers are similar on the origin and destination node (about 7000/s). Still, it could be better, but I see Liran is working on it :)

Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
#5 lore raw
In reply to Pierre Riteau

[email protected] wrote on 07/09/2009 19:40:51:

Quoted 15 lines · On 7 sept. 2009, at 11:24, [email protected] wrote:
Quoted 14 lines

This series adds support for live migration without shared storage, means
copy the storage while migrating. It was tested with KVM. Supports 2 ways
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of
allocated
data (user responsibility to verify that the same backend file
reside
on src and destination).

Quoted 8 lines
Quoted 8 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against

git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git

Quoted 14 lines
Quoted 4 lines

kvm-87

I'm trying blk migration (not incremental) between two machines connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire). While the load on the sending end is low (vmstat says ~2000 blocks in/ sec, and top says ~1% in io wait), on the receiving end I see almost 40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports ~14000 blocks out/sec.

Quoted 7 lines

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches. The guest is also running Debian Lenny and is idle io wise. I tried with both idle and full cpu utilization, it doesn't change anything.

--
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/

True I see that there is a problem with performance. Need to change the destination
side to write async to disk And in the source side to submit async read from disk at
least at the rate of the network bandwidth. I will resend a new patch series.

Thanks,
- Liran