Skip to contentSkip to conversation

Live migration without shared storage (improved version)

PATCH Superseded [email protected] 3 participants 5 messages Sep 10, 2009 – Sep 14, 2009

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Live migration without shared storage (improved version)

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
kvm-87

#1 lore raw
Quoted 11 lines · On 10 sept. 2009, at 12:26, [email protected] wrote:

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of
allocated
data (user responsibility to verify that the same backend file
reside
on src and destination).

Quoted 10 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
kvm-87

Just tried with the v2 of the third patch. Now throughput is much better when the disk image has already been allocated (zero'd with dd) on the receiving end.

However, you must have a memory leak somewhere: at the end of the migration of a 128MB VM, the destination machine was swapping and qemu was using nearly 2GB of resident memory (the size of the disk image was 2GB).

Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
#2 lore raw

[email protected] wrote on 10/09/2009 17:35:53:

Quoted 15 lines · On 10 sept. 2009, at 12:26, [email protected] wrote:
Quoted 14 lines

This series adds support for live migration without shared storage, means
copy the storage while migrating. It was tested with KVM. Supports 2 ways
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of
allocated
data (user responsibility to verify that the same backend file
reside
on src and destination).

Quoted 8 lines
Quoted 8 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against

git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git

Quoted 6 lines
Quoted 4 lines

kvm-87

Quoted 8 lines

Just tried with the v2 of the third patch. Now throughput is much better when the disk image has already been allocated (zero'd with dd) on the receiving end.

However, you must have a memory leak somewhere: at the end of the migration of a 128MB VM, the destination machine was swapping and qemu was using nearly 2GB of resident memory (the size of the disk image was 2GB).

It seems that there is a memory leak but from what I see it is a memory leak in
raw_aio_writev. Seems like bdrv_aio_writev does not call the callback on completion.
Therefore the buffer allocated for the aio_writev is not being freed. Anyone knows
about this bug?

- Liran

#3 lore raw
In reply to [email protected]
Quoted 9 lines · [email protected] wrote:

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Quoted 7 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
kvm-87

As you are posting to qemu-devel, you should base your work on top of qemu.git master and make sure that it still works as intended. Upstream has KVM support (that is normally even usable), so testing won't take much longer than with qemu-kvm.

Nice feature by the way! Are you (also) using it to generate backups of a VM without stopping it (for a noticeably long time)? That's what we have in mind ATM.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#4 lore raw

[email protected] wrote on 11/09/2009 14:25:29:

Quoted 2 lines
Quoted 1 line · [email protected] wrote:

This series adds support for live migration without shared storage,

means

Quoted 1 line
Quoted 1 line

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 4 lines
Quoted 4 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of

allocated

Quoted 1 line
Quoted 1 line

data (user responsibility to verify that the same backend file

reside

Quoted 10 lines
Quoted 5 lines

on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 1 line

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

Quoted 2 lines

The patches are against

git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git

Quoted 7 lines
Quoted 1 line

kvm-87

As you are posting to qemu-devel, you should base your work on top of qemu.git master and make sure that it still works as intended. Upstream has KVM support (that is normally even usable), so testing won't take much longer than with qemu-kvm.

I will post a patch based on qemu.git.

Quoted 4 lines

Nice feature by the way! Are you (also) using it to generate backups of a VM without stopping it (for a noticeably long time)? That's what we have in mind ATM.

Right now I am not using it for different scenario accept live migration. But it
can be done easily.

- Liran