Skip to contentSkip to conversation

Live migration without shared storage

PATCH In review [email protected] 5 participants 10 messages Nov 2, 2009 – Nov 24, 2009

lore
Share & tools
lore
#0 lore raw

v5 cover Live migration without shared storage

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

Changes from v4:
- Register block dirty tracking for each block device.
- Block.c will now manage a dirty bitmap per device once bdrv_set_dirty_tracking() is called. Bitmap is polled by the upper layer (block-migration.c).
- Global variabels moved to a global state structure allocated dynamically.
- Coding style issues.

Patch summary:

Makefile          |    1 +
block-migration.c |  558 +++++++++++++++++++++++++++++++++++++++++++++++++++++
block-migration.h |   29 +++
block.c           |   78 ++++++++-
block.h           |    8 +-
block_int.h       |    2 +
buffered_file.c   |   10 +-
hw/hw.h           |    9 +-
migration-exec.c  |    9 +-
migration-fd.c    |    7 +-
migration-tcp.c   |    7 +-
migration-unix.c  |    7 +-
migration.c       |   23 ++-
migration.h       |   20 ++-
monitor.c         |   36 +++-
qemu-monitor.hx   |   19 ++-
savevm.c          |   62 +++++--
sysemu.h          |    2 +-
vl.c              |    9 +-
19 files changed, 841 insertions(+), 55 deletions(-)
#1 lore raw
Quoted 9 lines · On 11/02/2009 03:40 PM, [email protected] wrote:

This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Quoted 5 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#2 lore raw

- Liran

Avi Kivity <[email protected]> wrote on 02/11/2009 20:47:34:

Quoted 2 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 1 line

This series adds support for live migration without shared storage,

means

Quoted 1 line
Quoted 1 line

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 20 lines
Quoted 10 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 1 line

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

Quoted 1 line

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

Quoted 4 lines

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

What do you think?

- Liran

#3 lore raw
Quoted 42 lines · Liran Schour

- Liran

Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:

Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines

This series adds support for live migration without shared storage,

means

Quoted 2 lines
Quoted 2 lines

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 23 lines
Quoted 11 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 2 lines

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

Quoted 1 line

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

Quoted 5 lines

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

Quoted 10 lines

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

Looks good.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 55 lines · Avi Kivity
Quoted 42 lines · Liran Schour

- Liran

Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:

Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines

This series adds support for live migration without shared storage,

means

Quoted 2 lines
Quoted 2 lines

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 23 lines
Quoted 11 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 2 lines

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

Quoted 1 line

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

Quoted 5 lines

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

Quoted 11 lines

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

Quoted 2 lines

Looks good.

Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#5 lore raw
Quoted 56 lines · Jan Kiszka
Quoted 55 lines · Avi Kivity
Quoted 53 lines · Liran Schour

- Liran

Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:

Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines

This series adds support for live migration without shared storage,

means

Quoted 2 lines
Quoted 2 lines

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 23 lines
Quoted 11 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 2 lines

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

Quoted 1 line

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

Quoted 5 lines

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

Quoted 8 lines
Quoted 2 lines

Looks good.

Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?

Jan

It was committed to the master branch a week ago.

Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
#6 lore raw
Quoted 58 lines · Pierre Riteau
Quoted 56 lines · Jan Kiszka
Quoted 55 lines · Avi Kivity
Quoted 41 lines · Liran Schour

- Liran

Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:

Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines

This series adds support for live migration without shared storage,

means

Quoted 2 lines
Quoted 2 lines

copy the storage while migrating. It was tested with KVM. Supports 2

ways

Quoted 22 lines
Quoted 11 lines

to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with

complete storage copy

Quoted 2 lines

(qemu) migrate -d blk inc tcp:0:4444 # for live migration with

incremental storage copy, storage is cow based.

I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.

Quoted 5 lines

I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.

Quoted 11 lines

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

Looks good.

Quoted 7 lines
Quoted 5 lines

Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?

Jan

It was committed to the master branch a week ago.

Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git. And the missing merge notifications contributed to this as well.

Anyway, good to see this upstream. /me goes testing.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#7 lore raw
Quoted 2 lines · Jan Kiszka

Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git.

They still need to be addressed. However, I wanted to do that work in the tree vs. outside of the tree. With all the qmp stuff happening, that seemed like the rationale thing to do.

Signature · 3 lines
Regards,

Anthony Liguori
#8 lore raw

"Anthony Liguori" <[email protected]> wrote on 24/11/2009 16:27:58:

Quoted 8 lines
Quoted 2 lines · Jan Kiszka

Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git.

They still need to be addressed. However, I wanted to do that work in the tree vs. outside of the tree. With all the qmp stuff happening, that seemed like the rationale thing to do.

What do you think about the following syntax:
migrate [-d] URI [block:default_copy_level:[<device:copy_level>,.....,<device:copy_level>]
copy_level: 0 - flat copy.
1 - copy only first COW level.
..
n - copy all COW levels.
default_copy_level: use this copy level for all devices that are not mentioned in the migrate command.

I intend to implement it this week, I am planning to implement on the first phase only copy levels 0 and 1.
It suppose to solve the major problem today that a guest has one block device that needs to be copied flat and the other block device that needs to be COW copy.

- Liran

#9 lore raw
In reply to [email protected]

[email protected] wrote on 02/11/2009 15:40:25:

Quoted 7 lines

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The up-to-date migrate command will work as follow:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d -b tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d -i tcp:0:4444 # for live migration with incremental storage copy,
storage is cow based.

- Liran