Live migration without shared storage
v5 cover Live migration without shared storage
This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).
Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.
Changes from v4:
- Register block dirty tracking for each block device.
- Block.c will now manage a dirty bitmap per device once bdrv_set_dirty_tracking() is called. Bitmap is polled by the upper layer (block-migration.c).
- Global variabels moved to a global state structure allocated dynamically.
- Coding style issues.
Patch summary:
Makefile | 1 + block-migration.c | 558 +++++++++++++++++++++++++++++++++++++++++++++++++++++ block-migration.h | 29 +++ block.c | 78 ++++++++- block.h | 8 +- block_int.h | 2 + buffered_file.c | 10 +- hw/hw.h | 9 +- migration-exec.c | 9 +- migration-fd.c | 7 +- migration-tcp.c | 7 +- migration-unix.c | 7 +- migration.c | 23 ++- migration.h | 20 ++- monitor.c | 36 +++- qemu-monitor.hx | 19 ++- savevm.c | 62 +++++-- sysemu.h | 2 +- vl.c | 9 +- 19 files changed, 841 insertions(+), 55 deletions(-)
Quoted 9 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
This series adds support for live migration without shared storage, means copy the storage while migrating. It was tested with KVM. Supports 2 ways to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).
Quoted 5 lines
Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
- Liran
Avi Kivity <[email protected]> wrote on 02/11/2009 20:47:34:
Quoted 2 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 1 line
This series adds support for live migration without shared storage,
means
Quoted 1 line
Quoted 1 line
copy the storage while migrating. It was tested with KVM. Supports 2
ways
Quoted 20 lines
Quoted 10 lines
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration withcomplete storage copy
Quoted 1 line
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with
incremental storage copy, storage is cow based.
Quoted 1 line
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
Quoted 4 lines
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices != BDRV_TYPE_HD will be ignored.
What do you think?
- Liran
Quoted 42 lines · Liran Schour
- Liran
Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:
Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines
This series adds support for live migration without shared storage,
means
Quoted 2 lines
Quoted 2 lines
copy the storage while migrating. It was tested with KVM. Supports 2
ways
Quoted 23 lines
Quoted 11 lines
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration withcomplete storage copy
Quoted 2 lines
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with
incremental storage copy, storage is cow based.
Quoted 1 line
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
Quoted 5 lines
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
Quoted 10 lines
I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. Andcopy_levelwill be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices !=BDRV_TYPE_HDwill be ignored.
Looks good.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 55 lines · Avi Kivity
Quoted 42 lines · Liran Schour
- Liran
Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:
Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines
This series adds support for live migration without shared storage,
means
Quoted 2 lines
Quoted 2 lines
copy the storage while migrating. It was tested with KVM. Supports 2
ways
Quoted 23 lines
Quoted 11 lines
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration withcomplete storage copy
Quoted 2 lines
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with
incremental storage copy, storage is cow based.
Quoted 1 line
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
Quoted 5 lines
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
Quoted 11 lines
I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. Andcopy_levelwill
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices !=BDRV_TYPE_HDwill be ignored.
Quoted 2 lines
Looks good.
Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 56 lines · Jan Kiszka
Quoted 55 lines · Avi Kivity
Quoted 53 lines · Liran Schour
- Liran
Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:
Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines
This series adds support for live migration without shared storage,
means
Quoted 2 lines
Quoted 2 lines
copy the storage while migrating. It was tested with KVM. Supports 2
ways
Quoted 23 lines
Quoted 11 lines
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration withcomplete storage copy
Quoted 2 lines
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with
incremental storage copy, storage is cow based.
Quoted 1 line
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
Quoted 5 lines
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. Andcopy_levelwill
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices !=BDRV_TYPE_HDwill be ignored.
Quoted 8 lines
Quoted 2 lines
Looks good.
Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?
Jan
It was committed to the master branch a week ago.
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
Quoted 58 lines · Pierre Riteau
Quoted 56 lines · Jan Kiszka
Quoted 55 lines · Avi Kivity
Quoted 41 lines · Liran Schour
- Liran
Avi Kivity<[email protected]> wrote on 02/11/2009 20:47:34:
Quoted 4 lines · On 11/02/2009 03:40 PM, [email protected] wrote:
Quoted 3 lines
This series adds support for live migration without shared storage,
means
Quoted 2 lines
Quoted 2 lines
copy the storage while migrating. It was tested with KVM. Supports 2
ways
Quoted 22 lines
Quoted 11 lines
to replicate the storage during migration:
1. Complete copy of storage to destination
2. Assuming the storage is cow based, copy only the allocated data, time of the migration will be linear with the amount of allocated data (user responsibility to verify that the same backend file reside on src and destination).Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration withcomplete storage copy
Quoted 2 lines
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with
incremental storage copy, storage is cow based.
I'd like to see the syntax generalized. For one, the guest may have several disks; an install image cdrom might be available as an nfs image but the main storage is local. Secondly, there can be several levels of cow and we want to control which one we copy.
Quoted 5 lines
I'll leave the exact details to the qpeople, but if we can specify a copy depth for each device, where 0=copy nothing, n=copy everything, 1=copy the last level (equivalent to -d blk inc) I think we'll have covered everything.
Quoted 11 lines
I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]] tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. Andcopy_levelwill
be integer or n, 0=flat copy of the device, 1=copy the last level ... n=copy all levels.
All HD devices that will not appear in the command will be treated as copy_level=0. All devices !=BDRV_TYPE_HDwill be ignored.Looks good.
Quoted 7 lines
Quoted 5 lines
Do I get this correctly: this series is now "only" blocked by the required rework of its user interface? Is there a schedule for this, a v6 roll out? Will it be in time for qemu-0.12?
Jan
It was committed to the master branch a week ago.
Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git. And the missing merge notifications contributed to this as well.
Anyway, good to see this upstream. /me goes testing.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 2 lines · Jan Kiszka
Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git.
They still need to be addressed. However, I wanted to do that work in the tree vs. outside of the tree. With all the qmp stuff happening, that seemed like the rationale thing to do.
Signature · 3 lines
Regards, Anthony Liguori
"Anthony Liguori" <[email protected]> wrote on 24/11/2009 16:27:58:
Quoted 8 lines
Quoted 2 lines · Jan Kiszka
Oh, indeed, thanks. Due to the fact that this discussion suggested that there are still open issues, I did not even checked git.
They still need to be addressed. However, I wanted to do that work in the tree vs. outside of the tree. With all the qmp stuff happening, that seemed like the rationale thing to do.
What do you think about the following syntax:
migrate [-d] URI [block:default_copy_level:[<device:copy_level>,.....,<device:copy_level>]copy_level: 0 - flat copy.
1 - copy only first COW level.
..
n - copy all COW levels.default_copy_level: use this copy level for all devices that are not mentioned in the migrate command.
I intend to implement it this week, I am planning to implement on the first phase only copy levels 0 and 1.
It suppose to solve the major problem today that a guest has one block device that needs to be copied flat and the other block device that needs to be COW copy.
- Liran
[email protected] wrote on 02/11/2009 15:40:25:
Quoted 7 lines
Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.
The up-to-date migrate command will work as follow:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration (qemu) migrate -d -b tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d -i tcp:0:4444 # for live migration with incremental storage copy,
storage is cow based.
- Liran