block migration: Fixes, cleanups and speedups
cover block migration: Fixes, cleanups and speedups
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restore
One patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. add bdrv_is_allocated support for the latter)
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
Find the patches also at git://git.kiszka.org/qemu.git queues/migration
Jan Kiszka (22):
migration: Fix use of file after release
migration: Catch multiple start commands
block migration: Fix coding style and whitespaces
block migration: Rework constants API
block migration: Cleanup dirty tracking code
block migration: Avoid large stack buffer
block migration: Avoid indirection of block_mig_state
block migration: Drop dead code
block migration: Switch device and block lists to QSIMPLEQ block migration: Initialize remaining BlkMigState fields
block migration: Clean up use of total_sectors
block migration: Consolidate mig_read_device_bulk into mig_save_device_bulk block migration: Consolidate block transmission
block migration: Add error handling/propagation
ram migration: Stop loading on error
live migration: Allow cleanup after cancellation or error block migration: Report overall migration progress
live migration: Propagate output monitor to callback handler block migration: Fix outgoing progress output
block migration: Report progress also via info migration
block migration: Add support for restore progress reporting block migration: Increase dirty chunk size to 1M
Pierre Riteau (1):
Import a simple queue implementation from NetBSD
block-migration.c | 737 ++++++++++++++++++++++++++--------------------------- block-migration.h | 16 +- block.c | 127 +++++----- block.h | 11 +- block_int.h | 3 +- hw/hw.h | 3 +- migration-exec.c | 12 +- migration-fd.c | 9 +- migration-tcp.c | 10 +- migration-unix.c | 10 +- migration.c | 52 +++-- migration.h | 13 +- qemu-queue.h | 109 ++++++++- savevm.c | 42 ++- sysemu.h | 9 +- vl.c | 13 +- 16 files changed, 653 insertions(+), 523 deletions(-)
PATCH 1/23 migration: Fix use of file after release
qemu_fclose frees the passed file structure, but do_migrate_set_speed may access it later on. Fix it by setting file NULL in migrate_fd_cleanup and checking for this.
---
migration.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
migration.c +3 -3
@@ -118,12 +118,11 @@ void do_migrate_set_speed(Monitor *mon, const QDict *qdict, QObject **ret_data) } max_throttle = (uint32_t )d; - s =migrate_to_fms (current_migration); -if (s) { + s =migrate_to_fms (current_migration); +if (s && s->file) {qemu_file_set_rate_limit (s->file, max_throttle); } - }/* amount of nanoseconds we are willing to wait for migration to be down.
@@ -209,6 +208,7 @@ void migrate_fd_cleanup(FdMigrationState *s)if (s->file) {dprintf ("closing file\n" );qemu_fclose (s->file); + s->file =NULL ; }if (s->fd !=-1 )
PATCH 2/23 migration: Catch multiple start commands
In reply to Jan Kiszka---
migration.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-)
migration.c +7 -1
@@ -58,7 +58,13 @@ void do_migrate(Monitor *mon, const QDict *qdict, QObject **ret_data)const char *p;int detach =qdict_get_int (qdict,"detach" );const char *uri =qdict_get_str (qdict,"uri" ); - + +if (current_migration && + current_migration->get_status (current_migration) ==MIG_STATE_ACTIVE ) { +monitor_printf (mon,"migration already in progress\n" ); +return ; + } +if (strstart (uri,"tcp:" , &p)) s = tcp_start_outgoing_migration(p, max_throttle, detach, (int)qdict_get_int(qdict, "blk"),
PATCH 3/23 block migration: Fix coding style and whitespaces
In reply to Jan KiszkaNo functional changes.
---
block-migration.c | 390 +++++++++++++++++++++++++---------------------------- block-migration.h | 10 - block.c | 49 ++++--- block.h | 4 - 4 files changed, 212 insertions(+), 241 deletions(-)
block-migration.c +184 -206
@@ -21,7 +21,7 @@#define SECTOR_SIZE (1 << SECTOR_BITS)#define SECTOR_MASK ~(SECTOR_SIZE - 1); -#define BLOCK_SIZE (block_mig_state->sectors_per_block << SECTOR_BITS) +#define BLOCK_SIZE (block_mig_state->sectors_per_block << SECTOR_BITS)#define BLK_MIG_FLAG_DEVICE_BLOCK 0x01#define BLK_MIG_FLAG_EOS 0x02
@@ -34,13 +34,23 @@//#define DEBUG_BLK_MIGRATION #ifdef DEBUG_BLK_MIGRATION -#define dprintf (fmt, ...) \ +#define dprintf (fmt, ...) \ do { printf("blk_migration: " fmt, ## __VA_ARGS__); } while (0)#else -#define dprintf (fmt, ...) \ +#define dprintf (fmt, ...) \ do { } while (0)#endif +typedef struct BlkMigDevState { +BlockDriverState *bs; +int bulk_completed; +int shared_base; +struct BlkMigDevState *next; +int64_t cur_sector; +int64_t total_sectors; +int64_t dirty; +}BlkMigDevState ; + typedef struct BlkMigBlock { uint8_t *buf; BlkMigDevState *bmds;
@@ -68,87 +78,85 @@ typedef struct BlkMigState {int64_t print_completion; } BlkMigState; -static BlkMigState *block_mig_state =NULL ; +static BlkMigState *block_mig_state =NULL ;static void blk_mig_read_cb (void *opaque,int ret) {BlkMigBlock *blk = opaque; - + blk->ret = ret; - +/* insert at the end */ -if (block_mig_state->last_blk ==NULL ) { +if (block_mig_state->last_blk ==NULL ) { block_mig_state->first_blk = blk; block_mig_state->last_blk = blk; }else { block_mig_state->last_blk->next = blk; block_mig_state->last_blk = blk; } - + block_mig_state->submitted--; block_mig_state->read_done++;assert (block_mig_state->submitted >=0 ); - -return ; }static int mig_read_device_bulk (QEMUFile *f,BlkMigDevState *bms) -{ +{int nr_sectors;int64_t total_sectors, cur_sector =0 ;BlockDriverState *bs = bms->bs;BlkMigBlock *blk; - + blk =qemu_malloc (sizeof (BlkMigBlock)); blk->buf =qemu_malloc (BLOCK_SIZE ); - + cur_sector = bms->cur_sector; total_sectors =bdrv_getlength (bs) >>SECTOR_BITS ; - -if (bms->shared_base) { -while (cur_sector < bms->total_sectors && - !bdrv_is_allocated (bms->bs, cur_sector, -MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { + +if (bms->shared_base) { +while (cur_sector < bms->total_sectors && + !bdrv_is_allocated (bms->bs, cur_sector, +MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { cur_sector += nr_sectors; } } - -if (cur_sector >= total_sectors) { + +if (cur_sector >= total_sectors) { bms->cur_sector = total_sectors;qemu_free (blk->buf);qemu_free (blk);return 1 ; } - -if (cur_sector >= block_mig_state->print_completion) { + +if (cur_sector >= block_mig_state->print_completion) {printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); - block_mig_state->print_completion += + block_mig_state->print_completion += (block_mig_state->sectors_per_block *10000 ); } - +/* we going to transfder BLOCK_SIZE any way even if it is not allocated */ nr_sectors = block_mig_state->sectors_per_block; cur_sector &= ~((int64_t )block_mig_state->sectors_per_block -1 ); - -if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { + +if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { nr_sectors = (total_sectors - cur_sector); } - + bms->cur_sector = cur_sector + nr_sectors; blk->sector = cur_sector; blk->bmds = bms; blk->next =NULL ; - + blk->iov.iov_base = blk->buf; blk->iov.iov_len = nr_sectors *SECTOR_SIZE ;qemu_iovec_init_external (&blk->qiov, &blk->iov,1 ); - + blk->aiocb =bdrv_aio_readv (bs, cur_sector, &blk->qiov, nr_sectors, blk_mig_read_cb, blk); - -if (!blk->aiocb) { + +if (!blk->aiocb) {printf ("Error reading sector %" PRId64"\n" , cur_sector);qemu_free (blk->buf);qemu_free (blk);
@@ -157,91 +165,87 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms)bdrv_reset_dirty (bms->bs, cur_sector, nr_sectors); block_mig_state->submitted++; - +return (bms->cur_sector >= total_sectors); }static int mig_save_device_bulk (QEMUFile *f,BlkMigDevState *bmds) -{ +{int len, nr_sectors;int64_t total_sectors = bmds->total_sectors, cur_sector =0 ;uint8_t *tmp_buf =NULL ;BlockDriverState *bs = bmds->bs; tmp_buf =qemu_malloc (BLOCK_SIZE ); - + cur_sector = bmds->cur_sector; - -if (bmds->shared_base) { -while (cur_sector < bmds->total_sectors && - !bdrv_is_allocated (bmds->bs, cur_sector, -MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { + +if (bmds->shared_base) { +while (cur_sector < bmds->total_sectors && + !bdrv_is_allocated (bmds->bs, cur_sector, +MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { cur_sector += nr_sectors; } } - -if (cur_sector >= total_sectors) { + +if (cur_sector >= total_sectors) { bmds->cur_sector = total_sectors;qemu_free (tmp_buf);return 1 ; } - -if (cur_sector >= block_mig_state->print_completion) { + +if (cur_sector >= block_mig_state->print_completion) {printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); - block_mig_state->print_completion += + block_mig_state->print_completion += (block_mig_state->sectors_per_block *10000 ); } - + cur_sector &= ~((int64_t )block_mig_state->sectors_per_block -1 ); - -/* we going to transfer -BLOCK_SIZE -any way even if it is not allocated */ + +/* we going to transfer BLOCK_SIZE any way even if it is not allocated */ nr_sectors = block_mig_state->sectors_per_block; - -if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { + +if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { nr_sectors = (total_sectors - cur_sector); } - -if (bdrv_read (bs, cur_sector, tmp_buf, nr_sectors) <0 ) { + +if (bdrv_read (bs, cur_sector, tmp_buf, nr_sectors) <0 ) {printf ("Error reading sector %" PRId64"\n" , cur_sector); }bdrv_reset_dirty (bs, cur_sector, nr_sectors); - -/* Device name */ -qemu_put_be64 (f,(cur_sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); - + +/* sector number and flags */ +qemu_put_be64 (f, (cur_sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); + +/* device name */ len =strlen (bs->device_name);qemu_put_byte (f, len);qemu_put_buffer (f, (uint8_t *)bs->device_name, len); - -qemu_put_buffer (f, tmp_buf, -BLOCK_SIZE ); - + +qemu_put_buffer (f, tmp_buf,BLOCK_SIZE ); + bmds->cur_sector = cur_sector + block_mig_state->sectors_per_block; - +qemu_free (tmp_buf); - +return (bmds->cur_sector >= total_sectors); }static void send_blk (QEMUFile *f,BlkMigBlock * blk) {int len; - -/* Device name */ -qemu_put_be64 (f,(blk->sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); - + +/* sector number and flags */ +qemu_put_be64 (f, (blk->sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); + +/* device name */ len =strlen (blk->bmds->bs->device_name);qemu_put_byte (f, len);qemu_put_buffer (f, (uint8_t *)blk->bmds->bs->device_name, len); - -qemu_put_buffer (f, blk->buf, -BLOCK_SIZE ); - -return ; + +qemu_put_buffer (f, blk->buf,BLOCK_SIZE ); }static void blk_mig_save_dev_info (QEMUFile *f,BlkMigDevState *bmds)
@@ -251,47 +255,43 @@ static void blk_mig_save_dev_info(QEMUFile *f, BlkMigDevState *bmds)static void set_dirty_tracking (int enable) {BlkMigDevState *bmds; -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { -bdrv_set_dirty_tracking (bmds->bs,enable); +for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +bdrv_set_dirty_tracking (bmds->bs, enable); } - -return ; }static void init_blk_migration (QEMUFile *f) {BlkMigDevState **pbmds, *bmds;BlockDriverState *bs; - +for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -if (bs->type ==BDRV_TYPE_HD ) { +if (bs->type ==BDRV_TYPE_HD ) { bmds =qemu_mallocz (sizeof (BlkMigDevState)); bmds->bs = bs; bmds->bulk_completed =0 ; bmds->total_sectors =bdrv_getlength (bs) >>SECTOR_BITS ; bmds->shared_base = block_mig_state->shared_base; - -if (bmds->shared_base) { -printf ("Start migration for %s with shared base image\n" , + +if (bmds->shared_base) { +printf ("Start migration for %s with shared base image\n" , bs->device_name); }else {printf ("Start full migration for %s\n" , bs->device_name); } - +/* insert at the end */ pbmds = &block_mig_state->bmds_first; -while (*pbmds !=NULL ) +while (*pbmds !=NULL ) { pbmds = &(*pbmds)->next; + } *pbmds = bmds; - +blk_mig_save_dev_info (f, bmds); - } - } - + } + block_mig_state->sectors_per_block =bdrv_get_sectors_per_chunk (); - -return ; }static int blk_mig_save_bulked_block (QEMUFile *f,int is_async)
@@ -299,14 +299,14 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)BlkMigDevState *bmds;for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { -if (bmds->bulk_completed ==0 ) { -if (is_async) { -if (mig_read_device_bulk (f, bmds) ==1 ) { +if (bmds->bulk_completed ==0 ) { +if (is_async) { +if (mig_read_device_bulk (f, bmds) ==1 ) {/* completed bulk section for this device */ bmds->bulk_completed =1 ; } }else { -if (mig_save_device_bulk (f,bmds) ==1 ) { +if (mig_save_device_bulk (f, bmds) ==1 ) {/* completed bulk section for this device */ bmds->bulk_completed =1 ; }
@@ -314,12 +314,11 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)return 1 ; } } - +/* we reached here means bulk is completed */ block_mig_state->bulk_completed =1 ; - +return 0 ; - }#define MAX_NUM_BLOCKS 4
@@ -330,143 +329,131 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)uint8_t buf[BLOCK_SIZE ];int64_t sector;int len; - -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { -for (sector =0 ; sector < bmds->cur_sector;) { - -if (bdrv_get_dirty (bmds->bs,sector)) { - -if (bdrv_read (bmds->bs, sector, buf, - block_mig_state->sectors_per_block) <0 ) { + +for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +for (sector =0 ; sector < bmds->cur_sector;) { +if (bdrv_get_dirty (bmds->bs, sector)) { +if (bdrv_read (bmds->bs, sector, buf, + block_mig_state->sectors_per_block) <0 ) { +/* FIXME: add error handling */ } - + +/* sector number and flags */ +qemu_put_be64 (f, (sector <<SECTOR_BITS ) + |BLK_MIG_FLAG_DEVICE_BLOCK ); +/* device name */ -qemu_put_be64 (f,(sector <<SECTOR_BITS ) - |BLK_MIG_FLAG_DEVICE_BLOCK ); - len =strlen (bmds->bs->device_name); -qemu_put_byte (f, len);qemu_put_buffer (f, (uint8_t *)bmds->bs->device_name, len); - -qemu_put_buffer (f, buf, - (block_mig_state->sectors_per_block * + +qemu_put_buffer (f, buf, + (block_mig_state->sectors_per_block *SECTOR_SIZE )); - -bdrv_reset_dirty (bmds->bs, sector, + +bdrv_reset_dirty (bmds->bs, sector, block_mig_state->sectors_per_block); - - sector += block_mig_state->sectors_per_block; - }else { -/* sector is clean */ - sector += block_mig_state->sectors_per_block; - } + } + sector += block_mig_state->sectors_per_block; } } - -return ; }static void flush_blks (QEMUFile * f) { -BlkMigBlock *blk, *tmp; - -dprintf ("%s Enter submitted %d read_done %d transfered\n" , __FUNCTION__, +BlkMigBlock *blk, *next; + +dprintf ("%s Enter submitted %d read_done %d transfered\n" , __FUNCTION__, submitted, read_done, transfered); - -for (blk = block_mig_state->first_blk; - blk !=NULL && !qemu_file_rate_limit (f); blk = tmp) { + +for (blk = block_mig_state->first_blk; + blk !=NULL && !qemu_file_rate_limit (f); + blk = next) {send_blk (f, blk); - - tmp = blk->next; + + next = blk->next;qemu_free (blk->buf);qemu_free (blk); - + block_mig_state->read_done--; block_mig_state->transferred++;assert (block_mig_state->read_done >=0 ); } block_mig_state->first_blk = blk; - -if (block_mig_state->first_blk ==NULL ) { + +if (block_mig_state->first_blk ==NULL ) { block_mig_state->last_blk =NULL ; } -dprintf ("%s Exit submitted %d read_done %d transferred%d\n" , __FUNCTION__, - block_mig_state->submitted, block_mig_state->read_done, +dprintf ("%s Exit submitted %d read_done %d transferred%d\n" , __FUNCTION__, + block_mig_state->submitted, block_mig_state->read_done, block_mig_state->transferred); - -return ; }static int is_stage2_completed (void ) {BlkMigDevState *bmds; - -if (block_mig_state->submitted >0 ) { + +if (block_mig_state->submitted >0 ) {return 0 ; } - +for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { -if (bmds->bulk_completed ==0 ) { +if (bmds->bulk_completed ==0 ) {return 0 ; } } - +return 1 ; }static int block_save_live (QEMUFile *f,int stage,void *opaque) { -int ret =1 ; - -dprintf ("Enter save live stage %d submitted %d transferred %d\n" , stage, +dprintf ("Enter save live stage %d submitted %d transferred %d\n" , stage, submitted, transferred); - -if (block_mig_state->blk_enable !=1 ) { + +if (block_mig_state->blk_enable !=1 ) {/* no need to migrate storage */ - -qemu_put_be64 (f,BLK_MIG_FLAG_EOS ); +qemu_put_be64 (f,BLK_MIG_FLAG_EOS );return 1 ; } - -if (stage ==1 ) { + +if (stage ==1 ) {init_blk_migration (f); - +/* start track dirty blocks */ set_dirty_tracking (1 ); - }flush_blks (f); - +/* control the rate of transfer */ -while ((block_mig_state->submitted + block_mig_state->read_done) * - (BLOCK_SIZE ) < - (qemu_file_get_rate_limit (f))) { - - ret =blk_mig_save_bulked_block (f,1 ); - -if (ret ==0 )/* no more bulk blocks for now*/ +while ((block_mig_state->submitted + + block_mig_state->read_done) *BLOCK_SIZE < +qemu_file_get_rate_limit (f)) { +if (blk_mig_save_bulked_block (f,1 ) ==0 ) { +/* no more bulk blocks for now */ break ; + } } - +flush_blks (f); - -if (stage ==3 ) { - -while (blk_mig_save_bulked_block (f,0 ) !=0 ); - + +if (stage ==3 ) { +while (blk_mig_save_bulked_block (f,0 ) !=0 ) { +/* empty */ + } +blk_mig_save_dirty_blocks (f); - +/* stop track dirty blocks */ -set_dirty_tracking (0 );; - -printf ("\nBlock migration completed\n" ); +set_dirty_tracking (0 ); + +printf ("\nBlock migration completed\n" ); } - -qemu_put_be64 (f,BLK_MIG_FLAG_EOS ); - + +qemu_put_be64 (f,BLK_MIG_FLAG_EOS ); +return ((stage ==2 ) &&is_stage2_completed ()); }
@@ -477,43 +464,39 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)int64_t addr;BlockDriverState *bs;uint8_t *buf; - + block_mig_state->sectors_per_block =bdrv_get_sectors_per_chunk (); buf =qemu_malloc (BLOCK_SIZE ); - +do { - addr =qemu_get_be64 (f); - + flags = addr & ~SECTOR_MASK ; addr &=SECTOR_MASK ; - -if (flags &BLK_MIG_FLAG_DEVICE_BLOCK ) { - + +if (flags &BLK_MIG_FLAG_DEVICE_BLOCK ) {/* get device name */ len =qemu_get_byte (f); - +qemu_get_buffer (f, (uint8_t *)device_name, len); device_name[len] ='\0' ; - + bs =bdrv_find (device_name); - -qemu_get_buffer (f, buf, -BLOCK_SIZE ); -if (bs !=NULL ) { - -bdrv_write (bs, (addr >>SECTOR_BITS ), + +qemu_get_buffer (f, buf,BLOCK_SIZE ); +if (bs !=NULL ) { +bdrv_write (bs, (addr >>SECTOR_BITS ), buf, block_mig_state->sectors_per_block); }else {printf ("Error unknown block device %s\n" , device_name); +/* FIXME: add error handling */ } - }else if (flags &BLK_MIG_FLAG_EOS ) { - - }else { + }else if (!(flags &BLK_MIG_FLAG_EOS )) {printf ("Unknown flags\n" ); +/* FIXME: add error handling */ } - }while (!(flags &BLK_MIG_FLAG_EOS )); - + }while (!(flags &BLK_MIG_FLAG_EOS )); +qemu_free (buf);return 0 ;
@@ -525,33 +508,28 @@ static void block_set_params(int blk_enable, int shared_base, void *opaque) block_mig_state->blk_enable = blk_enable; block_mig_state->shared_base = shared_base; - +/* shared base means that blk_enable = 1 */ block_mig_state->blk_enable |= shared_base; - -return ; }void blk_mig_info (void ) {BlockDriverState *bs; - +for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {printf ("Device %s\n" , bs->device_name); -if (bs->type ==BDRV_TYPE_HD ) { -printf ("device %s format %s\n" , +if (bs->type ==BDRV_TYPE_HD ) { +printf ("device %s format %s\n" , bs->device_name, bs->drv->format_name); } } }void blk_mig_init (void ) -{ - +{ block_mig_state =qemu_mallocz (sizeof (BlkMigState)); - -register_savevm_live ("block" ,0 ,1 , block_set_params, block_save_live, -NULL , block_load, block_mig_state); - +register_savevm_live ("block" ,0 ,1 , block_set_params, block_save_live, +NULL , block_load, block_mig_state); }
block-migration.h +0 -10
@@ -14,16 +14,6 @@#ifndef BLOCK_MIGRATION_H #define BLOCK_MIGRATION_H -typedef struct BlkMigDevState { -BlockDriverState *bs; -int bulk_completed; -int shared_base; -struct BlkMigDevState *next; -int64_t cur_sector; -int64_t total_sectors; -int64_t dirty; -}BlkMigDevState ; -void blk_mig_init (void );void blk_mig_info (void );#endif /* BLOCK_MIGRATION_H */
block.c +26 -23
112 lines not on this page · raw message
block.h +2 -2
@@ -190,7 +190,7 @@ int bdrv_load_vmstate(BlockDriverState *bs, uint8_t *buf,void bdrv_set_dirty_tracking (BlockDriverState *bs,int enable);int bdrv_get_dirty (BlockDriverState *bs,int64_t sector); -void bdrv_reset_dirty (BlockDriverState *bs,int64_t cur_sector, -int nr_sectors); +void bdrv_reset_dirty (BlockDriverState *bs,int64_t cur_sector, +int nr_sectors);int bdrv_get_sectors_per_chunk (void );#endif
PATCH 4/23 block migration: Rework constants API
In reply to Jan KiszkaInstead of duplicating the definition of constants or introducing trivial retrieval functions move the SECTOR constants into the public block API. This also obsoletes sector_per_block in BlkMigState.
---
block-migration.c | 63 +++++++++++++++++++++++------------------------------ block.c | 60 +++++++++++++++++++++----------------------------- block.h | 7 +++++- 3 files changed, 58 insertions(+), 72 deletions(-)
block-migration.c +27 -36
@@ -17,11 +17,7 @@#include "block-migration.h" #include <assert.h> -#define SECTOR_BITS 9 -#define SECTOR_SIZE (1 << SECTOR_BITS) -#define SECTOR_MASK ~(SECTOR_SIZE - 1); - -#define BLOCK_SIZE (block_mig_state->sectors_per_block << SECTOR_BITS) +#define BLOCK_SIZE (BDRV_SECTORS_PER_DIRTY_CHUNK << BDRV_SECTOR_BITS)#define BLK_MIG_FLAG_DEVICE_BLOCK 0x01#define BLK_MIG_FLAG_EOS 0x02
@@ -69,7 +65,6 @@ typedef struct BlkMigState {int no_dirty;QEMUFile *load_file;BlkMigDevState *bmds_first; -int sectors_per_block;BlkMigBlock *first_blk;BlkMigBlock *last_blk;int submitted;
@@ -111,7 +106,7 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) blk->buf =qemu_malloc (BLOCK_SIZE ); cur_sector = bms->cur_sector; - total_sectors =bdrv_getlength (bs) >>SECTOR_BITS ; + total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ;if (bms->shared_base) { while (cur_sector < bms->total_sectors &&
@@ -132,15 +127,15 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms)printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); block_mig_state->print_completion += - (block_mig_state->sectors_per_block *10000 ); + (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); } -/* we going to transfder BLOCK_SIZE any way even if it is not allocated */ - nr_sectors = block_mig_state->sectors_per_block; +/* we are going to transfer a full block even if it is not allocated */ + nr_sectors =BDRV_SECTORS_PER_DIRTY_CHUNK ; - cur_sector &= ~((int64_t )block_mig_state->sectors_per_block -1 ); + cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 ); -if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { +if (total_sectors - cur_sector <BDRV_SECTORS_PER_DIRTY_CHUNK ) { nr_sectors = (total_sectors - cur_sector); }
@@ -150,7 +145,7 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) blk->next =NULL ; blk->iov.iov_base = blk->buf; - blk->iov.iov_len = nr_sectors *SECTOR_SIZE ; + blk->iov.iov_len = nr_sectors *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&blk->qiov, &blk->iov,1 ); blk->aiocb = bdrv_aio_readv(bs, cur_sector, &blk->qiov,
@@ -198,15 +193,15 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); block_mig_state->print_completion += - (block_mig_state->sectors_per_block *10000 ); + (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); } - cur_sector &= ~((int64_t )block_mig_state->sectors_per_block -1 ); + cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 ); -/* we going to transfer BLOCK_SIZE any way even if it is not allocated */ - nr_sectors = block_mig_state->sectors_per_block; +/* we are going to transfer a full block even if it is not allocated */ + nr_sectors =BDRV_SECTORS_PER_DIRTY_CHUNK ; -if (total_sectors - cur_sector < block_mig_state->sectors_per_block) { +if (total_sectors - cur_sector <BDRV_SECTORS_PER_DIRTY_CHUNK ) { nr_sectors = (total_sectors - cur_sector); }
@@ -217,7 +212,8 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)bdrv_reset_dirty (bs, cur_sector, nr_sectors);/* sector number and flags */ -qemu_put_be64 (f, (cur_sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); +qemu_put_be64 (f, (cur_sector <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_DEVICE_BLOCK );/* device name */ len =strlen (bs->device_name);
@@ -226,7 +222,7 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)qemu_put_buffer (f, tmp_buf,BLOCK_SIZE ); - bmds->cur_sector = cur_sector + block_mig_state->sectors_per_block; + bmds->cur_sector = cur_sector +BDRV_SECTORS_PER_DIRTY_CHUNK ;qemu_free (tmp_buf);
@@ -238,7 +234,8 @@ static void send_blk(QEMUFile *f, BlkMigBlock * blk)int len;/* sector number and flags */ -qemu_put_be64 (f, (blk->sector <<SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK ); +qemu_put_be64 (f, (blk->sector <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_DEVICE_BLOCK );/* device name */ len =strlen (blk->bmds->bs->device_name);
@@ -270,7 +267,7 @@ static void init_blk_migration(QEMUFile *f) bmds =qemu_mallocz (sizeof (BlkMigDevState)); bmds->bs = bs; bmds->bulk_completed =0 ; - bmds->total_sectors =bdrv_getlength (bs) >>SECTOR_BITS ; + bmds->total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; bmds->shared_base = block_mig_state->shared_base;if (bmds->shared_base) {
@@ -290,8 +287,6 @@ static void init_blk_migration(QEMUFile *f)blk_mig_save_dev_info (f, bmds); } } - - block_mig_state->sectors_per_block =bdrv_get_sectors_per_chunk (); }static int blk_mig_save_bulked_block (QEMUFile *f,int is_async)
@@ -334,12 +329,12 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)for (sector =0 ; sector < bmds->cur_sector;) {if (bdrv_get_dirty (bmds->bs, sector)) {if (bdrv_read (bmds->bs, sector, buf, - block_mig_state->sectors_per_block) <0 ) { +BDRV_SECTORS_PER_DIRTY_CHUNK ) <0 ) {/* FIXME: add error handling */ }/* sector number and flags */ -qemu_put_be64 (f, (sector <<SECTOR_BITS ) +qemu_put_be64 (f, (sector <<BDRV_SECTOR_BITS ) |BLK_MIG_FLAG_DEVICE_BLOCK );/* device name */
@@ -347,14 +342,12 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)qemu_put_byte (f, len);qemu_put_buffer (f, (uint8_t *)bmds->bs->device_name, len); -qemu_put_buffer (f, buf, - (block_mig_state->sectors_per_block * -SECTOR_SIZE )); +qemu_put_buffer (f, buf,BLOCK_SIZE );bdrv_reset_dirty (bmds->bs, sector, - block_mig_state->sectors_per_block); +BDRV_SECTORS_PER_DIRTY_CHUNK ); } - sector += block_mig_state->sectors_per_block; + sector +=BDRV_SECTORS_PER_DIRTY_CHUNK ; } } }
@@ -465,14 +458,13 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)BlockDriverState *bs;uint8_t *buf; - block_mig_state->sectors_per_block =bdrv_get_sectors_per_chunk (); buf =qemu_malloc (BLOCK_SIZE ); do { addr = qemu_get_be64(f); - flags = addr & ~SECTOR_MASK; - addr &= SECTOR_MASK; + flags = addr & ~BDRV_SECTOR_MASK; + addr >>= BDRV_SECTOR_BITS; if (flags & BLK_MIG_FLAG_DEVICE_BLOCK) { /* get device name */
@@ -485,8 +477,7 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)qemu_get_buffer (f, buf,BLOCK_SIZE );if (bs !=NULL ) { -bdrv_write (bs, (addr >>SECTOR_BITS ), - buf, block_mig_state->sectors_per_block); +bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK ); }else {printf ("Error unknown block device %s\n" , device_name);/* FIXME: add error handling */
block.c +25 -35
183 lines not on this page · raw message
block.h +6 -1
@@ -41,6 +41,10 @@ typedef struct QEMUSnapshotInfo {#define BDRV_O_CACHE_MASK (BDRV_O_NOCACHE | BDRV_O_CACHE_WB) +#define BDRV_SECTOR_BITS 9 +#define BDRV_SECTOR_SIZE (1 << BDRV_SECTOR_BITS) +#define BDRV_SECTOR_MASK ~(BDRV_SECTOR_SIZE - 1); +void bdrv_info (Monitor *mon);void bdrv_info_stats (Monitor *mon);
@@ -188,9 +192,10 @@ int bdrv_save_vmstate(BlockDriverState *bs, const uint8_t *buf,int bdrv_load_vmstate (BlockDriverState *bs,uint8_t *buf,int64_t pos,int size); +#define BDRV_SECTORS_PER_DIRTY_CHUNK 8 +void bdrv_set_dirty_tracking (BlockDriverState *bs,int enable);int bdrv_get_dirty (BlockDriverState *bs,int64_t sector);void bdrv_reset_dirty (BlockDriverState *bs,int64_t cur_sector,int nr_sectors); -int bdrv_get_sectors_per_chunk (void );#endif
PATCH 5/23 block migration: Cleanup dirty tracking code
In reply to Jan KiszkaThis switches the dirty bitmap to a true bitmap, reducing its footprint (specifically in caches). It moreover fixes off-by-one bugs in set_dirty_bitmap (nb_sectors+1 were marked) and bdrv_get_dirty (limit check allowed one sector behind end of drive). And is drops redundant dirty_tracking field from BlockDriverState.
---
block.c | 44 ++++++++++++++++++++++++-------------------- block_int.h | 3 +-- 2 files changed, 25 insertions(+), 22 deletions(-)
block.c +24 -20
@@ -642,12 +642,21 @@ static void set_dirty_bitmap(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int dirty) { int64_t start, end; +unsigned long val, idx, bit; start = sector_num /BDRV_SECTORS_PER_DIRTY_CHUNK ; - end = (sector_num + nb_sectors) /BDRV_SECTORS_PER_DIRTY_CHUNK ; + end = (sector_num + nb_sectors -1 ) /BDRV_SECTORS_PER_DIRTY_CHUNK ;for (; start <= end; start++) { - bs->dirty_bitmap[start] = dirty; + idx = start / (sizeof (unsigned long ) *8 ); + bit = start % (sizeof (unsigned long ) *8 ); + val = bs->dirty_bitmap[idx]; +if (dirty) { + val |=1 << bit; + }else { + val &= ~(1 << bit); + } + bs->dirty_bitmap[idx] = val; } }
@@ -668,7 +677,7 @@ int bdrv_write(BlockDriverState *bs, int64_t sector_num,if (bdrv_check_request (bs, sector_num, nb_sectors))return -EIO ; -if (bs->dirty_tracking) { +if (bs->dirty_bitmap) {set_dirty_bitmap (bs, sector_num, nb_sectors,1 ); }
@@ -1218,7 +1227,7 @@ int bdrv_write_compressed(BlockDriverState *bs, int64_t sector_num,if (bdrv_check_request (bs, sector_num, nb_sectors))return -EIO ; -if (bs->dirty_tracking) { +if (bs->dirty_bitmap) {set_dirty_bitmap (bs, sector_num, nb_sectors,1 ); }
@@ -1419,7 +1428,7 @@ BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,if (bdrv_check_request (bs, sector_num, nb_sectors))return NULL ; -if (bs->dirty_tracking) { +if (bs->dirty_bitmap) {set_dirty_bitmap (bs, sector_num, nb_sectors,1 ); }
@@ -1965,23 +1974,17 @@ void bdrv_set_dirty_tracking(BlockDriverState *bs, int enable)int64_t bitmap_size;if (enable) { -if (bs->dirty_tracking ==0 ) { -int64_t i; -uint8_t test; - - bitmap_size = (bdrv_getlength (bs) >>BDRV_SECTOR_BITS ); - bitmap_size /=BDRV_SECTORS_PER_DIRTY_CHUNK ; - bitmap_size++; +if (!bs->dirty_bitmap) { + bitmap_size = (bdrv_getlength (bs) >>BDRV_SECTOR_BITS ) + +BDRV_SECTORS_PER_DIRTY_CHUNK *8 -1 ; + bitmap_size /=BDRV_SECTORS_PER_DIRTY_CHUNK *8 ; bs->dirty_bitmap =qemu_mallocz (bitmap_size); - - bs->dirty_tracking = enable; -for (i =0 ; i < bitmap_size; i++) test = bs->dirty_bitmap[i]; } }else { -if (bs->dirty_tracking !=0 ) { +if (bs->dirty_bitmap) {qemu_free (bs->dirty_bitmap); - bs->dirty_tracking = enable; + bs->dirty_bitmap =NULL ; } } }
@@ -1990,9 +1993,10 @@ int bdrv_get_dirty(BlockDriverState *bs, int64_t sector) {int64_t chunk = sector / (int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK ; -if (bs->dirty_bitmap !=NULL && - (sector <<BDRV_SECTOR_BITS ) <=bdrv_getlength (bs)) { -return bs->dirty_bitmap[chunk]; +if (bs->dirty_bitmap && + (sector <<BDRV_SECTOR_BITS ) <bdrv_getlength (bs)) { +return bs->dirty_bitmap[chunk / (sizeof (unsigned long ) *8 )] & + (1 << (chunk % (sizeof (unsigned long ) *8 ))); }else {return 0 ; }
block_int.h +1 -2
@@ -168,8 +168,7 @@ struct BlockDriverState {int cyls, heads, secs, translation;int type;char device_name[32 ]; -int dirty_tracking; -uint8_t *dirty_bitmap; +unsigned long *dirty_bitmap;BlockDriverState *next;void *private; };
PATCH 6/23 block migration: Avoid large stack buffer
In reply to Jan KiszkaMove a potentially large buffer from stack to heap.
---
block-migration.c | 14 +++++++++----- 1 files changed, 9 insertions(+), 5 deletions(-)
block-migration.c +9 -5
@@ -321,10 +321,12 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)static void blk_mig_save_dirty_blocks (QEMUFile *f) {BlkMigDevState *bmds; -uint8_t buf[BLOCK_SIZE ]; +uint8_t *buf;int64_t sector;int len; + buf =qemu_malloc (BLOCK_SIZE ); +for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) {for (sector =0 ; sector < bmds->cur_sector;) {if (bdrv_get_dirty (bmds->bs, sector)) {
@@ -350,6 +352,8 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f) sector +=BDRV_SECTORS_PER_DIRTY_CHUNK ; } } + +qemu_free (buf); }static void flush_blks (QEMUFile * f)
@@ -458,8 +462,6 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)BlockDriverState *bs;uint8_t *buf; - buf =qemu_malloc (BLOCK_SIZE ); - do { addr = qemu_get_be64(f);
@@ -475,6 +477,8 @@ static int block_load(QEMUFile *f, void *opaque, int version_id) bs =bdrv_find (device_name); + buf =qemu_malloc (BLOCK_SIZE ); +qemu_get_buffer (f, buf,BLOCK_SIZE );if (bs !=NULL ) {bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK );
@@ -482,14 +486,14 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)printf ("Error unknown block device %s\n" , device_name);/* FIXME: add error handling */ } + +qemu_free (buf); }else if (!(flags &BLK_MIG_FLAG_EOS )) {printf ("Unknown flags\n" );/* FIXME: add error handling */ } }while (!(flags &BLK_MIG_FLAG_EOS )); -qemu_free (buf); -return 0 ; }
PATCH 7/23 block migration: Avoid indirection of block_mig_state
In reply to Jan KiszkaNo need to push block_mig_state to the heap and, thus, establish an indirection.
---
block-migration.c | 91 ++++++++++++++++++++++++++--------------------------- 1 files changed, 44 insertions(+), 47 deletions(-)
block-migration.c +44 -47
@@ -73,7 +73,7 @@ typedef struct BlkMigState {int64_t print_completion; } BlkMigState; -static BlkMigState *block_mig_state =NULL ; +static BlkMigState block_mig_state;static void blk_mig_read_cb (void *opaque,int ret) {
@@ -82,17 +82,17 @@ static void blk_mig_read_cb(void *opaque, int ret) blk->ret = ret;/* insert at the end */ -if (block_mig_state->last_blk ==NULL ) { - block_mig_state->first_blk = blk; - block_mig_state->last_blk = blk; +if (block_mig_state.last_blk ==NULL ) { + block_mig_state.first_blk = blk; + block_mig_state.last_blk = blk; }else { - block_mig_state->last_blk->next = blk; - block_mig_state->last_blk = blk; + block_mig_state.last_blk->next = blk; + block_mig_state.last_blk = blk; } - block_mig_state->submitted--; - block_mig_state->read_done++; -assert (block_mig_state->submitted >=0 ); + block_mig_state.submitted--; + block_mig_state.read_done++; +assert (block_mig_state.submitted >=0 ); }static int mig_read_device_bulk (QEMUFile *f,BlkMigDevState *bms)
@@ -123,10 +123,10 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms)return 1 ; } -if (cur_sector >= block_mig_state->print_completion) { +if (cur_sector >= block_mig_state.print_completion) {printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); - block_mig_state->print_completion += + block_mig_state.print_completion += (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); }
@@ -159,7 +159,7 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) }bdrv_reset_dirty (bms->bs, cur_sector, nr_sectors); - block_mig_state->submitted++; + block_mig_state.submitted++;return (bms->cur_sector >= total_sectors); }
@@ -189,10 +189,10 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)return 1 ; } -if (cur_sector >= block_mig_state->print_completion) { +if (cur_sector >= block_mig_state.print_completion) {printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors);fflush (stdout); - block_mig_state->print_completion += + block_mig_state.print_completion += (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); }
@@ -252,7 +252,7 @@ static void blk_mig_save_dev_info(QEMUFile *f, BlkMigDevState *bmds)static void set_dirty_tracking (int enable) {BlkMigDevState *bmds; -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) {bdrv_set_dirty_tracking (bmds->bs, enable); } }
@@ -268,7 +268,7 @@ static void init_blk_migration(QEMUFile *f) bmds->bs = bs; bmds->bulk_completed =0 ; bmds->total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; - bmds->shared_base = block_mig_state->shared_base; + bmds->shared_base = block_mig_state.shared_base;if (bmds->shared_base) { printf("Start migration for %s with shared base image\n",
@@ -278,7 +278,7 @@ static void init_blk_migration(QEMUFile *f) }/* insert at the end */ - pbmds = &block_mig_state->bmds_first; + pbmds = &block_mig_state.bmds_first;while (*pbmds !=NULL ) { pbmds = &(*pbmds)->next; }
@@ -293,7 +293,7 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) {BlkMigDevState *bmds; -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) {if (bmds->bulk_completed ==0 ) {if (is_async) {if (mig_read_device_bulk (f, bmds) ==1 ) {
@@ -311,7 +311,7 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) }/* we reached here means bulk is completed */ - block_mig_state->bulk_completed =1 ; + block_mig_state.bulk_completed =1 ;return 0 ; }
@@ -327,7 +327,7 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f) buf =qemu_malloc (BLOCK_SIZE ); -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) {for (sector =0 ; sector < bmds->cur_sector;) {if (bdrv_get_dirty (bmds->bs, sector)) { if (bdrv_read(bmds->bs, sector, buf,
@@ -360,10 +360,11 @@ static void flush_blks(QEMUFile* f) {BlkMigBlock *blk, *next; -dprintf ("%s Enter submitted %d read_done %d transfered\n" , __FUNCTION__, - submitted, read_done, transfered); +dprintf ("%s Enter submitted %d read_done %d transferred %d\n" , + __FUNCTION__, block_mig_state.submitted, block_mig_state.read_done, + block_mig_state.transferred); -for (blk = block_mig_state->first_blk; +for (blk = block_mig_state.first_blk; blk !=NULL && !qemu_file_rate_limit (f); blk = next) {send_blk (f, blk);
@@ -372,30 +373,30 @@ static void flush_blks(QEMUFile* f)qemu_free (blk->buf);qemu_free (blk); - block_mig_state->read_done--; - block_mig_state->transferred++; -assert (block_mig_state->read_done >=0 ); + block_mig_state.read_done--; + block_mig_state.transferred++; +assert (block_mig_state.read_done >=0 ); } - block_mig_state->first_blk = blk; + block_mig_state.first_blk = blk; -if (block_mig_state->first_blk ==NULL ) { - block_mig_state->last_blk =NULL ; +if (block_mig_state.first_blk ==NULL ) { + block_mig_state.last_blk =NULL ; } -dprintf ("%s Exit submitted %d read_done %d transferred%d\n" , __FUNCTION__, - block_mig_state->submitted, block_mig_state->read_done, - block_mig_state->transferred); +dprintf ("%s Exit submitted %d read_done %d transferred %d\n" , __FUNCTION__, + block_mig_state.submitted, block_mig_state.read_done, + block_mig_state.transferred); }static int is_stage2_completed (void ) {BlkMigDevState *bmds; -if (block_mig_state->submitted >0 ) { +if (block_mig_state.submitted >0 ) {return 0 ; } -for (bmds = block_mig_state->bmds_first; bmds !=NULL ; bmds = bmds->next) { +for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) {if (bmds->bulk_completed ==0 ) {return 0 ; }
@@ -406,10 +407,10 @@ static int is_stage2_completed(void)static int block_save_live (QEMUFile *f,int stage,void *opaque) { -dprintf ("Enter save live stage %d submitted %d transferred %d\n" , stage, - submitted, transferred); +dprintf ("Enter save live stage %d submitted %d transferred %d\n" , + stage, block_mig_state.submitted, block_mig_state.transferred); -if (block_mig_state->blk_enable !=1 ) { +if (block_mig_state.blk_enable !=1 ) {/* no need to migrate storage */ qemu_put_be64 (f,BLK_MIG_FLAG_EOS );return 1 ;
@@ -425,8 +426,8 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)flush_blks (f);/* control the rate of transfer */ -while ((block_mig_state->submitted + - block_mig_state->read_done) *BLOCK_SIZE < +while ((block_mig_state.submitted + + block_mig_state.read_done) *BLOCK_SIZE <qemu_file_get_rate_limit (f)) {if (blk_mig_save_bulked_block (f,1 ) ==0 ) {/* no more bulk blocks for now */
@@ -499,13 +500,11 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)static void block_set_params (int blk_enable,int shared_base,void *opaque) { -assert (opaque == block_mig_state); - - block_mig_state->blk_enable = blk_enable; - block_mig_state->shared_base = shared_base; + block_mig_state.blk_enable = blk_enable; + block_mig_state.shared_base = shared_base;/* shared base means that blk_enable = 1 */ - block_mig_state->blk_enable |= shared_base; + block_mig_state.blk_enable |= shared_base; }void blk_mig_info (void )
@@ -523,8 +522,6 @@ void blk_mig_info(void)void blk_mig_init (void ) { - block_mig_state =qemu_mallocz (sizeof (BlkMigState)); -register_savevm_live ("block" ,0 ,1 , block_set_params, block_save_live, -NULL , block_load, block_mig_state); +NULL , block_load, &block_mig_state); }
PATCH 8/23 block migration: Drop dead code
In reply to Jan Kiszka---
block-migration.c | 24 ------------------------ block-migration.h | 2 +- 2 files changed, 1 insertions(+), 25 deletions(-)
block-migration.c +0 -24
@@ -59,11 +59,8 @@ typedef struct BlkMigBlock { } BlkMigBlock; typedef struct BlkMigState { - int bulk_completed; int blk_enable; int shared_base; - int no_dirty; - QEMUFile *load_file; BlkMigDevState *bmds_first; BlkMigBlock *first_blk; BlkMigBlock *last_blk;
@@ -245,10 +242,6 @@ static void send_blk(QEMUFile *f, BlkMigBlock * blk)qemu_put_buffer (f, blk->buf,BLOCK_SIZE ); } -static void blk_mig_save_dev_info (QEMUFile *f,BlkMigDevState *bmds) -{ -} -static void set_dirty_tracking (int enable) {BlkMigDevState *bmds;
@@ -283,8 +276,6 @@ static void init_blk_migration(QEMUFile *f) pbmds = &(*pbmds)->next; } *pbmds = bmds; - -blk_mig_save_dev_info (f, bmds); } } }
@@ -311,8 +302,6 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) }/* we reached here means bulk is completed */ - block_mig_state.bulk_completed =1 ; -return 0 ; }
@@ -507,19 +496,6 @@ static void block_set_params(int blk_enable, int shared_base, void *opaque) block_mig_state.blk_enable |= shared_base; } -void blk_mig_info (void ) -{ -BlockDriverState *bs; - -for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -printf ("Device %s\n" , bs->device_name); -if (bs->type ==BDRV_TYPE_HD ) { -printf ("device %s format %s\n" , - bs->device_name, bs->drv->format_name); - } - } -} -void blk_mig_init (void ) { register_savevm_live("block", 0, 1, block_set_params, block_save_live,
block-migration.h +1 -1
@@ -15,5 +15,5 @@#define BLOCK_MIGRATION_H void blk_mig_init (void ); -void blk_mig_info (void ); +#endif /* BLOCK_MIGRATION_H */
PATCH 9/23 Import a simple queue implementation from NetBSD
In reply to Jan KiszkaFrom: Pierre Riteau <[email protected]>
---
qemu-queue.h | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 105 insertions(+), 4 deletions(-)
qemu-queue.h +105 -4
@@ -1,8 +1,9 @@ -/* $NetBSD: queue.h,v 1.45.14.1 2007/07/18 20:13:24 liamjfoy Exp $ */ +/* $NetBSD: queue.h,v 1.52 2009/04/20 09:56:08 mschuett Exp $ */ /* * Qemu version: Copy from netbsd, removed debug code, removed some of -* the implementations. Left in lists, tail queues and circular queues. +* the implementations. Left in lists, simple queues, tail queues and +* circular queues. */ /*
@@ -40,8 +41,8 @@#define QEMU_SYS_QUEUE_H_ /* -* This file defines three types of data structures: -* lists, tail queues, and circular queues. +* This file defines four types of data structures: +* lists, simple queues, tail queues, and circular queues. * * A list is headed by a single forward pointer (or an array of forward * pointers for a hash table header). The elements are doubly linked
@@ -50,6 +51,13 @@* or after an existing element or at the head of the list. A list * may only be traversed in the forward direction. * +* A simple queue is headed by a pair of pointers, one the head of the +* list and the other to the tail of the list. The elements are singly +* linked to save space, so elements can only be removed from the +* head of the list. New elements can be added to the list after +* an existing element, at the head of the list, or at the end of the +* list. A simple queue may only be traversed in the forward direction. +* * A tail queue is headed by a pair of pointers, one to the head of the * list and the other to the tail of the list. The elements are doubly * linked so that an arbitrary element can be removed without a need to
@@ -140,6 +148,99 @@ struct { \/* +* Simple queue definitions. +*/ +#define QSIMPLEQ_HEAD (name, type) \ +struct name { \ + struct type *sqh_first;/* first element */ \ +struct type **sqh_last;/* addr of last next element */ \ +} + +#define QSIMPLEQ_HEAD_INITIALIZER (head) \ + { NULL, &(head).sqh_first } + +#define QSIMPLEQ_ENTRY(type) \ +struct { \ + struct type *sqe_next; /* next element */ \ +} + +/* +* Simple queue functions. +*/ +#define QSIMPLEQ_INIT(head) do { \ + (head)->sqh_first = NULL; \ + (head)->sqh_last = &(head)->sqh_first; \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_INSERT_HEAD(head, elm, field) do { \ + if (((elm)->field.sqe_next = (head)->sqh_first) == NULL) \ + (head)->sqh_last = &(elm)->field.sqe_next; \ + (head)->sqh_first = (elm); \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_INSERT_TAIL(head, elm, field) do { \ + (elm)->field.sqe_next = NULL; \ + *(head)->sqh_last = (elm); \ + (head)->sqh_last = &(elm)->field.sqe_next; \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_INSERT_AFTER(head, listelm, elm, field) do { \ + if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL) \ + (head)->sqh_last = &(elm)->field.sqe_next; \ + (listelm)->field.sqe_next = (elm); \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_REMOVE_HEAD(head, field) do { \ + if (((head)->sqh_first = (head)->sqh_first->field.sqe_next) == NULL)\ + (head)->sqh_last = &(head)->sqh_first; \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_REMOVE(head, elm, type, field) do { \ + if ((head)->sqh_first == (elm)) { \ + QSIMPLEQ_REMOVE_HEAD((head), field); \ + } else { \ + struct type *curelm = (head)->sqh_first; \ + while (curelm->field.sqe_next != (elm)) \ + curelm = curelm->field.sqe_next; \ + if ((curelm->field.sqe_next = \ + curelm->field.sqe_next->field.sqe_next) == NULL) \ + (head)->sqh_last = &(curelm)->field.sqe_next; \ + } \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_FOREACH (var, head, field) \ + for ((var) = ((head)->sqh_first); \ + (var); \ + (var) = ((var)->field.sqe_next)) + +#define QSIMPLEQ_FOREACH_SAFE (var, head, field, next) \ + for ((var) = ((head)->sqh_first); \ + (var) && ((next = ((var)->field.sqe_next)), 1); \ + (var) = (next)) + +#define QSIMPLEQ_CONCAT(head1, head2) do { \ + if (!QSIMPLEQ_EMPTY((head2))) { \ + *(head1)->sqh_last = (head2)->sqh_first; \ + (head1)->sqh_last = (head2)->sqh_last; \ + QSIMPLEQ_INIT((head2)); \ + } \ +} while (/*CONSTCOND*/0) + +#define QSIMPLEQ_LAST (head, type, field) \ + (QSIMPLEQ_EMPTY((head)) ? \ + NULL : \ + ((struct type *)(void *) \ + ((char *)((head)->sqh_last) - offsetof(struct type, field)))) + +/* +* Simple queue access methods. +*/ +#define QSIMPLEQ_EMPTY (head) ((head)->sqh_first == NULL) +#define QSIMPLEQ_FIRST (head) ((head)->sqh_first) +#define QSIMPLEQ_NEXT (elm, field) ((elm)->field.sqe_next) + + +/* * Tail queue definitions. */ #define Q_TAILQ_HEAD (name, type, qual) \
PATCH 10/23 block migration: Switch device and block lists to QSIMPLEQ
In reply to Jan KiszkaBased on the original patch by Pierre Riteau.
---
block-migration.c | 57 ++++++++++++++++++++--------------------------------- 1 files changed, 22 insertions(+), 35 deletions(-)
block-migration.c +22 -35
@@ -14,6 +14,7 @@#include "qemu-common.h" #include "block_int.h" #include "hw/hw.h" +#include "qemu-queue.h" #include "block-migration.h" #include <assert.h>
@@ -41,10 +42,10 @@ typedef struct BlkMigDevState {BlockDriverState *bs;int bulk_completed;int shared_base; -struct BlkMigDevState *next;int64_t cur_sector;int64_t total_sectors;int64_t dirty; +QSIMPLEQ_ENTRY (BlkMigDevState) entry; } BlkMigDevState; typedef struct BlkMigBlock {
@@ -55,15 +56,14 @@ typedef struct BlkMigBlock {QEMUIOVector qiov;BlockDriverAIOCB *aiocb;int ret; -struct BlkMigBlock *next; +QSIMPLEQ_ENTRY (BlkMigBlock) entry; } BlkMigBlock; typedef struct BlkMigState { int blk_enable; int shared_base; - BlkMigDevState *bmds_first; - BlkMigBlock *first_blk; - BlkMigBlock *last_blk; + QSIMPLEQ_HEAD(bmds_list, BlkMigDevState) bmds_list; + QSIMPLEQ_HEAD(blk_list, BlkMigBlock) blk_list; int submitted; int read_done; int transferred;
@@ -78,14 +78,7 @@ static void blk_mig_read_cb(void *opaque, int ret) blk->ret = ret; -/* insert at the end */ -if (block_mig_state.last_blk ==NULL ) { - block_mig_state.first_blk = blk; - block_mig_state.last_blk = blk; - }else { - block_mig_state.last_blk->next = blk; - block_mig_state.last_blk = blk; - } +QSIMPLEQ_INSERT_TAIL (&block_mig_state.blk_list, blk, entry); block_mig_state.submitted--; block_mig_state.read_done++;
@@ -139,7 +132,6 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) bms->cur_sector = cur_sector + nr_sectors; blk->sector = cur_sector; blk->bmds = bms; - blk->next =NULL ; blk->iov.iov_base = blk->buf; blk->iov.iov_len = nr_sectors *BDRV_SECTOR_SIZE ;
@@ -245,14 +237,15 @@ static void send_blk(QEMUFile *f, BlkMigBlock * blk)static void set_dirty_tracking (int enable) {BlkMigDevState *bmds; -for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) { + + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {bdrv_set_dirty_tracking (bmds->bs, enable); } }static void init_blk_migration (QEMUFile *f) { -BlkMigDevState **pbmds, *bmds; +BlkMigDevState *bmds;BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {
@@ -270,12 +263,7 @@ static void init_blk_migration(QEMUFile *f)printf ("Start full migration for %s\n" , bs->device_name); } -/* insert at the end */ - pbmds = &block_mig_state.bmds_first; -while (*pbmds !=NULL ) { - pbmds = &(*pbmds)->next; - } - *pbmds = bmds; +QSIMPLEQ_INSERT_TAIL (&block_mig_state.bmds_list, bmds, entry); } } }
@@ -284,7 +272,7 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) {BlkMigDevState *bmds; -for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) { + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {if (bmds->bulk_completed ==0 ) {if (is_async) {if (mig_read_device_bulk (f, bmds) ==1 ) {
@@ -316,7 +304,7 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f) buf =qemu_malloc (BLOCK_SIZE ); -for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) { + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {for (sector =0 ; sector < bmds->cur_sector;) {if (bdrv_get_dirty (bmds->bs, sector)) { if (bdrv_read(bmds->bs, sector, buf,
@@ -347,18 +335,19 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)static void flush_blks (QEMUFile * f) { -BlkMigBlock *blk, *next; +BlkMigBlock *blk;dprintf ("%s Enter submitted %d read_done %d transferred %d\n" , __FUNCTION__, block_mig_state.submitted, block_mig_state.read_done, block_mig_state.transferred); -for (blk = block_mig_state.first_blk; - blk !=NULL && !qemu_file_rate_limit (f); - blk = next) { +while ((blk =QSIMPLEQ_FIRST (&block_mig_state.blk_list)) !=NULL ) { +if (qemu_file_rate_limit (f)) { +break ; + }send_blk (f, blk); - next = blk->next; +QSIMPLEQ_REMOVE_HEAD (&block_mig_state.blk_list, entry);qemu_free (blk->buf);qemu_free (blk);
@@ -366,11 +355,6 @@ static void flush_blks(QEMUFile* f) block_mig_state.transferred++;assert (block_mig_state.read_done >=0 ); } - block_mig_state.first_blk = blk; - -if (block_mig_state.first_blk ==NULL ) { - block_mig_state.last_blk =NULL ; - } dprintf("%s Exit submitted %d read_done %d transferred %d\n", __FUNCTION__, block_mig_state.submitted, block_mig_state.read_done,
@@ -385,7 +369,7 @@ static int is_stage2_completed(void)return 0 ; } -for (bmds = block_mig_state.bmds_first; bmds !=NULL ; bmds = bmds->next) { + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {if (bmds->bulk_completed ==0 ) {return 0 ; }
@@ -498,6 +482,9 @@ static void block_set_params(int blk_enable, int shared_base, void *opaque)void blk_mig_init (void ) { +QSIMPLEQ_INIT (&block_mig_state.bmds_list); +QSIMPLEQ_INIT (&block_mig_state.blk_list); +register_savevm_live ("block" ,0 ,1 , block_set_params, block_save_live,NULL , block_load, &block_mig_state); }
PATCH 11/23 block migration: Initialize remaining BlkMigState fields
In reply to Jan KiszkaIn case we restart a migration, submitted, read_done, transferred, and print_completion need to be reinitialized to 0.
---
block-migration.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
block-migration.c +5 -0
@@ -248,6 +248,11 @@ static void init_blk_migration(QEMUFile *f)BlkMigDevState *bmds;BlockDriverState *bs; + block_mig_state.submitted =0 ; + block_mig_state.read_done =0 ; + block_mig_state.transferred =0 ; + block_mig_state.print_completion =0 ; +for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {if (bs->type ==BDRV_TYPE_HD ) { bmds =qemu_mallocz (sizeof (BlkMigDevState));
PATCH 12/23 block migration: Clean up use of total_sectors
In reply to Jan KiszkaWe already save total_sectors in BlkMigDevState, let's use this value during the migration and avoid to recalculate it needlessly.
---
block-migration.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
block-migration.c +3 -3
@@ -96,10 +96,10 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) blk->buf =qemu_malloc (BLOCK_SIZE ); cur_sector = bms->cur_sector; - total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; + total_sectors = bms->total_sectors;if (bms->shared_base) { -while (cur_sector < bms->total_sectors && +while (cur_sector < total_sectors && !bdrv_is_allocated (bms->bs, cur_sector,MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { cur_sector += nr_sectors;
@@ -165,7 +165,7 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds) cur_sector = bmds->cur_sector;if (bmds->shared_base) { -while (cur_sector < bmds->total_sectors && +while (cur_sector < total_sectors && !bdrv_is_allocated (bmds->bs, cur_sector,MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { cur_sector += nr_sectors;
PATCH 13/23 block migration: Consolidate mig_read_device_bulk into mig_save_device_bulk
In reply to Jan KiszkaBoth functions share a lot of code, so make them one.
---
block-migration.c | 149 ++++++++++++++++++----------------------------------- 1 files changed, 50 insertions(+), 99 deletions(-)
block-migration.c +50 -99
@@ -85,31 +85,25 @@ static void blk_mig_read_cb(void *opaque, int ret)assert (block_mig_state.submitted >=0 ); } -static int mig_read_device_bulk (QEMUFile *f,BlkMigDevState *bms) +static int mig_save_device_bulk (QEMUFile *f,BlkMigDevState *bmds,int is_async) { -int nr_sectors; -int64_t total_sectors, cur_sector =0 ; -BlockDriverState *bs = bms->bs; +int64_t total_sectors = bmds->total_sectors; +int64_t cur_sector = bmds->cur_sector; +BlockDriverState *bs = bmds->bs; +int len, nr_sectors;BlkMigBlock *blk; +uint8_t *tmp_buf; - blk =qemu_malloc (sizeof (BlkMigBlock)); - blk->buf =qemu_malloc (BLOCK_SIZE ); - - cur_sector = bms->cur_sector; - total_sectors = bms->total_sectors; - -if (bms->shared_base) { +if (bmds->shared_base) {while (cur_sector < total_sectors && - !bdrv_is_allocated (bms->bs, cur_sector, -MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { + !bdrv_is_allocated (bs, cur_sector,MAX_IS_ALLOCATED_SEARCH , + &nr_sectors)) { cur_sector += nr_sectors; } }if (cur_sector >= total_sectors) { - bms->cur_sector = total_sectors; -qemu_free (blk->buf); -qemu_free (blk); + bmds->cur_sector = total_sectors;return 1 ; }
@@ -120,101 +114,65 @@ static int mig_read_device_bulk(QEMUFile *f, BlkMigDevState *bms) (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); } + cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 ); +/* we are going to transfer a full block even if it is not allocated */ nr_sectors =BDRV_SECTORS_PER_DIRTY_CHUNK ; - cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 ); -if (total_sectors - cur_sector <BDRV_SECTORS_PER_DIRTY_CHUNK ) { - nr_sectors = (total_sectors - cur_sector); + nr_sectors = total_sectors - cur_sector; } - bms->cur_sector = cur_sector + nr_sectors; - blk->sector = cur_sector; - blk->bmds = bms; +if (is_async) { + blk =qemu_malloc (sizeof (BlkMigBlock)); + blk->buf =qemu_malloc (BLOCK_SIZE ); - blk->iov.iov_base = blk->buf; - blk->iov.iov_len = nr_sectors *BDRV_SECTOR_SIZE ; -qemu_iovec_init_external (&blk->qiov, &blk->iov,1 ); + bmds->cur_sector = cur_sector + nr_sectors; + blk->sector = cur_sector; + blk->bmds = bmds; - blk->aiocb =bdrv_aio_readv (bs, cur_sector, &blk->qiov, - nr_sectors, blk_mig_read_cb, blk); + blk->iov.iov_base = blk->buf; + blk->iov.iov_len = nr_sectors *BDRV_SECTOR_SIZE ; +qemu_iovec_init_external (&blk->qiov, &blk->iov,1 ); -if (!blk->aiocb) { -printf ("Error reading sector %" PRId64"\n" , cur_sector); -qemu_free (blk->buf); -qemu_free (blk); -return 0 ; - } - -bdrv_reset_dirty (bms->bs, cur_sector, nr_sectors); - block_mig_state.submitted++; - -return (bms->cur_sector >= total_sectors); -} + blk->aiocb =bdrv_aio_readv (bs, cur_sector, &blk->qiov, + nr_sectors, blk_mig_read_cb, blk); -static int mig_save_device_bulk (QEMUFile *f,BlkMigDevState *bmds) -{ -int len, nr_sectors; -int64_t total_sectors = bmds->total_sectors, cur_sector =0 ; -uint8_t *tmp_buf =NULL ; -BlockDriverState *bs = bmds->bs; +if (!blk->aiocb) { +printf ("Error reading sector %" PRId64"\n" , cur_sector); +qemu_free (blk->buf); +qemu_free (blk); +return 0 ; + } - tmp_buf =qemu_malloc (BLOCK_SIZE ); +bdrv_reset_dirty (bs, cur_sector, nr_sectors); + block_mig_state.submitted++; - cur_sector = bmds->cur_sector; + }else { + tmp_buf =qemu_malloc (BLOCK_SIZE ); -if (bmds->shared_base) { -while (cur_sector < total_sectors && - !bdrv_is_allocated (bmds->bs, cur_sector, -MAX_IS_ALLOCATED_SEARCH , &nr_sectors)) { - cur_sector += nr_sectors; +if (bdrv_read (bs, cur_sector, tmp_buf, nr_sectors) <0 ) { +printf ("Error reading sector %" PRId64"\n" , cur_sector); } - } -if (cur_sector >= total_sectors) { - bmds->cur_sector = total_sectors; -qemu_free (tmp_buf); -return 1 ; - } +bdrv_reset_dirty (bs, cur_sector, nr_sectors); -if (cur_sector >= block_mig_state.print_completion) { -printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors); -fflush (stdout); - block_mig_state.print_completion += - (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); - } +/* sector number and flags */ +qemu_put_be64 (f, (cur_sector <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_DEVICE_BLOCK ); - cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 ); +/* device name */ + len =strlen (bs->device_name); +qemu_put_byte (f, len); +qemu_put_buffer (f, (uint8_t *)bs->device_name, len); -/* we are going to transfer a full block even if it is not allocated */ - nr_sectors =BDRV_SECTORS_PER_DIRTY_CHUNK ; +qemu_put_buffer (f, tmp_buf,BLOCK_SIZE ); -if (total_sectors - cur_sector <BDRV_SECTORS_PER_DIRTY_CHUNK ) { - nr_sectors = (total_sectors - cur_sector); - } + bmds->cur_sector = cur_sector +BDRV_SECTORS_PER_DIRTY_CHUNK ; -if (bdrv_read (bs, cur_sector, tmp_buf, nr_sectors) <0 ) { -printf ("Error reading sector %" PRId64"\n" , cur_sector); +qemu_free (tmp_buf); } -bdrv_reset_dirty (bs, cur_sector, nr_sectors); - -/* sector number and flags */ -qemu_put_be64 (f, (cur_sector <<BDRV_SECTOR_BITS ) - |BLK_MIG_FLAG_DEVICE_BLOCK ); - -/* device name */ - len =strlen (bs->device_name); -qemu_put_byte (f, len); -qemu_put_buffer (f, (uint8_t *)bs->device_name, len); - -qemu_put_buffer (f, tmp_buf,BLOCK_SIZE ); - - bmds->cur_sector = cur_sector +BDRV_SECTORS_PER_DIRTY_CHUNK ; - -qemu_free (tmp_buf); -return (bmds->cur_sector >= total_sectors); }
@@ -279,16 +237,9 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {if (bmds->bulk_completed ==0 ) { -if (is_async) { -if (mig_read_device_bulk (f, bmds) ==1 ) { -/* completed bulk section for this device */ - bmds->bulk_completed =1 ; - } - }else { -if (mig_save_device_bulk (f, bmds) ==1 ) { -/* completed bulk section for this device */ - bmds->bulk_completed =1 ; - } +if (mig_save_device_bulk (f, bmds, is_async) ==1 ) { +/* completed bulk section for this device */ + bmds->bulk_completed =1 ; }return 1 ; }
PATCH 14/23 block migration: Consolidate block transmission
In reply to Jan KiszkaBased on the original patch by Pierre Riteau: Use a common blk_send function to transmit a block.
---
block-migration.c | 104 ++++++++++++++++++++--------------------------------- 1 files changed, 39 insertions(+), 65 deletions(-)
block-migration.c +39 -65
@@ -72,6 +72,22 @@ typedef struct BlkMigState {static BlkMigState block_mig_state; +static void blk_send (QEMUFile *f,BlkMigBlock * blk) +{ +int len; + +/* sector number and flags */ +qemu_put_be64 (f, (blk->sector <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_DEVICE_BLOCK ); + +/* device name */ + len =strlen (blk->bmds->bs->device_name); +qemu_put_byte (f, len); +qemu_put_buffer (f, (uint8_t *)blk->bmds->bs->device_name, len); + +qemu_put_buffer (f, blk->buf,BLOCK_SIZE ); +} +static void blk_mig_read_cb (void *opaque,int ret) {BlkMigBlock *blk = opaque;
@@ -90,9 +106,8 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async)int64_t total_sectors = bmds->total_sectors;int64_t cur_sector = bmds->cur_sector;BlockDriverState *bs = bmds->bs; -int len, nr_sectors;BlkMigBlock *blk; -uint8_t *tmp_buf; +int nr_sectors;if (bmds->shared_base) { while (cur_sector < total_sectors &&
@@ -123,73 +138,40 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async) nr_sectors = total_sectors - cur_sector; } -if (is_async) { - blk =qemu_malloc (sizeof (BlkMigBlock)); - blk->buf =qemu_malloc (BLOCK_SIZE ); - - bmds->cur_sector = cur_sector + nr_sectors; - blk->sector = cur_sector; - blk->bmds = bmds; + blk =qemu_malloc (sizeof (BlkMigBlock)); + blk->buf =qemu_malloc (BLOCK_SIZE ); + blk->bmds = bmds; + blk->sector = cur_sector; +if (is_async) { blk->iov.iov_base = blk->buf; blk->iov.iov_len = nr_sectors *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&blk->qiov, &blk->iov,1 ); blk->aiocb =bdrv_aio_readv (bs, cur_sector, &blk->qiov, nr_sectors, blk_mig_read_cb, blk); -if (!blk->aiocb) {printf ("Error reading sector %" PRId64"\n" , cur_sector);qemu_free (blk->buf);qemu_free (blk);return 0 ; } - -bdrv_reset_dirty (bs, cur_sector, nr_sectors); block_mig_state.submitted++; - }else { - tmp_buf =qemu_malloc (BLOCK_SIZE ); - -if (bdrv_read (bs, cur_sector, tmp_buf, nr_sectors) <0 ) { +if (bdrv_read (bs, cur_sector, blk->buf, nr_sectors) <0 ) {printf ("Error reading sector %" PRId64"\n" , cur_sector); +return 0 ; } +blk_send (f, blk); -bdrv_reset_dirty (bs, cur_sector, nr_sectors); - -/* sector number and flags */ -qemu_put_be64 (f, (cur_sector <<BDRV_SECTOR_BITS ) - |BLK_MIG_FLAG_DEVICE_BLOCK ); - -/* device name */ - len =strlen (bs->device_name); -qemu_put_byte (f, len); -qemu_put_buffer (f, (uint8_t *)bs->device_name, len); - -qemu_put_buffer (f, tmp_buf,BLOCK_SIZE ); - - bmds->cur_sector = cur_sector +BDRV_SECTORS_PER_DIRTY_CHUNK ; - -qemu_free (tmp_buf); +qemu_free (blk->buf); +qemu_free (blk); } -return (bmds->cur_sector >= total_sectors); -} - -static void send_blk (QEMUFile *f,BlkMigBlock * blk) -{ -int len; - -/* sector number and flags */ -qemu_put_be64 (f, (blk->sector <<BDRV_SECTOR_BITS ) - |BLK_MIG_FLAG_DEVICE_BLOCK ); +bdrv_reset_dirty (bs, cur_sector, nr_sectors); + bmds->cur_sector = cur_sector + nr_sectors; -/* device name */ - len =strlen (blk->bmds->bs->device_name); -qemu_put_byte (f, len); -qemu_put_buffer (f, (uint8_t *)blk->bmds->bs->device_name, len); - -qemu_put_buffer (f, blk->buf,BLOCK_SIZE ); +return (bmds->cur_sector >= total_sectors); }static void set_dirty_tracking (int enable)
@@ -254,30 +236,22 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)static void blk_mig_save_dirty_blocks (QEMUFile *f) {BlkMigDevState *bmds; -uint8_t *buf; +BlkMigBlock blk;int64_t sector; -int len; - buf =qemu_malloc (BLOCK_SIZE ); + blk.buf =qemu_malloc (BLOCK_SIZE ); QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {for (sector =0 ; sector < bmds->cur_sector;) {if (bdrv_get_dirty (bmds->bs, sector)) { -if (bdrv_read (bmds->bs, sector, buf, +if (bdrv_read (bmds->bs, sector, blk.buf,BDRV_SECTORS_PER_DIRTY_CHUNK ) <0 ) { +printf ("Error reading sector %" PRId64"\n" , sector);/* FIXME: add error handling */ } - -/* sector number and flags */ -qemu_put_be64 (f, (sector <<BDRV_SECTOR_BITS ) - |BLK_MIG_FLAG_DEVICE_BLOCK ); - -/* device name */ - len =strlen (bmds->bs->device_name); -qemu_put_byte (f, len); -qemu_put_buffer (f, (uint8_t *)bmds->bs->device_name, len); - -qemu_put_buffer (f, buf,BLOCK_SIZE ); + blk.bmds = bmds; + blk.sector = sector; +blk_send (f, &blk);bdrv_reset_dirty (bmds->bs, sector,BDRV_SECTORS_PER_DIRTY_CHUNK );
@@ -286,7 +260,7 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f) } } -qemu_free (buf); +qemu_free (blk.buf); }static void flush_blks (QEMUFile * f)
@@ -301,7 +275,7 @@ static void flush_blks(QEMUFile* f)if (qemu_file_rate_limit (f)) {break ; } -send_blk (f, blk); +blk_send (f, blk);QSIMPLEQ_REMOVE_HEAD (&block_mig_state.blk_list, entry);qemu_free (blk->buf);
PATCH 15/23 block migration: Add error handling/propagation
In reply to Jan Kiszka---
block-migration.c | 57 ++++++++++++++++++++++++++++++++++++++--------------- 1 files changed, 41 insertions(+), 16 deletions(-)
block-migration.c +41 -16
@@ -151,16 +151,12 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async) blk->aiocb =bdrv_aio_readv (bs, cur_sector, &blk->qiov, nr_sectors, blk_mig_read_cb, blk);if (!blk->aiocb) { -printf ("Error reading sector %" PRId64"\n" , cur_sector); -qemu_free (blk->buf); -qemu_free (blk); -return 0 ; + goto error; } block_mig_state.submitted++; }else {if (bdrv_read (bs, cur_sector, blk->buf, nr_sectors) <0 ) { -printf ("Error reading sector %" PRId64"\n" , cur_sector); -return 0 ; + goto error; }blk_send (f, blk);
@@ -172,6 +168,13 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async) bmds->cur_sector = cur_sector + nr_sectors;return (bmds->cur_sector >= total_sectors); + +error: +printf ("Error reading sector %" PRId64"\n" , cur_sector); +qemu_file_set_error (f); +qemu_free (blk->buf); +qemu_free (blk); +return 0 ; }static void set_dirty_tracking (int enable)
@@ -247,7 +250,9 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)if (bdrv_read (bmds->bs, sector, blk.buf,BDRV_SECTORS_PER_DIRTY_CHUNK ) <0 ) {printf ("Error reading sector %" PRId64"\n" , sector); -/* FIXME: add error handling */ +qemu_file_set_error (f); +qemu_free (blk.buf); +return ; } blk.bmds = bmds; blk.sector = sector;
@@ -275,6 +280,10 @@ static void flush_blks(QEMUFile* f)if (qemu_file_rate_limit (f)) {break ; } +if (blk->ret <0 ) { +qemu_file_set_error (f); +break ; + }blk_send (f, blk);QSIMPLEQ_REMOVE_HEAD (&block_mig_state.blk_list, entry);
@@ -328,6 +337,11 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)flush_blks (f); +if (qemu_file_has_error (f)) { +set_dirty_tracking (0 ); +return 0 ; + } +/* control the rate of transfer */ while ((block_mig_state.submitted + block_mig_state.read_done) * BLOCK_SIZE <
@@ -340,6 +354,11 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)flush_blks (f); +if (qemu_file_has_error (f)) { +set_dirty_tracking (0 ); +return 0 ; + } +if (stage ==3 ) {while (blk_mig_save_bulked_block (f,0 ) !=0 ) {/* empty */
@@ -350,6 +369,10 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)/* stop track dirty blocks */ set_dirty_tracking (0 ); +if (qemu_file_has_error (f)) { +return 0 ; + } +printf ("\nBlock migration completed\n" ); }
@@ -375,26 +398,28 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)if (flags &BLK_MIG_FLAG_DEVICE_BLOCK ) {/* get device name */ len =qemu_get_byte (f); -qemu_get_buffer (f, (uint8_t *)device_name, len); device_name[len] ='\0' ; bs =bdrv_find (device_name); +if (!bs) { +fprintf (stderr,"Error unknown block device %s\n" , + device_name); +return -EINVAL ; + } buf =qemu_malloc (BLOCK_SIZE );qemu_get_buffer (f, buf,BLOCK_SIZE ); -if (bs !=NULL ) { -bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK ); - }else { -printf ("Error unknown block device %s\n" , device_name); -/* FIXME: add error handling */ - } +bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK );qemu_free (buf); }else if (!(flags &BLK_MIG_FLAG_EOS )) { -printf ("Unknown flags\n" ); -/* FIXME: add error handling */ +fprintf (stderr,"Unknown flags\n" ); +return -EINVAL ; + } +if (qemu_file_has_error (f)) { +return -EIO ; } }while (!(flags &BLK_MIG_FLAG_EOS ));
PATCH 16/23 ram migration: Stop loading on error
In reply to Jan KiszkaBesides catching real errors, this also allows to interrrupt the qemu process during restore.
---
vl.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-)
vl.c +5 -1
@@ -3017,8 +3017,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) madvise(qemu_get_ram_ptr(addr), TARGET_PAGE_SIZE, MADV_DONTNEED); }#endif - }else if (flags &RAM_SAVE_FLAG_PAGE ) + }else if (flags &RAM_SAVE_FLAG_PAGE ) {qemu_get_buffer (f,qemu_get_ram_ptr (addr),TARGET_PAGE_SIZE ); + } +if (qemu_file_has_error (f)) { +return -EIO ; + } }while (!(flags &RAM_SAVE_FLAG_EOS ));return 0 ;
PATCH 17/23 live migration: Allow cleanup after cancellation or error
In reply to Jan KiszkaIntroduce qemu_savevm_state_cancel and inject a stage -1 to cancel a live migration. This gives the involved subsystems a chance to clean up dynamically allocated resources. Namely, the block migration layer can now free its device descriptors and pending blocks.
---
block-migration.c | 36 ++++++++++++++++++++++++++++++------ migration.c | 1 + savevm.c | 19 +++++++++++++++++-- sysemu.h | 1 + vl.c | 5 +++++ 5 files changed, 54 insertions(+), 8 deletions(-)
block-migration.c +30 -6
@@ -317,11 +317,37 @@ static int is_stage2_completed(void)return 1 ; } +static void blk_mig_cleanup (void ) +{ +BlkMigDevState *bmds, *next_bmds; +BlkMigBlock *blk, *next_blk; + + QTAILQ_FOREACH_SAFE(bmds, &block_mig_state.dev_list, entry, next_bmds) { +QTAILQ_REMOVE (&block_mig_state.dev_list, bmds, entry); +qemu_free (bmds); + } + + QTAILQ_FOREACH_SAFE(blk, &block_mig_state.blk_list, entry, next_blk) { +QTAILQ_REMOVE (&block_mig_state.blk_list, blk, entry); +qemu_free (blk->buf); +qemu_free (blk); + } + +set_dirty_tracking (0 ); + +printf ("\n" ); +} +static int block_save_live (QEMUFile *f,int stage,void *opaque) {dprintf ("Enter save live stage %d submitted %d transferred %d\n" , stage, block_mig_state.submitted, block_mig_state.transferred); +if (stage <0 ) { +blk_mig_cleanup (); +return 0 ; + } +if (block_mig_state.blk_enable !=1 ) {/* no need to migrate storage */ qemu_put_be64 (f,BLK_MIG_FLAG_EOS );
@@ -338,7 +364,7 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)flush_blks (f);if (qemu_file_has_error (f)) { -set_dirty_tracking (0 ); +blk_mig_cleanup ();return 0 ; }
@@ -355,7 +381,7 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque)flush_blks (f);if (qemu_file_has_error (f)) { -set_dirty_tracking (0 ); +blk_mig_cleanup ();return 0 ; }
@@ -365,15 +391,13 @@ static int block_save_live(QEMUFile *f, int stage, void *opaque) }blk_mig_save_dirty_blocks (f); - -/* stop track dirty blocks */ -set_dirty_tracking (0 ); +blk_mig_cleanup ();if (qemu_file_has_error (f)) {return 0 ; } -printf ("\nBlock migration completed\n" ); +printf ("Block migration completed\n" ); }qemu_put_be64 (f,BLK_MIG_FLAG_EOS );
migration.c +1 -0
@@ -324,6 +324,7 @@ void migrate_fd_cancel(MigrationState *mig_state)dprintf ("cancelling migration\n" ); s->state =MIG_STATE_CANCELLED ; +qemu_savevm_state_cancel (s->file);migrate_fd_cleanup (s); }
savevm.c +17 -2
@@ -1271,8 +1271,10 @@ int qemu_savevm_state_begin(QEMUFile *f, int blk_enable, int shared) se->save_live_state (f,QEMU_VM_SECTION_START , se->opaque); } -if (qemu_file_has_error (f)) +if (qemu_file_has_error (f)) { +qemu_savevm_state_cancel (f);return -EIO ; + }return 0 ; }
@@ -1296,8 +1298,10 @@ int qemu_savevm_state_iterate(QEMUFile *f)if (ret)return 1 ; -if (qemu_file_has_error (f)) +if (qemu_file_has_error (f)) { +qemu_savevm_state_cancel (f);return -EIO ; + }return 0 ; }
@@ -1346,6 +1350,17 @@ int qemu_savevm_state_complete(QEMUFile *f)return 0 ; } +void qemu_savevm_state_cancel (QEMUFile *f) +{ +SaveStateEntry *se; + + QTAILQ_FOREACH(se, &savevm_handlers, entry) { +if (se->save_live_state) { + se->save_live_state (f,-1 , se->opaque); + } + } +} +int qemu_savevm_state (QEMUFile *f) {int saved_vm_running;
sysemu.h +1 -0
@@ -64,6 +64,7 @@ void main_loop_wait(int timeout);int qemu_savevm_state_begin (QEMUFile *f,int blk_enable,int shared);int qemu_savevm_state_iterate (QEMUFile *f);int qemu_savevm_state_complete (QEMUFile *f); +void qemu_savevm_state_cancel (QEMUFile *f);int qemu_savevm_state (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f);
vl.c +5 -0
@@ -2935,6 +2935,11 @@ static int ram_save_live(QEMUFile *f, int stage, void *opaque)double bwidth =0 ;uint64_t expected_time =0 ; +if (stage <0 ) { +cpu_physical_memory_set_dirty_tracking (0 ); +return 0 ; + } +if (cpu_physical_sync_dirty_bitmap (0 ,TARGET_PHYS_ADDR_MAX ) !=0 ) {qemu_file_set_error (f);return 0 ;
PATCH 18/23 block migration: Report overall migration progress
In reply to Jan KiszkaSo far progress reporting only works for the first block device. Fix this by keeping an overall sum of sectors to be migratated, calculating the sum of all processed sectors, and finally basing the progress display on those values.
---
block-migration.c | 46 ++++++++++++++++++++++++++++++---------------- 1 files changed, 30 insertions(+), 16 deletions(-)
block-migration.c +30 -16
@@ -43,6 +43,7 @@ typedef struct BlkMigDevState {int bulk_completed;int shared_base;int64_t cur_sector; +int64_t completed_sectors;int64_t total_sectors;int64_t dirty;QSIMPLEQ_ENTRY (BlkMigDevState) entry;
@@ -67,6 +68,7 @@ typedef struct BlkMigState {int submitted;int read_done;int transferred; +int64_t total_sector_sum;int64_t print_completion; } BlkMigState;
@@ -118,16 +120,11 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async) }if (cur_sector >= total_sectors) { - bmds->cur_sector = total_sectors; + bmds->cur_sector = bmds->completed_sectors = total_sectors;return 1 ; } -if (cur_sector >= block_mig_state.print_completion) { -printf ("Completed %" PRId64" %%\r" , cur_sector *100 / total_sectors); -fflush (stdout); - block_mig_state.print_completion += - (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); - } + bmds->completed_sectors = cur_sector; cur_sector &= ~((int64_t )BDRV_SECTORS_PER_DIRTY_CHUNK -1 );
@@ -194,6 +191,7 @@ static void init_blk_migration(QEMUFile *f) block_mig_state.submitted =0 ; block_mig_state.read_done =0 ; block_mig_state.transferred =0 ; + block_mig_state.total_sector_sum =0 ; block_mig_state.print_completion =0 ;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {
@@ -202,8 +200,11 @@ static void init_blk_migration(QEMUFile *f) bmds->bs = bs; bmds->bulk_completed =0 ; bmds->total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; + bmds->completed_sectors =0 ; bmds->shared_base = block_mig_state.shared_base; + block_mig_state.total_sector_sum += bmds->total_sectors; +if (bmds->shared_base) {printf ("Start migration for %s with shared base image\n" , bs->device_name);
@@ -218,7 +219,9 @@ static void init_blk_migration(QEMUFile *f)static int blk_mig_save_bulked_block (QEMUFile *f,int is_async) { +int64_t completed_sector_sum =0 ;BlkMigDevState *bmds; +int ret =0 ; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {if (bmds->bulk_completed ==0 ) {
@@ -226,12 +229,23 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)/* completed bulk section for this device */ bmds->bulk_completed =1 ; } -return 1 ; + completed_sector_sum += bmds->completed_sectors; + ret =1 ; +break ; + }else { + completed_sector_sum += bmds->completed_sectors; } } -/* we reached here means bulk is completed */ -return 0 ; +if (completed_sector_sum >= block_mig_state.print_completion) { +printf ("Completed %" PRId64" %%\r" , + completed_sector_sum *100 / block_mig_state.total_sector_sum); +fflush (stdout); + block_mig_state.print_completion += + (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); + } + +return ret; }#define MAX_NUM_BLOCKS 4
@@ -319,16 +333,16 @@ static int is_stage2_completed(void)static void blk_mig_cleanup (void ) { -BlkMigDevState *bmds, *next_bmds; -BlkMigBlock *blk, *next_blk; +BlkMigDevState *bmds; +BlkMigBlock *blk; - QTAILQ_FOREACH_SAFE(bmds, &block_mig_state.dev_list, entry, next_bmds) { -QTAILQ_REMOVE (&block_mig_state.dev_list, bmds, entry); +while ((bmds =QSIMPLEQ_FIRST (&block_mig_state.bmds_list)) !=NULL ) { +QSIMPLEQ_REMOVE_HEAD (&block_mig_state.bmds_list, entry);qemu_free (bmds); } - QTAILQ_FOREACH_SAFE(blk, &block_mig_state.blk_list, entry, next_blk) { -QTAILQ_REMOVE (&block_mig_state.blk_list, blk, entry); +while ((blk =QSIMPLEQ_FIRST (&block_mig_state.blk_list)) !=NULL ) { +QSIMPLEQ_REMOVE_HEAD (&block_mig_state.blk_list, entry);qemu_free (blk->buf);qemu_free (blk); }
PATCH 19/23 live migration: Propagate output monitor to callback handler
In reply to Jan KiszkaIn order to allow proper progress reporting to the monitor that initiated the migration, forward the monitor reference through the migration layer down to SaveLiveStateHandler.
---
block-migration.c | 2 +- hw/hw.h | 3 ++- migration-exec.c | 12 +++++++----- migration-fd.c | 9 +++++---- migration-tcp.c | 10 ++++++---- migration-unix.c | 10 ++++++---- migration.c | 30 ++++++++++++++++-------------- migration.h | 13 ++++++++----- savevm.c | 31 ++++++++++++++++--------------- sysemu.h | 10 +++++----- vl.c | 2 +- 11 files changed, 73 insertions(+), 59 deletions(-)
block-migration.c +1 -1
@@ -352,7 +352,7 @@ static void blk_mig_cleanup(void)printf ("\n" ); } -static int block_save_live (QEMUFile *f,int stage,void *opaque) +static int block_save_live (Monitor *mon,QEMUFile *f,int stage,void *opaque) {dprintf ("Enter save live stage %d submitted %d transferred %d\n" , stage, block_mig_state.submitted, block_mig_state.transferred);
hw/hw.h +2 -1
@@ -244,7 +244,8 @@ int64_t qemu_fseek(QEMUFile *f, int64_t pos, int whence);typedef void SaveSetParamsHandler (int blk_enable,int shared,void * opaque);typedef void SaveStateHandler (QEMUFile *f,void *opaque); -typedef int SaveLiveStateHandler (QEMUFile *f,int stage,void *opaque); +typedef int SaveLiveStateHandler (Monitor *mon,QEMUFile *f,int stage, +void *opaque);typedef int LoadStateHandler (QEMUFile *f,void *opaque,int version_id); int register_savevm(const char *idstr,
migration-exec.c +7 -5
@@ -52,7 +52,8 @@ static int exec_close(FdMigrationState *s)return 0 ; } -MigrationState *exec_start_outgoing_migration(const char *command, +MigrationState *exec_start_outgoing_migration(Monitor *mon, + const char *command, int64_t bandwidth_limit, int detach, int blk,
@@ -88,13 +89,14 @@ MigrationState *exec_start_outgoing_migration(const char *command, s->mig_state.blk = blk; s->mig_state.shared = inc; - + s->state =MIG_STATE_ACTIVE ; - s->mon_resume =NULL ; + s->mon =NULL ; s->bandwidth_limit = bandwidth_limit; -if (!detach) -migrate_fd_monitor_suspend (s); +if (!detach) { +migrate_fd_monitor_suspend (s, mon); + }migrate_fd_connect (s);return &s->mig_state;
migration-fd.c +5 -4
@@ -82,13 +82,14 @@ MigrationState *fd_start_outgoing_migration(Monitor *mon, s->mig_state.blk = blk; s->mig_state.shared = inc; - + s->state =MIG_STATE_ACTIVE ; - s->mon_resume =NULL ; + s->mon =NULL ; s->bandwidth_limit = bandwidth_limit; -if (!detach) -migrate_fd_monitor_suspend (s); +if (!detach) { +migrate_fd_monitor_suspend (s, mon); + }migrate_fd_connect (s);return &s->mig_state;
migration-tcp.c +6 -4
@@ -76,7 +76,8 @@ static void tcp_wait_for_connect(void *opaque) } } -MigrationState *tcp_start_outgoing_migration(const char *host_port, +MigrationState *tcp_start_outgoing_migration(Monitor *mon, + const char *host_port, int64_t bandwidth_limit, int detach, int blk,
@@ -102,7 +103,7 @@ MigrationState *tcp_start_outgoing_migration(const char *host_port, s->mig_state.shared = inc; s->state =MIG_STATE_ACTIVE ; - s->mon_resume =NULL ; + s->mon =NULL ; s->bandwidth_limit = bandwidth_limit; s->fd =socket (PF_INET ,SOCK_STREAM ,0 );if (s->fd ==-1 ) {
@@ -112,8 +113,9 @@ MigrationState *tcp_start_outgoing_migration(const char *host_port,socket_set_nonblock (s->fd); -if (!detach) -migrate_fd_monitor_suspend (s); +if (!detach) { +migrate_fd_monitor_suspend (s, mon); + } do { ret = connect(s->fd, (struct sockaddr *)&addr, sizeof(addr));
migration-unix.c +6 -4
@@ -75,7 +75,8 @@ static void unix_wait_for_connect(void *opaque) } } -MigrationState *unix_start_outgoing_migration(const char *path, +MigrationState *unix_start_outgoing_migration(Monitor *mon, + const char *path, int64_t bandwidth_limit, int detach, int blk,
@@ -101,7 +102,7 @@ MigrationState *unix_start_outgoing_migration(const char *path, s->mig_state.shared = inc; s->state =MIG_STATE_ACTIVE ; - s->mon_resume =NULL ; + s->mon =NULL ; s->bandwidth_limit = bandwidth_limit; s->fd =socket (PF_UNIX ,SOCK_STREAM ,0 );if (s->fd <0 ) {
@@ -111,8 +112,9 @@ MigrationState *unix_start_outgoing_migration(const char *path,socket_set_nonblock (s->fd); -if (!detach) -migrate_fd_monitor_suspend (s); +if (!detach) { +migrate_fd_monitor_suspend (s, mon); + } do { ret = connect(s->fd, (struct sockaddr *)&addr, sizeof(addr));
migration.c +16 -14
@@ -66,16 +66,16 @@ void do_migrate(Monitor *mon, const QDict *qdict, QObject **ret_data) }if (strstart (uri,"tcp:" , &p)) - s =tcp_start_outgoing_migration (p, max_throttle, detach, + s =tcp_start_outgoing_migration (mon, p, max_throttle, detach, (int )qdict_get_int (qdict,"blk" ), (int )qdict_get_int (qdict,"inc" ));#if !defined(WIN32 ) else if (strstart(uri, "exec:", &p)) - s =exec_start_outgoing_migration (p, max_throttle, detach, + s =exec_start_outgoing_migration (mon, p, max_throttle, detach, (int )qdict_get_int (qdict,"blk" ), (int )qdict_get_int (qdict,"inc" )); else if (strstart(uri, "unix:", &p)) - s =unix_start_outgoing_migration (p, max_throttle, detach, + s =unix_start_outgoing_migration (mon, p, max_throttle, detach, (int )qdict_get_int (qdict,"blk" ), (int )qdict_get_int (qdict,"inc" )); else if (strstart(uri, "fd:", &p))
@@ -190,14 +190,15 @@ void do_info_migrate(Monitor *mon)/* shared migration helpers */ -void migrate_fd_monitor_suspend (FdMigrationState *s) +void migrate_fd_monitor_suspend (FdMigrationState *s,Monitor *mon) { - s->mon_resume = cur_mon; -if (monitor_suspend (cur_mon) ==0 ) + s->mon = mon; +if (monitor_suspend (mon) ==0 ) {dprintf ("suspending monitor\n" ); -else -monitor_printf (cur_mon,"terminal does not allow synchronous " + }else { +monitor_printf (mon,"terminal does not allow synchronous " "migration, continuing detached\n" ); + } }void migrate_fd_error (FdMigrationState *s)
@@ -221,8 +222,9 @@ void migrate_fd_cleanup(FdMigrationState *s)close (s->fd);/* Don't resume monitor until we've flushed all of the buffers */ -if (s->mon_resume) -monitor_resume (s->mon_resume); +if (s->mon) { +monitor_resume (s->mon); + } s->fd =-1 ; }
@@ -265,7 +267,7 @@ void migrate_fd_connect(FdMigrationState *s) migrate_fd_close);dprintf ("beginning savevm\n" ); - ret =qemu_savevm_state_begin (s->file, s->mig_state.blk, + ret =qemu_savevm_state_begin (s->mon, s->file, s->mig_state.blk, s->mig_state.shared);if (ret <0 ) {dprintf ("failed, %d\n" , ret);
@@ -286,7 +288,7 @@ void migrate_fd_put_ready(void *opaque) }dprintf ("iterate\n" ); -if (qemu_savevm_state_iterate (s->file) ==1 ) { +if (qemu_savevm_state_iterate (s->mon, s->file) ==1 ) {int state;int old_vm_running = vm_running;
@@ -295,7 +297,7 @@ void migrate_fd_put_ready(void *opaque)qemu_aio_flush ();bdrv_flush_all (); -if ((qemu_savevm_state_complete (s->file)) <0 ) { +if ((qemu_savevm_state_complete (s->mon, s->file)) <0 ) {if (old_vm_running) {vm_start (); }
@@ -324,7 +326,7 @@ void migrate_fd_cancel(MigrationState *mig_state)dprintf ("cancelling migration\n" ); s->state =MIG_STATE_CANCELLED ; -qemu_savevm_state_cancel (s->file); +qemu_savevm_state_cancel (s->mon, s->file);migrate_fd_cleanup (s); }
migration.h +8 -5
@@ -42,7 +42,7 @@ struct FdMigrationStateint64_t bandwidth_limit;QEMUFile *file;int fd; -Monitor *mon_resume; +Monitor *mon;int state;int (*get_error)(struct FdMigrationState *);int (*close)(struct FdMigrationState *);
@@ -66,7 +66,8 @@ void do_info_migrate(Monitor *mon);int exec_start_incoming_migration (const char *host_port); -MigrationState *exec_start_outgoing_migration(const char *host_port, +MigrationState *exec_start_outgoing_migration(Monitor *mon, + const char *host_port, int64_t bandwidth_limit, int detach, int blk,
@@ -74,7 +75,8 @@ MigrationState *exec_start_outgoing_migration(const char *host_port,int tcp_start_incoming_migration (const char *host_port); -MigrationState *tcp_start_outgoing_migration(const char *host_port, +MigrationState *tcp_start_outgoing_migration(Monitor *mon, + const char *host_port, int64_t bandwidth_limit, int detach, int blk,
@@ -82,7 +84,8 @@ MigrationState *tcp_start_outgoing_migration(const char *host_port,int unix_start_incoming_migration (const char *path); -MigrationState *unix_start_outgoing_migration(const char *path, +MigrationState *unix_start_outgoing_migration(Monitor *mon, + const char *path, int64_t bandwidth_limit, int detach, int blk,
@@ -97,7 +100,7 @@ MigrationState *fd_start_outgoing_migration(Monitor *mon,int blk, int inc); -void migrate_fd_monitor_suspend (FdMigrationState *s); +void migrate_fd_monitor_suspend (FdMigrationState *s,Monitor *mon);void migrate_fd_error (FdMigrationState *s);
savevm.c +16 -15
@@ -1236,7 +1236,8 @@ static void vmstate_save(QEMUFile *f, SaveStateEntry *se)#define QEMU_VM_SECTION_END 0x03#define QEMU_VM_SECTION_FULL 0x04 -int qemu_savevm_state_begin (QEMUFile *f,int blk_enable,int shared) +int qemu_savevm_state_begin (Monitor *mon,QEMUFile *f,int blk_enable, +int shared) {SaveStateEntry *se;
@@ -1268,18 +1269,18 @@ int qemu_savevm_state_begin(QEMUFile *f, int blk_enable, int shared)qemu_put_be32 (f, se->instance_id);qemu_put_be32 (f, se->version_id); - se->save_live_state (f,QEMU_VM_SECTION_START , se->opaque); + se->save_live_state (mon, f,QEMU_VM_SECTION_START , se->opaque); }if (qemu_file_has_error (f)) { -qemu_savevm_state_cancel (f); +qemu_savevm_state_cancel (mon, f);return -EIO ; }return 0 ; } -int qemu_savevm_state_iterate (QEMUFile *f) +int qemu_savevm_state_iterate (Monitor *mon,QEMUFile *f) {SaveStateEntry *se;int ret =1 ;
@@ -1292,21 +1293,21 @@ int qemu_savevm_state_iterate(QEMUFile *f)qemu_put_byte (f,QEMU_VM_SECTION_PART );qemu_put_be32 (f, se->section_id); - ret &= !!se->save_live_state (f,QEMU_VM_SECTION_PART , se->opaque); + ret &= !!se->save_live_state (mon, f,QEMU_VM_SECTION_PART , se->opaque); }if (ret)return 1 ;if (qemu_file_has_error (f)) { -qemu_savevm_state_cancel (f); +qemu_savevm_state_cancel (mon, f);return -EIO ; }return 0 ; } -int qemu_savevm_state_complete (QEMUFile *f) +int qemu_savevm_state_complete (Monitor *mon,QEMUFile *f) {SaveStateEntry *se;
@@ -1318,7 +1319,7 @@ int qemu_savevm_state_complete(QEMUFile *f)qemu_put_byte (f,QEMU_VM_SECTION_END );qemu_put_be32 (f, se->section_id); - se->save_live_state (f,QEMU_VM_SECTION_END , se->opaque); + se->save_live_state (mon, f,QEMU_VM_SECTION_END , se->opaque); } QTAILQ_FOREACH(se, &savevm_handlers, entry) {
@@ -1350,18 +1351,18 @@ int qemu_savevm_state_complete(QEMUFile *f)return 0 ; } -void qemu_savevm_state_cancel (QEMUFile *f) +void qemu_savevm_state_cancel (Monitor *mon,QEMUFile *f) {SaveStateEntry *se; QTAILQ_FOREACH(se, &savevm_handlers, entry) {if (se->save_live_state) { - se->save_live_state (f,-1 , se->opaque); + se->save_live_state (mon, f,-1 , se->opaque); } } } -int qemu_savevm_state (QEMUFile *f) +static int qemu_savevm_state (Monitor *mon,QEMUFile *f) {int saved_vm_running;int ret;
@@ -1371,17 +1372,17 @@ int qemu_savevm_state(QEMUFile *f)bdrv_flush_all (); - ret =qemu_savevm_state_begin (f,0 ,0 ); + ret =qemu_savevm_state_begin (mon, f,0 ,0 );if (ret <0 ) goto out;do { - ret =qemu_savevm_state_iterate (f); + ret =qemu_savevm_state_iterate (mon, f);if (ret <0 ) goto out; }while (ret ==0 ); - ret =qemu_savevm_state_complete (f); + ret =qemu_savevm_state_complete (mon, f); out:if (qemu_file_has_error (f))
@@ -1670,7 +1671,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)monitor_printf (mon,"Could not open VM state file\n" ); goto the_end; } - ret =qemu_savevm_state (f); + ret =qemu_savevm_state (mon, f); vm_state_size =qemu_ftell (f);qemu_fclose (f);if (ret <0 ) {
sysemu.h +5 -5
@@ -61,11 +61,11 @@ void qemu_announce_self(void);void main_loop_wait (int timeout); -int qemu_savevm_state_begin (QEMUFile *f,int blk_enable,int shared); -int qemu_savevm_state_iterate (QEMUFile *f); -int qemu_savevm_state_complete (QEMUFile *f); -void qemu_savevm_state_cancel (QEMUFile *f); -int qemu_savevm_state (QEMUFile *f); +int qemu_savevm_state_begin (Monitor *mon,QEMUFile *f,int blk_enable, +int shared); +int qemu_savevm_state_iterate (Monitor *mon,QEMUFile *f); +int qemu_savevm_state_complete (Monitor *mon,QEMUFile *f); +void qemu_savevm_state_cancel (Monitor *mon,QEMUFile *f);int qemu_loadvm_state (QEMUFile *f);void qemu_errors_to_file (FILE *fp);
vl.c +1 -1
@@ -2928,7 +2928,7 @@ uint64_t ram_bytes_total(void)return last_ram_offset; } -static int ram_save_live (QEMUFile *f,int stage,void *opaque) +static int ram_save_live (Monitor *mon,QEMUFile *f,int stage,void *opaque) {ram_addr_t addr;uint64_t bytes_transferred_last;
PATCH 20/23 block migration: Fix outgoing progress output
In reply to Jan KiszkaReport progress of an outgoing live migration to the monitor instead of stdout.
---
block-migration.c | 54 +++++++++++++++++++++++++++++------------------------ 1 files changed, 30 insertions(+), 24 deletions(-)
block-migration.c +30 -24
@@ -15,6 +15,7 @@#include "block_int.h" #include "hw/hw.h" #include "qemu-queue.h" +#include "monitor.h" #include "block-migration.h" #include <assert.h>
@@ -103,7 +104,8 @@ static void blk_mig_read_cb(void *opaque, int ret)assert (block_mig_state.submitted >=0 ); } -static int mig_save_device_bulk (QEMUFile *f,BlkMigDevState *bmds,int is_async) +static int mig_save_device_bulk (Monitor *mon,QEMUFile *f, +BlkMigDevState *bmds,int is_async) {int64_t total_sectors = bmds->total_sectors;int64_t cur_sector = bmds->cur_sector;
@@ -167,7 +169,7 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds, int is_async)return (bmds->cur_sector >= total_sectors); error: -printf ("Error reading sector %" PRId64"\n" , cur_sector); +monitor_printf (mon,"Error reading sector %" PRId64"\n" , cur_sector);qemu_file_set_error (f);qemu_free (blk->buf);qemu_free (blk);
@@ -183,7 +185,7 @@ static void set_dirty_tracking(int enable) } } -static void init_blk_migration (QEMUFile *f) +static void init_blk_migration (Monitor *mon,QEMUFile *f) {BlkMigDevState *bmds;BlockDriverState *bs;
@@ -206,10 +208,12 @@ static void init_blk_migration(QEMUFile *f) block_mig_state.total_sector_sum += bmds->total_sectors;if (bmds->shared_base) { -printf ("Start migration for %s with shared base image\n" , - bs->device_name); +monitor_printf (mon,"Start migration for %s with shared base " +"image\n" , + bs->device_name); }else { -printf ("Start full migration for %s\n" , bs->device_name); +monitor_printf (mon,"Start full migration for %s\n" , + bs->device_name); }QSIMPLEQ_INSERT_TAIL (&block_mig_state.bmds_list, bmds, entry);
@@ -217,7 +221,7 @@ static void init_blk_migration(QEMUFile *f) } } -static int blk_mig_save_bulked_block (QEMUFile *f,int is_async) +static int blk_mig_save_bulked_block (Monitor *mon,QEMUFile *f,int is_async) {int64_t completed_sector_sum =0 ;BlkMigDevState *bmds;
@@ -225,7 +229,7 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {if (bmds->bulk_completed ==0 ) { -if (mig_save_device_bulk (f, bmds, is_async) ==1 ) { +if (mig_save_device_bulk (mon, f, bmds, is_async) ==1 ) {/* completed bulk section for this device */ bmds->bulk_completed =1 ; }
@@ -238,9 +242,10 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async) }if (completed_sector_sum >= block_mig_state.print_completion) { -printf ("Completed %" PRId64" %%\r" , - completed_sector_sum *100 / block_mig_state.total_sector_sum); -fflush (stdout); +monitor_printf (mon,"Completed %" PRId64" %%\r" , + completed_sector_sum *100 / + block_mig_state.total_sector_sum); +monitor_flush (mon); block_mig_state.print_completion += (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); }
@@ -250,7 +255,7 @@ static int blk_mig_save_bulked_block(QEMUFile *f, int is_async)#define MAX_NUM_BLOCKS 4 -static void blk_mig_save_dirty_blocks (QEMUFile *f) +static void blk_mig_save_dirty_blocks (Monitor *mon,QEMUFile *f) {BlkMigDevState *bmds;BlkMigBlock blk;
@@ -263,7 +268,8 @@ static void blk_mig_save_dirty_blocks(QEMUFile *f)if (bdrv_get_dirty (bmds->bs, sector)) {if (bdrv_read (bmds->bs, sector, blk.buf,BDRV_SECTORS_PER_DIRTY_CHUNK ) <0 ) { -printf ("Error reading sector %" PRId64"\n" , sector); +monitor_printf (mon,"Error reading sector %" PRId64"\n" , + sector);qemu_file_set_error (f);qemu_free (blk.buf);return ;
@@ -331,7 +337,7 @@ static int is_stage2_completed(void)return 1 ; } -static void blk_mig_cleanup (void ) +static void blk_mig_cleanup (Monitor *mon) {BlkMigDevState *bmds;BlkMigBlock *blk;
@@ -349,7 +355,7 @@ static void blk_mig_cleanup(void)set_dirty_tracking (0 ); -printf ("\n" ); +monitor_printf (mon,"\n" ); }static int block_save_live (Monitor *mon,QEMUFile *f,int stage,void *opaque)
@@ -358,7 +364,7 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque) stage, block_mig_state.submitted, block_mig_state.transferred);if (stage <0 ) { -blk_mig_cleanup (); +blk_mig_cleanup (mon);return 0 ; }
@@ -369,7 +375,7 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque) }if (stage ==1 ) { -init_blk_migration (f); +init_blk_migration (mon, f);/* start track dirty blocks */ set_dirty_tracking (1 );
@@ -378,7 +384,7 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)flush_blks (f);if (qemu_file_has_error (f)) { -blk_mig_cleanup (); +blk_mig_cleanup (mon);return 0 ; }
@@ -386,7 +392,7 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)while ((block_mig_state.submitted + block_mig_state.read_done) *BLOCK_SIZE <qemu_file_get_rate_limit (f)) { -if (blk_mig_save_bulked_block (f,1 ) ==0 ) { +if (blk_mig_save_bulked_block (mon, f,1 ) ==0 ) {/* no more bulk blocks for now */ break ; }
@@ -395,23 +401,23 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)flush_blks (f);if (qemu_file_has_error (f)) { -blk_mig_cleanup (); +blk_mig_cleanup (mon);return 0 ; }if (stage ==3 ) { -while (blk_mig_save_bulked_block (f,0 ) !=0 ) { +while (blk_mig_save_bulked_block (mon, f,0 ) !=0 ) {/* empty */ } -blk_mig_save_dirty_blocks (f); -blk_mig_cleanup (); +blk_mig_save_dirty_blocks (mon, f); +blk_mig_cleanup (mon);if (qemu_file_has_error (f)) {return 0 ; } -printf ("Block migration completed\n" ); +monitor_printf (mon,"Block migration completed\n" ); }qemu_put_be64 (f,BLK_MIG_FLAG_EOS );
PATCH 21/23 block migration: Report progress also via info migration
In reply to Jan Kiszka---
block-migration.c | 32 ++++++++++++++++++++++++++++++++ block-migration.h | 4 ++++ migration.c | 9 +++++++++ 3 files changed, 45 insertions(+), 0 deletions(-)
block-migration.c +32 -0
@@ -91,6 +91,38 @@ static void blk_send(QEMUFile *f, BlkMigBlock * blk)qemu_put_buffer (f, blk->buf,BLOCK_SIZE ); } +int blk_mig_active (void ) +{ +return !QSIMPLEQ_EMPTY (&block_mig_state.bmds_list); +} + +uint64_t blk_mig_bytes_transferred (void ) +{ +BlkMigDevState *bmds; +uint64_t sum =0 ; + + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + sum += bmds->completed_sectors; + } +return sum <<BDRV_SECTOR_BITS ; +} + +uint64_t blk_mig_bytes_remaining (void ) +{ +return blk_mig_bytes_total () -blk_mig_bytes_transferred (); +} + +uint64_t blk_mig_bytes_total (void ) +{ +BlkMigDevState *bmds; +uint64_t sum =0 ; + + QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + sum += bmds->total_sectors; + } +return sum <<BDRV_SECTOR_BITS ; +} +static void blk_mig_read_cb (void *opaque,int ret) {BlkMigBlock *blk = opaque;
block-migration.h +4 -0
@@ -15,5 +15,9 @@#define BLOCK_MIGRATION_H void blk_mig_init (void ); +int blk_mig_active (void ); +uint64_t blk_mig_bytes_transferred (void ); +uint64_t blk_mig_bytes_remaining (void ); +uint64_t blk_mig_bytes_total (void );#endif /* BLOCK_MIGRATION_H */
migration.c +9 -0
@@ -18,6 +18,7 @@#include "sysemu.h" #include "block.h" #include "qemu_socket.h" +#include "block-migration.h" //#define DEBUG_MIGRATION
@@ -174,6 +175,14 @@ void do_info_migrate(Monitor *mon)monitor_printf (mon,"transferred ram: %" PRIu64" kbytes\n" ,ram_bytes_transferred () >>10 );monitor_printf (mon,"remaining ram: %" PRIu64" kbytes\n" ,ram_bytes_remaining () >>10 );monitor_printf (mon,"total ram: %" PRIu64" kbytes\n" ,ram_bytes_total () >>10 ); +if (blk_mig_active ()) { +monitor_printf (mon,"transferred disk: %" PRIu64" kbytes\n" , +blk_mig_bytes_transferred () >>10 ); +monitor_printf (mon,"remaining disk: %" PRIu64" kbytes\n" , +blk_mig_bytes_remaining () >>10 ); +monitor_printf (mon,"total disk: %" PRIu64" kbytes\n" , +blk_mig_bytes_total () >>10 ); + }break ;case MIG_STATE_COMPLETED :monitor_printf (mon,"completed\n" );
PATCH 22/23 block migration: Add support for restore progress reporting
In reply to Jan KiszkaInject progress report in percentage into the block live stream. This can be read out and displayed easily on restore.
---
block-migration.c | 31 +++++++++++++++++++++++-------- 1 files changed, 23 insertions(+), 8 deletions(-)
block-migration.c +23 -8
@@ -23,6 +23,7 @@#define BLK_MIG_FLAG_DEVICE_BLOCK 0x01#define BLK_MIG_FLAG_EOS 0x02 +#define BLK_MIG_FLAG_PROGRESS 0x04#define MAX_IS_ALLOCATED_SEARCH 65536#define MAX_BLOCKS_READ 10000
@@ -70,7 +71,7 @@ typedef struct BlkMigState {int read_done;int transferred;int64_t total_sector_sum; -int64_t print_completion; +int prev_progress; } BlkMigState;static BlkMigState block_mig_state;
@@ -226,7 +227,7 @@ static void init_blk_migration(Monitor *mon, QEMUFile *f) block_mig_state.read_done =0 ; block_mig_state.transferred =0 ; block_mig_state.total_sector_sum =0 ; - block_mig_state.print_completion =0 ; + block_mig_state.prev_progress =-1 ;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {if (bs->type ==BDRV_TYPE_HD ) {
@@ -257,6 +258,7 @@ static int blk_mig_save_bulked_block(Monitor *mon, QEMUFile *f, int is_async) {int64_t completed_sector_sum =0 ;BlkMigDevState *bmds; +int progress;int ret =0 ; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
@@ -273,13 +275,13 @@ static int blk_mig_save_bulked_block(Monitor *mon, QEMUFile *f, int is_async) } } -if (completed_sector_sum >= block_mig_state.print_completion) { -monitor_printf (mon,"Completed %" PRId64" %%\r" , - completed_sector_sum *100 / - block_mig_state.total_sector_sum); + progress = completed_sector_sum *100 / block_mig_state.total_sector_sum; +if (progress != block_mig_state.prev_progress) { + block_mig_state.prev_progress = progress; +qemu_put_be64 (f, (progress <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_PROGRESS ); +monitor_printf (mon,"Completed %d %%\r" , progress);monitor_flush (mon); - block_mig_state.print_completion += - (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); }return ret;
@@ -445,6 +447,9 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)blk_mig_save_dirty_blocks (mon, f);blk_mig_cleanup (mon); +/* report completion */ +qemu_put_be64 (f, (100 <<BDRV_SECTOR_BITS ) |BLK_MIG_FLAG_PROGRESS ); +if (qemu_file_has_error (f)) {return 0 ; }
@@ -459,12 +464,18 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)static int block_load (QEMUFile *f,void *opaque,int version_id) { +static int banner_printed;int len, flags;char device_name[256 ];int64_t addr;BlockDriverState *bs;uint8_t *buf; +if (!banner_printed) { +printf ("Receiving block device images\n" ); + banner_printed =1 ; + } + do { addr = qemu_get_be64(f);
@@ -490,6 +501,10 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK );qemu_free (buf); + }else if (flags &BLK_MIG_FLAG_PROGRESS ) { +printf ("Completed %d %%%c" , (int )addr, + (addr ==100 ) ?'\n' :'\r' ); +fflush (stdout); }else if (!(flags &BLK_MIG_FLAG_EOS )) {fprintf (stderr,"Unknown flags\n" );return -EINVAL ;
PATCH v2 22/23 block migration: Add support for restore progress reporting
Inject progress report in percentage into the block live stream. This can be read out and displayed easily on restore.
---
Changes in v2:
- Print banner only if there is really some block device to restore
block-migration.c | 30 ++++++++++++++++++++++-------- 1 files changed, 22 insertions(+), 8 deletions(-)
block-migration.c +22 -8
@@ -23,6 +23,7 @@#define BLK_MIG_FLAG_DEVICE_BLOCK 0x01#define BLK_MIG_FLAG_EOS 0x02 +#define BLK_MIG_FLAG_PROGRESS 0x04#define MAX_IS_ALLOCATED_SEARCH 65536#define MAX_BLOCKS_READ 10000
@@ -70,7 +71,7 @@ typedef struct BlkMigState {int read_done;int transferred;int64_t total_sector_sum; -int64_t print_completion; +int prev_progress; } BlkMigState;static BlkMigState block_mig_state;
@@ -226,7 +227,7 @@ static void init_blk_migration(Monitor *mon, QEMUFile *f) block_mig_state.read_done =0 ; block_mig_state.transferred =0 ; block_mig_state.total_sector_sum =0 ; - block_mig_state.print_completion =0 ; + block_mig_state.prev_progress =-1 ;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {if (bs->type ==BDRV_TYPE_HD ) {
@@ -257,6 +258,7 @@ static int blk_mig_save_bulked_block(Monitor *mon, QEMUFile *f, int is_async) {int64_t completed_sector_sum =0 ;BlkMigDevState *bmds; +int progress;int ret =0 ; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
@@ -273,13 +275,13 @@ static int blk_mig_save_bulked_block(Monitor *mon, QEMUFile *f, int is_async) } } -if (completed_sector_sum >= block_mig_state.print_completion) { -monitor_printf (mon,"Completed %" PRId64" %%\r" , - completed_sector_sum *100 / - block_mig_state.total_sector_sum); + progress = completed_sector_sum *100 / block_mig_state.total_sector_sum; +if (progress != block_mig_state.prev_progress) { + block_mig_state.prev_progress = progress; +qemu_put_be64 (f, (progress <<BDRV_SECTOR_BITS ) + |BLK_MIG_FLAG_PROGRESS ); +monitor_printf (mon,"Completed %d %%\r" , progress);monitor_flush (mon); - block_mig_state.print_completion += - (BDRV_SECTORS_PER_DIRTY_CHUNK *10000 ); }return ret;
@@ -445,6 +447,9 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)blk_mig_save_dirty_blocks (mon, f);blk_mig_cleanup (mon); +/* report completion */ +qemu_put_be64 (f, (100 <<BDRV_SECTOR_BITS ) |BLK_MIG_FLAG_PROGRESS ); +if (qemu_file_has_error (f)) {return 0 ; }
@@ -459,6 +464,7 @@ static int block_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)static int block_load (QEMUFile *f,void *opaque,int version_id) { +static int banner_printed;int len, flags;char device_name[256 ];int64_t addr;
@@ -490,6 +496,14 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)bdrv_write (bs, addr, buf,BDRV_SECTORS_PER_DIRTY_CHUNK );qemu_free (buf); + }else if (flags &BLK_MIG_FLAG_PROGRESS ) { +if (!banner_printed) { +printf ("Receiving block device images\n" ); + banner_printed =1 ; + } +printf ("Completed %d %%%c" , (int )addr, + (addr ==100 ) ?'\n' :'\r' ); +fflush (stdout); }else if (!(flags &BLK_MIG_FLAG_EOS )) {fprintf (stderr,"Unknown flags\n" );return -EINVAL ;
[Qemu-devel] Re: [PATCH v2 22/23] block migration: Add support for restore progress reporting
Quoted 2 lines · Jan Kiszka
Inject progress report in percentage into the block live stream. This can be read out and displayed easily on restore.
I guess that this patch only reports percentage for the initial bulk copy of the image.
I haven't tested this scenario, but the next phase, sending dirty blocks, can be quite long too if the guest does a lot of I/O. Won't it give a wrong impression to the user when qemu says "Completed 100%" but disk migration continues catching up for a while?
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
[Qemu-devel] Re: [PATCH v2 22/23] block migration: Add support for restore progress reporting
Quoted 9 lines · Pierre Riteau
Quoted 2 lines · Jan Kiszka
Inject progress report in percentage into the block live stream. This can be read out and displayed easily on restore.
I guess that this patch only reports percentage for the initial bulk copy of the image.
I haven't tested this scenario, but the next phase, sending dirty blocks, can be quite long too if the guest does a lot of I/O. Won't it give a wrong impression to the user when qemu says "Completed 100%" but disk migration continues catching up for a while?
I does give a wrong impression (as there is also a wrong behavior) ATM.
But the plan is to update the number of pending blocks during the sync. Theoretically we could even go backwards with this progress value if (much) more blocks become dirty than we are able to write over a certain period.
Effectively, the total disk sizes increases during the migration due to dirty blocks being added. Instead of carrying this updated number over to the receiving side, I want to let the sender do the calculation and only transfer the result inside the stream (as this is only about visualization).
Jan
PS: Think I just found the e1000 migration issue, which turned out to affect all NICs.
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
PATCH 23/23 block migration: Increase dirty chunk size to 1M
In reply to Jan Kiszka4K is too small for efficiently saving and restoring multi-GB block devices.
---
block.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
block.h +1 -1
@@ -192,7 +192,7 @@ int bdrv_save_vmstate(BlockDriverState *bs, const uint8_t *buf,int bdrv_load_vmstate (BlockDriverState *bs,uint8_t *buf,int64_t pos,int size); -#define BDRV_SECTORS_PER_DIRTY_CHUNK 8 +#define BDRV_SECTORS_PER_DIRTY_CHUNK 2048void bdrv_set_dirty_tracking (BlockDriverState *bs,int enable);int bdrv_get_dirty (BlockDriverState *bs,int64_t sector);
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)
Quoted 8 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
That really needs to happen no later than the end of this week.
So Pierre/Liran, what do you think about Jan's series?
Regards,
Anthony Liguori
Quoted 35 lines · Anthony Liguori
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)Quoted 8 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
Quoted 7 lines
That really needs to happen no later than the end of this week.
So Pierre/Liran, what do you think about Jan's series?
Regards,
Anthony Liguori
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
- We use bdrv_aio_readv() to read blocks from the disk. This function increments rd_bytes and rd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.
- We may need to call bdrv_reset_dirty() _before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress = completed_sector_sum * 100 / block_mig_state.total_sector_sum;"
Apart from that, it works quite fine. Still a few things to cleanup (e.g. unused constants) but much better than before. However, I haven't tested the incremental transfer support at all yet. It's on my todo list.
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
Quoted 9 lines · Pierre Riteau
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
- We usebdrv_aio_readv()to read blocks from the disk. This function incrementsrd_bytesandrd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.
- We may need to callbdrv_reset_dirty()_before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress =completed_sector_sum* 100 / block_mig_state.total_sector_sum;"Apart from that, it works quite fine. Still a few things to cleanup (e.g. unused constants) but much better than before. However, I haven't tested the incremental transfer support at all yet. It's on my todo list.
BTW, of these four issues, only the last one is related to Jan changes. The others were there before. This is for saying that I strongly agree with the patch series.
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
[Qemu-devel] [PATCH 24/23] block migration: Skip zero-sized disks
Quoted 9 lines · Pierre Riteau
Quoted 7 lines · Pierre Riteau
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
- We usebdrv_aio_readv()to read blocks from the disk. This function incrementsrd_bytesandrd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.
- We may need to callbdrv_reset_dirty()_before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress =completed_sector_sum* 100 / block_mig_state.total_sector_sum;"
Quoted 6 lines
Quoted 2 lines
Apart from that, it works quite fine. Still a few things to cleanup (e.g. unused constants) but much better than before. However, I haven't tested the incremental transfer support at all yet. It's on my todo list.
BTW, of these four issues, only the last one is related to Jan changes. The others were there before. This is for saying that I strongly agree with the patch series.
That should catch it (yet untested, though).
Jan
----->
block migration: Skip zero-sized disks
No need to migrate emptiness (risking divide by zero later on).
---
block-migration.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-)
block-migration.c +8 -2
@@ -222,6 +222,7 @@ static void init_blk_migration(Monitor *mon, QEMUFile *f) {BlkMigDevState *bmds;BlockDriverState *bs; +int64_t sectors; block_mig_state.submitted =0 ; block_mig_state.read_done =0 ;
@@ -231,14 +232,19 @@ static void init_blk_migration(Monitor *mon, QEMUFile *f)for (bs = bdrv_first; bs !=NULL ; bs = bs->next) {if (bs->type ==BDRV_TYPE_HD ) { + sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; +if (sectors ==0 ) { +continue ; + } + bmds =qemu_mallocz (sizeof (BlkMigDevState)); bmds->bs = bs; bmds->bulk_completed =0 ; - bmds->total_sectors =bdrv_getlength (bs) >>BDRV_SECTOR_BITS ; + bmds->total_sectors = sectors; bmds->completed_sectors =0 ; bmds->shared_base = block_mig_state.shared_base; - block_mig_state.total_sector_sum += bmds->total_sectors; + block_mig_state.total_sector_sum += sectors;if (bmds->shared_base) { monitor_printf(mon, "Start migration for %s with shared base "
Quoted 40 lines · Pierre Riteau
Quoted 38 lines · Anthony Liguori
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)Quoted 8 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
That really needs to happen no later than the end of this week.
So Pierre/Liran, what do you think about Jan's series?
Quoted 8 lines
Quoted 3 lines
Regards,
Anthony Liguori
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
Saw that too. I think the monitor commands simply do very primitive option parsing so far. Should be addressed if the final format comes with this issue as well.
Quoted 1 line
- We use
bdrv_aio_readv()to read blocks from the disk. This function incrementsrd_bytesandrd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.
Ack.
Quoted 1 line
- We may need to call
bdrv_reset_dirty()_before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)
Can you elaborate? Even in case of multi-threaded qemu, the iomutex should protect us here.
Quoted 1 line
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress =
completed_sector_sum* 100 / block_mig_state.total_sector_sum;"
Although I don't see their use, it should be handled gracefully, likely by skipping such disks.
Quoted 4 lines
Apart from that, it works quite fine. Still a few things to cleanup (e.g. unused constants) but much better than before. However, I haven't tested the incremental transfer support at all yet. It's on my todo list.
Jan
Quoted 54 lines · Jan Kiszka
Quoted 40 lines · Pierre Riteau
Quoted 38 lines · Anthony Liguori
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)Quoted 8 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
That really needs to happen no later than the end of this week.
So Pierre/Liran, what do you think about Jan's series?
Quoted 8 lines
Quoted 3 lines
Regards,
Anthony Liguori
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
Saw that too. I think the monitor commands simply do very primitive option parsing so far. Should be addressed if the final format comes with this issue as well.
Quoted 8 lines
Quoted 1 line
- We use
bdrv_aio_readv()to read blocks from the disk. This function incrementsrd_bytesandrd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.Ack.
Quoted 1 line
- We may need to call
bdrv_reset_dirty()_before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)Can you elaborate? Even in case of multi-threaded qemu, the iomutex should protect us here.
I only said that because I remember seeing this kind of behavior, but with ram migration on kvm. As I'm not familiar with the I/O emulation in qemu, if you say that it's OK, no problem.
By multi-threaded, are you talking about the IO thread feature?
Quoted 4 lines
Quoted 1 line
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress =
completed_sector_sum* 100 / block_mig_state.total_sector_sum;"Although I don't see their use, it should be handled gracefully, likely by skipping such disks.
From a patch by Stefan Weil a few weeks ago:
Quoted 9 lines
Images with disk size 0 may be used for
VM snapshots, but not to save normal block data.It is possible to create such images using
qemu-img, but opening them later fails.So even "qemu-img info image.qcow2" is not
possible for an image created with
"qemu-img create -f qcow2 image.qcow2 0".
I'm not sure if that concerns us...
Signature · 1 line
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
Quoted 55 lines · Pierre Riteau
Quoted 53 lines · Jan Kiszka
Quoted 40 lines · Pierre Riteau
Quoted 38 lines · Anthony Liguori
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)Quoted 8 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
That really needs to happen no later than the end of this week.
So Pierre/Liran, what do you think about Jan's series?
Quoted 7 lines
Quoted 3 lines
Regards,
Anthony Liguori
I'm currently testing these patches. Here are a few issues I noticed, before I forget about them.
- "migrate -d -b tcp:dest:port" works, but "migrate -b -d tcp:dest:port" doesn't, although "help migrate" doesn't really specify ordering as important. But anyway I think Liran is working on a new version of the command.
Saw that too. I think the monitor commands simply do very primitive option parsing so far. Should be addressed if the final format comes with this issue as well.
Quoted 1 line
- We use
bdrv_aio_readv()to read blocks from the disk. This function incrementsrd_bytesandrd_ops, which are reported by "info blockstats". I don't think this read operations should appear in VM activity, especially if this interface is used by libvirt to report VM stats (and draw graphs in virt-manager, etc.). Same for write stats.
Quoted 8 lines
Quoted 5 lines
Ack.
Quoted 1 line
- We may need to call
bdrv_reset_dirty()_before_ sending the data, to be sure the block is not rewritten in the meantime (maybe it's an issue only with kvm?)Can you elaborate? Even in case of multi-threaded qemu, the iomutex should protect us here.
I only said that because I remember seeing this kind of behavior, but with ram migration on kvm. As I'm not familiar with the I/O emulation in qemu, if you say that it's OK, no problem.
RAM is different as RAM access need not be synchronized across the vcpus and the iothread.
Quoted 2 lines
By multi-threaded, are you talking about the IO thread feature?
Yes (which also includes per vcpu threads).
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
- I seem to remember that disk images with 0 size are now possible. I'm afraid we will hit a divide by zero in this case: "progress =
completed_sector_sum* 100 / block_mig_state.total_sector_sum;"Although I don't see their use, it should be handled gracefully, likely by skipping such disks.
From a patch by Stefan Weil a few weeks ago:
Quoted 3 lines
Images with disk size 0 may be used for
VM snapshots, but not to save normal block data.
Quoted 9 lines
Quoted 6 lines
It is possible to create such images using
qemu-img, but opening them later fails.So even "qemu-img info image.qcow2" is not
possible for an image created with
"qemu-img create -f qcow2 image.qcow2 0".I'm not sure if that concerns us...
Good point. Then my add-on patch is definitely required.
Jan
Quoted 26 lines · Anthony Liguori
Quoted 25 lines · Jan Kiszka
This series is a larger rework of the block migration support qemu recently gained. Besides lots of code refactorings the major changes are:
- Faster restore due to larger block sizes (even if the target disk is unallocated)
- Off-by-one fixes in the block dirty tracking code
- Allow for multiple migrations (after cancellation or if migrating into a backup image)
- Proper error handling
- Progress reporting fixes: report to monitor instead of stdout, report sum of multiple disks
- Report disk migration progress via 'info migrate'
- Progress report during restoreOne patch is directly taken from Pierre Riteau queue [1] who happend to work on the some topic the last days, two more are derived from his commits.
These patches make block migration usable for us. Still, there are two more major improvements on my wish/todo list:
- Respect specified maximum migration downtime (will require tracking of the number of dirty blocks + some coordination with ram migration)
- Do not transfere unallocated disk space (also for raw images, ie. addbdrv_is_allocatedsupport for the latter)
Quoted 4 lines
Quoted 4 lines
In an off-list chat, Liran additionally brought up the topic that RAM migration should not start too early so that we avoid re-transmitting dirty pages over and over again while the disk image is slowly beamed over.
Putting "slightly" more load on the guest while trying to migrate, this issue became a very visible one here. We definitely need some ordering between block and ram migration, but I'm not sure how to achieve this best.
Quoted 6 lines
Quoted 4 lines
I hope we can join our efforts to resolve the open topics quickly, the critical ones ideally before the merge window closes.
That really needs to happen no later than the end of this week.
Most critical is surely the vmstate format. From my current POV, none of the open issues should force us to change the format.
So when all goes wrong, we should even be able to fix remaining issues of this brand new feature inside the stable series. Still, the earlier we resolve the urging ones (migration order is now my favorite), the better.
Quoted 6 lines
So Pierre/Liran, what do you think about Jan's series?
Regards,
Anthony Liguori
Jan