COLO: integrate colo frame with block replication and COLO proxy
v10 cover COLO: integrate colo frame with block replication and COLO proxy
COLO Frame, block replication and COLO proxy(colo-compare,filter-mirror, filter-redirector,filter-rewriter) have been exist in qemu
for long time, it's time to integrate these three parts to make COLO really works.
In this series, we have some optimizations for COLO frame, including separating the process of saving ram and device state, using an COLO_EXIT event to notify users that VM exits COLO, for these parts, most of them have been reviewed long time ago in old version, but since this series have just rebased on upstream which had merged a new series of migration, parts of pathes in this series deserve review again.
We use notifier/callback method for COLO compare to notify COLO frame about net packets inconsistent event, and add a handle_event method for NetFilterClass to help COLO frame to notify filters and colo-compare about checkpoint/failover event, it is flexible.
For the neweset version, please refer to:
https://github.com/zhangckid/qemu/tree/qemu-colo-18jul22
Please review, thanks.
V10:
- Rebase on upstream.
- Remove the "active" in COLOState.
- Fix some comments.
V9:
- Rebased on upstream codes.
- Addressed Jason's comments add TCP state machine track in filter-rewriter.
- Fix some bug in colo-compare.
- Fix typo.
- Add filter-rewriter failover handle.
- Add net client type check in colo-compare.
- Add COLO state diagram.
- Addressed Markus and Daive's comments.
V8:
- Rebased on upstream codes.
- Addressed Markus's comments in patch 10/17.
- Addressed Markus's comments in patch 11/17.
- Removed some comments in patch 4/17.
- Moved the "migration_bitmap_clear_dirty()" to suitable position in patch 9/17.
- Rewrote the patch 07/17 to address Davie's comments.
- Moved the "qemu_savevm_live_state" out of the qemu_mutex_lock_iothread.
- Fixed the bug that in some status COLO vm crash with segmentation fault.
V7:
- Addressed Markus's comments in 11/17.
- Rebased on upstream.
V6:
- Addressed Eric Blake's comments, use the enum to feedback in patch 11/17.
- Fixed QAPI command separator problem in patch 11/17.
Zhang Chen (16):
filter-rewriter: Add TCP state machine and fix memory leak in
connection_track_table
colo-compare: implement the process of checkpoint
colo-compare: use notifier to notify packets comparing result
COLO: integrate colo compare with colo frame
COLO: Add block replication into colo process
COLO: Remove colo_state migration struct
COLO: Load dirty pages into SVM's RAM cache firstly ram/COLO: Record the dirty pages that SVM received
COLO: Flush memory data from ram cache qapi/migration.json: Rename COLO unknown mode to none mode.
qapi: Add new command to query colo status
savevm: split the process of different stages for loadvm/savevm net/net.c: Add net client type check function for COLO
filter: Add handle_event method for NetFilterClass
filter-rewriter: handle checkpoint and failover event
docs: Add COLO status diagram to COLO-FT.txt
zhanghailiang (4):
qmp event: Add COLO_EXIT event to notify users while exited COLO
COLO: flush host dirty ram from cache
COLO: notify net filters about checkpoint/failover event
COLO: quick failover process by kick COLO thread
docs/COLO-FT.txt | 34 ++++++ include/exec/ram_addr.h | 1 + include/migration/colo.h | 11 +- include/net/filter.h | 5 + include/net/net.h | 1 + migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 -------------- migration/colo-failover.c | 2 +- migration/colo.c | 212 ++++++++++++++++++++++++++++++++++++-- migration/migration.c | 44 +++++++- migration/ram.c | 163 ++++++++++++++++++++++++++++- migration/ram.h | 4 + migration/savevm.c | 53 ++++++++-- migration/savevm.h | 5 + migration/trace-events | 3 + net/colo-compare.c | 120 +++++++++++++++++++-- net/colo-compare.h | 24 +++++ net/colo.c | 10 +- net/colo.h | 11 +- net/filter-rewriter.c | 162 +++++++++++++++++++++++++++-- net/filter.c | 17 +++ net/net.c | 42 ++++++++ qapi/migration.json | 80 +++++++++++++- vl.c | 2 - 24 files changed, 945 insertions(+), 139 deletions(-) delete mode 100644 migration/colo-comm.c create mode 100644 net/colo-compare.h
Signature · 1 line
2.17.1
PATCH v10 1/20 filter-rewriter: Add TCP state machine and fix memory leak in connection_track_table
We add almost full TCP state machine in filter-rewriter, except TCPS_LISTEN and some simplify in VM active close FIN states.
After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak.
Let't track the state of net connection, if it is closed, its related resources will be cleared up.
--- net/colo.c | 2 +- net/colo.h | 9 ++-- net/filter-rewriter.c | 105 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 100 insertions(+), 16 deletions(-)
net/colo.c +1 -1
@@ -137,7 +137,7 @@ Connection *connection_new(ConnectionKey *key) conn->ip_proto = key->ip_proto; conn->processing = false; conn->offset =0 ; - conn->syn_flag =0 ; + conn->tcp_state =TCPS_CLOSED ; conn->pack =0 ; conn->sack =0 ;g_queue_init (&conn->primary_list);
net/colo.h +4 -5
@@ -18,6 +18,7 @@#include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384
@@ -81,11 +82,9 @@ typedef struct Connection {uint32_t sack;/* offset = secondary_seq - primary_seq */ tcp_seq offset; -/* -* we use this flag update offset func -* run once in independent tcp connection -*/ -int syn_flag; + +int tcp_state;/* TCP FSM state */ +tcp_seq fin_ack_seq;/* the seq of 'fin=1,ack=1' */ } Connection;uint32_t connection_key_hash (const void *opaque);
net/filter-rewriter.c +95 -10
175 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v10 2/20 colo-compare: implement the process of checkpoint
In reply to Zhang ChenWhile do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine.
--- include/migration/colo.h | 6 ++++ net/colo-compare.c | 78 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 net/colo-compare.h
include/migration/colo.h +6 -0
@@ -16,6 +16,12 @@#include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { +COLO_EVENT_NONE , +COLO_EVENT_CHECKPOINT , +COLO_EVENT_FAILOVER , +}; +void colo_info_init (void );void migrate_start_colo_process (MigrationState *s);
net/colo-compare.c +78 -0
145 lines not on this page · raw message
net/colo-compare.h new file +22 -0
@@ -0,0 +1,22 @@ +/* +* COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) +* (a.k.a. Fault Tolerance or Continuous Replication) +* +* Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. +* Copyright (c) 2017 FUJITSU LIMITED +* Copyright (c) 2017 Intel Corporation +* +* Authors: +* zhanghailiang <[email protected]> +* Zhang Chen <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or +* later. See the COPYING file in the top-level directory. +*/ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event (void *opaque,int event,Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */
Signature · 1 line
2.17.1
PATCH v10 3/20 colo-compare: use notifier to notify packets comparing result
In reply to Zhang ChenIt's a good idea to use notifier to notify COLO frame of inconsistent packets comparing.
--- net/colo-compare.c | 37 ++++++++++++++++++++++++++----------- net/colo-compare.h | 2 ++ 2 files changed, 28 insertions(+), 11 deletions(-)
net/colo-compare.c +26 -11
@@ -29,6 +29,7 @@#include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare"#define COLO_COMPARE (obj) \
@@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =QTAILQ_HEAD_INITIALIZER (net_compares); +static NotifierList colo_compare_notifiers = +NOTIFIER_LIST_INITIALIZER (colo_compare_notifiers); +#define COMPARE_READ_LEN_MAX NET_BUFSIZE#define MAX_QUEUE_SIZE 1024
@@ -427,10 +431,7 @@ sec:qemu_hexdump ((char *)spkt->data, stderr,"colo-compare spkt" , spkt->size); -/* -* colo_compare_inconsistent_notify(); -* TODO: notice to checkpoint(); -*/ +colo_compare_inconsistency_notify (); } }
@@ -561,8 +562,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistency_notify (void ) +{ +notifier_list_notify (&colo_compare_notifiers, +migrate_get_current ()); +} + +void colo_compare_register_notifier (Notifier *notify) +{ +notifier_list_add (&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier (Notifier *notify) +{ +notifier_remove (notify); +} +static int colo_old_packet_check_one_conn (Connection *conn, -void *user_data) +void *user_data) {GList *result =NULL ;int64_t check_time =REGULAR_PACKET_CHECK_MS ;
@@ -573,10 +590,7 @@ static int colo_old_packet_check_one_conn(Connection *conn,if (result) {/* Do checkpoint will flush old packet */ -/* -* TODO: Notify colo frame to do checkpoint. -* colo_compare_inconsistent_notify(); -*/ +colo_compare_inconsistency_notify ();return 0 ; }
@@ -620,11 +634,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn,/* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it -* until next comparison. +* until next comparison. If the packets in the list are +* timeout, it will trigger a checkpoint request. */ trace_colo_compare_main ("packet different" );g_queue_push_head (&conn->primary_list, pkt); -/* TODO: colo_notify_checkpoint();*/ +colo_compare_inconsistency_notify ();break ; } }
net/colo-compare.h +2 -0
@@ -18,5 +18,7 @@#define QEMU_COLO_COMPARE_H void colo_notify_compares_event (void *opaque,int event,Error **errp); +void colo_compare_register_notifier (Notifier *notify); +void colo_compare_unregister_notifier (Notifier *notify);#endif /* QEMU_COLO_COMPARE_H */
Signature · 1 line
2.17.1
PATCH v10 4/20 COLO: integrate colo compare with colo frame
In reply to Zhang ChenFor COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs.
So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100.
Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here.
--- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 6 ++---- 2 files changed, 42 insertions(+), 6 deletions(-)
migration/colo.c +40 -2
@@ -25,8 +25,11 @@#include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier;#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } +colo_notify_compares_event (NULL ,COLO_EVENT_CHECKPOINT , &local_err); +if (local_err) { + goto out; + } +/* Disable block migration */ migrate_set_block_enabled (false, &local_err);qemu_savevm_state_header (fb);
@@ -400,6 +408,11 @@ out:return ret; } +static void colo_compare_notify_checkpoint (Notifier *notifier,void *data) +{ +colo_checkpoint_notify (data); +} +static void colo_process_checkpoint (MigrationState *s) {QIOChannelBuffer *bioc;
@@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; +colo_compare_register_notifier (&packets_compare_notifier); +/* * Wait for Secondary finish loading VM states and enter COLO * restore.
@@ -461,11 +477,21 @@ out:qemu_fclose (fb); } -timer_del (s->colo_delay_timer); -/* Hope this not to be too long to wait here */ qemu_sem_wait (&s->colo_exit_sem);qemu_sem_destroy (&s->colo_exit_sem); + +/* +* It is safe to unregister notifier after failover finished. +* Besides, colo_delay_timer and colo_checkpoint_sem can't be +* released befor unregister notifier, or there will be use-after-free +* error. +*/ +colo_compare_unregister_notifier (&packets_compare_notifier); +timer_del (s->colo_delay_timer); +timer_free (s->colo_delay_timer); +qemu_sem_destroy (&s->colo_checkpoint_sem); +/* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that
@@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb =qemu_fopen_channel_input (QIO_CHANNEL (bioc));object_unref (OBJECT (bioc)); +qemu_mutex_lock_iothread (); +vm_start (); +trace_colo_vm_state_change ("stop" ,"run" ); +qemu_mutex_unlock_iothread (); +colo_send_message (mis->to_src_file,COLO_MESSAGE_CHECKPOINT_READY , &local_err);if (local_err) {
@@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +qemu_mutex_lock_iothread (); +vm_stop_force_state (RUN_STATE_COLO ); +trace_colo_vm_state_change ("run" ,"stop" ); +qemu_mutex_unlock_iothread (); +/* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message (mis->to_src_file,COLO_MESSAGE_CHECKPOINT_REPLY , &local_err);
@@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; +vm_start (); +trace_colo_vm_state_change ("stop" ,"run" );qemu_mutex_unlock_iothread ();if (failover_get_state () ==FAILOVER_STATUS_RELAUNCH ) {
migration/migration.c +2 -4
@@ -75,10 +75,8 @@/* Migration XBZRLE default cache size */ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) -/* The delay time (in ms) between two COLO checkpoints -* Note: Please change this default value to 10000 when we support hybrid mode. -*/ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +/* The delay time (in ms) between two COLO checkpoints */ +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100)#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2#define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16
Signature · 1 line
2.17.1
PATCH v10 5/20 COLO: Add block replication into colo process
In reply to Zhang ChenMake sure master start block replication after slave's block replication started.
Besides, we need to activate VM's blocks before goes into COLO state.
--- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+)
migration/colo.c +43 -0
@@ -27,6 +27,7 @@#include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading;static Notifier packets_compare_notifier;
@@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) {int old_state;MigrationIncomingState *mis =migration_incoming_get_current (); +Error *local_err =NULL ;/* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM.
@@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void)migrate_set_state (&mis->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED ); +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + } +if (!autostart) {error_report ("\"-S\" qemu option will be ignored in secondary side" );/* recover runstate to normal migration finish state */
@@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) {MigrationState *s =migrate_get_current ();int old_state; +Error *local_err =NULL ;migrate_set_state (&s->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED );
@@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state));return ; } + +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + local_err =NULL ; + } +/* Notify COLO thread that failover work is finished */ qemu_sem_post (&s->colo_exit_sem); }
@@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s,qemu_savevm_state_header (fb);qemu_savevm_state_setup (fb);qemu_mutex_lock_iothread (); +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }qemu_savevm_state_complete_precopy (fb, false, false);qemu_mutex_unlock_iothread ();
@@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_PRIMARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +vm_start ();qemu_mutex_unlock_iothread ();trace_colo_vm_state_change ("stop" ,"run" );
@@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_SECONDARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }vm_start ();trace_colo_vm_state_change ("stop" ,"run" );qemu_mutex_unlock_iothread ();
@@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +replication_get_error_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +/* discard colo disk buffer */ +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } + vmstate_loading = false;vm_start ();trace_colo_vm_state_change ("stop" ,"run" );
migration/migration.c +9 -0
@@ -385,6 +385,7 @@ static void process_incoming_migration_co(void *opaque)MigrationIncomingState *mis =migration_incoming_get_current ();PostcopyState ps;int ret; +Error *local_err =NULL ;assert (mis->from_src_file); mis->largest_page_size =qemu_ram_pagesize_largest ();
@@ -416,6 +417,14 @@ static void process_incoming_migration_co(void *opaque)/* we get COLO info, and know if we are in COLO mode */ if (!ret &&migration_incoming_enable_colo ()) { +/* Make sure all file formats flush their mutable metadata */ +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE , +MIGRATION_STATUS_FAILED ); +error_report_err (local_err); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE );
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH V10 05/20] COLO: Add block replication into colo process
Quoted 69 lines · * Zhang Chen ([email protected]) wrote:
Make sure master start block replication after slave's block replication started.
Besides, we need to activate VM's blocks before goes into COLO state.
Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>--- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+)migration/colo.c +15
-0@@ -27,6 +27,7 @@#include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading;static Notifier packets_compare_notifier;@@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) {int old_state;MigrationIncomingState *mis =migration_incoming_get_current (); +Error *local_err =NULL ;/* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void)migrate_set_state (&mis->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED ); +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + } +if (!autostart) {error_report ("\"-S\" qemu option will be ignored in secondary side" );/* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) {MigrationState *s =migrate_get_current ();int old_state; +Error *local_err =NULL ;migrate_set_state (&s->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED );@@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state));return ; } + +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + local_err =NULL ; + } +/* Notify COLO thread that failover work is finished */ qemu_sem_post (&s->colo_exit_sem); }
migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 378
Quoted 8 lines
migration/colo.c +5
-0@@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s,qemu_savevm_state_header (fb);qemu_savevm_state_setup (fb);qemu_mutex_lock_iothread (); +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }
In docs/block-replication.txt it says:
b. replication_do_checkpoint_all()
This interface is called after all VM state is transferred to Secondary QEMU. The Disk buffer will be dropped in this interface. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread.
but we're making this call before the call below that actually transfers all the state. Which is right?
Other than that I think it's OK.
Dave
Quoted 77 lines
migration/colo.c +23
-0qemu_savevm_state_complete_precopy (fb, false, false);qemu_mutex_unlock_iothread ();@@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_PRIMARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +vm_start ();qemu_mutex_unlock_iothread ();trace_colo_vm_state_change ("stop" ,"run" );@@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_SECONDARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }vm_start ();trace_colo_vm_state_change ("stop" ,"run" );qemu_mutex_unlock_iothread ();@@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +replication_get_error_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +/* discard colo disk buffer */ +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } + vmstate_loading = false;vm_start ();trace_colo_vm_state_change ("stop" ,"run" );migration/migration.c +9
-0@@ -385,6 +385,7 @@ static void process_incoming_migration_co(void *opaque)MigrationIncomingState *mis =migration_incoming_get_current ();PostcopyState ps;int ret; +Error *local_err =NULL ;assert (mis->from_src_file); mis->largest_page_size =qemu_ram_pagesize_largest ();@@ -416,6 +417,14 @@ static void process_incoming_migration_co(void *opaque)/* we get COLO info, and know if we are in COLO mode */ if (!ret &&migration_incoming_enable_colo ()) { +/* Make sure all file formats flush their mutable metadata */ +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE , +MIGRATION_STATUS_FAILED ); +error_report_err (local_err); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE );Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 05/20] COLO: Add block replication into colo process
Quoted 89 lines · Dr. David Alan Gilbert
Quoted 27 lines · * Zhang Chen ([email protected]) wrote:
Make sure master start block replication after slave's block replication started.
Besides, we need to activate VM's blocks before goes into COLO state.
Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>--- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+)migration/colo.c +1
-0@@ -27,6 +27,7 @@#include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading;static Notifier packets_compare_notifier;migration/colo.c static void secondary_vm_do_failover(void) · line 62
Quoted 6 lines
migration/colo.c +1
-0@@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) {int old_state;MigrationIncomingState *mis =migration_incoming_get_current (); +Error *local_err =NULL ;/* Can not do failover during the process of VM's loading VMstate, Or
migration/colo.c static void secondary_vm_do_failover(void) · line 84
Quoted 12 lines
migration/colo.c +5
-0* it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void)migrate_set_state (&mis->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED ); +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + } +if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondaryside");
migration/colo.c static int
Quoted 24 lines
migration/colo.c +8
-0/* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) {MigrationState *s =migrate_get_current ();int old_state; +Error *local_err =NULL ;migrate_set_state (&s->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED );@@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state));return ; } + +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + local_err =NULL ; + } +/* Notify COLO thread that failover work is finished */ qemu_sem_post (&s->colo_exit_sem); }@@ -356,6 +371,11 @@ static intcolo_do_checkpoint_transaction(MigrationState *s,
migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 378
Quoted 8 lines
migration/colo.c +5
-0qemu_savevm_state_header (fb);qemu_savevm_state_setup (fb);qemu_mutex_lock_iothread (); +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }In
docs/block-replication.txtit says:
b.replication_do_checkpoint_all()
This interface is called after all VM state is transferred to Secondary QEMU. The Disk buffer will be dropped in this interface. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread.
Quoted 3 lines
but we're making this call before the call below that actually transfers all the state. Which is right?
Hi Dave,
The "docs/block-replication.txt" means we should call the replication_do_checkpoint_all() after VM state is transferred in secondary node,
and in primary node this function no need to call before transfers all the state.
Thanks
Zhang Chen
Quoted 88 lines
Other than that I think it's OK.
Dave
migration/colo.c static void colo_process_checkpoint(MigrationState
Quoted 4 lines
migration/colo.c +0
-0qemu_savevm_state_complete_precopy (fb, false, false);qemu_mutex_unlock_iothread ();@@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState*s)
migration/migration.c static void process_incoming_migration_co(void
Quoted 48 lines
migration/colo.c +23
-0object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_PRIMARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +vm_start ();qemu_mutex_unlock_iothread ();trace_colo_vm_state_change ("stop" ,"run" );@@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_SECONDARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }vm_start ();trace_colo_vm_state_change ("stop" ,"run" );qemu_mutex_unlock_iothread ();@@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +replication_get_error_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +/* discard colo disk buffer */ +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } + vmstate_loading = false;vm_start ();trace_colo_vm_state_change ("stop" ,"run" );migration/migration.c +0
-0@@ -385,6 +385,7 @@ static void process_incoming_migration_co(void*opaque)
migration/migration.c static void process_incoming_migration_co(void
Quoted 8 lines
migration/migration.c +1
-0MigrationIncomingState *mis =migration_incoming_get_current ();PostcopyState ps;int ret; +Error *local_err =NULL ;assert (mis->from_src_file); mis->largest_page_size =qemu_ram_pagesize_largest ();@@ -416,6 +417,14 @@ static void process_incoming_migration_co(void*opaque)
Quoted 15 lines
migration/migration.c +8
-1/* we get COLO info, and know if we are in COLO mode */ if (!ret &&migration_incoming_enable_colo ()) { +/* Make sure all file formats flush their mutable metadata */ +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE , +MIGRATION_STATUS_FAILED ); +error_report_err (local_err); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE ); --Quoted 2 lines
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 05/20] COLO: Add block replication into colo process
Quoted 96 lines · * Zhang Chen ([email protected]) wrote:
Quoted 89 lines · Dr. David Alan Gilbert
Quoted 27 lines · * Zhang Chen ([email protected]) wrote:
Make sure master start block replication after slave's block replication started.
Besides, we need to activate VM's blocks before goes into COLO state.
Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>--- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+)migration/colo.c +1
-0@@ -27,6 +27,7 @@#include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading;static Notifier packets_compare_notifier;migration/colo.c static void secondary_vm_do_failover(void) · line 62
Quoted 6 lines
migration/colo.c +1
-0@@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) {int old_state;MigrationIncomingState *mis =migration_incoming_get_current (); +Error *local_err =NULL ;/* Can not do failover during the process of VM's loading VMstate, Or
migration/colo.c static void secondary_vm_do_failover(void) · line 84
Quoted 12 lines
migration/colo.c +5
-0* it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void)migrate_set_state (&mis->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED ); +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + } +if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondaryside");
migration/colo.c static int
Quoted 24 lines
migration/colo.c +8
-0/* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) {MigrationState *s =migrate_get_current ();int old_state; +Error *local_err =NULL ;migrate_set_state (&s->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED );@@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state));return ; } + +replication_stop_all (true, &local_err); +if (local_err) { +error_report_err (local_err); + local_err =NULL ; + } +/* Notify COLO thread that failover work is finished */ qemu_sem_post (&s->colo_exit_sem); }@@ -356,6 +371,11 @@ static intcolo_do_checkpoint_transaction(MigrationState *s,
migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 378
Quoted 8 lines
migration/colo.c +5
-0qemu_savevm_state_header (fb);qemu_savevm_state_setup (fb);qemu_mutex_lock_iothread (); +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }In
docs/block-replication.txtit says:
b.replication_do_checkpoint_all()
This interface is called after all VM state is transferred to Secondary QEMU. The Disk buffer will be dropped in this interface. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread.Quoted 3 lines
but we're making this call before the call below that actually transfers all the state. Which is right?
Quoted 7 lines
Hi Dave,
The "
docs/block-replication.txt" means we should call thereplication_do_checkpoint_all()after VM state is transferred in secondary node,
and in primary node this function no need to call before transfers all the state.
OK, it might be worth clarifying the docs.
Dave
Quoted 93 lines
Thanks
Zhang ChenQuoted 88 lines
Other than that I think it's OK.
Dave
migration/colo.c static void colo_process_checkpoint(MigrationState
Quoted 4 lines
migration/colo.c +0
-0qemu_savevm_state_complete_precopy (fb, false, false);qemu_mutex_unlock_iothread ();@@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState*s)
migration/migration.c static void process_incoming_migration_co(void
Quoted 48 lines
migration/colo.c +23
-0object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_PRIMARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +vm_start ();qemu_mutex_unlock_iothread ();trace_colo_vm_state_change ("stop" ,"run" );@@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque)object_unref (OBJECT (bioc));qemu_mutex_lock_iothread (); +replication_start_all (REPLICATION_MODE_SECONDARY , &local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + }vm_start ();trace_colo_vm_state_change ("stop" ,"run" );qemu_mutex_unlock_iothread ();@@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +replication_get_error_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +/* discard colo disk buffer */ +replication_do_checkpoint_all (&local_err); +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } + vmstate_loading = false;vm_start ();trace_colo_vm_state_change ("stop" ,"run" );migration/migration.c +0
-0@@ -385,6 +385,7 @@ static void process_incoming_migration_co(void*opaque)
migration/migration.c static void process_incoming_migration_co(void
Quoted 8 lines
migration/migration.c +1
-0MigrationIncomingState *mis =migration_incoming_get_current ();PostcopyState ps;int ret; +Error *local_err =NULL ;assert (mis->from_src_file); mis->largest_page_size =qemu_ram_pagesize_largest ();@@ -416,6 +417,14 @@ static void process_incoming_migration_co(void*opaque)
Quoted 15 lines
migration/migration.c +8
-1/* we get COLO info, and know if we are in COLO mode */ if (!ret &&migration_incoming_enable_colo ()) { +/* Make sure all file formats flush their mutable metadata */ +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE , +MIGRATION_STATUS_FAILED ); +error_report_err (local_err); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE ); --Quoted 2 lines
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v10 6/20 COLO: Remove colo_state migration struct
In reply to Zhang ChenWe need to know if migration is going into COLO state for incoming side before start normal migration.
Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not.
--- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 17 +++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 57 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c
include/migration/colo.h +3 -2
@@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s);bool migration_in_colo_state (void );/* loadvm */ -bool migration_incoming_enable_colo (void ); -void migration_incoming_exit_colo (void ); +void migration_incoming_enable_colo (void ); +void migration_incoming_disable_colo (void ); +bool migration_incoming_colo_enabled (void );void *colo_process_incoming_thread (void *opaque);bool migration_incoming_in_colo_state (void );
migration/Makefile.objs +1 -1
@@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o
migration/colo-comm.c deleted +0 -76
76 lines not on this page · raw message
migration/colo.c +12 -1
@@ -152,6 +152,17 @@ static void primary_vm_do_failover(void)qemu_sem_post (&s->colo_exit_sem); } +COLOMode get_colo_mode (void ) +{ +if (migration_in_colo_state ()) { +return COLO_MODE_PRIMARY ; + }else if (migration_incoming_in_colo_state ()) { +return COLO_MODE_SECONDARY ; + }else { +return COLO_MODE_UNKNOWN ; + } +} +void colo_do_failover (MigrationState *s) {/* Make sure VM stopped while failover happened. */
@@ -745,7 +756,7 @@ out:if (mis->to_src_file) {qemu_fclose (mis->to_src_file); } -migration_incoming_exit_colo (); +migration_incoming_disable_colo ();return NULL ; }
migration/migration.c +22 -1
@@ -295,6 +295,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname,return migrate_send_rp_message (mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled (void ) +{ +return migration_colo_enabled; +} + +void migration_incoming_disable_colo (void ) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo (void ) +{ + migration_colo_enabled = true; +} +void qemu_start_incoming_migration (const char *uri,Error **errp) {const char *p;
@@ -416,7 +432,7 @@ static void process_incoming_migration_co(void *opaque) }/* we get COLO info, and know if we are in COLO mode */ -if (!ret &&migration_incoming_enable_colo ()) { +if (!ret &&migration_incoming_colo_enabled ()) {/* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all (&local_err);if (local_err) {
@@ -2950,6 +2966,11 @@ static void *migration_thread(void *opaque)qemu_savevm_send_postcopy_advise (s->to_dst_file); } +if (migrate_colo_enabled ()) { +/* Notify migration destination that we enable COLO */ +qemu_savevm_send_colo_enable (s->to_dst_file); + } +qemu_savevm_state_setup (s->to_dst_file); s->setup_time =qemu_clock_get_ms (QEMU_CLOCK_HOST ) - setup_start;
migration/savevm.c +17 -0
@@ -56,6 +56,7 @@#include "io/channel-file.h" #include "sysemu/replay.h" #include "qjson.h" +#include "migration/colo.h" #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035
@@ -82,6 +83,7 @@ enum qemu_vm_cmd {were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED ,/* Send a wrapped stream within this stream */ +MIG_CMD_ENABLE_COLO ,/* Enable COLO */ MIG_CMD_POSTCOPY_RESUME ,/* resume postcopy on dest */ MIG_CMD_RECV_BITMAP ,/* Request for recved bitmap on dst */ MIG_CMD_MAX
@@ -841,6 +843,12 @@ static void qemu_savevm_command_send(QEMUFile *f,qemu_fflush (f); } +void qemu_savevm_send_colo_enable (QEMUFile *f) +{ +trace_savevm_send_colo_enable (); +qemu_savevm_command_send (f,MIG_CMD_ENABLE_COLO ,0 ,NULL ); +} +void qemu_savevm_send_ping (QEMUFile *f,uint32_t value) {uint32_t buf;
@@ -1918,6 +1926,12 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis,return 0 ; } +static int loadvm_process_enable_colo (MigrationIncomingState *mis) +{ +migration_incoming_enable_colo (); +return 0 ; +} +/* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return
@@ -1997,6 +2011,9 @@ static int loadvm_process_command(QEMUFile *f)case MIG_CMD_RECV_BITMAP :return loadvm_handle_recv_bitmap (mis, len); + +case MIG_CMD_ENABLE_COLO : +return loadvm_process_enable_colo (mis); }return 0 ;
migration/savevm.h +1 -0
@@ -55,6 +55,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f);void qemu_loadvm_state_cleanup (void );
migration/trace-events +1 -0
@@ -37,6 +37,7 @@ savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" +savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" savevm_state_setup(void) "" savevm_state_resume_prepare(void) ""
vl.c +0 -2
@@ -4392,8 +4392,6 @@ int main(int argc, char **argv, char **envp)#endif } -colo_info_init (); -if (net_init_clients (&err) <0 ) {error_report_err (err);exit (1 );
Signature · 1 line
2.17.1
PATCH v10 7/20 COLO: Load dirty pages into SVM's RAM cache firstly
In reply to Zhang ChenWe should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM.
We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state.
--- include/exec/ram_addr.h | 1 + migration/migration.c | 6 +++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 93 insertions(+), 3 deletions(-)
include/exec/ram_addr.h +1 -0
@@ -27,6 +27,7 @@ struct RAMBlock {struct rcu_head rcu;struct MemoryRegion *mr;uint8_t *host; +uint8_t *colo_cache;/* For colo, VM's ram cache */ ram_addr_t offset;ram_addr_t used_length;ram_addr_t max_length;
migration/migration.c +6 -0
@@ -441,6 +441,10 @@ static void process_incoming_migration_co(void *opaque)error_report_err (local_err);exit (EXIT_FAILURE ); } +if (colo_init_ram_cache () <0 ) { +error_report ("Init ram cache failed" ); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE );
@@ -449,6 +453,8 @@ static void process_incoming_migration_co(void *opaque)/* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join (&mis->colo_incoming_thread); +/* We hold the global iothread lock, so it is safe here */ +colo_release_ram_cache (); }if (ret <0 ) {
migration/ram.c +81 -2
118 lines not on this page · raw message
migration/ram.h +4 -0
@@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name);int ram_dirty_bitmap_reload (MigrationState *s,RAMBlock *rb); +/* ram cache */ +int colo_init_ram_cache (void ); +void colo_release_ram_cache (void ); +#endif
migration/savevm.c +1 -1
@@ -1929,7 +1929,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis,static int loadvm_process_enable_colo (MigrationIncomingState *mis) {migration_incoming_enable_colo (); -return 0 ; +return colo_init_ram_cache (); }/*
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH V10 07/20] COLO: Load dirty pages into SVM's RAM cache firstly
Quoted 10 lines · * Zhang Chen ([email protected]) wrote:
We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM.
We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state.
Quoted 3 lines
Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>
As mentioned before you do need to update this to skip non-migratable blocks; there are also two blank-lines inserted in this patch which should just go in the appropriate patch.
However, with those minor things fixed:
Quoted 200 lines
--- include/exec/ram_addr.h | 1 + migration/migration.c | 6 +++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 93 insertions(+), 3 deletions(-)include/exec/ram_addr.h +1
-0@@ -27,6 +27,7 @@ struct RAMBlock {struct rcu_head rcu;struct MemoryRegion *mr;uint8_t *host; +uint8_t *colo_cache;/* For colo, VM's ram cache */ ram_addr_t offset;ram_addr_t used_length;ram_addr_t max_length;migration/migration.c +6
-0@@ -441,6 +441,10 @@ static void process_incoming_migration_co(void *opaque)error_report_err (local_err);exit (EXIT_FAILURE ); } +if (colo_init_ram_cache () <0 ) { +error_report ("Init ram cache failed" ); +exit (EXIT_FAILURE ); + } mis->migration_incoming_co =qemu_coroutine_self ();qemu_thread_create (&mis->colo_incoming_thread,"COLO incoming" , colo_process_incoming_thread, mis,QEMU_THREAD_JOINABLE );@@ -449,6 +453,8 @@ static void process_incoming_migration_co(void *opaque)/* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join (&mis->colo_incoming_thread); +/* We hold the global iothread lock, so it is safe here */ +colo_release_ram_cache (); }if (ret <0 ) {migration/ram.c +81
-2@@ -3314,6 +3314,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,return block->host + offset; } +static inline void *colo_cache_from_block_offset (RAMBlock *block, +ram_addr_t offset) +{ +if (!offset_in_ramblock (block, offset)) { +return NULL ; + } +if (!block->colo_cache) { +error_report ("%s: colo_cache is NULL in block :%s" , + __func__, block->idstr); +return NULL ; + } +return block->colo_cache + offset; +} +/** * ram_handle_compressed: handle the zero page case * @@ -3518,6 +3532,58 @@ static void decompress_data_with_multi_threads(QEMUFile *f,qemu_mutex_unlock (&decomp_done_lock); } +/* +* colo cache: this is for secondary VM, we cache the whole +* memory of the secondary VM, it is need to hold the global lock +* to call this helper. +*/ +int colo_init_ram_cache (void ) +{ +RAMBlock *block; + +rcu_read_lock (); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =qemu_anon_ram_alloc (block->used_length, +NULL , + false); +if (!block->colo_cache) { +error_report ("%s: Can't alloc memory for COLO cache of block %s," +"size 0x" RAM_ADDR_FMT , __func__, block->idstr, + block->used_length); + goto out_locked; + } +memcpy (block->colo_cache, block->host, block->used_length); + } +rcu_read_unlock (); +return 0 ; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +if (block->colo_cache) { +qemu_anon_ram_free (block->colo_cache, block->used_length); + block->colo_cache =NULL ; + } + } + +rcu_read_unlock (); +return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache (void ) +{ +RAMBlock *block; + +rcu_read_lock (); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +if (block->colo_cache) { +qemu_anon_ram_free (block->colo_cache, block->used_length); + block->colo_cache =NULL ; + } + } +rcu_read_unlock (); +} +/** * ram_load_setup: Setup RAM for migration incoming side * @@ -3534,6 +3600,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque)xbzrle_load_setup ();ramblock_recv_map_init (); +return 0 ; }@@ -3547,6 +3614,7 @@ static int ram_load_cleanup(void *opaque)g_free (rb->receivedmap); rb->receivedmap =NULL ; } +return 0 ; }@@ -3784,13 +3852,24 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {RAMBlock *block =ram_block_from_stream (f, flags); - host =host_from_ram_block_offset (block, addr); +/* +* After going into COLO, we should load the Page into colo_cache. +*/ +if (migration_incoming_in_colo_state ()) { + host =colo_cache_from_block_offset (block, addr); + }else { + host =host_from_ram_block_offset (block, addr); + }if (!host) {error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); ret = -EINVAL ;break ; } -ramblock_recv_bitmap_set (block, host); + +if (!migration_incoming_in_colo_state ()) { +ramblock_recv_bitmap_set (block, host); + } +trace_ram_load_loop (block->idstr, (uint64_t )addr, flags, host); }migration/ram.h +4
-0@@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name);int ram_dirty_bitmap_reload (MigrationState *s,RAMBlock *rb); +/* ram cache */ +int colo_init_ram_cache (void ); +void colo_release_ram_cache (void ); +#endif migration/savevm.c +1
-1@@ -1929,7 +1929,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis,static int loadvm_process_enable_colo (MigrationIncomingState *mis) {migration_incoming_enable_colo (); -return 0 ; +return colo_init_ram_cache (); }/* Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v10 8/20 ram/COLO: Record the dirty pages that SVM received
In reply to Zhang ChenWe record the address of the dirty pages that received, it will help flushing pages that cached into SVM.
Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint.
--- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
migration/ram.c +33 -0
@@ -3325,6 +3325,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr);return NULL ; } + +/* +* During colo checkpoint, we need bitmap of these migrated pages. +* It help us to decide which pages in ram cache should be flushed +* into VM's RAM later. +*/ +if (!test_and_set_bit (offset >>TARGET_PAGE_BITS , block->bmap)) { + ram_state->migration_dirty_pages++; + }return block->colo_cache + offset; }
@@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void)memcpy (block->colo_cache, block->host, block->used_length); }rcu_read_unlock (); +/* +* Record the dirty pages that sent by PVM, we use this dirty bitmap together +* with to decide which page in cache should be flushed into SVM's RAM. Here +* we use the same name 'ram_bitmap' as for migration. +*/ +if (ram_bytes_total ()) { +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +unsigned long pages = block->max_length >>TARGET_PAGE_BITS ; + + block->bmap =bitmap_new (pages); +bitmap_set (block->bmap,0 , pages); + } + } + ram_state =g_new0 (RAMState,1 ); + ram_state->migration_dirty_pages =0 ; +return 0 ; out_locked:
@@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) {RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +g_free (block->bmap); + block->bmap =NULL ; + }rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {
@@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } }rcu_read_unlock (); +g_free (ram_state); + ram_state =NULL ; }/**
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH V10 08/20] ram/COLO: Record the dirty pages that SVM received
Quoted 38 lines · * Zhang Chen ([email protected]) wrote:
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM.
Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint.
Signed-off-by zhanghailiang <[email protected]>Reviewed-by Dr. David Alan Gilbert <[email protected]>--- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)migration/ram.c +9
-0@@ -3325,6 +3325,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr);return NULL ; } + +/* +* During colo checkpoint, we need bitmap of these migrated pages. +* It help us to decide which pages in ram cache should be flushed +* into VM's RAM later. +*/ +if (!test_and_set_bit (offset >>TARGET_PAGE_BITS , block->bmap)) { + ram_state->migration_dirty_pages++; + }return block->colo_cache + offset; }@@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void)memcpy (block->colo_cache, block->host, block->used_length); }
migration/ram.c int colo_init_ram_cache(void) · line 3575
Quoted 10 lines
migration/ram.c +9
-0rcu_read_unlock (); +/* +* Record the dirty pages that sent by PVM, we use this dirty bitmap together +* with to decide which page in cache should be flushed into SVM's RAM. Here +* we use the same name 'ram_bitmap' as for migration. +*/ +if (ram_bytes_total ()) { +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
I think those need updating to check for 'qemu_ram_is_migratable' - it might be worth moving RAMBLOCK_FOREACH_MIGRATABLE.
migration/ram.c int colo_init_ram_cache(void) · line 3579
Quoted 4 lines
migration/ram.c +4
-0+unsigned long pages = block->max_length >>TARGET_PAGE_BITS ; + + block->bmap =bitmap_new (pages); +bitmap_set (block->bmap,0 , pages);
Would it make sense to use the 'receivedmap' that was recently added for other uses rather than needing your own?
Dave
Quoted 31 lines
migration/ram.c +11
-0+ } + } + ram_state =g_new0 (RAMState,1 ); + ram_state->migration_dirty_pages =0 ; +return 0 ; out_locked:@@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) {RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +g_free (block->bmap); + block->bmap =NULL ; + }rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {@@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } }rcu_read_unlock (); +g_free (ram_state); + ram_state =NULL ; }/** Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 08/20] ram/COLO: Record the dirty pages that SVM received
On Sat, Jul 28, 2018 at 2:51 AM, Dr. David Alan Gilbert <[email protected]
Quoted 55 lines
wrote:
migration/ram.c static inline void *colo_cache_from_block_offset(RAMBlock
Quoted 20 lines · * Zhang Chen ([email protected]) wrote:
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM.
Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint.
Signed-off-by zhanghailiang <[email protected]>Reviewed-by Dr. David Alan Gilbert <[email protected]>--- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)migration/ram.c +0
-0@@ -3325,6 +3325,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock*block,
Quoted 16 lines
migration/ram.c +9
-0__func__, block->idstr);return NULL ; } + +/* +* During colo checkpoint, we need bitmap of these migrated pages. +* It help us to decide which pages in ram cache should be flushed +* into VM's RAM later. +*/ +if (!test_and_set_bit (offset >>TARGET_PAGE_BITS , block->bmap)) { + ram_state->migration_dirty_pages++; + }return block->colo_cache + offset; }migration/ram.c int colo_init_ram_cache(void) · line 3568
Quoted 5 lines
migration/ram.c +2
-0@@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void)memcpy (block->colo_cache, block->host, block->used_length); }rcu_read_unlock (); +/* +* Record the dirty pages that sent by PVM, we use this dirty bitmap together
migration/ram.c int colo_init_ram_cache(void) · line 3569
Quoted 1 line
migration/ram.c +1
-0+* with to decide which page in cache should be flushed into SVM's RAM. Here
migration/ram.c int colo_init_ram_cache(void) · line 3575
Quoted 6 lines
migration/ram.c +6
-0+* we use the same name 'ram_bitmap' as for migration. +*/ +if (ram_bytes_total ()) { +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
Quoted 3 lines
I think those need updating to check for '
qemu_ram_is_migratable' - it might be worth movingRAMBLOCK_FOREACH_MIGRATABLE.
OK, I will change to the " RAMBLOCK_FOREACH_MIGRATABLE " in next version.
Quoted 9 lines
migration/ram.c int colo_init_ram_cache(void) · line 3579
Quoted 5 lines
migration/ram.c +4
-0+unsigned long pages = block->max_length >>TARGET_PAGE_BITS ; + + block->bmap =bitmap_new (pages); +bitmap_set (block->bmap,0 , pages);Would it make sense to use the 'receivedmap' that was recently added for other uses rather than needing your own?
Maybe we can do this job as optimization in the future, currently we want to make this series merged into upstream first.
Thanks
Zhang Chen
Quoted 37 lines
Dave
Quoted 29 lines
migration/ram.c +11
-1+ } + } + ram_state =g_new0 (RAMState,1 ); + ram_state->migration_dirty_pages =0 ; +return 0 ; out_locked:@@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) {RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +g_free (block->bmap); + block->bmap =NULL ; + }rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {@@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } }rcu_read_unlock (); +g_free (ram_state); + ram_state =NULL ; }/** -- Quoted 2 lines
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 08/20] ram/COLO: Record the dirty pages that SVM received
Quoted 70 lines · * Zhang Chen ([email protected]) wrote:
On Sat, Jul 28, 2018 at 2:51 AM, Dr. David Alan Gilbert <[email protected]
Quoted 55 lines
wrote:
migration/ram.c static inline void *colo_cache_from_block_offset(RAMBlock
Quoted 20 lines · * Zhang Chen ([email protected]) wrote:
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM.
Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint.
Signed-off-by zhanghailiang <[email protected]>Reviewed-by Dr. David Alan Gilbert <[email protected]>--- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)migration/ram.c +0
-0@@ -3325,6 +3325,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock*block,
Quoted 16 lines
migration/ram.c +9
-0__func__, block->idstr);return NULL ; } + +/* +* During colo checkpoint, we need bitmap of these migrated pages. +* It help us to decide which pages in ram cache should be flushed +* into VM's RAM later. +*/ +if (!test_and_set_bit (offset >>TARGET_PAGE_BITS , block->bmap)) { + ram_state->migration_dirty_pages++; + }return block->colo_cache + offset; }migration/ram.c int colo_init_ram_cache(void) · line 3568
Quoted 5 lines
migration/ram.c +2
-0@@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void)memcpy (block->colo_cache, block->host, block->used_length); }rcu_read_unlock (); +/* +* Record the dirty pages that sent by PVM, we use this dirty bitmap together
migration/ram.c int colo_init_ram_cache(void) · line 3569
Quoted 1 line
migration/ram.c +1
-0+* with to decide which page in cache should be flushed into SVM's RAM. Here
migration/ram.c int colo_init_ram_cache(void) · line 3575
Quoted 6 lines
migration/ram.c +6
-0+* we use the same name 'ram_bitmap' as for migration. +*/ +if (ram_bytes_total ()) { +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {Quoted 3 lines
I think those need updating to check for '
qemu_ram_is_migratable' - it might be worth movingRAMBLOCK_FOREACH_MIGRATABLE.OK, I will change to the "
RAMBLOCK_FOREACH_MIGRATABLE" in next version.Quoted 6 lines
Quoted 5 lines
migration/ram.c +4
-0+unsigned long pages = block->max_length >>TARGET_PAGE_BITS ; + + block->bmap =bitmap_new (pages); +bitmap_set (block->bmap,0 , pages);
Quoted 6 lines
Quoted 3 lines
Would it make sense to use the 'receivedmap' that was recently added for other uses rather than needing your own?
Maybe we can do this job as optimization in the future, currently we want to make this series merged into upstream first.
Sure; I think it's getting there; I don't see anything too bad in the migration bits as is; hopefully Jason can check the networking set again and Markus/Eric check the QMP side of things.
Dave
Quoted 41 lines
Thanks
Zhang ChenQuoted 37 lines
Dave
Quoted 29 lines
migration/ram.c +11
-1+ } + } + ram_state =g_new0 (RAMState,1 ); + ram_state->migration_dirty_pages =0 ; +return 0 ; out_locked:@@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) {RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +g_free (block->bmap); + block->bmap =NULL ; + }rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {@@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } }rcu_read_unlock (); +g_free (ram_state); + ram_state =NULL ; }/** -- Quoted 2 lines
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 08/20] ram/COLO: Record the dirty pages that SVM received
Quoted 83 lines · Dr. David Alan Gilbert
Quoted 1 line · * Zhang Chen ([email protected]) wrote:
On Sat, Jul 28, 2018 at 2:51 AM, Dr. David Alan Gilbert <
Quoted 18 lines
Quoted 18 lines
wrote:
Quoted 15 lines · * Zhang Chen ([email protected]) wrote:
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM.
Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint.
Signed-off-by zhanghailiang <[email protected]>Reviewed-by Dr. David Alan Gilbert <[email protected]>--- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)Quoted 5 lines
Quoted 5 lines
migration/ram.c static inline void
Quoted 5 lines
migration/ram.c +0
-0@@ -3325,6 +3325,15 @@ static inline void*colo_cache_from_block_offset(RAMBlock
Quoted 7 lines
Quoted 7 lines
*block,
Quoted 6 lines
migration/ram.c +3
-0__func__, block->idstr);return NULL ; } + +/* +* During colo checkpoint, we need bitmap of these migrated pages.
Quoted 1 line
Quoted 1 line
migration/ram.c static inline void · line 3331
Quoted 1 line
migration/ram.c +1
-0+* It help us to decide which pages in ram cache should be flushed
Quoted 3 lines
Quoted 3 lines
migration/ram.c static inline void *colo_cache_from_block_offset(RAMBlock *block,
Quoted 3 lines
migration/ram.c +3
-0+* into VM's RAM later. +*/ +if (!test_and_set_bit (offset >>TARGET_PAGE_BITS , block->bmap)){
Quoted 11 lines
Quoted 11 lines
migration/ram.c int colo_init_ram_cache(void) · line 3568
Quoted 11 lines
migration/ram.c +4
-0+ ram_state->migration_dirty_pages++; + }return block->colo_cache + offset; }@@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void)memcpy (block->colo_cache, block->host, block->used_length); }rcu_read_unlock (); +/* +* Record the dirty pages that sent by PVM, we use this dirty bitmap
Quoted 2 lines
Quoted 2 lines
together
Quoted 1 line
migration/ram.c +1
-0+* with to decide which page in cache should be flushed into SVM's
Quoted 8 lines
Quoted 8 lines
RAM. Here
Quoted 6 lines
migration/ram.c +6
-0+* we use the same name 'ram_bitmap' as for migration. +*/ +if (ram_bytes_total ()) { +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {Quoted 10 lines
Quoted 3 lines
I think those need updating to check for '
qemu_ram_is_migratable' - it might be worth movingRAMBLOCK_FOREACH_MIGRATABLE.OK, I will change to the "
RAMBLOCK_FOREACH_MIGRATABLE" in next version.Quoted 2 lines
Quoted 2 lines
migration/ram.c +1
-0+ unsigned long pages = block->max_length >>
TARGET_PAGE_BITS;Quoted 8 lines
Quoted 7 lines
migration/ram.c int colo_init_ram_cache(void) · line 3579
Quoted 3 lines
migration/ram.c +3
-0+ + block->bmap =bitmap_new (pages); +bitmap_set (block->bmap,0 , pages);Would it make sense to use the 'receivedmap' that was recently added for other uses rather than needing your own?
Quoted 7 lines
Quoted 2 lines
Maybe we can do this job as optimization in the future, currently we want to make this series merged into upstream first.
Sure; I think it's getting there; I don't see anything too bad in the migration bits as is; hopefully Jason can check the networking set again and Markus/Eric check the QMP side of things.
Hi Dave,
Yes, I will update new version for this series.
Thank you for your help with the COLO project.
Zhang Chen
Quoted 47 lines
Dave
Quoted 41 lines
Thanks
Zhang ChenQuoted 37 lines
Dave
Quoted 29 lines
migration/ram.c +11
-1+ } + } + ram_state =g_new0 (RAMState,1 ); + ram_state->migration_dirty_pages =0 ; +return 0 ; out_locked:@@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) {RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +g_free (block->bmap); + block->bmap =NULL ; + }rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {@@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } }rcu_read_unlock (); +g_free (ram_state); + ram_state =NULL ; }/** -- Quoted 2 lines
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v10 9/20 COLO: Flush memory data from ram cache
In reply to Zhang ChenDuring the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint.
Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way:
Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's.
Besides, we must ensure flush RAM cache before load device state.
--- migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 39 insertions(+)
migration/ram.c +37 -0
@@ -3830,6 +3830,39 @@ static bool postcopy_is_running(void)return ps >=POSTCOPY_INCOMING_LISTENING && ps <POSTCOPY_INCOMING_END ; } +/* +* Flush content of RAM cache into SVM's memory. +* Only flush the pages that be dirtied by PVM or SVM or both. +*/ +static void colo_flush_ram_cache (void ) +{ +RAMBlock *block =NULL ; +void *dst_host; +void *src_host; +unsigned long offset =0 ; + +trace_colo_flush_ram_cache_begin (ram_state->migration_dirty_pages); +rcu_read_lock (); + block =QLIST_FIRST_RCU (&ram_list.blocks); + +while (block) { + offset =migration_bitmap_find_dirty (ram_state, block, offset); + +if (offset <<TARGET_PAGE_BITS >= block->used_length) { + offset =0 ; + block =QLIST_NEXT_RCU (block, next); + }else { +migration_bitmap_clear_dirty (ram_state, block, offset); + dst_host = block->host + (offset <<TARGET_PAGE_BITS ); + src_host = block->colo_cache + (offset <<TARGET_PAGE_BITS ); +memcpy (dst_host, src_host,TARGET_PAGE_SIZE ); + } + } + +rcu_read_unlock (); +trace_colo_flush_ram_cache_end (); +} +static int ram_load (QEMUFile *f,void *opaque,int version_id) {int flags =0 , ret =0 , invalid_flags =0 ;
@@ -4006,6 +4039,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |=wait_for_decompress_done ();rcu_read_unlock ();trace_ram_load_complete (ret, seq_iter); + +if (!ret &&migration_incoming_in_colo_state ()) { +colo_flush_ram_cache (); + }return ret; }
migration/trace-events +2 -0
@@ -102,6 +102,8 @@ ram_dirty_bitmap_sync_start(void) "" ram_dirty_bitmap_sync_wait(void) "" ram_dirty_bitmap_sync_complete(void) "" ram_state_resume_prepare(uint64_t v) "%" PRId64 +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) ""
Signature · 1 line
2.17.1
PATCH v10 10/20 qmp event: Add COLO_EXIT event to notify users while exited COLO
In reply to Zhang ChenFrom: zhanghailiang <[email protected]>
If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately.
If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode.
--- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)
migration/colo.c +31 -0
@@ -28,6 +28,7 @@#include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading;static Notifier packets_compare_notifier;
@@ -514,6 +515,23 @@ out:qemu_fclose (fb); } +/* +* There are only two reasons we can get here, some error happened +* or the user triggered failover. +*/ +switch (failover_get_state ()) { +case FAILOVER_STATUS_NONE : +qapi_event_send_colo_exit (COLO_MODE_PRIMARY , +COLO_EXIT_REASON_ERROR ,NULL ); +break ; +case FAILOVER_STATUS_REQUIRE : +qapi_event_send_colo_exit (COLO_MODE_PRIMARY , +COLO_EXIT_REASON_REQUEST ,NULL ); +break ; +default : +abort (); + } +/* Hope this not to be too long to wait here */ qemu_sem_wait (&s->colo_exit_sem);qemu_sem_destroy (&s->colo_exit_sem);
@@ -745,6 +763,19 @@ out:error_report_err (local_err); } +switch (failover_get_state ()) { +case FAILOVER_STATUS_NONE : +qapi_event_send_colo_exit (COLO_MODE_SECONDARY , +COLO_EXIT_REASON_ERROR ,NULL ); +break ; +case FAILOVER_STATUS_REQUIRE : +qapi_event_send_colo_exit (COLO_MODE_SECONDARY , +COLO_EXIT_REASON_REQUEST ,NULL ); +break ; +default : +abort (); + } +if (fb) {qemu_fclose (fb); }
qapi/migration.json +38 -0
@@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. This can't occur in the +# COLO_EXIT event, only in the result of query-colo-status. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 3.0 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: #
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH V10 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO
Quoted 13 lines · Zhang Chen
From: zhanghailiang <[email protected]>
If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately.
If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode.Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Markus Armbruster <[email protected]>
Quoted 4 lines
--- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)
At this point in the release cycle, this series feels like enough of a new feature (rather than a bug fix) that it is probably not appropriate for 3.0, which means...
Quoted 6 lines
+++
b/qapi/migration.jsonqapi/migration.json +1
-0@@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +##
qapi/migration.json line 913
Quoted 10 lines
qapi/migration.json +10
-0+# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0
...this and other references should be updated to 3.1.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH V10 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO
Quoted 36 lines · Eric Blake
Quoted 13 lines · Zhang Chen
From: zhanghailiang <[email protected]>
If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately.
If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode.Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Markus Armbruster <[email protected]>Quoted 5 lines
--- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)At this point in the release cycle, this series feels like enough of a new feature (rather than a bug fix) that it is probably not appropriate for 3.0, which means...
+++
b/qapi/migration.jsonQuoted 9 lines
qapi/migration.json +5
-0@@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +#
Quoted 8 lines
qapi/migration.json
Quoted 6 lines
qapi/migration.json +5
-0+# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0...this and other references should be updated to 3.1.
OK, I will fix it and resend this series.
Thanks
Zhang Chen
Quoted 7 lines
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH V10 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO
Quoted 48 lines · * Zhang Chen ([email protected]) wrote:
Quoted 36 lines · Eric Blake
Quoted 13 lines · Zhang Chen
From: zhanghailiang <[email protected]>
If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately.
If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode.Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Markus Armbruster <[email protected]>Quoted 5 lines
--- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)At this point in the release cycle, this series feels like enough of a new feature (rather than a bug fix) that it is probably not appropriate for 3.0, which means...
+++
b/qapi/migration.jsonQuoted 9 lines
qapi/migration.json +5
-0@@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +#Quoted 8 lines
qapi/migration.json
Quoted 6 lines
qapi/migration.json +5
-0+# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0...this and other references should be updated to 3.1.
Quoted 1 line
OK, I will fix it and resend this series.
No need to resend now; resend after the review of the other bits in the series; I'm sure there will be other minor things; anyway we can always fix up version nunmbers in merge.
Dave
Quoted 11 lines
Thanks
Zhang ChenQuoted 7 lines
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH V10 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO
Quoted 50 lines · Dr. David Alan Gilbert
Quoted 23 lines · * Zhang Chen ([email protected]) wrote:
Quoted 21 lines · Eric Blake
Quoted 18 lines · Zhang Chen
From: zhanghailiang <[email protected]>
If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately.
If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode.Signed-off-by zhanghailiang <[email protected]>Signed-off-by Li Zhijian <[email protected]>Signed-off-by Zhang Chen <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Markus Armbruster <[email protected]>--- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)Quoted 1 line
Quoted 1 line
At this point in the release cycle, this series feels like enough of a
new
Quoted 24 lines
Quoted 22 lines
feature (rather than a bug fix) that it is probably not appropriate for 3.0, which means...
+++
b/qapi/migration.jsonQuoted 5 lines
qapi/migration.json +1
-0@@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT:qapi/migration.json line 913
Quoted 10 lines
qapi/migration.json +9
-0+# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0...this and other references should be updated to 3.1.
Quoted 6 lines
Quoted 1 line
OK, I will fix it and resend this series.
No need to resend now; resend after the review of the other bits in the series; I'm sure there will be other minor things; anyway we can always fix up version nunmbers in merge.
OK, I got it.
Thanks
Zhang Chen
Quoted 17 lines
Dave
Quoted 11 lines
Thanks
Zhang ChenQuoted 7 lines
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v10 11/20 qapi/migration.json: Rename COLO unknown mode to none mode.
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
Suggested by Markus Armbruster rename COLO unknown mode to none mode.
--- migration/colo-failover.c | 2 +- migration/colo.c | 2 +- qapi/migration.json | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-)
migration/colo-failover.c +1 -1
@@ -77,7 +77,7 @@ FailoverStatus failover_get_state(void)void qmp_x_colo_lost_heartbeat (Error **errp) { -if (get_colo_mode () ==COLO_MODE_UNKNOWN ) { +if (get_colo_mode () ==COLO_MODE_NONE ) {error_setg (errp,QERR_FEATURE_DISABLED ,"colo" );return ; }
migration/colo.c +1 -1
@@ -160,7 +160,7 @@ COLOMode get_colo_mode(void) }else if (migration_incoming_in_colo_state ()) {return COLO_MODE_SECONDARY ; }else { -return COLO_MODE_UNKNOWN ; +return COLO_MODE_NONE ; } }
qapi/migration.json +5 -5
@@ -867,18 +867,18 @@ ## # @COLOMode: # -# The colo mode +# The COLO current mode. # -# @unknown: unknown mode +# @none: COLO is disabled. # -# @primary: master side +# @primary: COLO node in primary side. # -# @secondary: slave side +# @secondary: COLO node in slave side. # # Since: 2.8 ## { 'enum': 'COLOMode', - 'data': [ 'unknown', 'primary', 'secondary'] } + 'data': [ 'none', 'primary', 'secondary'] } ## # @FailoverStatus:
Signature · 1 line
2.17.1
PATCH v10 12/20 qapi: Add new command to query colo status
In reply to Zhang ChenLibvirt or other high level software can use this command query colo status. You can test this command like that:
{'execute':'query-colo-status'}
--- migration/colo.c | 21 +++++++++++++++++++++ qapi/migration.json | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+)
migration/colo.c +21 -0
@@ -29,6 +29,7 @@#include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading;static Notifier packets_compare_notifier;
@@ -237,6 +238,26 @@ void qmp_xen_colo_do_checkpoint(Error **errp)#endif } +COLOStatus *qmp_query_colo_status (Error **errp) +{ +COLOStatus *s =g_new0 (COLOStatus,1 ); + + s->mode =get_colo_mode (); + +switch (failover_get_state ()) { +case FAILOVER_STATUS_NONE : + s->reason =COLO_EXIT_REASON_NONE ; +break ; +case FAILOVER_STATUS_REQUIRE : + s->reason =COLO_EXIT_REASON_REQUEST ; +break ; +default : + s->reason =COLO_EXIT_REASON_ERROR ; + } + +return s; +} +static void colo_send_message (QEMUFile *f,COLOMessage msg,Error **errp) {
qapi/migration.json +32 -0
@@ -1251,6 +1251,38 @@ ## { 'command': 'xen-colo-do-checkpoint' } +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: COLO running mode. If COLO is running, this field will return +# 'primary' or 'secondary'. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "active": true, "reason": "request" } } +# +# Since: 3.0 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } + ## # @migrate-recover: #
Signature · 1 line
2.17.1
PATCH v10 13/20 savevm: split the process of different stages for loadvm/savevm
In reply to Zhang ChenThere are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections.
We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state.
So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration.
Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header().
--- migration/colo.c | 41 ++++++++++++++++++++++++++++++++--------- migration/savevm.c | 36 +++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 65 insertions(+), 16 deletions(-)
migration/colo.c +32 -9
@@ -30,6 +30,7 @@#include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading;static Notifier packets_compare_notifier;
@@ -401,23 +402,34 @@ static int colo_do_checkpoint_transaction(MigrationState *s,/* Disable block migration */ migrate_set_block_enabled (false, &local_err); -qemu_savevm_state_header (fb); -qemu_savevm_state_setup (fb);qemu_mutex_lock_iothread ();replication_do_checkpoint_all (&local_err);if (local_err) {qemu_mutex_unlock_iothread (); goto out; } -qemu_savevm_state_complete_precopy (fb, false, false); -qemu_mutex_unlock_iothread (); - -qemu_fflush (fb);colo_send_message (s->to_dst_file,COLO_MESSAGE_VMSTATE_SEND , &local_err);if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } +/* Note: device state is saved into buffer */ + ret =qemu_save_device_state (fb); + +qemu_mutex_unlock_iothread (); +if (ret <0 ) { goto out; } +/* +* Only save VM's live state, which not including device state. +* TODO: We may need a timeout mechanism to prevent COLO process +* to be blocked here. +*/ +qemu_savevm_live_state (s->to_dst_file); + +qemu_fflush (fb); +/* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read.
@@ -635,6 +647,7 @@ void *colo_process_incoming_thread(void *opaque)uint64_t total_size;uint64_t value;Error *local_err =NULL ; +int ret;qemu_sem_init (&mis->colo_incoming_sem,0 );
@@ -707,6 +720,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +qemu_mutex_lock_iothread (); +cpu_synchronize_all_pre_loadvm (); + ret =qemu_loadvm_state_main (mis->from_src_file, mis); +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { +error_report ("Load VM's live state (ram) error" ); + goto out; + } + value =colo_receive_message_value (mis->from_src_file,COLO_MESSAGE_VMSTATE_SIZE , &local_err);if (local_err) {
@@ -738,10 +761,10 @@ void *colo_process_incoming_thread(void *opaque) }qemu_mutex_lock_iothread (); -qemu_system_reset (SHUTDOWN_CAUSE_NONE ); vmstate_loading = true; -if (qemu_loadvm_state (fb) <0 ) { -error_report ("COLO: loadvm failed" ); + ret =qemu_load_device_state (fb); +if (ret <0 ) { +error_report ("COLO: load device state failed" );qemu_mutex_unlock_iothread (); goto out; }
migration/savevm.c +29 -7
@@ -1378,13 +1378,21 @@ done:return ret; } -static int qemu_save_device_state (QEMUFile *f) +void qemu_savevm_live_state (QEMUFile *f) { -SaveStateEntry *se; +/* save QEMU_VM_SECTION_END section */ +qemu_savevm_state_complete_precopy (f, true, false); +qemu_put_byte (f,QEMU_VM_EOF ); +} -qemu_put_be32 (f,QEMU_VM_FILE_MAGIC ); -qemu_put_be32 (f,QEMU_VM_FILE_VERSION ); +int qemu_save_device_state (QEMUFile *f) +{ +SaveStateEntry *se; +if (!migration_in_colo_state ()) { +qemu_put_be32 (f,QEMU_VM_FILE_MAGIC ); +qemu_put_be32 (f,QEMU_VM_FILE_VERSION ); + }cpu_synchronize_all_states (); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -1440,8 +1448,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis); -/* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything
@@ -2241,7 +2247,7 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis)return true; } -static int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis) +int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis) {uint8_t section_type;int ret =0 ;
@@ -2410,6 +2416,22 @@ int qemu_loadvm_state(QEMUFile *f)return ret; } +int qemu_load_device_state (QEMUFile *f) +{ +MigrationIncomingState *mis =migration_incoming_get_current (); +int ret; + +/* Load QEMU_VM_SECTION_FULL section */ + ret =qemu_loadvm_state_main (f, mis); +if (ret <0 ) { +error_report ("Failed to load device state: %d" , ret); +return ret; + } + +cpu_synchronize_all_post_init (); +return 0 ; +} +int save_snapshot (const char *name,Error **errp) {BlockDriverState *bs, *bs1;
migration/savevm.h +4 -0
@@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,uint64_t *start_list, uint64_t *length_list);void qemu_savevm_send_colo_enable (QEMUFile *f); +void qemu_savevm_live_state (QEMUFile *f); +int qemu_save_device_state (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f);void qemu_loadvm_state_cleanup (void ); +int qemu_loadvm_state_main (QEMUFile *f,MigrationIncomingState *mis); +int qemu_load_device_state (QEMUFile *f);#endif
Signature · 1 line
2.17.1
PATCH v10 14/20 COLO: flush host dirty ram from cache
In reply to Zhang ChenFrom: zhanghailiang <[email protected]>
Don't need to flush all VM's ram from cache, only
flush the dirty pages since last checkpoint
--- migration/ram.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
migration/ram.c +10 -0
@@ -3581,6 +3581,7 @@ int colo_init_ram_cache(void) } ram_state =g_new0 (RAMState,1 ); ram_state->migration_dirty_pages =0 ; +memory_global_dirty_log_start ();return 0 ;
@@ -3601,10 +3602,12 @@ void colo_release_ram_cache(void) {RAMBlock *block; +memory_global_dirty_log_stop (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {g_free (block->bmap); block->bmap =NULL ; } +rcu_read_lock (); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {if (block->colo_cache) {
@@ -3841,6 +3844,13 @@ static void colo_flush_ram_cache(void)void *src_host;unsigned long offset =0 ; +memory_global_dirty_log_sync (); +rcu_read_lock (); +RAMBLOCK_FOREACH_MIGRATABLE (block) { +migration_bitmap_sync_range (ram_state, block,0 , block->used_length); + } +rcu_read_unlock (); +trace_colo_flush_ram_cache_begin (ram_state->migration_dirty_pages);rcu_read_lock (); block =QLIST_FIRST_RCU (&ram_list.blocks);
Signature · 1 line
2.17.1
PATCH v10 15/20 net/net.c: Add net client type check function for COLO
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
We add is_colo_support_client_type() to check the net client type for COLO-compare. Currently we just support TAP.
Suggested by Jason.
--- include/net/net.h | 1 + net/colo-compare.c | 5 +++++ net/net.c | 14 ++++++++++++++ 3 files changed, 20 insertions(+)
include/net/net.h +1 -0
@@ -212,6 +212,7 @@ void hmp_host_net_add(Monitor *mon, const QDict *qdict);void hmp_host_net_remove (Monitor *mon,const QDict *qdict);void netdev_add (QemuOpts *opts,Error **errp);void qmp_netdev_add (QDict *qdict,QObject **ret,Error **errp); +bool is_colo_support_client_type (void );int net_hub_id_for_client (NetClientState *nc,int *id);NetClientState *net_hub_port_find (int hub_id);
net/colo-compare.c +5 -0
@@ -996,6 +996,11 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)return ; } +if (!is_colo_support_client_type ()) { +error_setg (errp,"COLO-compare: Net client type is not supported" ); +return ; + } +net_socket_rs_init (&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr);net_socket_rs_init (&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr);
net/net.c +14 -0
@@ -1733,3 +1733,17 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size)assert (size ==0 );return 0 ; } + +/* Currently, COLO just support TAP */ +bool is_colo_support_client_type (void ) +{ +NetClientState *nc; + + QTAILQ_FOREACH(nc, &net_clients, next) { +if (nc->info->type !=NET_CLIENT_DRIVER_TAP ) { +return false; + } + } + +return true; +}
Signature · 1 line
2.17.1
PATCH v10 16/20 filter: Add handle_event method for NetFilterClass
In reply to Zhang ChenFilter needs to process the event of checkpoint/failover or other event passed by COLO frame.
--- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+)
include/net/filter.h +5 -0
@@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc,typedef void (FilterStatusChanged ) (NetFilterState *nf,Error **errp); +typedef void (FilterHandleEvent ) (NetFilterState *nf,int event,Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class;
@@ -45,6 +47,7 @@ typedef struct NetFilterClass {FilterSetup *setup;FilterCleanup *cleanup;FilterStatusChanged *status_changed; +FilterHandleEvent *handle_event;/* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass;
@@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,int iovcnt, void *opaque); +void colo_notify_filters_event (int event,Error **errp); +#endif /* QEMU_NET_FILTER_H */
net/filter.c +17 -0
@@ -17,6 +17,8 @@#include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter (NetFilterState *nf) {
@@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj)g_free (nf->netdev_id); } +static void dummy_handle_event (NetFilterState *nf,int event,Error **errp) +{ +switch (event) { +case COLO_EVENT_CHECKPOINT : +break ; +case COLO_EVENT_FAILOVER : +object_property_set_str (OBJECT (nf),"off" ,"status" , errp); +break ; +default : +break ; + } +} +static void netfilter_class_init (ObjectClass *oc,void *data) {UserCreatableClass *ucc =USER_CREATABLE_CLASS (oc); +NetFilterClass *nfc =NETFILTER_CLASS (oc); ucc->complete = netfilter_complete; + nfc->handle_event = dummy_handle_event; }static const TypeInfo netfilter_info = {
net/net.c +28 -0
@@ -1331,6 +1331,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event (int event,Error **errp) +{ +NetClientState *nc, *peer; +NetClientDriver type; +NetFilterState *nf; +NetFilterClass *nfc =NULL ; +Error *local_err =NULL ; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer = nc->peer; + type = nc->info->type; +if (!peer || type !=NET_CLIENT_DRIVER_TAP ) { +continue ; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =NETFILTER_GET_CLASS (OBJECT (nf)); +if (!nfc->handle_event) { +continue ; + } + nfc->handle_event (nf, event, &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } + } + } +} +void qmp_set_link (const char *name,bool up,Error **errp) {NetClientState *ncs[MAX_QUEUE_NUM ];
Signature · 1 line
2.17.1
PATCH v10 17/20 filter-rewriter: handle checkpoint and failover event
In reply to Zhang ChenAfter one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter will into failover mode that needn't handle the new TCP connection.
--- net/colo-compare.c | 12 ++++----- net/colo.c | 8 ++++++ net/colo.h | 2 ++ net/filter-rewriter.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 6 deletions(-)
net/colo-compare.c +6 -6
@@ -116,6 +116,12 @@ enum { SECONDARY_IN, }; +static void colo_compare_inconsistency_notify (void ) +{ +notifier_list_notify (&colo_compare_notifiers, +migrate_get_current ()); +} + static int compare_chr_send(CompareState *s, const uint8_t *buf, uint32_t size,
@@ -562,12 +568,6 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } -static void colo_compare_inconsistency_notify (void ) -{ -notifier_list_notify (&colo_compare_notifiers, -migrate_get_current ()); -} -void colo_compare_register_notifier (Notifier *notify) {notifier_list_add (&colo_compare_notifiers, notify);
net/colo.c +8 -0
@@ -221,3 +221,11 @@ Connection *connection_get(GHashTable *connection_track_table,return conn; } + +bool connection_has_tracked (GHashTable *connection_track_table, +ConnectionKey *key) +{ +Connection *conn =g_hash_table_lookup (connection_track_table, key); + +return conn ? true : false; +}
net/colo.h +2 -0
@@ -98,6 +98,8 @@ void connection_destroy(void *opaque);Connection *connection_get (GHashTable *connection_track_table,ConnectionKey *key,GQueue *conn_list); +bool connection_has_tracked (GHashTable *connection_track_table, +ConnectionKey *key);void connection_hashtable_reset (GHashTable *connection_track_table);Packet *packet_new (const void *data,int size,int vnet_hdr_len);void packet_destroy (void *opaque,void *user_data);
net/filter-rewriter.c +57 -0
105 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v10 18/20 COLO: notify net filters about checkpoint/failover event
In reply to Zhang ChenFrom: zhanghailiang <[email protected]>
Notify all net filters about the checkpoint and failover event.
--- migration/colo.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
migration/colo.c +15 -0
@@ -31,6 +31,7 @@#include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading;static Notifier packets_compare_notifier;
@@ -83,6 +84,12 @@ static void secondary_vm_do_failover(void)error_report_err (local_err); } +/* Notify all filters of all NIC to do checkpoint */ +colo_notify_filters_event (COLO_EVENT_FAILOVER , &local_err); +if (local_err) { +error_report_err (local_err); + } +if (!autostart) {error_report ("\"-S\" qemu option will be ignored in secondary side" );/* recover runstate to normal migration finish state */
@@ -781,6 +788,14 @@ void *colo_process_incoming_thread(void *opaque) goto out; } +/* Notify all filters of all NIC to do checkpoint */ +colo_notify_filters_event (COLO_EVENT_CHECKPOINT , &local_err); + +if (local_err) { +qemu_mutex_unlock_iothread (); + goto out; + } + vmstate_loading = false;vm_start ();trace_colo_vm_state_change ("stop" ,"run" );
Signature · 1 line
2.17.1
PATCH v10 19/20 COLO: quick failover process by kick COLO thread
In reply to Zhang ChenFrom: zhanghailiang <[email protected]>
COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process.
--- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+)
migration/colo.c +8 -0
@@ -131,6 +131,11 @@ static void primary_vm_do_failover(void)migrate_set_state (&s->state,MIGRATION_STATUS_COLO ,MIGRATION_STATUS_COMPLETED ); +/* +* kick COLO thread which might wait at +* qemu_sem_wait(&s->colo_checkpoint_sem). +*/ +colo_checkpoint_notify (migrate_get_current ());/* * Wake up COLO thread which may blocked in recv() or send(),
@@ -539,6 +544,9 @@ static void colo_process_checkpoint(MigrationState *s)qemu_sem_wait (&s->colo_checkpoint_sem); +if (s->state !=MIGRATION_STATUS_COLO ) { + goto out; + } ret =colo_do_checkpoint_transaction (s, bioc, fb);if (ret <0 ) { goto out;
Signature · 1 line
2.17.1
PATCH v10 20/20 docs: Add COLO status diagram to COLO-FT.txt
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
This diagram make user better understand COLO.
Suggested by Markus Armbruster.
--- docs/COLO-FT.txt | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
docs/COLO-FT.txt +34 -0
@@ -110,6 +110,40 @@ Note: HeartBeat has not been implemented yet, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command. +== COLO operation status == + ++-----------------+ +| | +| Start COLO | +| | ++--------+--------+ + | + | Main qmp command: + | migrate-set-capabilities with x-colo + | migrate + | + v ++--------+--------+ +| | +| COLO running | +| | ++--------+--------+ + | + | Main qmp command: + | x-colo-lost-heartbeat + | or + | some error happened + v ++--------+--------+ +| | send qmp event: +| COLO failover | COLO_EXIT +| | ++-----------------+ + +COLO use the qmp command switching and report operation status. +The diagram just write the main qmp command, you can get the detail +in test procedure. + == Test procedure == 1. Startup qemu Primary:
Signature · 1 line
2.17.1