migration: include timing and RAM stats on destination when query-migrate
cover migration: include timing and RAM stats on destination when query-migrate
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.
The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.
On dst, {"execute":"query-migrate"}
* As-is:
{
"return": {
"status": "completed"
}
* To-be:
{
"return": {
"status": "completed",
"total-time": 94,
"ram": {
"total": 554508288,
"pages-per-second": 1440234,
"page-size": 4096,
"remaining": 0,
"mbps": 97.955404255319152,
"transferred": 1150976,
"duplicate": 135101,
"normal-bytes": 1150976,
"normal": 281
}
}
}Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 migration: track timing and received pages in MigrationIncomingState
From: Trieu Huynh <[email protected]>
Add start_time, total_time, received_normal_pages, received_zero_pages, and received_xbzrle_pages to MigrationIncomingState.
start_time is recorded when the incoming state transitions to ACTIVE; total_time is recorded when it transitions to COMPLETED. The page counters are incremented in ram_load_precopy() as each page type is received from the source.
These fields will be used by a subsequent patch to populate fill_destination_migration_info() for the migration COMPLETED case.
--- migration/migration.c | 2 ++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ 3 files changed, 22 insertions(+)
migration/migration.c +2 -0
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)*/ migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_COMPLETED ); + mis->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) - mis->start_time;migration_incoming_state_destroy (); }
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)migrate_set_state (&mis->state,MIGRATION_STATUS_SETUP ,MIGRATION_STATUS_ACTIVE ); + mis->start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); mis->loadvm_co =qemu_coroutine_self (); ret =qemu_loadvm_state (mis->from_src_file, &local_err); mis->loadvm_co =NULL ;
migration/migration.h +17 -0
@@ -176,6 +176,23 @@ struct MigrationIncomingState {/* PostCopyFD's for external userfaultfds & handlers of shared memory */ GArray *postcopy_remote_fds; +/* +* Timestamps for reporting migration duration via query-migrate on the +* destination. start_time is recorded when the state moves to ACTIVE; +* total_time is recorded when it moves to COMPLETED. +*/ +int64_t start_time; +int64_t total_time; + +/* +* Page counters for reporting RAM statistics via query-migrate on the +* destination. Incremented in ram_load_precopy() as each page type +* is received. +*/ +uint64_t received_normal_pages; +uint64_t received_zero_pages; +uint64_t received_xbzrle_pages; +MigrationStatus state;/*
migration/ram.c +3 -0
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)break ; }ram_handle_zero (host,TARGET_PAGE_SIZE ); + mis->received_zero_pages++;break ;case RAM_SAVE_FLAG_PAGE :qemu_get_buffer (f, host,TARGET_PAGE_SIZE ); + mis->received_normal_pages++;break ;case RAM_SAVE_FLAG_XBZRLE :
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f) ret = -EINVAL ;break ; } + mis->received_xbzrle_pages++;break ;case RAM_SAVE_FLAG_MULTIFD_FLUSH :multifd_recv_sync_main ();
Signature · 1 line
2.43.0
PATCH 2/2 migration: expose RAM stats and timing on destination via query-migrate
In reply to Trieu HuynhFrom: Trieu Huynh <[email protected]>
MigrationStats had all-required QAPI fields, so partially populating info->ram on the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.
Make those source-only fields optional with '*' in the QAPI schema. Update populate_ram_info() to set them via has_* setters so source output is unchanged.
As per schema's changes, fill_destination_migration_info() can now populate info->ram for the COMPLETED case with the fields that are meaningful on the receive side:
* total, remaining, page-size (RAM layout)
* transferred, normal, normal-bytes (bytes/pages received)
* duplicate (zero pages received)
* mbps (throughput: transferred*8
/time_ms/1000) * pages-per-second (total pages / duration in
seconds)
Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.
On dst, {"execute":"query-migrate"}
* As-is:
{
"return": {
"status": "completed"
}
* To-be:
{
"return": {
"status": "completed",
"total-time": 94,
"ram": {
"total": 554508288,
"pages-per-second": 1440234,
"page-size": 4096,
"remaining": 0,
"mbps": 97.955404255319152,
"transferred": 1150976,
"duplicate": 135101,
"normal-bytes": 1150976,
"normal": 281
}
}
}--- migration/migration.c | 35 +++++++++++++++++++++++++++++++++++ qapi/migration.json | 14 +++++++------- 2 files changed, 42 insertions(+), 7 deletions(-)
migration/migration.c +35 -0
@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal =qatomic_read (&mig_stats.normal_pages); info->ram->normal_bytes = info->ram->normal * page_size; info->ram->mbps = s->mbps; + info->ram->has_dirty_sync_count = true; info->ram->dirty_sync_count =qatomic_read (&mig_stats.dirty_sync_count); + info->ram->has_dirty_sync_missed_zero_copy = true; info->ram->dirty_sync_missed_zero_copy =qatomic_read (&mig_stats.dirty_sync_missed_zero_copy); + info->ram->has_postcopy_requests = true; info->ram->postcopy_requests =qatomic_read (&mig_stats.postcopy_requests); info->ram->page_size = page_size; + info->ram->has_multifd_bytes = true; info->ram->multifd_bytes =qatomic_read (&mig_stats.multifd_bytes); info->ram->pages_per_second = s->pages_per_second; + info->ram->has_precopy_bytes = true; info->ram->precopy_bytes =qatomic_read (&mig_stats.precopy_bytes); + info->ram->has_downtime_bytes = true; info->ram->downtime_bytes =qatomic_read (&mig_stats.downtime_bytes); + info->ram->has_postcopy_bytes = true; info->ram->postcopy_bytes =qatomic_read (&mig_stats.postcopy_bytes);if (migrate_xbzrle ()) {
@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)if (s->state !=MIGRATION_STATUS_COMPLETED ) { info->ram->remaining =ram_bytes_remaining (); + info->ram->has_dirty_pages_rate = true; info->ram->dirty_pages_rate =qatomic_read (&mig_stats.dirty_pages_rate); }
@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)case MIGRATION_STATUS_COMPLETED : info->has_status = true;fill_destination_postcopy_migration_info (info); +if (mis->total_time >0 ) { + info->has_total_time = true; + info->total_time = mis->total_time; + } + { +size_t page_size =qemu_target_page_size (); +uint64_t normal = mis->received_normal_pages; +uint64_t zero = mis->received_zero_pages; +uint64_t xbzrle = mis->received_xbzrle_pages; + + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->total =ram_bytes_total (); + info->ram->remaining =0 ; + info->ram->normal = normal + xbzrle; + info->ram->normal_bytes = (normal + xbzrle) * page_size; + info->ram->duplicate = zero; + info->ram->transferred = (normal + xbzrle) * page_size; + info->ram->page_size = page_size; + +if (info->has_total_time) { + info->ram->mbps = info->ram->transferred *8.0 + / info->total_time /1000.0 ; + info->ram->pages_per_second = (normal + xbzrle + zero) + *1000.0 / info->total_time; + } +/* source-only optional omitted from output */ + }break ;default :return ;
qapi/migration.json +7 -7
@@ -68,13 +68,13 @@ 'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' , 'duplicate': 'int', 'normal': 'int', - 'normal-bytes': 'int', 'dirty-pages-rate': 'int', - 'mbps': 'number', 'dirty-sync-count': 'int', - 'postcopy-requests': 'int', 'page-size': 'int', - 'multifd-bytes': 'uint64', 'pages-per-second': 'uint64', - 'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64', - 'postcopy-bytes': 'uint64', - 'dirty-sync-missed-zero-copy': 'uint64' } } + 'normal-bytes': 'int', '*dirty-pages-rate': 'int', + 'mbps': 'number', '*dirty-sync-count': 'int', + '*postcopy-requests': 'int', 'page-size': 'int', + '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64', + '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64', + '*postcopy-bytes': 'uint64', + '*dirty-sync-missed-zero-copy': 'uint64' } } ## # @XBZRLECacheStats:
Signature · 1 line
2.43.0
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate
Quoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)
Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.
+CC Daniel and Claudio
Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState
In reply to Trieu HuynhHello,
Quoted 46 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
Add
start_time,total_time,received_normal_pages,received_zero_pages, andreceived_xbzrle_pagesto MigrationIncomingState.
start_timeis recorded when the incoming state transitions to ACTIVE;total_timeis recorded when it transitions to COMPLETED. The page counters are incremented inram_load_precopy()as each page type is received from the source.These fields will be used by a subsequent patch to populate
fill_destination_migration_info()for the migration COMPLETED case.Signed-off-by Trieu Huynh <[email protected]>--- migration/migration.c | 2 ++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ 3 files changed, 22 insertions(+)migration/migration.c +2
-0@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)*/ migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_COMPLETED ); + mis->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) - mis->start_time;migration_incoming_state_destroy (); }@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)migrate_set_state (&mis->state,MIGRATION_STATUS_SETUP ,MIGRATION_STATUS_ACTIVE ); + mis->start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); mis->loadvm_co =qemu_coroutine_self (); ret =qemu_loadvm_state (mis->from_src_file, &local_err); mis->loadvm_co =NULL ;migration/migration.h +0
-0
migration/migration.h struct MigrationIncomingState { · line 185
Quoted 10 lines
migration/migration.h +7
-0@@ -176,6 +176,23 @@ struct MigrationIncomingState {/* PostCopyFD's for external userfaultfds & handlers of shared memory */ GArray *postcopy_remote_fds; +/* +* Timestamps for reporting migration duration via query-migrate on the +* destination. start_time is recorded when the state moves to ACTIVE; +* total_time is recorded when it moves to COMPLETED. +*/ +int64_t start_time; +int64_t total_time;
nit: since this is a timestamp and these are directly linked to state changes, why not call these:
int64_t migration_active_ts;int64_t migration_completed_ts;
or something similar?
Thank you,
Claudio
Quoted 38 lines
migration/migration.h +10
-0+ +/* +* Page counters for reporting RAM statistics via query-migrate on the +* destination. Incremented in ram_load_precopy() as each page type +* is received. +*/ +uint64_t received_normal_pages; +uint64_t received_zero_pages; +uint64_t received_xbzrle_pages; +MigrationStatus state;/* migration/ram.c +3
-0@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)break ; }ram_handle_zero (host,TARGET_PAGE_SIZE ); + mis->received_zero_pages++;break ;case RAM_SAVE_FLAG_PAGE :qemu_get_buffer (f, host,TARGET_PAGE_SIZE ); + mis->received_normal_pages++;break ;case RAM_SAVE_FLAG_XBZRLE :@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f) ret = -EINVAL ;break ; } + mis->received_xbzrle_pages++;break ;case RAM_SAVE_FLAG_MULTIFD_FLUSH :multifd_recv_sync_main ();
Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState
Quoted 63 lines · Claudio Fontana
Hello,
Quoted 46 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
Add
start_time,total_time,received_normal_pages,received_zero_pages, andreceived_xbzrle_pagesto MigrationIncomingState.
start_timeis recorded when the incoming state transitions to ACTIVE;total_timeis recorded when it transitions to COMPLETED. The page counters are incremented inram_load_precopy()as each page type is received from the source.These fields will be used by a subsequent patch to populate
fill_destination_migration_info()for the migration COMPLETED case.Signed-off-by Trieu Huynh <[email protected]>--- migration/migration.c | 2 ++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ 3 files changed, 22 insertions(+)migration/migration.c +2
-0@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)*/ migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_COMPLETED ); + mis->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) - mis->start_time;migration_incoming_state_destroy (); }@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)migrate_set_state (&mis->state,MIGRATION_STATUS_SETUP ,MIGRATION_STATUS_ACTIVE ); + mis->start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); mis->loadvm_co =qemu_coroutine_self (); ret =qemu_loadvm_state (mis->from_src_file, &local_err); mis->loadvm_co =NULL ;migration/migration.h +0
-0migration/migration.h struct MigrationIncomingState { · line 185
Quoted 10 lines
migration/migration.h +7
-0@@ -176,6 +176,23 @@ struct MigrationIncomingState {/* PostCopyFD's for external userfaultfds & handlers of shared memory */ GArray *postcopy_remote_fds; +/* +* Timestamps for reporting migration duration via query-migrate on the +* destination. start_time is recorded when the state moves to ACTIVE; +* total_time is recorded when it moves to COMPLETED. +*/ +int64_t start_time; +int64_t total_time;nit: since this is a timestamp and these are directly linked to state changes, why not call these:
Quoted 2 lines
int64_tmigration_active_ts;int64_tmigration_completed_ts;
I got the second one wrong, it is not a timestamp, that is indeed a time. So clarity in the name seems to be more than a nit. What about:
int64_t migration_active_ts; /* ms */int64_t migration_completed_time; /* ms */
Thanks,
Claudio
Quoted 40 lines
Quoted 39 lines
migration/migration.h +10
-0+ +/* +* Page counters for reporting RAM statistics via query-migrate on the +* destination. Incremented in ram_load_precopy() as each page type +* is received. +*/ +uint64_t received_normal_pages; +uint64_t received_zero_pages; +uint64_t received_xbzrle_pages; +MigrationStatus state;/* migration/ram.c +3
-0@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)break ; }ram_handle_zero (host,TARGET_PAGE_SIZE ); + mis->received_zero_pages++;break ;case RAM_SAVE_FLAG_PAGE :qemu_get_buffer (f, host,TARGET_PAGE_SIZE ); + mis->received_normal_pages++;break ;case RAM_SAVE_FLAG_XBZRLE :@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f) ret = -EINVAL ;break ; } + mis->received_xbzrle_pages++;break ;case RAM_SAVE_FLAG_MULTIFD_FLUSH :multifd_recv_sync_main ();
Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState
Quoted 64 lines · Claudio Fontana
Quoted 63 lines · Claudio Fontana
Hello,
Quoted 46 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
Add
start_time,total_time,received_normal_pages,received_zero_pages, andreceived_xbzrle_pagesto MigrationIncomingState.
start_timeis recorded when the incoming state transitions to ACTIVE;total_timeis recorded when it transitions to COMPLETED. The page counters are incremented inram_load_precopy()as each page type is received from the source.These fields will be used by a subsequent patch to populate
fill_destination_migration_info()for the migration COMPLETED case.Signed-off-by Trieu Huynh <[email protected]>--- migration/migration.c | 2 ++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ 3 files changed, 22 insertions(+)migration/migration.c +2
-0@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)*/ migrate_set_state (&mis->state,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_COMPLETED ); + mis->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) - mis->start_time;migration_incoming_state_destroy (); }@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)migrate_set_state (&mis->state,MIGRATION_STATUS_SETUP ,MIGRATION_STATUS_ACTIVE ); + mis->start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); mis->loadvm_co =qemu_coroutine_self (); ret =qemu_loadvm_state (mis->from_src_file, &local_err); mis->loadvm_co =NULL ;migration/migration.h +0
-0migration/migration.h struct MigrationIncomingState { · line 185
Quoted 10 lines
migration/migration.h +7
-0@@ -176,6 +176,23 @@ struct MigrationIncomingState {/* PostCopyFD's for external userfaultfds & handlers of shared memory */ GArray *postcopy_remote_fds; +/* +* Timestamps for reporting migration duration via query-migrate on the +* destination. start_time is recorded when the state moves to ACTIVE; +* total_time is recorded when it moves to COMPLETED. +*/ +int64_t start_time; +int64_t total_time;nit: since this is a timestamp and these are directly linked to state changes, why not call these:
Quoted 9 lines
Quoted 2 lines
int64_tmigration_active_ts;int64_tmigration_completed_ts;I got the second one wrong, it is not a timestamp, that is indeed a time. So clarity in the name seems to be more than a nit. What about:
int64_tmigration_active_ts; /* ms */int64_tmigration_completed_time; /* ms */
make sense to me. will update in v2. thank you.
Quoted 45 lines
Thanks,
Claudio
Quoted 40 lines
Quoted 39 lines
migration/migration.h +10
-0+ +/* +* Page counters for reporting RAM statistics via query-migrate on the +* destination. Incremented in ram_load_precopy() as each page type +* is received. +*/ +uint64_t received_normal_pages; +uint64_t received_zero_pages; +uint64_t received_xbzrle_pages; +MigrationStatus state;/* migration/ram.c +3
-0@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)break ; }ram_handle_zero (host,TARGET_PAGE_SIZE ); + mis->received_zero_pages++;break ;case RAM_SAVE_FLAG_PAGE :qemu_get_buffer (f, host,TARGET_PAGE_SIZE ); + mis->received_normal_pages++;break ;case RAM_SAVE_FLAG_XBZRLE :@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f) ret = -EINVAL ;break ; } + mis->received_xbzrle_pages++;break ;case RAM_SAVE_FLAG_MULTIFD_FLUSH :multifd_recv_sync_main ();
BRs,
Trieu Huynh
Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate
In reply to Trieu HuynhHello Trieu,
makes sense to me in general, but maintainers can give you a more proper review on this,
I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:
Quoted 10 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
MigrationStats had all-required QAPI fields, so partially populating
info->ramon the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.Make those source-only fields optional with '*' in the QAPI schema. Update
populate_ram_info()to set them via has_* setters so source output is unchanged.
Quoted 5 lines
As per schema's changes,
fill_destination_migration_info()can now populateinfo->ramfor the COMPLETED case with the fields that are meaningful on the receive side:* total, remaining, page-size (RAM layout)
Does 'remaining' make sense if the migration is completed? Don't we just assume 0? But maybe it makes sense to keep it for simplicity/consistency.
Quoted 2 lines
* transferred, normal, normal-bytes (bytes/pages received)
* duplicate (zero pages received)
it is interesting that zero pages are called "duplicate".
Quoted 127 lines
* mbps (throughput: transferred*8
/time_ms/1000)* pages-per-second (total pages / duration in
seconds)Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.
On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Signed-off-by Trieu Huynh <[email protected]>--- migration/migration.c | 35 +++++++++++++++++++++++++++++++++++ qapi/migration.json | 14 +++++++------- 2 files changed, 42 insertions(+), 7 deletions(-)migration/migration.c +35
-0@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal =qatomic_read (&mig_stats.normal_pages); info->ram->normal_bytes = info->ram->normal * page_size; info->ram->mbps = s->mbps; + info->ram->has_dirty_sync_count = true; info->ram->dirty_sync_count =qatomic_read (&mig_stats.dirty_sync_count); + info->ram->has_dirty_sync_missed_zero_copy = true; info->ram->dirty_sync_missed_zero_copy =qatomic_read (&mig_stats.dirty_sync_missed_zero_copy); + info->ram->has_postcopy_requests = true; info->ram->postcopy_requests =qatomic_read (&mig_stats.postcopy_requests); info->ram->page_size = page_size; + info->ram->has_multifd_bytes = true; info->ram->multifd_bytes =qatomic_read (&mig_stats.multifd_bytes); info->ram->pages_per_second = s->pages_per_second; + info->ram->has_precopy_bytes = true; info->ram->precopy_bytes =qatomic_read (&mig_stats.precopy_bytes); + info->ram->has_downtime_bytes = true; info->ram->downtime_bytes =qatomic_read (&mig_stats.downtime_bytes); + info->ram->has_postcopy_bytes = true; info->ram->postcopy_bytes =qatomic_read (&mig_stats.postcopy_bytes);if (migrate_xbzrle ()) {@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)if (s->state !=MIGRATION_STATUS_COMPLETED ) { info->ram->remaining =ram_bytes_remaining (); + info->ram->has_dirty_pages_rate = true; info->ram->dirty_pages_rate =qatomic_read (&mig_stats.dirty_pages_rate); }@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)case MIGRATION_STATUS_COMPLETED : info->has_status = true;fill_destination_postcopy_migration_info (info); +if (mis->total_time >0 ) { + info->has_total_time = true; + info->total_time = mis->total_time; + } + { +size_t page_size =qemu_target_page_size (); +uint64_t normal = mis->received_normal_pages; +uint64_t zero = mis->received_zero_pages; +uint64_t xbzrle = mis->received_xbzrle_pages; + + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->total =ram_bytes_total (); + info->ram->remaining =0 ; + info->ram->normal = normal + xbzrle; + info->ram->normal_bytes = (normal + xbzrle) * page_size; + info->ram->duplicate = zero; + info->ram->transferred = (normal + xbzrle) * page_size; + info->ram->page_size = page_size; + +if (info->has_total_time) { + info->ram->mbps = info->ram->transferred *8.0 + / info->total_time /1000.0 ; + info->ram->pages_per_second = (normal + xbzrle + zero) + *1000.0 / info->total_time; + } +/* source-only optional omitted from output */ + }break ;default :return ;qapi/migration.json +0
-7@@ -68,13 +68,13 @@ 'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' , 'duplicate': 'int', 'normal': 'int', - 'normal-bytes': 'int', 'dirty-pages-rate': 'int', - 'mbps': 'number', 'dirty-sync-count': 'int', - 'postcopy-requests': 'int', 'page-size': 'int', - 'multifd-bytes': 'uint64', 'pages-per-second': 'uint64', - 'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64', - 'postcopy-bytes': 'uint64', - 'dirty-sync-missed-zero-copy': 'uint64' } }
qapi/migration.json line 80
Quoted 10 lines
qapi/migration.json +7
-0+ 'normal-bytes': 'int', '*dirty-pages-rate': 'int', + 'mbps': 'number', '*dirty-sync-count': 'int', + '*postcopy-requests': 'int', 'page-size': 'int', + '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64', + '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64', + '*postcopy-bytes': 'uint64', + '*dirty-sync-missed-zero-copy': 'uint64' } } ## # @XBZRLECacheStats:
This looks good and makes sense to me,
but looking forward to more authoritative voices.
Does this work build ok on all archs? Any tests that could be extended to cover these new stats, maybe tests/qtest/migration/ ?
Thank you!
Claudio
Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate
Quoted 16 lines · Claudio Fontana
Hello Trieu,
makes sense to me in general, but maintainers can give you a more proper review on this,
I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:Quoted 10 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
MigrationStats had all-required QAPI fields, so partially populating
info->ramon the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.Make those source-only fields optional with '*' in the QAPI schema. Update
populate_ram_info()to set them via has_* setters so source output is unchanged.
Quoted 9 lines
Quoted 5 lines
As per schema's changes,
fill_destination_migration_info()can now populateinfo->ramfor the COMPLETED case with the fields that are meaningful on the receive side:* total, remaining, page-size (RAM layout)
Does 'remaining' make sense if the migration is completed? Don't we just assume 0? But maybe it makes sense to keep it for simplicity/consistency.
Indeed, it set to 0 when migration status is completed, just to be consistent.
Quoted 5 lines
Quoted 2 lines
* transferred, normal, normal-bytes (bytes/pages received)
* duplicate (zero pages received)it is interesting that zero pages are called "duplicate".
Yes, it is. IIUC, in migration stats, duplicate = "pages we didn't need to send because dest's memory is zero-initialized on startup, so any all-zero src page is already "duplicate" of what the dst has.
Quoted 135 lines
Quoted 135 lines
* mbps (throughput: transferred*8
/time_ms/1000)* pages-per-second (total pages / duration in
seconds)Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.
On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Signed-off-by Trieu Huynh <[email protected]>--- migration/migration.c | 35 +++++++++++++++++++++++++++++++++++ qapi/migration.json | 14 +++++++------- 2 files changed, 42 insertions(+), 7 deletions(-)migration/migration.c +35
-0@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal =qatomic_read (&mig_stats.normal_pages); info->ram->normal_bytes = info->ram->normal * page_size; info->ram->mbps = s->mbps; + info->ram->has_dirty_sync_count = true; info->ram->dirty_sync_count =qatomic_read (&mig_stats.dirty_sync_count); + info->ram->has_dirty_sync_missed_zero_copy = true; info->ram->dirty_sync_missed_zero_copy =qatomic_read (&mig_stats.dirty_sync_missed_zero_copy); + info->ram->has_postcopy_requests = true; info->ram->postcopy_requests =qatomic_read (&mig_stats.postcopy_requests); info->ram->page_size = page_size; + info->ram->has_multifd_bytes = true; info->ram->multifd_bytes =qatomic_read (&mig_stats.multifd_bytes); info->ram->pages_per_second = s->pages_per_second; + info->ram->has_precopy_bytes = true; info->ram->precopy_bytes =qatomic_read (&mig_stats.precopy_bytes); + info->ram->has_downtime_bytes = true; info->ram->downtime_bytes =qatomic_read (&mig_stats.downtime_bytes); + info->ram->has_postcopy_bytes = true; info->ram->postcopy_bytes =qatomic_read (&mig_stats.postcopy_bytes);if (migrate_xbzrle ()) {@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)if (s->state !=MIGRATION_STATUS_COMPLETED ) { info->ram->remaining =ram_bytes_remaining (); + info->ram->has_dirty_pages_rate = true; info->ram->dirty_pages_rate =qatomic_read (&mig_stats.dirty_pages_rate); }@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)case MIGRATION_STATUS_COMPLETED : info->has_status = true;fill_destination_postcopy_migration_info (info); +if (mis->total_time >0 ) { + info->has_total_time = true; + info->total_time = mis->total_time; + } + { +size_t page_size =qemu_target_page_size (); +uint64_t normal = mis->received_normal_pages; +uint64_t zero = mis->received_zero_pages; +uint64_t xbzrle = mis->received_xbzrle_pages; + + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->total =ram_bytes_total (); + info->ram->remaining =0 ; + info->ram->normal = normal + xbzrle; + info->ram->normal_bytes = (normal + xbzrle) * page_size; + info->ram->duplicate = zero; + info->ram->transferred = (normal + xbzrle) * page_size; + info->ram->page_size = page_size; + +if (info->has_total_time) { + info->ram->mbps = info->ram->transferred *8.0 + / info->total_time /1000.0 ; + info->ram->pages_per_second = (normal + xbzrle + zero) + *1000.0 / info->total_time; + } +/* source-only optional omitted from output */ + }break ;default :return ;qapi/migration.json +7
-7@@ -68,13 +68,13 @@ 'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' , 'duplicate': 'int', 'normal': 'int', - 'normal-bytes': 'int', 'dirty-pages-rate': 'int', - 'mbps': 'number', 'dirty-sync-count': 'int', - 'postcopy-requests': 'int', 'page-size': 'int', - 'multifd-bytes': 'uint64', 'pages-per-second': 'uint64', - 'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64', - 'postcopy-bytes': 'uint64', - 'dirty-sync-missed-zero-copy': 'uint64' } } + 'normal-bytes': 'int', '*dirty-pages-rate': 'int', + 'mbps': 'number', '*dirty-sync-count': 'int', + '*postcopy-requests': 'int', 'page-size': 'int', + '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64', + '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64', + '*postcopy-bytes': 'uint64', + '*dirty-sync-missed-zero-copy': 'uint64' } }
Quoted 6 lines
qapi/migration.json line 80
Quoted 2 lines
qapi/migration.json +0
-0## # @XBZRLECacheStats:This looks good and makes sense to me,
but looking forward to more authoritative voices.
Thanks, let's waiting.
Quoted 1 line
Does this work build ok on all archs? Any tests that could be extended to cover these new stats,
Actually, I have tested x86_64 but not other archs. The patch are arch-independent IIUC, so I expected it will work without impact any existing callers on other archs. Anw, I will check it.
Quoted 2 lines
maybe tests/qtest/migration/ ?
Thanks for the point. I think can reuse existing test_precopy_tcp_plain. Will try to add it in v2.
Quoted 4 lines
Thank you!
Claudio
BRs,
Trieu Huynh
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrateQuoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Thanks,
Signature · 1 line
Peter Xu
Quoted 71 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrateQuoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)Quoted 5 lines
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
Quoted 9 lines
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.
Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFF
So one might say:
flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
Thanks,
Claudio
Quoted 27 lines · Claudio Fontana
Quoted 26 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
Quoted 8 lines
Quoted 2 lines
[..]
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Quoted 6 lines
Quoted 2 lines
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:
Quoted 25 lines
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.Something similar to (and potentially combined with) virDomainMigrateFlags
VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFFSo one might say:
flags =
VIR_MIGRATE_LIVE|VIR_MIGRATE_PERSIST_DEST|VIR_MIGRATE_UNDEFINE_SOURCE|VIR_MIGRATE_PAUSE|VIR_MIGRATE_NO_SOURCE_SHUTOFFnew_domain=virDomainMigrate3(old_domain, conn, params, nparams, flags)and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
Thanks,
Claudio
Quoted 33 lines · Fabian Grünbichler
Quoted 32 lines · Claudio Fontana
Quoted 19 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
Quoted 8 lines
Quoted 2 lines
[..]
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
Quoted 9 lines
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
Quoted 8 lines
Quoted 1 line
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:
Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?
I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?
Quoted 27 lines
Quoted 26 lines
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.Something similar to (and potentially combined with) virDomainMigrateFlags
VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFFSo one might say:
flags =
VIR_MIGRATE_LIVE|VIR_MIGRATE_PERSIST_DEST|VIR_MIGRATE_UNDEFINE_SOURCE|VIR_MIGRATE_PAUSE|VIR_MIGRATE_NO_SOURCE_SHUTOFFnew_domain=virDomainMigrate3(old_domain, conn, params, nparams, flags)and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
Thanks,
Claudio
Quoted 41 lines · Claudio Fontana
Quoted 40 lines · Fabian Grünbichler
Quoted 27 lines · Claudio Fontana
Quoted 26 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
Quoted 8 lines
Quoted 2 lines
[..]
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Quoted 6 lines
Quoted 2 lines
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:
Quoted 4 lines
Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?
not really - the source node is driving the migration anyway, so it can also do the polling + logging. there is no management task running on the target node/side, after the initial spawning of the VM process.
Quoted 1 line
I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrateQuoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)Quoted 5 lines
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Quoted 6 lines
Quoted 2 lines
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.
Quoted 11 lines
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.Something similar to (and potentially combined with) virDomainMigrateFlags
VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFFSo one might say:
Quoted 8 lines
flags =
VIR_MIGRATE_LIVE|VIR_MIGRATE_PERSIST_DEST|VIR_MIGRATE_UNDEFINE_SOURCE|VIR_MIGRATE_PAUSE|VIR_MIGRATE_NO_SOURCE_SHUTOFFnew_domain=virDomainMigrate3(old_domain, conn, params, nparams, flags)and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.
If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.
Thanks,
Quoted 5 lines
Thanks,
Claudio
Signature · 1 line
Peter Xu
Quoted 116 lines · Peter Xu
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrateQuoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)Quoted 5 lines
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Quoted 6 lines
Quoted 2 lines
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.
Quoted 11 lines
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.Something similar to (and potentially combined with) virDomainMigrateFlags
VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFFSo one might say:
Quoted 8 lines
flags =
VIR_MIGRATE_LIVE|VIR_MIGRATE_PERSIST_DEST|VIR_MIGRATE_UNDEFINE_SOURCE|VIR_MIGRATE_PAUSE|VIR_MIGRATE_NO_SOURCE_SHUTOFFnew_domain=virDomainMigrate3(old_domain, conn, params, nparams, flags)and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.
Quoted 6 lines
If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.
Hello Peter,
One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU to query_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?
Quoted 12 lines
Thanks,
Quoted 5 lines
Thanks,
Claudio
Signature · 1 line
Peter Xu
BRs,
Trieu Huynh
Quoted 1 line · Trieu Huynh
Hello Peter,
Hi, Trieu,
Quoted 6 lines
One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU toquery_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?
Yes it's in general valid, but it also depends. There's a reason that QEMU supported load snapshot over the years and nobody was asking for it. I believe it's because people simply don't need it.
Load snapshot, unlike generic form of live migration, doesn't have much uncertainty on its own. It should either succeed after a while, or fail with an error. People should not expect to need to query anything.
It will likely, and should, hold true even if with async load snapshot in the future.
It is only valid in that it might be useful for developers who works on snapshot, especially tuning the performance. But then it also means we shouldn't expose it to an user API (QMP is in this case). For dev only reportings, we should rely on trace points unless we have good reason to make it visible to users.
In general, we don't need to add unnecessary API maint burden to ourselves when not needed.
Thanks,
Signature · 1 line
Peter Xu
Quoted 23 lines · Peter Xu
Quoted 1 line · Trieu Huynh
Hello Peter,
Hi, Trieu,
Quoted 6 lines
One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU toquery_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?Yes it's in general valid, but it also depends. There's a reason that QEMU supported load snapshot over the years and nobody was asking for it. I believe it's because people simply don't need it.
Load snapshot, unlike generic form of live migration, doesn't have much uncertainty on its own. It should either succeed after a while, or fail with an error. People should not expect to need to query anything.
It will likely, and should, hold true even if with async load snapshot in the future.
Quoted 8 lines
It is only valid in that it might be useful for developers who works on snapshot, especially tuning the performance. But then it also means we shouldn't expose it to an user API (QMP is in this case). For dev only reportings, we should rely on trace points unless we have good reason to make it visible to users.
In general, we don't need to add unnecessary API maint burden to ourselves when not needed.
Thanks for your detailed feedback, I appreciate it.
Taking your comments from the other thread into account, I might drop this series for now. As I don't have a deep understanding of libvirt or similar management tools, the specific use case remains a bit ambiguous to me. I’ll leave this series/patches here for now in case Claudio and other experts want to chime in or if further concerns arise. Thanks again!
Quoted 6 lines
Thanks,
Signature · 1 line
Peter Xu
Brs,
Trieu Huynh
Quoted 116 lines · Peter Xu
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh
From: Trieu Huynh <[email protected]>
When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.
This series fixes the gap in two commits:
* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.
* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate
info->ramwith only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because
populate_ram_info()sets all optional fields via has_* setters, so the source output is identical to before.On dst, {"execute":"query-migrate"}
* As-is:{ "return": { "status": "completed" } * To-be: { "return": { "status": "completed", "total-time": 94, "ram": { "total": 554508288, "pages-per-second": 1440234, "page-size": 4096, "remaining": 0, "mbps": 97.955404255319152, "transferred": 1150976, "duplicate": 135101, "normal-bytes": 1150976, "normal": 281 } } }Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrateQuoted 5 lines
migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++ migration/migration.h | 17 +++++++++++++++++ migration/ram.c | 3 +++ qapi/migration.json | 14 +++++++------- 4 files changed, 64 insertions(+), 7 deletions(-)Quoted 5 lines
Hi, remember to copy the interested people in your series. Using
get_maintainersis correct, but when there's already a discussion about the topic it's good to add some CCs manually.+CC Daniel and Claudio
The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.
Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?
Quoted 6 lines
Quoted 2 lines
Thanks,
I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.
Quoted 11 lines
So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.Something similar to (and potentially combined with) virDomainMigrateFlags
VIR_MIGRATE_PAUSE, for example:
VIR_MIGRATE_NO_SOURCE_SHUTOFFSo one might say:
Quoted 8 lines
flags =
VIR_MIGRATE_LIVE|VIR_MIGRATE_PERSIST_DEST|VIR_MIGRATE_UNDEFINE_SOURCE|VIR_MIGRATE_PAUSE|VIR_MIGRATE_NO_SOURCE_SHUTOFFnew_domain=virDomainMigrate3(old_domain, conn, params, nparams, flags)and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.
Quoted 6 lines
If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.
I came to the thread to suggest this, but I see you already mentioned it. I'm not sure what you mean by "forward", but I was actually thinking "migrate". Do we have precedent for migrating a QAPI object? Do we want to have it? =) Put that MigrationInfo at the end of the stream and we'd have a nice symmetric interface for query-migrate in both src and dst.
(we have plans to deal with QEMU_VM_VMDESCRIPTION at some point anyway, we could maybe rework that end section into something useful)
Quoted 8 lines
Thanks,
Quoted 5 lines
Thanks,
Claudio
Quoted 5 lines · Fabiano Rosas
I came to the thread to suggest this, but I see you already mentioned it. I'm not sure what you mean by "forward", but I was actually thinking "migrate". Do we have precedent for migrating a QAPI object? Do we want to have it? =) Put that MigrationInfo at the end of the stream and we'd have a nice symmetric interface for query-migrate in both src and dst.
Do we need to "migrate" it, say, using VMSD data structures?
We could, but I'm not yet sure if it's a good idea, especially if we should make that as part of ABI. It may add some unnecessary burden to us.
Quoted 3 lines
(we have plans to deal with
QEMU_VM_VMDESCRIPTIONat some point anyway, we could maybe rework that end section into something useful)
That might still be a bit early phase to contain all the stats info. For example, the real downtime needs to be calculated until src receiving the ACK, that should be after dest receive QEMU_VM_VMDESCRIPTION, IIRC. So if we want to at least forward the downtime info, that might be too early.
In all cases, I still think we should evaluate libvirt solution before any other explorations.. because so far we still don't know why it can't be done, and it's the simplest afaiu.
Signature · 1 line
Peter Xu