Skip to contentSkip to conversation

migration: include timing and RAM stats on destination when query-migrate

PATCH In review Trieu Huynh 5 participants 20 messages Apr 5 – Apr 23

lore
Share & tools
lore
#0 lore raw

cover migration: include timing and RAM stats on destination when query-migrate

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/2 migration: track timing and received pages in MigrationIncomingState

From: Trieu Huynh <[email protected]>

Add start_time, total_time, received_normal_pages, received_zero_pages, and received_xbzrle_pages to MigrationIncomingState.

start_time is recorded when the incoming state transitions to ACTIVE; total_time is recorded when it transitions to COMPLETED. The page counters are incremented in ram_load_precopy() as each page type is received from the source.

These fields will be used by a subsequent patch to populate fill_destination_migration_info() for the migration COMPLETED case.

---
 migration/migration.c |  2 ++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 3 files changed, 22 insertions(+)
migration/migration.c +2 -0
index 5c9aaa6e58..17c9a8b344 100644
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
+    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
     migration_incoming_state_destroy();
 }
 
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
migration/migration.h +17 -0
index b6888daced..cd51ae452c 100644
@@ -176,6 +176,23 @@ struct MigrationIncomingState {
     /* PostCopyFD's for external userfaultfds & handlers of shared memory */
     GArray   *postcopy_remote_fds;
 
+    /*
+     * Timestamps for reporting migration duration via query-migrate on the
+     * destination. start_time is recorded when the state moves to ACTIVE;
+     * total_time is recorded when it moves to COMPLETED.
+     */
+    int64_t start_time;
+    int64_t total_time;
+
+    /*
+     * Page counters for reporting RAM statistics via query-migrate on the
+     * destination.  Incremented in ram_load_precopy() as each page type
+     * is received.
+     */
+    uint64_t received_normal_pages;
+    uint64_t received_zero_pages;
+    uint64_t received_xbzrle_pages;
+
     MigrationStatus state;
 
     /*
migration/ram.c +3 -0
index 979751f61b..6646e5daad 100644
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
                 break;
             }
             ram_handle_zero(host, TARGET_PAGE_SIZE);
+            mis->received_zero_pages++;
             break;
 
         case RAM_SAVE_FLAG_PAGE:
             qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
+            mis->received_normal_pages++;
             break;
 
         case RAM_SAVE_FLAG_XBZRLE:
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
                 ret = -EINVAL;
                 break;
             }
+            mis->received_xbzrle_pages++;
             break;
         case RAM_SAVE_FLAG_MULTIFD_FLUSH:
             multifd_recv_sync_main();
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

Hello,

Quoted 46 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

Add start_time, total_time, received_normal_pages, received_zero_pages, and received_xbzrle_pages to MigrationIncomingState.

start_time is recorded when the incoming state transitions to ACTIVE; total_time is recorded when it transitions to COMPLETED. The page counters are incremented in ram_load_precopy() as each page type is received from the source.

These fields will be used by a subsequent patch to populate fill_destination_migration_info() for the migration COMPLETED case.

---
 migration/migration.c |  2 ++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 3 files changed, 22 insertions(+)
migration/migration.c +2 -0
index 5c9aaa6e58..17c9a8b344 100644
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
+    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
     migration_incoming_state_destroy();
 }
 
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
migration/migration.h +0 -0
index b6888daced..cd51ae452c 100644

migration/migration.h struct MigrationIncomingState { · line 185

Quoted 10 lines
migration/migration.h +7 -0
@@ -176,6 +176,23 @@ struct MigrationIncomingState {
     /* PostCopyFD's for external userfaultfds & handlers of shared memory */
     GArray   *postcopy_remote_fds;
 
+    /*
+     * Timestamps for reporting migration duration via query-migrate on the
+     * destination. start_time is recorded when the state moves to ACTIVE;
+     * total_time is recorded when it moves to COMPLETED.
+     */
+    int64_t start_time;
+    int64_t total_time;

nit: since this is a timestamp and these are directly linked to state changes, why not call these:

int64_t migration_active_ts;
int64_t migration_completed_ts;

or something similar?

Thank you,

Claudio

Quoted 38 lines
migration/migration.h +10 -0
+
+    /*
+     * Page counters for reporting RAM statistics via query-migrate on the
+     * destination.  Incremented in ram_load_precopy() as each page type
+     * is received.
+     */
+    uint64_t received_normal_pages;
+    uint64_t received_zero_pages;
+    uint64_t received_xbzrle_pages;
+
     MigrationStatus state;
 
     /*
migration/ram.c +3 -0
index 979751f61b..6646e5daad 100644
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
                 break;
             }
             ram_handle_zero(host, TARGET_PAGE_SIZE);
+            mis->received_zero_pages++;
             break;
 
         case RAM_SAVE_FLAG_PAGE:
             qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
+            mis->received_normal_pages++;
             break;
 
         case RAM_SAVE_FLAG_XBZRLE:
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
                 ret = -EINVAL;
                 break;
             }
+            mis->received_xbzrle_pages++;
             break;
         case RAM_SAVE_FLAG_MULTIFD_FLUSH:
             multifd_recv_sync_main();
#3 lore raw

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

Quoted 63 lines · Claudio Fontana

Hello,

Quoted 46 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

Add start_time, total_time, received_normal_pages, received_zero_pages, and received_xbzrle_pages to MigrationIncomingState.

start_time is recorded when the incoming state transitions to ACTIVE; total_time is recorded when it transitions to COMPLETED. The page counters are incremented in ram_load_precopy() as each page type is received from the source.

These fields will be used by a subsequent patch to populate fill_destination_migration_info() for the migration COMPLETED case.

---
 migration/migration.c |  2 ++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 3 files changed, 22 insertions(+)
migration/migration.c +2 -0
index 5c9aaa6e58..17c9a8b344 100644
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
+    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
     migration_incoming_state_destroy();
 }
 
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
migration/migration.h +0 -0
index b6888daced..cd51ae452c 100644

migration/migration.h struct MigrationIncomingState { · line 185

Quoted 10 lines
migration/migration.h +7 -0
@@ -176,6 +176,23 @@ struct MigrationIncomingState {
     /* PostCopyFD's for external userfaultfds & handlers of shared memory */
     GArray   *postcopy_remote_fds;
 
+    /*
+     * Timestamps for reporting migration duration via query-migrate on the
+     * destination. start_time is recorded when the state moves to ACTIVE;
+     * total_time is recorded when it moves to COMPLETED.
+     */
+    int64_t start_time;
+    int64_t total_time;

nit: since this is a timestamp and these are directly linked to state changes, why not call these:

Quoted 2 lines

int64_t migration_active_ts;
int64_t migration_completed_ts;

I got the second one wrong, it is not a timestamp, that is indeed a time. So clarity in the name seems to be more than a nit. What about:

int64_t migration_active_ts; /* ms */
int64_t migration_completed_time; /* ms */

Thanks,

Claudio

Quoted 40 lines
Quoted 39 lines
migration/migration.h +10 -0
+
+    /*
+     * Page counters for reporting RAM statistics via query-migrate on the
+     * destination.  Incremented in ram_load_precopy() as each page type
+     * is received.
+     */
+    uint64_t received_normal_pages;
+    uint64_t received_zero_pages;
+    uint64_t received_xbzrle_pages;
+
     MigrationStatus state;
 
     /*
migration/ram.c +3 -0
index 979751f61b..6646e5daad 100644
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
                 break;
             }
             ram_handle_zero(host, TARGET_PAGE_SIZE);
+            mis->received_zero_pages++;
             break;
 
         case RAM_SAVE_FLAG_PAGE:
             qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
+            mis->received_normal_pages++;
             break;
 
         case RAM_SAVE_FLAG_XBZRLE:
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
                 ret = -EINVAL;
                 break;
             }
+            mis->received_xbzrle_pages++;
             break;
         case RAM_SAVE_FLAG_MULTIFD_FLUSH:
             multifd_recv_sync_main();
#4 lore raw

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

Quoted 64 lines · Claudio Fontana
Quoted 63 lines · Claudio Fontana

Hello,

Quoted 46 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

Add start_time, total_time, received_normal_pages, received_zero_pages, and received_xbzrle_pages to MigrationIncomingState.

start_time is recorded when the incoming state transitions to ACTIVE; total_time is recorded when it transitions to COMPLETED. The page counters are incremented in ram_load_precopy() as each page type is received from the source.

These fields will be used by a subsequent patch to populate fill_destination_migration_info() for the migration COMPLETED case.

---
 migration/migration.c |  2 ++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 3 files changed, 22 insertions(+)
migration/migration.c +2 -0
index 5c9aaa6e58..17c9a8b344 100644
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
+    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
     migration_incoming_state_destroy();
 }
 
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
migration/migration.h +0 -0
index b6888daced..cd51ae452c 100644

migration/migration.h struct MigrationIncomingState { · line 185

Quoted 10 lines
migration/migration.h +7 -0
@@ -176,6 +176,23 @@ struct MigrationIncomingState {
     /* PostCopyFD's for external userfaultfds & handlers of shared memory */
     GArray   *postcopy_remote_fds;
 
+    /*
+     * Timestamps for reporting migration duration via query-migrate on the
+     * destination. start_time is recorded when the state moves to ACTIVE;
+     * total_time is recorded when it moves to COMPLETED.
+     */
+    int64_t start_time;
+    int64_t total_time;

nit: since this is a timestamp and these are directly linked to state changes, why not call these:

Quoted 9 lines
Quoted 2 lines

int64_t migration_active_ts;
int64_t migration_completed_ts;

I got the second one wrong, it is not a timestamp, that is indeed a time. So clarity in the name seems to be more than a nit. What about:

int64_t migration_active_ts; /* ms */
int64_t migration_completed_time; /* ms */

make sense to me. will update in v2. thank you.

Quoted 45 lines

Thanks,

Claudio

Quoted 40 lines
Quoted 39 lines
migration/migration.h +10 -0
+
+    /*
+     * Page counters for reporting RAM statistics via query-migrate on the
+     * destination.  Incremented in ram_load_precopy() as each page type
+     * is received.
+     */
+    uint64_t received_normal_pages;
+    uint64_t received_zero_pages;
+    uint64_t received_xbzrle_pages;
+
     MigrationStatus state;
 
     /*
migration/ram.c +3 -0
index 979751f61b..6646e5daad 100644
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
                 break;
             }
             ram_handle_zero(host, TARGET_PAGE_SIZE);
+            mis->received_zero_pages++;
             break;
 
         case RAM_SAVE_FLAG_PAGE:
             qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
+            mis->received_normal_pages++;
             break;
 
         case RAM_SAVE_FLAG_XBZRLE:
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
                 ret = -EINVAL;
                 break;
             }
+            mis->received_xbzrle_pages++;
             break;
         case RAM_SAVE_FLAG_MULTIFD_FLUSH:
             multifd_recv_sync_main();

BRs,
Trieu Huynh

#5 lore raw

PATCH 2/2 migration: expose RAM stats and timing on destination via query-migrate

In reply to Trieu Huynh

From: Trieu Huynh <[email protected]>

MigrationStats had all-required QAPI fields, so partially populating info->ram on the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.

Make those source-only fields optional with '*' in the QAPI schema. Update populate_ram_info() to set them via has_* setters so source output is unchanged.

As per schema's changes, fill_destination_migration_info() can now populate info->ram for the COMPLETED case with the fields that are meaningful on the receive side:

  * total, remaining, page-size         (RAM layout)
  * transferred, normal, normal-bytes   (bytes/pages received)
  * duplicate                           (zero pages received)
  * mbps                                (throughput: transferred*8
                                         /time_ms/1000)

* pages-per-second (total pages / duration in
seconds)

Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}
---
 migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
 qapi/migration.json   | 14 +++++++-------
 2 files changed, 42 insertions(+), 7 deletions(-)
migration/migration.c +35 -0
index 17c9a8b344..925d29890d 100644
@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal = qatomic_read(&mig_stats.normal_pages);
     info->ram->normal_bytes = info->ram->normal * page_size;
     info->ram->mbps = s->mbps;
+    info->ram->has_dirty_sync_count = true;
     info->ram->dirty_sync_count =
         qatomic_read(&mig_stats.dirty_sync_count);
+    info->ram->has_dirty_sync_missed_zero_copy = true;
     info->ram->dirty_sync_missed_zero_copy =
         qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
+    info->ram->has_postcopy_requests = true;
     info->ram->postcopy_requests =
         qatomic_read(&mig_stats.postcopy_requests);
     info->ram->page_size = page_size;
+    info->ram->has_multifd_bytes = true;
     info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
     info->ram->pages_per_second = s->pages_per_second;
+    info->ram->has_precopy_bytes = true;
     info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
+    info->ram->has_downtime_bytes = true;
     info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
+    info->ram->has_postcopy_bytes = true;
     info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
 
     if (migrate_xbzrle()) {
@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
 
     if (s->state != MIGRATION_STATUS_COMPLETED) {
         info->ram->remaining = ram_bytes_remaining();
+        info->ram->has_dirty_pages_rate = true;
         info->ram->dirty_pages_rate =
            qatomic_read(&mig_stats.dirty_pages_rate);
     }
@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
     case MIGRATION_STATUS_COMPLETED:
         info->has_status = true;
         fill_destination_postcopy_migration_info(info);
+        if (mis->total_time > 0) {
+            info->has_total_time = true;
+            info->total_time = mis->total_time;
+        }
+        {
+            size_t page_size = qemu_target_page_size();
+            uint64_t normal  = mis->received_normal_pages;
+            uint64_t zero    = mis->received_zero_pages;
+            uint64_t xbzrle  = mis->received_xbzrle_pages;
+
+            info->ram = g_malloc0(sizeof(*info->ram));
+            info->ram->total        = ram_bytes_total();
+            info->ram->remaining    = 0;
+            info->ram->normal       = normal + xbzrle;
+            info->ram->normal_bytes = (normal + xbzrle) * page_size;
+            info->ram->duplicate    = zero;
+            info->ram->transferred  = (normal + xbzrle) * page_size;
+            info->ram->page_size    = page_size;
+
+            if (info->has_total_time) {
+                info->ram->mbps = info->ram->transferred * 8.0
+                                  / info->total_time / 1000.0;
+                info->ram->pages_per_second = (normal + xbzrle + zero)
+                                              * 1000.0 / info->total_time;
+            }
+            /* source-only optional omitted from output */
+        }
         break;
     default:
         return;
qapi/migration.json +7 -7
index 7134d4ce47..a695d04a22 100644
@@ -68,13 +68,13 @@
   'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
            'duplicate': 'int',
            'normal': 'int',
-           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
-           'mbps': 'number', 'dirty-sync-count': 'int',
-           'postcopy-requests': 'int', 'page-size': 'int',
-           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
-           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
-           'postcopy-bytes': 'uint64',
-           'dirty-sync-missed-zero-copy': 'uint64' } }
+           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
+           'mbps': 'number', '*dirty-sync-count': 'int',
+           '*postcopy-requests': 'int', 'page-size': 'int',
+           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
+           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
+           '*postcopy-bytes': 'uint64',
+           '*dirty-sync-missed-zero-copy': 'uint64' } }
 
 ##
 # @XBZRLECacheStats:
Signature · 1 line
2.43.0
#6 lore raw

Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate

Hello Trieu,

makes sense to me in general, but maintainers can give you a more proper review on this,
I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:

Quoted 10 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

MigrationStats had all-required QAPI fields, so partially populating info->ram on the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.

Make those source-only fields optional with '*' in the QAPI schema. Update populate_ram_info() to set them via has_* setters so source output is unchanged.

Quoted 5 lines

As per schema's changes, fill_destination_migration_info() can now populate info->ram for the COMPLETED case with the fields that are meaningful on the receive side:

* total, remaining, page-size (RAM layout)

Does 'remaining' make sense if the migration is completed? Don't we just assume 0? But maybe it makes sense to keep it for simplicity/consistency.

Quoted 2 lines

* transferred, normal, normal-bytes (bytes/pages received)
* duplicate (zero pages received)

it is interesting that zero pages are called "duplicate".

Quoted 127 lines

* mbps (throughput: transferred*8

                                         /time_ms/1000)

* pages-per-second (total pages / duration in
seconds)

Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}
---
 migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
 qapi/migration.json   | 14 +++++++-------
 2 files changed, 42 insertions(+), 7 deletions(-)
migration/migration.c +35 -0
index 17c9a8b344..925d29890d 100644
@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal = qatomic_read(&mig_stats.normal_pages);
     info->ram->normal_bytes = info->ram->normal * page_size;
     info->ram->mbps = s->mbps;
+    info->ram->has_dirty_sync_count = true;
     info->ram->dirty_sync_count =
         qatomic_read(&mig_stats.dirty_sync_count);
+    info->ram->has_dirty_sync_missed_zero_copy = true;
     info->ram->dirty_sync_missed_zero_copy =
         qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
+    info->ram->has_postcopy_requests = true;
     info->ram->postcopy_requests =
         qatomic_read(&mig_stats.postcopy_requests);
     info->ram->page_size = page_size;
+    info->ram->has_multifd_bytes = true;
     info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
     info->ram->pages_per_second = s->pages_per_second;
+    info->ram->has_precopy_bytes = true;
     info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
+    info->ram->has_downtime_bytes = true;
     info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
+    info->ram->has_postcopy_bytes = true;
     info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
 
     if (migrate_xbzrle()) {
@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
 
     if (s->state != MIGRATION_STATUS_COMPLETED) {
         info->ram->remaining = ram_bytes_remaining();
+        info->ram->has_dirty_pages_rate = true;
         info->ram->dirty_pages_rate =
            qatomic_read(&mig_stats.dirty_pages_rate);
     }
@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
     case MIGRATION_STATUS_COMPLETED:
         info->has_status = true;
         fill_destination_postcopy_migration_info(info);
+        if (mis->total_time > 0) {
+            info->has_total_time = true;
+            info->total_time = mis->total_time;
+        }
+        {
+            size_t page_size = qemu_target_page_size();
+            uint64_t normal  = mis->received_normal_pages;
+            uint64_t zero    = mis->received_zero_pages;
+            uint64_t xbzrle  = mis->received_xbzrle_pages;
+
+            info->ram = g_malloc0(sizeof(*info->ram));
+            info->ram->total        = ram_bytes_total();
+            info->ram->remaining    = 0;
+            info->ram->normal       = normal + xbzrle;
+            info->ram->normal_bytes = (normal + xbzrle) * page_size;
+            info->ram->duplicate    = zero;
+            info->ram->transferred  = (normal + xbzrle) * page_size;
+            info->ram->page_size    = page_size;
+
+            if (info->has_total_time) {
+                info->ram->mbps = info->ram->transferred * 8.0
+                                  / info->total_time / 1000.0;
+                info->ram->pages_per_second = (normal + xbzrle + zero)
+                                              * 1000.0 / info->total_time;
+            }
+            /* source-only optional omitted from output */
+        }
         break;
     default:
         return;
qapi/migration.json +0 -7
index 7134d4ce47..a695d04a22 100644
@@ -68,13 +68,13 @@
   'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
            'duplicate': 'int',
            'normal': 'int',
-           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
-           'mbps': 'number', 'dirty-sync-count': 'int',
-           'postcopy-requests': 'int', 'page-size': 'int',
-           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
-           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
-           'postcopy-bytes': 'uint64',
-           'dirty-sync-missed-zero-copy': 'uint64' } }

qapi/migration.json line 80

Quoted 10 lines
qapi/migration.json +7 -0
+           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
+           'mbps': 'number', '*dirty-sync-count': 'int',
+           '*postcopy-requests': 'int', 'page-size': 'int',
+           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
+           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
+           '*postcopy-bytes': 'uint64',
+           '*dirty-sync-missed-zero-copy': 'uint64' } }
 
 ##
 # @XBZRLECacheStats:

This looks good and makes sense to me,
but looking forward to more authoritative voices.

Does this work build ok on all archs? Any tests that could be extended to cover these new stats, maybe tests/qtest/migration/ ?

Thank you!

Claudio

#7 lore raw

Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate

Quoted 16 lines · Claudio Fontana

Hello Trieu,

makes sense to me in general, but maintainers can give you a more proper review on this,
I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:

Quoted 10 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

MigrationStats had all-required QAPI fields, so partially populating info->ram on the destination would force display of source-only metrics (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.

Make those source-only fields optional with '*' in the QAPI schema. Update populate_ram_info() to set them via has_* setters so source output is unchanged.

Quoted 9 lines
Quoted 5 lines

As per schema's changes, fill_destination_migration_info() can now populate info->ram for the COMPLETED case with the fields that are meaningful on the receive side:

* total, remaining, page-size (RAM layout)

Does 'remaining' make sense if the migration is completed? Don't we just assume 0? But maybe it makes sense to keep it for simplicity/consistency.

Indeed, it set to 0 when migration status is completed, just to be consistent.

Quoted 5 lines
Quoted 2 lines

* transferred, normal, normal-bytes (bytes/pages received)
* duplicate (zero pages received)

it is interesting that zero pages are called "duplicate".

Yes, it is. IIUC, in migration stats, duplicate = "pages we didn't need to send because dest's memory is zero-initialized on startup, so any all-zero src page is already "duplicate" of what the dst has.

Quoted 135 lines
Quoted 135 lines

* mbps (throughput: transferred*8

                                         /time_ms/1000)

* pages-per-second (total pages / duration in
seconds)

Others optional fields (dirty-sync-count, precopy-bytes, downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests, dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and are absent from the destination output.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}
---
 migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
 qapi/migration.json   | 14 +++++++-------
 2 files changed, 42 insertions(+), 7 deletions(-)
migration/migration.c +35 -0
index 17c9a8b344..925d29890d 100644
@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal = qatomic_read(&mig_stats.normal_pages);
     info->ram->normal_bytes = info->ram->normal * page_size;
     info->ram->mbps = s->mbps;
+    info->ram->has_dirty_sync_count = true;
     info->ram->dirty_sync_count =
         qatomic_read(&mig_stats.dirty_sync_count);
+    info->ram->has_dirty_sync_missed_zero_copy = true;
     info->ram->dirty_sync_missed_zero_copy =
         qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
+    info->ram->has_postcopy_requests = true;
     info->ram->postcopy_requests =
         qatomic_read(&mig_stats.postcopy_requests);
     info->ram->page_size = page_size;
+    info->ram->has_multifd_bytes = true;
     info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
     info->ram->pages_per_second = s->pages_per_second;
+    info->ram->has_precopy_bytes = true;
     info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
+    info->ram->has_downtime_bytes = true;
     info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
+    info->ram->has_postcopy_bytes = true;
     info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
 
     if (migrate_xbzrle()) {
@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
 
     if (s->state != MIGRATION_STATUS_COMPLETED) {
         info->ram->remaining = ram_bytes_remaining();
+        info->ram->has_dirty_pages_rate = true;
         info->ram->dirty_pages_rate =
            qatomic_read(&mig_stats.dirty_pages_rate);
     }
@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
     case MIGRATION_STATUS_COMPLETED:
         info->has_status = true;
         fill_destination_postcopy_migration_info(info);
+        if (mis->total_time > 0) {
+            info->has_total_time = true;
+            info->total_time = mis->total_time;
+        }
+        {
+            size_t page_size = qemu_target_page_size();
+            uint64_t normal  = mis->received_normal_pages;
+            uint64_t zero    = mis->received_zero_pages;
+            uint64_t xbzrle  = mis->received_xbzrle_pages;
+
+            info->ram = g_malloc0(sizeof(*info->ram));
+            info->ram->total        = ram_bytes_total();
+            info->ram->remaining    = 0;
+            info->ram->normal       = normal + xbzrle;
+            info->ram->normal_bytes = (normal + xbzrle) * page_size;
+            info->ram->duplicate    = zero;
+            info->ram->transferred  = (normal + xbzrle) * page_size;
+            info->ram->page_size    = page_size;
+
+            if (info->has_total_time) {
+                info->ram->mbps = info->ram->transferred * 8.0
+                                  / info->total_time / 1000.0;
+                info->ram->pages_per_second = (normal + xbzrle + zero)
+                                              * 1000.0 / info->total_time;
+            }
+            /* source-only optional omitted from output */
+        }
         break;
     default:
         return;
qapi/migration.json +7 -7
index 7134d4ce47..a695d04a22 100644
@@ -68,13 +68,13 @@
   'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
            'duplicate': 'int',
            'normal': 'int',
-           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
-           'mbps': 'number', 'dirty-sync-count': 'int',
-           'postcopy-requests': 'int', 'page-size': 'int',
-           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
-           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
-           'postcopy-bytes': 'uint64',
-           'dirty-sync-missed-zero-copy': 'uint64' } }
+           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
+           'mbps': 'number', '*dirty-sync-count': 'int',
+           '*postcopy-requests': 'int', 'page-size': 'int',
+           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
+           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
+           '*postcopy-bytes': 'uint64',
+           '*dirty-sync-missed-zero-copy': 'uint64' } }
 
Quoted 6 lines

qapi/migration.json line 80

Quoted 2 lines
qapi/migration.json +0 -0
 ##
 # @XBZRLECacheStats:

This looks good and makes sense to me,
but looking forward to more authoritative voices.

Thanks, let's waiting.

Quoted 1 line

Does this work build ok on all archs? Any tests that could be extended to cover these new stats,

Actually, I have tested x86_64 but not other archs. The patch are arch-independent IIUC, so I expected it will work without impact any existing callers on other archs. Anw, I will check it.

Quoted 2 lines

maybe tests/qtest/migration/ ?

Thanks for the point. I think can reuse existing test_precopy_tcp_plain. Will try to add it in v2.

Quoted 4 lines

Thank you!

Claudio

BRs,
Trieu Huynh

#8 lore raw
In reply to Trieu Huynh
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

#9 lore raw
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Thanks,

Signature · 1 line
Peter Xu
#10 lore raw
Quoted 71 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

Quoted 9 lines

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

Thanks,

Claudio

#11 lore raw
Quoted 27 lines · Claudio Fontana
Quoted 26 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

Quoted 8 lines
Quoted 2 lines

[..]

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Quoted 6 lines
Quoted 2 lines

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:

https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

Quoted 25 lines

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

Thanks,

Claudio

#12 lore raw
Quoted 33 lines · Fabian Grünbichler
Quoted 32 lines · Claudio Fontana
Quoted 19 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

Quoted 8 lines
Quoted 2 lines

[..]

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

Quoted 9 lines

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

Quoted 8 lines
Quoted 1 line

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:

https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?

Quoted 27 lines
Quoted 26 lines

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

Thanks,

Claudio

#13 lore raw
Quoted 41 lines · Claudio Fontana
Quoted 40 lines · Fabian Grünbichler
Quoted 27 lines · Claudio Fontana
Quoted 26 lines · Peter Xu
Quoted 9 lines · Fabiano Rosas
Quoted 6 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

Quoted 8 lines
Quoted 2 lines

[..]

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Quoted 6 lines
Quoted 2 lines

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

that is correct, we call query-migrate in a loop while migrating both to log progress and to manage migration parameters, and manually switch execution and ownership over to the target node once the migration has converged:

Quoted 4 lines
Quoted 1 line

https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

not really - the source node is driving the migration anyway, so it can also do the polling + logging. there is no management task running on the target node/side, after the initial spawning of the VM process.

Quoted 1 line

I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?

#14 lore raw
In reply to Claudio Fontana
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Quoted 6 lines
Quoted 2 lines

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.

Quoted 11 lines

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

Quoted 8 lines

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.

If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.

Thanks,

Quoted 5 lines

Thanks,

Claudio

Signature · 1 line
Peter Xu
#15 lore raw
Quoted 116 lines · Peter Xu
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Quoted 6 lines
Quoted 2 lines

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.

Quoted 11 lines

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

Quoted 8 lines

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.

Quoted 6 lines

If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.

Hello Peter,
One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU to query_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?

Quoted 12 lines

Thanks,

Quoted 5 lines

Thanks,

Claudio

Signature · 1 line
Peter Xu

BRs,
Trieu Huynh

#16 lore raw
Quoted 1 line · Trieu Huynh

Hello Peter,

Hi, Trieu,

Quoted 6 lines

One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU to query_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?

Yes it's in general valid, but it also depends. There's a reason that QEMU supported load snapshot over the years and nobody was asking for it. I believe it's because people simply don't need it.

Load snapshot, unlike generic form of live migration, doesn't have much uncertainty on its own. It should either succeed after a while, or fail with an error. People should not expect to need to query anything.

It will likely, and should, hold true even if with async load snapshot in the future.

It is only valid in that it might be useful for developers who works on snapshot, especially tuning the performance. But then it also means we shouldn't expose it to an user API (QMP is in this case). For dev only reportings, we should rely on trace points unless we have good reason to make it visible to users.

In general, we don't need to add unnecessary API maint burden to ourselves when not needed.

Thanks,

Signature · 1 line
Peter Xu
#17 lore raw
Quoted 23 lines · Peter Xu
Quoted 1 line · Trieu Huynh

Hello Peter,

Hi, Trieu,

Quoted 6 lines

One specific use case where the source is completely unavailable is loading from a migration snapshot (non-live).
In this scenario, there is no active source QEMU to query_for_stats. The dst (IMHO) is the only entity that can report how long the loading took and how much data was actually processed from the snapshot file. This use case is a little bit niche but still a valid one though, isn't it?

Yes it's in general valid, but it also depends. There's a reason that QEMU supported load snapshot over the years and nobody was asking for it. I believe it's because people simply don't need it.

Load snapshot, unlike generic form of live migration, doesn't have much uncertainty on its own. It should either succeed after a while, or fail with an error. People should not expect to need to query anything.

It will likely, and should, hold true even if with async load snapshot in the future.

Quoted 8 lines

It is only valid in that it might be useful for developers who works on snapshot, especially tuning the performance. But then it also means we shouldn't expose it to an user API (QMP is in this case). For dev only reportings, we should rely on trace points unless we have good reason to make it visible to users.

In general, we don't need to add unnecessary API maint burden to ourselves when not needed.

Thanks for your detailed feedback, I appreciate it.
Taking your comments from the other thread into account, I might drop this series for now. As I don't have a deep understanding of libvirt or similar management tools, the specific use case remains a bit ambiguous to me. I’ll leave this series/patches here for now in case Claudio and other experts want to chime in or if further concerns arise. Thanks again!

Quoted 6 lines

Thanks,

Signature · 1 line
Peter Xu

Brs,
Trieu Huynh

#18 lore raw
In reply to Peter Xu
Quoted 116 lines · Peter Xu
Quoted 79 lines · Claudio Fontana
Quoted 78 lines · Peter Xu
Quoted 64 lines · Fabiano Rosas
Quoted 56 lines · Trieu Huynh

From: Trieu Huynh <[email protected]>

When query-migrate is called on the *destination* QEMU after a precopy migration completes it returns only {"status": "completed"} — no timing, no RAM statistics. The source correctly returns total-time, downtime, setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI schema so the destination can populate info->ram with only the fields that are meaningful on the receive side. Fields computed from tracked counters (transferred, normal, duplicate, mbps, pages-per-second) are shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are not set are simply absent from the output. Source-side callers are unaffected because populate_ram_info() sets all optional fields via has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:

{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
migration: track timing and received pages in MigrationIncomingState
migration: expose RAM stats and timing on destination via
query-migrate

Quoted 5 lines
 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)
Quoted 5 lines

Hi, remember to copy the interested people in your series. Using get_maintainers is correct, but when there's already a discussion about the topic it's good to add some CCs manually.

+CC Daniel and Claudio

The request is a valid one. Said that, redo accounting on both sides seem to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies and report it somewhere? I recall these info were captured somewhere, do we at least dump them into logs?

Quoted 6 lines
Quoted 2 lines

Thanks,

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

The problem is here from QEMU's perspective we have all the information, and "not quit" is the default behavior for src QEMU. It means by default all info is available.. this is true for no matter what upper mgmt is in use.

Quoted 11 lines

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed, so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

Quoted 8 lines

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer, then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

I still think it'll be great we only do accounting once, and above sounds like a libvirt question that I don't know the answer. Adding Jiri and libvirt list in case we can collect some inputs.

Quoted 6 lines

If you're looking for this feature, IMHO it'll be great if you can have a look at least from libvirt side to see if there's any blocker for collecting this info from src, then we can evaluate the rest. E.g. there's at least another option to forward statistic data from src to dest. Then we don't worry on mismatched accounting, or when fixing / add new accounting we don't need to do it both sides, which will be awkward.

I came to the thread to suggest this, but I see you already mentioned it. I'm not sure what you mean by "forward", but I was actually thinking "migrate". Do we have precedent for migrating a QAPI object? Do we want to have it? =) Put that MigrationInfo at the end of the stream and we'd have a nice symmetric interface for query-migrate in both src and dst.

(we have plans to deal with QEMU_VM_VMDESCRIPTION at some point anyway, we could maybe rework that end section into something useful)

Quoted 8 lines

Thanks,

Quoted 5 lines

Thanks,

Claudio

#19 lore raw
Quoted 5 lines · Fabiano Rosas

I came to the thread to suggest this, but I see you already mentioned it. I'm not sure what you mean by "forward", but I was actually thinking "migrate". Do we have precedent for migrating a QAPI object? Do we want to have it? =) Put that MigrationInfo at the end of the stream and we'd have a nice symmetric interface for query-migrate in both src and dst.

Do we need to "migrate" it, say, using VMSD data structures?

We could, but I'm not yet sure if it's a good idea, especially if we should make that as part of ABI. It may add some unnecessary burden to us.

Quoted 3 lines

(we have plans to deal with QEMU_VM_VMDESCRIPTION at some point anyway, we could maybe rework that end section into something useful)

That might still be a bit early phase to contain all the stats info. For example, the real downtime needs to be calculated until src receiving the ACK, that should be after dest receive QEMU_VM_VMDESCRIPTION, IIRC. So if we want to at least forward the downtime info, that might be too early.

In all cases, I still think we should evaluate libvirt solution before any other explorations.. because so far we still don't know why it can't be done, and it's the simplest afaiu.

Signature · 1 line
Peter Xu