Skip to contentSkip to conversation

migration/colo: small cleanup series

PATCH Superseded Peter Xu 3 participants 28 messages Jan 21 – Jan 26

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

cover migration/colo: small cleanup series

CI: https://gitlab.com/peterx/qemu/-/pipelines/2277445319

While reading COLO in the past two days, I got a few small patches to clean up here and there. I also ran this with the COLO qtests [*] and it ran all fine.

Comments welcomed, thanks.

[*] https://lore.kernel.org/r/[email protected]

Peter Xu (10):
migration: Introduce qemu_savevm_send_* helpers
migration: Use qemu_savevm_send_header() in qemu_save_device_state()
migration: Remove one migration_in_colo_state() occurance migration/savevm: Remove SaveStateEntry.is_ram migration/colo: Unwrap qemu_savevm_live_state() migration/colo: Remove call to send switchover start event
colo: Forbid VM resume during checkpointing migration/colo: Use the RAM iterable helper directly migration/colo: Move qemu_fflush() closer to its user for fb
migration: Drop iterable_only in qemu_savevm_state_complete_precopy

 migration/savevm.h    |  4 +-
 migration/colo.c      |  9 ++---
 migration/migration.c |  2 +-
 migration/savevm.c    | 91 +++++++++++++++++++++----------------------
 monitor/qmp-cmds.c    |  3 ++
 5 files changed, 55 insertions(+), 54 deletions(-)
Signature · 1 line
2.50.1
author
#1 lore raw

PATCH 1/10 migration: Introduce qemu_savevm_send_* helpers

Split qemu_savevm_state_header() into two parts. This paves way for a
reuse elsewhere.

---
 migration/savevm.h |  1 +
 migration/savevm.c | 58 +++++++++++++++++++++++++++-------------------
 2 files changed, 35 insertions(+), 24 deletions(-)
migration/savevm.h +1 -0
index 125a2507b7..5d815af742 100644
@@ -37,6 +37,7 @@ int qemu_savevm_state_prepare(Error **errp);
 int qemu_savevm_state_setup(QEMUFile *f, Error **errp);
 bool qemu_savevm_state_guest_unplug_pending(void);
 int qemu_savevm_state_resume_prepare(MigrationState *s);
+void qemu_savevm_send_header(QEMUFile *f);
 void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy);
 void qemu_savevm_state_cleanup(void);
migration/savevm.c +34 -24

72 lines not on this page · raw message

Signature · 1 line
2.50.1
Reviewed-by on v1 1/10
#2 lore raw

Re: [PATCH 01/10] migration: Introduce qemu_savevm_send_* helpers

Quoted 87 lines · Peter Xu

Split qemu_savevm_state_header() into two parts. This paves way for a
reuse elsewhere.

---
 migration/savevm.h |  1 +
 migration/savevm.c | 58 +++++++++++++++++++++++++++-------------------
 2 files changed, 35 insertions(+), 24 deletions(-)
migration/savevm.h +1 -0
index 125a2507b7..5d815af742 100644
@@ -37,6 +37,7 @@ int qemu_savevm_state_prepare(Error **errp);
 int qemu_savevm_state_setup(QEMUFile *f, Error **errp);
 bool qemu_savevm_state_guest_unplug_pending(void);
 int qemu_savevm_state_resume_prepare(MigrationState *s);
+void qemu_savevm_send_header(QEMUFile *f);
 void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy);
 void qemu_savevm_state_cleanup(void);
migration/savevm.c +26 -24
index 3dc812a7bb..e26656cca3 100644
@@ -1282,38 +1282,48 @@ void qemu_savevm_non_migratable_list(strList **reasons)
     }
 }
 
-void qemu_savevm_state_header(QEMUFile *f)
+void qemu_savevm_send_header(QEMUFile *f)
 {
-    MigrationState *s = migrate_get_current();
-    JSONWriter *vmdesc = s->vmdesc;
-    Error *local_err = NULL;
-
     trace_savevm_state_header();
     qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
     qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+}
 
-    if (s->send_configuration) {
-        qemu_put_byte(f, QEMU_VM_CONFIGURATION);
+static void qemu_savevm_send_configuration(MigrationState *s, QEMUFile *f)
+{
+    JSONWriter *vmdesc = s->vmdesc;
+    Error *local_err = NULL;
 
-        if (vmdesc) {
-            /*
-             * This starts the main json object and is paired with the
-             * json_writer_end_object in
-             * qemu_savevm_state_complete_precopy_non_iterable
-             */
-            json_writer_start_object(vmdesc, NULL);
-            json_writer_start_object(vmdesc, "configuration");
-        }
+    qemu_put_byte(f, QEMU_VM_CONFIGURATION);
 
-        vmstate_save_state(f, &vmstate_configuration, &savevm_state,
-                           vmdesc, &local_err);
-        if (local_err) {
-            error_report_err(local_err);
-        }
+    if (vmdesc) {
+        /*
+         * This starts the main json object and is paired with the
+         * json_writer_end_object in
+         * qemu_savevm_state_complete_precopy_non_iterable
+         */
+        json_writer_start_object(vmdesc, NULL);
+        json_writer_start_object(vmdesc, "configuration");
+    }
 
-        if (vmdesc) {
-            json_writer_end_object(vmdesc);
-        }
+    vmstate_save_state(f, &vmstate_configuration, &savevm_state,
+                       vmdesc, &local_err);
+    if (local_err) {
+        error_report_err(local_err);
+    }
+
+    if (vmdesc) {
+        json_writer_end_object(vmdesc);
+    }
+}

migration/savevm.c void qemu_savevm_non_migratable_list(strList **reasons) · line 1328

Quoted 10 lines
migration/savevm.c +8 -0
+
+void qemu_savevm_state_header(QEMUFile *f)
+{
+    MigrationState *s = migrate_get_current();
+
+    qemu_savevm_send_header(f);
+    if (s->send_configuration) {
+        qemu_savevm_send_configuration(s, f);
     }
 }
author
#3 lore raw

PATCH 2/10 migration: Use qemu_savevm_send_header() in qemu_save_device_state()

In reply to Peter Xu

Reduces duplication of the other path where we also send the same header.

---
 migration/savevm.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
migration/savevm.c +1 -2
index e26656cca3..64bf445c98 100644
@@ -1872,8 +1872,7 @@ int qemu_save_device_state(QEMUFile *f)
     SaveStateEntry *se;
 
     if (!migration_in_colo_state()) {
-        qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
-        qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+        qemu_savevm_send_header(f);
     }
     cpu_synchronize_all_states();
 
Signature · 1 line
2.50.1
Reviewed-by on v1 2/10
#4 lore raw

Re: [PATCH 02/10] migration: Use qemu_savevm_send_header() in qemu_save_device_state()

Quoted 12 lines · Peter Xu

Reduces duplication of the other path where we also send the same header.

---
 migration/savevm.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
migration/savevm.c +0 -0
index e26656cca3..64bf445c98 100644

migration/savevm.c int qemu_save_device_state(QEMUFile *f) · line 1877

Quoted 8 lines
migration/savevm.c +1 -2
@@ -1872,8 +1872,7 @@ int qemu_save_device_state(QEMUFile *f)
     SaveStateEntry *se;
 
     if (!migration_in_colo_state()) {
-        qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
-        qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+        qemu_savevm_send_header(f);
     }
     cpu_synchronize_all_states();
author
#5 lore raw

PATCH 3/10 migration: Remove one migration_in_colo_state() occurance

In reply to Peter Xu

Move the send header operation directly into Xen's QMP command, as COLO doesn't need it.

---
 migration/savevm.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
migration/savevm.c +1 -3
index 64bf445c98..61e873d90c 100644
@@ -1871,9 +1871,6 @@ int qemu_save_device_state(QEMUFile *f)
     Error *local_err = NULL;
     SaveStateEntry *se;
 
-    if (!migration_in_colo_state()) {
-        qemu_savevm_send_header(f);
-    }
     cpu_synchronize_all_states();
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -3335,6 +3332,7 @@ void qmp_xen_save_devices_state(const char *filename, bool has_live, bool live,
     qio_channel_set_name(QIO_CHANNEL(ioc), "migration-xen-save-state");
     f = qemu_file_new_output(QIO_CHANNEL(ioc));
     object_unref(OBJECT(ioc));
+    qemu_savevm_send_header(f);
     ret = qemu_save_device_state(f);
     if (ret < 0 || qemu_fclose(f) < 0) {
         error_setg(errp, "saving Xen device state failed");
Signature · 1 line
2.50.1
Reviewed-by on v1 3/10
#6 lore raw

Re: [PATCH 03/10] migration: Remove one migration_in_colo_state() occurance

Quoted 4 lines · Peter Xu

Move the send header operation directly into Xen's QMP command, as COLO doesn't need it.

We should make an effort to unify qemu_save_device_state and qemu_savevm_state_complete_precopy_non_iterable, they're almost the same.

Quoted 26 lines
---
 migration/savevm.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
migration/savevm.c +1 -3
index 64bf445c98..61e873d90c 100644
@@ -1871,9 +1871,6 @@ int qemu_save_device_state(QEMUFile *f)
     Error *local_err = NULL;
     SaveStateEntry *se;
 
-    if (!migration_in_colo_state()) {
-        qemu_savevm_send_header(f);
-    }
     cpu_synchronize_all_states();
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -3335,6 +3332,7 @@ void qmp_xen_save_devices_state(const char *filename, bool has_live, bool live,
     qio_channel_set_name(QIO_CHANNEL(ioc), "migration-xen-save-state");
     f = qemu_file_new_output(QIO_CHANNEL(ioc));
     object_unref(OBJECT(ioc));
+    qemu_savevm_send_header(f);
     ret = qemu_save_device_state(f);
     if (ret < 0 || qemu_fclose(f) < 0) {
         error_setg(errp, "saving Xen device state failed");
author
#7 lore raw

Re: [PATCH 03/10] migration: Remove one migration_in_colo_state() occurance

Quoted 8 lines · Fabiano Rosas
Quoted 4 lines · Peter Xu

Move the send header operation directly into Xen's QMP command, as COLO doesn't need it.

Thanks!

Quoted 4 lines

We should make an effort to unify qemu_save_device_state and qemu_savevm_state_complete_precopy_non_iterable, they're almost the same.

Let me give it a shot.

Signature · 1 line
Peter Xu
author
#8 lore raw

PATCH 4/10 migration/savevm: Remove SaveStateEntry.is_ram

In reply to Peter Xu

It's neither accurate nor necessary. Use a proper helper to detect if it's an iterable savevm state entry instead.

---
 migration/savevm.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
migration/savevm.c +8 -7
index 61e873d90c..f1cd8c913d 100644
@@ -249,7 +249,6 @@ typedef struct SaveStateEntry {
     const VMStateDescription *vmsd;
     void *opaque;
     CompatEntry *compat;
-    int is_ram;
 } SaveStateEntry;
 
 typedef struct SaveState {
@@ -816,10 +815,6 @@ int register_savevm_live(const char *idstr,
     se->ops = ops;
     se->opaque = opaque;
     se->vmsd = NULL;
-    /* if this is a live_savem then set is_ram */
-    if (ops->save_setup != NULL) {
-        se->is_ram = 1;
-    }
 
     pstrcat(se->idstr, sizeof(se->idstr), idstr);
 
@@ -1866,6 +1861,12 @@ void qemu_savevm_live_state(QEMUFile *f)
     qemu_put_byte(f, QEMU_VM_EOF);
 }
 
+/* Is a save state entry iterable (e.g. RAM)? */
+static bool qemu_savevm_se_iterable(SaveStateEntry *se)
+{
+    return se->ops && se->ops->save_setup;
+}
+
 int qemu_save_device_state(QEMUFile *f)
 {
     Error *local_err = NULL;
@@ -1876,7 +1877,7 @@ int qemu_save_device_state(QEMUFile *f)
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         int ret;
 
-        if (se->is_ram) {
+        if (qemu_savevm_se_iterable(se)) {
             continue;
         }
         ret = vmstate_save(f, se, NULL, &local_err);
@@ -2648,7 +2649,7 @@ qemu_loadvm_section_start_full(QEMUFile *f, uint8_t type, Error **errp)
     se->load_section_id = section_id;
 
     /* Validate if it is a device's state */
-    if (xen_enabled() && se->is_ram) {
+    if (xen_enabled() && qemu_savevm_se_iterable(se)) {
         error_setg(errp, "loadvm: %s RAM loading not allowed on Xen", idstr);
         return -EINVAL;
     }
Signature · 1 line
2.50.1
#9 lore raw

Re: [PATCH 04/10] migration/savevm: Remove SaveStateEntry.is_ram

Quoted 2 lines · Peter Xu

It's neither accurate nor necessary. Use a proper helper to detect if it's an iterable savevm state entry instead.

Could you explain what exactly is_ram was supposed to mean?

Quoted 30 lines
---
 migration/savevm.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
migration/savevm.c +0 -5
index 61e873d90c..f1cd8c913d 100644
@@ -249,7 +249,6 @@ typedef struct SaveStateEntry {
     const VMStateDescription *vmsd;
     void *opaque;
     CompatEntry *compat;
-    int is_ram;
 } SaveStateEntry;
 
 typedef struct SaveState {
@@ -816,10 +815,6 @@ int register_savevm_live(const char *idstr,
     se->ops = ops;
     se->opaque = opaque;
     se->vmsd = NULL;
-    /* if this is a live_savem then set is_ram */
-    if (ops->save_setup != NULL) {
-        se->is_ram = 1;
-    }
 
     pstrcat(se->idstr, sizeof(se->idstr), idstr);
 

migration/savevm.c void qemu_savevm_live_state(QEMUFile *f) · line 1867

Quoted 7 lines
migration/savevm.c +4 -0
@@ -1866,6 +1861,12 @@ void qemu_savevm_live_state(QEMUFile *f)
     qemu_put_byte(f, QEMU_VM_EOF);
 }
 
+/* Is a save state entry iterable (e.g. RAM)? */
+static bool qemu_savevm_se_iterable(SaveStateEntry *se)
+{
+    return se->ops && se->ops->save_setup;

So it could be iterable and not have .save_live_iterate?

Also, what's the deal with slirp? AIUI, it would be considered is_ram?

migration/savevm.c int qemu_save_device_state(QEMUFile *f) · line 1880

Quoted 11 lines
migration/savevm.c +3 -1
+}
+
 int qemu_save_device_state(QEMUFile *f)
 {
     Error *local_err = NULL;
@@ -1876,7 +1877,7 @@ int qemu_save_device_state(QEMUFile *f)
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         int ret;
 
-        if (se->is_ram) {
+        if (qemu_savevm_se_iterable(se)) {

Hmm, qemu_savevm_state_setup has:

    if (!se->ops || !se->ops->save_setup) {
            continue;
    }

therefore:

    if (!qemu_savevm_se_iterable(se)) {
            continue;
    }

Maybe it would be more practical to put these in a separate list already while registering.

Quoted 12 lines
migration/savevm.c +1 -1
             continue;
         }
         ret = vmstate_save(f, se, NULL, &local_err);
@@ -2648,7 +2649,7 @@ qemu_loadvm_section_start_full(QEMUFile *f, uint8_t type, Error **errp)
     se->load_section_id = section_id;
 
     /* Validate if it is a device's state */
-    if (xen_enabled() && se->is_ram) {
+    if (xen_enabled() && qemu_savevm_se_iterable(se)) {
         error_setg(errp, "loadvm: %s RAM loading not allowed on Xen", idstr);
         return -EINVAL;
     }
author
#10 lore raw

Re: [PATCH 04/10] migration/savevm: Remove SaveStateEntry.is_ram

Quoted 6 lines · Fabiano Rosas
Quoted 2 lines · Peter Xu

It's neither accurate nor necessary. Use a proper helper to detect if it's an iterable savevm state entry instead.

Could you explain what exactly is_ram was supposed to mean?

IIUC when introduced it was trying to capture the ram only, and that's also why I said it's inaccurate, because ram is not the only thing that can provide a save_setup() nowadays..

Quoted 34 lines
Quoted 34 lines
---
 migration/savevm.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
migration/savevm.c +0 -5
index 61e873d90c..f1cd8c913d 100644
@@ -249,7 +249,6 @@ typedef struct SaveStateEntry {
     const VMStateDescription *vmsd;
     void *opaque;
     CompatEntry *compat;
-    int is_ram;
 } SaveStateEntry;
 
 typedef struct SaveState {
@@ -816,10 +815,6 @@ int register_savevm_live(const char *idstr,
     se->ops = ops;
     se->opaque = opaque;
     se->vmsd = NULL;
-    /* if this is a live_savem then set is_ram */
-    if (ops->save_setup != NULL) {
-        se->is_ram = 1;
-    }
 
     pstrcat(se->idstr, sizeof(se->idstr), idstr);
 
@@ -1866,6 +1861,12 @@ void qemu_savevm_live_state(QEMUFile *f)
     qemu_put_byte(f, QEMU_VM_EOF);
 }
 
Quoted 8 lines

migration/savevm.c void qemu_savevm_live_state(QEMUFile *f) · line 1867

Quoted 4 lines
migration/savevm.c +4 -0
+/* Is a save state entry iterable (e.g. RAM)? */
+static bool qemu_savevm_se_iterable(SaveStateEntry *se)
+{
+    return se->ops && se->ops->save_setup;

So it could be iterable and not have .save_live_iterate?

Also, what's the deal with slirp? AIUI, it would be considered is_ram?

I guess no. Either in term of when it's introduced, or in theory of how I understand this piece of code should work.. Because savevm_slirp_state doesn't provide save_setup().

Note that this patch didn't really change the fact on how it works, even if I _think_ this is broken.. I want to leave it for later, making sure this patch (while doing the cleanup) doesn't introduce functional changes.

So I expect things to break if e.g. COLO is used with VFIO devices (that supports migrations), where it also provides save_setup(), but actually COLO will not properly sync VFIO states. Likely nobody is using it anyway.

It's also the same to Xen's xen-save-devices-state QMP command. It looks to me Xen is playing some "split savevm" work here, however I don't know whether it'll always workout these days when there're special devices that provide save_setup().

Quoted 19 lines

migration/savevm.c int qemu_save_device_state(QEMUFile *f) · line 1880

Quoted 12 lines
migration/savevm.c +3 -1
+}
+
 int qemu_save_device_state(QEMUFile *f)
 {
     Error *local_err = NULL;
@@ -1876,7 +1877,7 @@ int qemu_save_device_state(QEMUFile *f)
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         int ret;
 
-        if (se->is_ram) {
+        if (qemu_savevm_se_iterable(se)) {

Hmm, qemu_savevm_state_setup has:

    if (!se->ops || !se->ops->save_setup) {
            continue;
    }
Quoted 8 lines

therefore:

    if (!qemu_savevm_se_iterable(se)) {
            continue;
    }

Maybe it would be more practical to put these in a separate list already while registering.

This is a question about performance, my gut feeling is looping over with one queue would be fine for now, but even if not, it'll be further question to ask after above (on correctness..).

So I plan to be focused on the cleanup part for now in this series..

Quoted 14 lines
Quoted 13 lines
migration/savevm.c +1 -1
             continue;
         }
         ret = vmstate_save(f, se, NULL, &local_err);
@@ -2648,7 +2649,7 @@ qemu_loadvm_section_start_full(QEMUFile *f, uint8_t type, Error **errp)
     se->load_section_id = section_id;
 
     /* Validate if it is a device's state */
-    if (xen_enabled() && se->is_ram) {
+    if (xen_enabled() && qemu_savevm_se_iterable(se)) {
         error_setg(errp, "loadvm: %s RAM loading not allowed on Xen", idstr);
         return -EINVAL;
     }
Signature · 1 line
Peter Xu
Reviewed-by
#11 lore raw

Re: [PATCH 04/10] migration/savevm: Remove SaveStateEntry.is_ram

Quoted 53 lines · Peter Xu
Quoted 6 lines · Fabiano Rosas
Quoted 2 lines · Peter Xu

It's neither accurate nor necessary. Use a proper helper to detect if it's an iterable savevm state entry instead.

Could you explain what exactly is_ram was supposed to mean?

IIUC when introduced it was trying to capture the ram only, and that's also why I said it's inaccurate, because ram is not the only thing that can provide a save_setup() nowadays..

Quoted 41 lines
Quoted 31 lines
---
 migration/savevm.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
migration/savevm.c +0 -5
index 61e873d90c..f1cd8c913d 100644
@@ -249,7 +249,6 @@ typedef struct SaveStateEntry {
     const VMStateDescription *vmsd;
     void *opaque;
     CompatEntry *compat;
-    int is_ram;
 } SaveStateEntry;
 
 typedef struct SaveState {
@@ -816,10 +815,6 @@ int register_savevm_live(const char *idstr,
     se->ops = ops;
     se->opaque = opaque;
     se->vmsd = NULL;
-    /* if this is a live_savem then set is_ram */
-    if (ops->save_setup != NULL) {
-        se->is_ram = 1;
-    }
 
     pstrcat(se->idstr, sizeof(se->idstr), idstr);
 

migration/savevm.c void qemu_savevm_live_state(QEMUFile *f) · line 1867

Quoted 7 lines
migration/savevm.c +4 -0
@@ -1866,6 +1861,12 @@ void qemu_savevm_live_state(QEMUFile *f)
     qemu_put_byte(f, QEMU_VM_EOF);
 }
 
+/* Is a save state entry iterable (e.g. RAM)? */
+static bool qemu_savevm_se_iterable(SaveStateEntry *se)
+{
+    return se->ops && se->ops->save_setup;

So it could be iterable and not have .save_live_iterate?

Quoted 6 lines
Quoted 1 line

Also, what's the deal with slirp? AIUI, it would be considered is_ram?

I guess no. Either in term of when it's introduced, or in theory of how I understand this piece of code should work.. Because savevm_slirp_state doesn't provide save_setup().

Oops, I misread save_state vs. save_setup.

Quoted 4 lines

Note that this patch didn't really change the fact on how it works, even if I _think_ this is broken.. I want to leave it for later, making sure this patch (while doing the cleanup) doesn't introduce functional changes.

That's always fine.

Quoted 9 lines

So I expect things to break if e.g. COLO is used with VFIO devices (that supports migrations), where it also provides save_setup(), but actually COLO will not properly sync VFIO states. Likely nobody is using it anyway.

It's also the same to Xen's xen-save-devices-state QMP command. It looks to me Xen is playing some "split savevm" work here, however I don't know whether it'll always workout these days when there're special devices that provide save_setup().

Hm, ok. Let's leave it then.

Quoted 25 lines
Quoted 25 lines

migration/savevm.c int qemu_save_device_state(QEMUFile *f) · line 1880

Quoted 12 lines
migration/savevm.c +3 -1
+}
+
 int qemu_save_device_state(QEMUFile *f)
 {
     Error *local_err = NULL;
@@ -1876,7 +1877,7 @@ int qemu_save_device_state(QEMUFile *f)
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         int ret;
 
-        if (se->is_ram) {
+        if (qemu_savevm_se_iterable(se)) {

Hmm, qemu_savevm_state_setup has:

    if (!se->ops || !se->ops->save_setup) {
            continue;
    }

therefore:

    if (!qemu_savevm_se_iterable(se)) {
            continue;
    }
Quoted 9 lines
Quoted 2 lines

Maybe it would be more practical to put these in a separate list already while registering.

This is a question about performance, my gut feeling is looping over with one queue would be fine for now, but even if not, it'll be further question to ask after above (on correctness..).

So I plan to be focused on the cleanup part for now in this series..

author
#12 lore raw

PATCH 5/10 migration/colo: Unwrap qemu_savevm_live_state()

In reply to Peter Xu

It's only used in COLO path and only contains two calls. Unwrap the function. It paves way for further reduce special COLO paths on sync.

---
 migration/savevm.h | 1 -
 migration/colo.c   | 3 ++-
 migration/savevm.c | 7 -------
 3 files changed, 2 insertions(+), 9 deletions(-)
migration/savevm.h +0 -1
index 5d815af742..528607f09e 100644
@@ -64,7 +64,6 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
                                            uint64_t *start_list,
                                            uint64_t *length_list);
 void qemu_savevm_send_colo_enable(QEMUFile *f);
-void qemu_savevm_live_state(QEMUFile *f);
 int qemu_save_device_state(QEMUFile *f);
 
 int qemu_loadvm_state(QEMUFile *f, Error **errp);
migration/colo.c +2 -1
index db783f6fa7..e05736ecf0 100644
@@ -471,7 +471,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
      * TODO: We may need a timeout mechanism to prevent COLO process
      * to be blocked here.
      */
-    qemu_savevm_live_state(s->to_dst_file);
+    qemu_savevm_state_complete_precopy(s->to_dst_file, true);
+    qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
     qemu_fflush(fb);
 
migration/savevm.c +0 -7
index f1cd8c913d..529cf310e0 100644
@@ -1854,13 +1854,6 @@ cleanup:
     return ret;
 }
 
-void qemu_savevm_live_state(QEMUFile *f)
-{
-    /* save QEMU_VM_SECTION_END section */
-    qemu_savevm_state_complete_precopy(f, true);
-    qemu_put_byte(f, QEMU_VM_EOF);
-}
-
 /* Is a save state entry iterable (e.g. RAM)? */
 static bool qemu_savevm_se_iterable(SaveStateEntry *se)
 {
Signature · 1 line
2.50.1
Reviewed-by on v1 5/10
#13 lore raw

Re: [PATCH 05/10] migration/colo: Unwrap qemu_savevm_live_state()

Quoted 27 lines · Peter Xu

It's only used in COLO path and only contains two calls. Unwrap the function. It paves way for further reduce special COLO paths on sync.

---
 migration/savevm.h | 1 -
 migration/colo.c   | 3 ++-
 migration/savevm.c | 7 -------
 3 files changed, 2 insertions(+), 9 deletions(-)
migration/savevm.h +0 -1
index 5d815af742..528607f09e 100644
@@ -64,7 +64,6 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
                                            uint64_t *start_list,
                                            uint64_t *length_list);
 void qemu_savevm_send_colo_enable(QEMUFile *f);
-void qemu_savevm_live_state(QEMUFile *f);
 int qemu_save_device_state(QEMUFile *f);
 
 int qemu_loadvm_state(QEMUFile *f, Error **errp);
migration/colo.c +0 -0
index db783f6fa7..e05736ecf0 100644

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 474

Quoted 5 lines
migration/colo.c +1 -1
@@ -471,7 +471,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
      * TODO: We may need a timeout mechanism to prevent COLO process
      * to be blocked here.
      */
-    qemu_savevm_live_state(s->to_dst_file);
+    qemu_savevm_state_complete_precopy(s->to_dst_file, true);

Could maybe drop this complete from the name eventually.

Quoted 12 lines
migration/colo.c +1 -0
+    qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
     qemu_fflush(fb);
 
migration/savevm.c +0 -0
index f1cd8c913d..529cf310e0 100644
@@ -1854,13 +1854,6 @@ cleanup:
     return ret;
 }
 

migration/savevm.c cleanup: · line 1859

Quoted 10 lines
migration/savevm.c +0 -7
-void qemu_savevm_live_state(QEMUFile *f)
-{
-    /* save QEMU_VM_SECTION_END section */
-    qemu_savevm_state_complete_precopy(f, true);
-    qemu_put_byte(f, QEMU_VM_EOF);
-}
-
 /* Is a save state entry iterable (e.g. RAM)? */
 static bool qemu_savevm_se_iterable(SaveStateEntry *se)
 {
author
#14 lore raw

Re: [PATCH 05/10] migration/colo: Unwrap qemu_savevm_live_state()

Quoted 23 lines · Fabiano Rosas
Quoted 21 lines · Peter Xu

It's only used in COLO path and only contains two calls. Unwrap the function. It paves way for further reduce special COLO paths on sync.

---
 migration/savevm.h | 1 -
 migration/colo.c   | 3 ++-
 migration/savevm.c | 7 -------
 3 files changed, 2 insertions(+), 9 deletions(-)
migration/savevm.h +0 -1
index 5d815af742..528607f09e 100644
@@ -64,7 +64,6 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
                                            uint64_t *start_list,
                                            uint64_t *length_list);
 void qemu_savevm_send_colo_enable(QEMUFile *f);
-void qemu_savevm_live_state(QEMUFile *f);
 int qemu_save_device_state(QEMUFile *f);
 
Quoted 13 lines

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 474

Quoted 11 lines
migration/savevm.h +0 -0
 int qemu_loadvm_state(QEMUFile *f, Error **errp);
migration/colo.c +1 -1
index db783f6fa7..e05736ecf0 100644
@@ -471,7 +471,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
      * TODO: We may need a timeout mechanism to prevent COLO process
      * to be blocked here.
      */
-    qemu_savevm_live_state(s->to_dst_file);
+    qemu_savevm_state_complete_precopy(s->to_dst_file, true);

Could maybe drop this complete from the name eventually.

Maybe it's good to keep it? As it describes it's in completion phase. For instance, for iterable devices we invoke different hooks when it's completing (save_complete()) v.s. iterating (save_iterate()).

Quoted 13 lines
Quoted 13 lines
migration/colo.c +1 -0
+    qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
     qemu_fflush(fb);
 
migration/savevm.c +0 -0
index f1cd8c913d..529cf310e0 100644
@@ -1854,13 +1854,6 @@ cleanup:
     return ret;
 }
 
Quoted 12 lines

migration/savevm.c cleanup: · line 1859

Quoted 10 lines
migration/savevm.c +0 -7
-void qemu_savevm_live_state(QEMUFile *f)
-{
-    /* save QEMU_VM_SECTION_END section */
-    qemu_savevm_state_complete_precopy(f, true);
-    qemu_put_byte(f, QEMU_VM_EOF);
-}
-
 /* Is a save state entry iterable (e.g. RAM)? */
 static bool qemu_savevm_se_iterable(SaveStateEntry *se)
 {

I took this. :)

Signature · 1 line
Peter Xu
author
#15 lore raw

PATCH 6/10 migration/colo: Remove call to send switchover start event

In reply to Peter Xu

COLO (in case of periodically checkpointing) already have switchover happened before hand. This switchover_start feature never applies to COLO. Remove it.

---
 migration/colo.c | 2 --
 1 file changed, 2 deletions(-)
migration/colo.c +0 -2
index e05736ecf0..c344943173 100644
@@ -453,8 +453,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
         goto out;
     }
 
-    qemu_savevm_maybe_send_switchover_start(s->to_dst_file);
-
     /* Note: device state is saved into buffer */
     ret = qemu_save_device_state(fb);
 
Signature · 1 line
2.50.1
Reviewed-by on v1 6/10
#16 lore raw

Re: [PATCH 06/10] migration/colo: Remove call to send switchover start event

Quoted 14 lines · Peter Xu

COLO (in case of periodically checkpointing) already have switchover happened before hand. This switchover_start feature never applies to COLO. Remove it.

---
 migration/colo.c | 2 --
 1 file changed, 2 deletions(-)
migration/colo.c +0 -0
index e05736ecf0..c344943173 100644

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 457

Quoted 7 lines
migration/colo.c +0 -2
@@ -453,8 +453,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
         goto out;
     }
 
-    qemu_savevm_maybe_send_switchover_start(s->to_dst_file);
-
     /* Note: device state is saved into buffer */
     ret = qemu_save_device_state(fb);
author
#17 lore raw

PATCH 7/10 colo: Forbid VM resume during checkpointing

In reply to Peter Xu

COLO will stop the VM during each checkpoint on either PVM or SVM.

Accidentally resuming the VM during the window might be fatal because it may cause the RAM and devices state to misalign, corrupting the checkpoint.

Hence forbid VM resume during the process.

---
 monitor/qmp-cmds.c | 3 +++
 1 file changed, 3 insertions(+)
monitor/qmp-cmds.c +3 -0
index 1ca44fbd72..0c409c27dc 100644
@@ -84,6 +84,9 @@ void qmp_cont(Error **errp)
     } else if (runstate_check(RUN_STATE_FINISH_MIGRATE)) {
         error_setg(errp, "Migration is not finalized yet");
         return;
+    } else if (runstate_check(RUN_STATE_COLO)) {
+        error_setg(errp, "COLO checkpoint in progress");
+        return;
     }
 
     for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
Signature · 1 line
2.50.1
Reviewed-by on v1 7/10
#18 lore raw

Re: [PATCH 07/10] colo: Forbid VM resume during checkpointing

Quoted 17 lines · Peter Xu

COLO will stop the VM during each checkpoint on either PVM or SVM.

Accidentally resuming the VM during the window might be fatal because it may cause the RAM and devices state to misalign, corrupting the checkpoint.

Hence forbid VM resume during the process.

---
 monitor/qmp-cmds.c | 3 +++
 1 file changed, 3 insertions(+)
monitor/qmp-cmds.c +0 -0
index 1ca44fbd72..0c409c27dc 100644

monitor/qmp-cmds.c void qmp_cont(Error **errp) · line 92

Quoted 9 lines
monitor/qmp-cmds.c +3 -0
@@ -84,6 +84,9 @@ void qmp_cont(Error **errp)
     } else if (runstate_check(RUN_STATE_FINISH_MIGRATE)) {
         error_setg(errp, "Migration is not finalized yet");
         return;
+    } else if (runstate_check(RUN_STATE_COLO)) {
+        error_setg(errp, "COLO checkpoint in progress");
+        return;
     }
 
     for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
author
#19 lore raw

PATCH 8/10 migration/colo: Use the RAM iterable helper directly

In reply to Peter Xu

qemu_savevm_state_complete_precopy() has a weird parameter called "iterable_only". It's needed because COLO saves device states in advance.

To make dropping that weird parameter easier, let COLO directly use the RAM iterator helper instead, which should make the code easier to read too.

---
 migration/colo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
migration/colo.c +1 -1
index c344943173..f92803dd29 100644
@@ -469,7 +469,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
      * TODO: We may need a timeout mechanism to prevent COLO process
      * to be blocked here.
      */
-    qemu_savevm_state_complete_precopy(s->to_dst_file, true);
+    qemu_savevm_state_complete_precopy_iterable(s->to_dst_file, false);
     qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
     qemu_fflush(fb);
Signature · 1 line
2.50.1
Reviewed-by on v1 8/10
#20 lore raw

Re: [PATCH 08/10] migration/colo: Use the RAM iterable helper directly

Quoted 16 lines · Peter Xu

qemu_savevm_state_complete_precopy() has a weird parameter called "iterable_only". It's needed because COLO saves device states in advance.

To make dropping that weird parameter easier, let COLO directly use the RAM iterator helper instead, which should make the code easier to read too.

---
 migration/colo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
migration/colo.c +0 -0
index c344943173..f92803dd29 100644

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 475

Quoted 8 lines
migration/colo.c +1 -1
@@ -469,7 +469,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
      * TODO: We may need a timeout mechanism to prevent COLO process
      * to be blocked here.
      */
-    qemu_savevm_state_complete_precopy(s->to_dst_file, true);
+    qemu_savevm_state_complete_precopy_iterable(s->to_dst_file, false);
     qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
     qemu_fflush(fb);
author
#21 lore raw

PATCH 9/10 migration/colo: Move qemu_fflush() closer to its user for fb

In reply to Peter Xu

COLO caches all device states in a buffer channel `fb'. Flush it right before pushing the buffer to make sure everything needed will be flushed, add a comment explaning it.

---
 migration/colo.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
migration/colo.c +2 -2
index f92803dd29..e2307f1907 100644
@@ -472,8 +472,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
     qemu_savevm_state_complete_precopy_iterable(s->to_dst_file, false);
     qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
-    qemu_fflush(fb);
-
     /*
      * We need the size of the VMstate data in Secondary side,
      * With which we can decide how much data should be read.
@@ -484,6 +482,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
         goto out;
     }
 
+    /* Flush to make sure everything lands bioc->data */
+    qemu_fflush(fb);
     qemu_put_buffer(s->to_dst_file, bioc->data, bioc->usage);
     ret = qemu_fflush(s->to_dst_file);
     if (ret < 0) {
Signature · 1 line
2.50.1
Reviewed-by on v1 9/10
#22 lore raw

Re: [PATCH 09/10] migration/colo: Move qemu_fflush() closer to its user for fb

Quoted 23 lines · Peter Xu

COLO caches all device states in a buffer channel `fb'. Flush it right before pushing the buffer to make sure everything needed will be flushed, add a comment explaning it.

---
 migration/colo.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
migration/colo.c +0 -2
index f92803dd29..e2307f1907 100644
@@ -472,8 +472,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
     qemu_savevm_state_complete_precopy_iterable(s->to_dst_file, false);
     qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
-    qemu_fflush(fb);
-
     /*
      * We need the size of the VMstate data in Secondary side,
      * With which we can decide how much data should be read.

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 488

Quoted 7 lines
migration/colo.c +2 -0
@@ -484,6 +482,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
         goto out;
     }
 
+    /* Flush to make sure everything lands bioc->data */
+    qemu_fflush(fb);
     qemu_put_buffer(s->to_dst_file, bioc->data, bioc->usage);
     ret = qemu_fflush(s->to_dst_file);

Wait, aren't we wasting a memcpy here? Isn't this the same, but without the copy of bioc->data to s->to_dst_file->buf:

qemu_fflush(fb); // ensure bioc->data is populated
qemu_fflush(s->to_dst_file); // send buffered data from migration stream
qemu_put_buffer_async(s->to_dst_file, bioc->data, ...); // point s->to_dst_file->iov to bioc->data
qemu_fflush(s->to_dst_file); // send s->to_dst_file->iov contents

Quoted 1 line
migration/colo.c +0 -0
     if (ret < 0) {
author
#23 lore raw

Re: [PATCH 09/10] migration/colo: Move qemu_fflush() closer to its user for fb

Quoted 35 lines · Fabiano Rosas
Quoted 23 lines · Peter Xu

COLO caches all device states in a buffer channel `fb'. Flush it right before pushing the buffer to make sure everything needed will be flushed, add a comment explaning it.

---
 migration/colo.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
migration/colo.c +0 -2
index f92803dd29..e2307f1907 100644
@@ -472,8 +472,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
     qemu_savevm_state_complete_precopy_iterable(s->to_dst_file, false);
     qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
 
-    qemu_fflush(fb);
-
     /*
      * We need the size of the VMstate data in Secondary side,
      * With which we can decide how much data should be read.

migration/colo.c static int colo_do_checkpoint_transaction(MigrationState *s, · line 488

Quoted 7 lines
migration/colo.c +2 -0
@@ -484,6 +482,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
         goto out;
     }
 
+    /* Flush to make sure everything lands bioc->data */
+    qemu_fflush(fb);
     qemu_put_buffer(s->to_dst_file, bioc->data, bioc->usage);
     ret = qemu_fflush(s->to_dst_file);
Quoted 7 lines

Wait, aren't we wasting a memcpy here? Isn't this the same, but without the copy of bioc->data to s->to_dst_file->buf:

qemu_fflush(fb); // ensure bioc->data is populated
qemu_fflush(s->to_dst_file); // send buffered data from migration stream
qemu_put_buffer_async(s->to_dst_file, bioc->data, ...); // point s->to_dst_file->iov to bioc->data
qemu_fflush(s->to_dst_file); // send s->to_dst_file->iov contents

Good point! Same to postcopy..

I quickly measured a small VM in postcopy path, async put can shrink the time needed (plus the flush) from 91us to 38us. Not a huge difference, but I don't see why we don't do it either. I'll test more and add those when I repost.

Signature · 1 line
Peter Xu
author
#24 lore raw

PATCH 10/10 migration: Drop iterable_only in qemu_savevm_state_complete_precopy

In reply to Peter Xu

Now after removing the special case in COLO, we can drop this parameter.

---
 migration/savevm.h    |  2 +-
 migration/migration.c |  2 +-
 migration/savevm.c    | 12 +++++-------
 3 files changed, 7 insertions(+), 9 deletions(-)
migration/savevm.h +1 -1
index 528607f09e..ea01ca63ec 100644
@@ -42,7 +42,7 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy);
 void qemu_savevm_state_cleanup(void);
 void qemu_savevm_state_complete_postcopy(QEMUFile *f);
-int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only);
+int qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_pending_exact(uint64_t *must_precopy,
                                      uint64_t *can_postcopy);
 void qemu_savevm_state_pending_estimate(uint64_t *must_precopy,
migration/migration.c +1 -1
index 1bcde301f7..5d21d6dc11 100644
@@ -3018,7 +3018,7 @@ static int migration_completion_precopy(MigrationState *s)
         goto out_unlock;
     }
 
-    ret = qemu_savevm_state_complete_precopy(s->to_dst_file, false);
+    ret = qemu_savevm_state_complete_precopy(s->to_dst_file);
 out_unlock:
     bql_unlock();
     return ret;
migration/savevm.c +5 -7
index 529cf310e0..1bf1037ce1 100644
@@ -1717,7 +1717,7 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
     return 0;
 }
 
-int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
+int qemu_savevm_state_complete_precopy(QEMUFile *f)
 {
     int ret;
 
@@ -1726,11 +1726,9 @@ int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
         return ret;
     }
 
-    if (!iterable_only) {
-        ret = qemu_savevm_state_complete_precopy_non_iterable(f, false);
-        if (ret) {
-            return ret;
-        }
+    ret = qemu_savevm_state_complete_precopy_non_iterable(f, false);
+    if (ret) {
+        return ret;
     }
 
     return qemu_fflush(f);
@@ -1831,7 +1829,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
     ret = qemu_file_get_error(f);
     if (ret == 0) {
         qemu_savevm_maybe_send_switchover_start(f);
-        qemu_savevm_state_complete_precopy(f, false);
+        qemu_savevm_state_complete_precopy(f);
         ret = qemu_file_get_error(f);
     }
     if (ret != 0) {
Signature · 1 line
2.50.1
Reviewed-by on v1 10/10
#25 lore raw

Re: [PATCH 10/10] migration: Drop iterable_only in qemu_savevm_state_complete_precopy

Quoted 64 lines · Peter Xu

Now after removing the special case in COLO, we can drop this parameter.

---
 migration/savevm.h    |  2 +-
 migration/migration.c |  2 +-
 migration/savevm.c    | 12 +++++-------
 3 files changed, 7 insertions(+), 9 deletions(-)
migration/savevm.h +1 -1
index 528607f09e..ea01ca63ec 100644
@@ -42,7 +42,7 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy);
 void qemu_savevm_state_cleanup(void);
 void qemu_savevm_state_complete_postcopy(QEMUFile *f);
-int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only);
+int qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_pending_exact(uint64_t *must_precopy,
                                      uint64_t *can_postcopy);
 void qemu_savevm_state_pending_estimate(uint64_t *must_precopy,
migration/migration.c +1 -1
index 1bcde301f7..5d21d6dc11 100644
@@ -3018,7 +3018,7 @@ static int migration_completion_precopy(MigrationState *s)
         goto out_unlock;
     }
 
-    ret = qemu_savevm_state_complete_precopy(s->to_dst_file, false);
+    ret = qemu_savevm_state_complete_precopy(s->to_dst_file);
 out_unlock:
     bql_unlock();
     return ret;
migration/savevm.c +4 -6
index 529cf310e0..1bf1037ce1 100644
@@ -1717,7 +1717,7 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
     return 0;
 }
 
-int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
+int qemu_savevm_state_complete_precopy(QEMUFile *f)
 {
     int ret;
 
@@ -1726,11 +1726,9 @@ int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
         return ret;
     }
 
-    if (!iterable_only) {
-        ret = qemu_savevm_state_complete_precopy_non_iterable(f, false);
-        if (ret) {
-            return ret;
-        }
+    ret = qemu_savevm_state_complete_precopy_non_iterable(f, false);
+    if (ret) {
+        return ret;
     }
 
     return qemu_fflush(f);

migration/savevm.c static int qemu_savevm_state(QEMUFile *f, Error **errp) · line 1835

Quoted 8 lines
migration/savevm.c +1 -1
@@ -1831,7 +1829,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
     ret = qemu_file_get_error(f);
     if (ret == 0) {
         qemu_savevm_maybe_send_switchover_start(f);
-        qemu_savevm_state_complete_precopy(f, false);
+        qemu_savevm_state_complete_precopy(f);
         ret = qemu_file_get_error(f);
     }
     if (ret != 0) {
#26 lore raw
In reply to Peter Xu
Quoted 8 lines · On Wed, 21 Jan 2026 17:33:25 -0500 Peter Xu <[email protected]> wrote:

CI: https://gitlab.com/peterx/qemu/-/pipelines/2277445319

While reading COLO in the past two days, I got a few small patches to clean up here and there. I also ran this with the COLO qtests [*] and it ran all fine.

Comments welcomed, thanks.

[*] https://lore.kernel.org/r/[email protected]

Hmm,
It fails for me on top of current master, but works fine wthout this patchset:
# Running /x86_64/migration/colo/plain/primary_failover
# Using machine type: pc-i440fx-11.0

# starting QEMU: exec ./qemu-system-x86_64 -qtest unix:/tmp/qtest-3157.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3157.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -run-with exit-with-parent=on -accel kvm -accel tcg -machine pc-i440fx-11.0, -name source,debug-threads=on -machine memory-backend=mig.mem -object memory-backend-ram,id=mig.mem,size=150M,share=off -serial file:/tmp/migration-test-OZFSJ3/src_serial -drive if=none,id=d0,file=/tmp/migration-test-OZFSJ3/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 -snapshot  -accel qtest
# starting QEMU: exec ./qemu-system-x86_64 -qtest unix:/tmp/qtest-3157.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3157.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -run-with exit-with-parent=on -accel kvm -accel tcg -machine pc-i440fx-11.0, -name target,debug-threads=on -machine memory-backend=mig.mem -object memory-backend-ram,id=mig.mem,size=150M,share=off -serial file:/tmp/migration-test-OZFSJ3/dest_serial -incoming tcp:127.0.0.1:0  -drive if=none,id=d0,file=/tmp/migration-test-OZFSJ3/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 -snapshot  -accel qtest
qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:03.0/e1000': Loading VM subsection 'e1000/full_mac_state' in 'e1000' failed: -5: Failed to load e1000/full_mac_state state: stream error: -5
Quoted 20 lines

Peter Xu (10):
migration: Introduce qemu_savevm_send_* helpers
migration: Use qemu_savevm_send_header() in qemu_save_device_state()
migration: Remove one migration_in_colo_state() occurance migration/savevm: Remove SaveStateEntry.is_ram migration/colo: Unwrap qemu_savevm_live_state() migration/colo: Remove call to send switchover start event
colo: Forbid VM resume during checkpointing migration/colo: Use the RAM iterable helper directly migration/colo: Move qemu_fflush() closer to its user for fb
migration: Drop iterable_only in qemu_savevm_state_complete_precopy

 migration/savevm.h    |  4 +-
 migration/colo.c      |  9 ++---
 migration/migration.c |  2 +-
 migration/savevm.c    | 91 +++++++++++++++++++++----------------------
 monitor/qmp-cmds.c    |  3 ++
 5 files changed, 55 insertions(+), 54 deletions(-)
author
#27 lore raw
Quoted 12 lines · Lukas Straub
Quoted 8 lines · On Wed, 21 Jan 2026 17:33:25 -0500 Peter Xu <[email protected]> wrote:

CI: https://gitlab.com/peterx/qemu/-/pipelines/2277445319

While reading COLO in the past two days, I got a few small patches to clean up here and there. I also ran this with the COLO qtests [*] and it ran all fine.

Comments welcomed, thanks.

[*] https://lore.kernel.org/r/[email protected]

Quoted 7 lines

Hmm,
It fails for me on top of current master, but works fine wthout this patchset:
# Running /x86_64/migration/colo/plain/primary_failover
# Using machine type: pc-i440fx-11.0

# starting QEMU: exec ./qemu-system-x86_64 -qtest unix:/tmp/qtest-3157.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3157.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -run-with exit-with-parent=on -accel kvm -accel tcg -machine pc-i440fx-11.0, -name source,debug-threads=on -machine memory-backend=mig.mem -object memory-backend-ram,id=mig.mem,size=150M,share=off -serial file:/tmp/migration-test-OZFSJ3/src_serial -drive if=none,id=d0,file=/tmp/migration-test-OZFSJ3/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 -snapshot  -accel qtest
# starting QEMU: exec ./qemu-system-x86_64 -qtest unix:/tmp/qtest-3157.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3157.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -run-with exit-with-parent=on -accel kvm -accel tcg -machine pc-i440fx-11.0, -name target,debug-threads=on -machine memory-backend=mig.mem -object memory-backend-ram,id=mig.mem,size=150M,share=off -serial file:/tmp/migration-test-OZFSJ3/dest_serial -incoming tcp:127.0.0.1:0  -drive if=none,id=d0,file=/tmp/migration-test-OZFSJ3/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 -snapshot  -accel qtest
qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:03.0/e1000': Loading VM subsection 'e1000/full_mac_state' in 'e1000' failed: -5: Failed to load e1000/full_mac_state state: stream error: -5

Somehow I didn't hit it previously, but at least when using your latest patches it hangs for me (rather than failing..).

Patch 9 is broken, because I overlooked COLO_MESSAGE_VMSTATE_SIZE also needs to reference bioc->usage.

I'll fix patch 9 when repost. Thanks for the report.

--
Peter Xu