Skip to contentSkip to conversation

migration: Support gtree migration

PATCH Superseded Eric Auger 3 participants 4 messages Oct 3, 2019

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

PATCH v2 1/1 migration: Support gtree migration

Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.

When putting the items, the tree is traversed in sorted order by g_tree_foreach.

On the get() path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of functions must be stored within the VMStateField in a new opaque pointer.

Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.

Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).

Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.

---

v1 -> v2:
- fix compilation issues reported by robots
- fixed init of VMSD array
- direct key now dumped as a 32b
- removed useless cast from/to pointer

---
 include/migration/vmstate.h |  31 +++
 migration/trace-events      |   6 +
 migration/vmstate-types.c   | 133 +++++++++++
 tests/test-vmstate.c        | 427 ++++++++++++++++++++++++++++++++++++
 4 files changed, 597 insertions(+)
include/migration/vmstate.h +31 -0
index 1fbfd099dd..4d9698eaab 100644
@@ -171,6 +171,7 @@ struct VMStateField {
     int version_id;
     int struct_version_id;
     bool (*field_exists)(void *opaque, int version_id);
+    void *data;
 };
 
 struct VMStateDescription {
@@ -224,6 +225,7 @@ extern const VMStateInfo vmstate_info_unused_buffer;
 extern const VMStateInfo vmstate_info_tmp;
 extern const VMStateInfo vmstate_info_bitmap;
 extern const VMStateInfo vmstate_info_qtailq;
+extern const VMStateInfo vmstate_info_gtree;
 
 #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
 /*
@@ -754,6 +756,35 @@ extern const VMStateInfo vmstate_info_qtailq;
     .start        = offsetof(_type, _next),                              \
 }
 
+typedef struct GTreeInitData {
+    GCompareDataFunc key_compare_func;
+    gpointer key_compare_data;
+    GDestroyNotify key_destroy_func;
+    GDestroyNotify value_destroy_func;
+} GTreeInitData;
+
+/*
+ * For migrating a GTree.
+ * _vmsd: Start address of the 2 element array containing the key vmsd
+ *        and the data vmsd
+ * _key_type: type of the key
+ * _val_type: type of the value
+ * _data: pointer to a GTreeInitData struct
+ * if _data is NULL, the target tree must have been properly initialized
+ */
+#define VMSTATE_GTREE_V(_field, _state, _version, _vmsd,                       \
+                        _key_type, _val_type, _data)                           \
+{                                                                              \
+    .name         = (stringify(_field)),                                       \
+    .version_id   = (_version),                                                \
+    .vmsd         = (_vmsd),                                                   \
+    .info         = &vmstate_info_gtree,                                       \
+    .start        = sizeof(_key_type),                                         \
+    .size         = sizeof(_val_type),                                         \
+    .offset       = offsetof(_state, _field),                                  \
+    .data         = (_data)                                                    \
+}
+
 /* _f : field name
    _f_n : num of elements field_name
    _n : num of elements
migration/trace-events +6 -0
index 858d415d56..3e2775bd3b 100644
@@ -71,6 +71,12 @@ get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
 put_qtailq(const char *name, int version_id) "%s v%d"
 put_qtailq_end(const char *name, const char *reason) "%s %s"
 
+get_gtree(const char *key_name, const char *val_name, int version) "%s/%s %d"
+get_gtree_fail(const char *key_name, const char *reason, int val) "%s %s/%d"
+get_gtree_succeed(const char *key_name, const char *val_name, const char *reason, int val) "%s/%s %s/%d"
+put_gtree(const char *key_name, int key_version, const char *val_name, int val_version) "%s(%d)/%s(%d)"
+put_gtree_end(const char *key_name, const char *val_name, const char *reason) "%s/%s %s"
+
 # qemu-file.c
 qemu_file_fclose(void) ""
 
migration/vmstate-types.c +133 -0

142 lines not on this page · raw message

tests/test-vmstate.c +427 -0

445 lines not on this page · raw message

Signature · 1 line
2.20.1
#1 lore raw
Quoted 32 lines · Eric Auger

Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.

When putting the items, the tree is traversed in sorted order by g_tree_foreach.

On the get() path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of functions must be stored within the VMStateField in a new opaque pointer.

Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.

Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).

Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.

---

migration/vmstate-types.c +0 -0
index bee658a1b2..06c4663de6 100644

migration/vmstate-types.c line 20

Quoted 4 lines
migration/vmstate-types.c +1 -0
@@ -17,6 +17,7 @@
 #include "qemu/error-report.h"
 #include "qemu/queue.h"
 #include "trace.h"
+#include <glib.h>

Not required, everything in QEMU includes this already.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
In reply to Eric Auger
Quoted 10 lines · Eric Auger

Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.

When putting the items, the tree is traversed in sorted order by g_tree_foreach.

On the get() path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of

             ^^

or.

Quoted 2 lines

functions must be stored within the VMStateField in a new opaque pointer.

I am not fully convinced that the automatic mode is needed. Especially the ->data field. I *fear* it being abused for other cases.

Quoted 3 lines

Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.

There is a pre_load() function for the VMState that creates this. it can be used to initualize the field value, no? That way the data part is not needed. I think this will make the code less complex, what do you think?

Quoted 2 lines

Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).

I am wondering if for this, it is better to add two VMSTATE (at least at the macro level). SIMPLE_TREE, and TREE, or whataver oyou want to call it. But I haven't fully looked into it.

The other general "consideration" that I have is that there is neither of:
- marker between elements
- number of elements
- total size/size of elements.

That makes completelly impractical to "walk" the migration stream without understanding exactyl what is there. (Now, to be fair, there are other parts of qemu that are like that. PCI cames to mind.)

Quoted 2 lines

Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.

Really nice to have the tests. Thanks.

include/migration/vmstate.h struct VMStateField { · line 175

Quoted 13 lines
include/migration/vmstate.h +1 -0
index 1fbfd099dd..4d9698eaab 100644
@@ -171,6 +171,7 @@ struct VMStateField {
     int version_id;
     int struct_version_id;
     bool (*field_exists)(void *opaque, int version_id);
+    void *data;
 };

This is the bit that I don't really like :p

include/migration/vmstate.h extern const VMStateInfo vmstate_info_qtailq; · line 764

Quoted 7 lines
include/migration/vmstate.h +6 -0
+typedef struct GTreeInitData {
+    GCompareDataFunc key_compare_func;
+    gpointer key_compare_data;
+    GDestroyNotify key_destroy_func;
+    GDestroyNotify value_destroy_func;
+} GTreeInitData;

My understanding is that if you do this on the pre_load() function, you don't need this at all.

Quoted 22 lines
migration/vmstate-types.c +7 -0
index bee658a1b2..06c4663de6 100644
@@ -17,6 +17,7 @@
 #include "qemu/error-report.h"
 #include "qemu/queue.h"
 #include "trace.h"
+#include <glib.h>
 
 /* bool */
 
@@ -691,3 +692,135 @@ const VMStateInfo vmstate_info_qtailq = {
     .get  = get_qtailq,
     .put  = put_qtailq,
 };
+
+struct put_gtree_data {
+    QEMUFile *f;
+    const VMStateField *field;
+    QJSON *vmdesc;
+};

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 710

Quoted 10 lines
migration/vmstate-types.c +10 -0
+
+static gboolean put_gtree_elem(gpointer key, gpointer value, gpointer data)
+{
+    struct put_gtree_data *capsule = (struct put_gtree_data *)data;
+    const VMStateField *field = capsule->field;
+    QEMUFile *f = capsule->f;
+    const VMStateDescription *key_vmsd = &field->vmsd[0];
+    const VMStateDescription *data_vmsd = &field->vmsd[1];
+
+    qemu_put_byte(f, true);

Ok. there is a marker O:-)

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 719

Quoted 9 lines
migration/vmstate-types.c +9 -0
+
+    /* put the key */
+    if (!key_vmsd->fields) {
+        qemu_put_be32(f, GPOINTER_TO_UINT(key));
+    } else {
+        if (vmstate_save_state(f, key_vmsd, key, capsule->vmdesc)) {
+            return true;
+        }
+    }

But it is magic to know if it is a simple or complex key.

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 793

Quoted 11 lines
migration/vmstate-types.c +11 -0
+    if (field->data) {
+        init_data = (GTreeInitData *)field->data;
+        tree = g_tree_new_full(init_data->key_compare_func,
+                               init_data->key_compare_data,
+                               init_data->key_destroy_func,
+                               init_data->value_destroy_func);
+        *pval = tree;
+    } else {
+        /* tree is externally allocated */
+        tree = *pval;
+    }

This can be simplified while we are at it.

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 795

Quoted 1 line
migration/vmstate-types.c +1 -0
+    while (qemu_get_byte(f)) {

If we get out of sync, for any reason, we have no way to recover. The only way to recover is that we don't get a "false" in this position.

Later, Juan.

author
#3 lore raw

Hi Juan,

Quoted 13 lines · Juan Quintela
Quoted 10 lines · Eric Auger

Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.

When putting the items, the tree is traversed in sorted order by g_tree_foreach.

On the get() path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of

             ^^
Quoted 7 lines

or.

Quoted 2 lines

functions must be stored within the VMStateField in a new opaque pointer.

I am not fully convinced that the automatic mode is needed. Especially the ->data field. I *fear* it being abused for other cases.

OK. I implemented your suggestion, ie. using preload and it does the job. So I don't need that field anymore.

Quoted 9 lines
Quoted 4 lines

Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.

There is a pre_load() function for the VMState that creates this. it can be used to initualize the field value, no? That way the data part is not needed. I think this will make the code less complex, what do you think?

agreed

Quoted 7 lines
Quoted 3 lines

Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).

I am wondering if for this, it is better to add two VMSTATE (at least at the macro level). SIMPLE_TREE, and TREE, or whataver oyou want to call it. But I haven't fully looked into it.

I don't have a strong opinion here. At the moment I test the key_vmsd->field and if it is NULL this means the key is a direct one.

Otherwise we could have 2 macros, a single info, but 2 different field names. the name would allow to know the nature of the key.

Quoted 4 lines

The other general "consideration" that I have is that there is neither of:
- marker between elements

so we have one

Quoted 1 line

- number of elements

you don't have it

Quoted 1 line

- total size/size of elements.

you just have the size of the key and the size of the value at the moment. I could easily add the number of nodes.

Quoted 29 lines

That makes completelly impractical to "walk" the migration stream without understanding exactyl what is there. (Now, to be fair, there are other parts of qemu that are like that. PCI cames to mind.)

Quoted 2 lines

Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.

Really nice to have the tests. Thanks.

include/migration/vmstate.h struct VMStateField { · line 175

Quoted 13 lines
include/migration/vmstate.h +1 -0
index 1fbfd099dd..4d9698eaab 100644
@@ -171,6 +171,7 @@ struct VMStateField {
     int version_id;
     int struct_version_id;
     bool (*field_exists)(void *opaque, int version_id);
+    void *data;
 };

This is the bit that I don't really like :p

Quoted 1 line
Quoted 9 lines

include/migration/vmstate.h extern const VMStateInfo vmstate_info_qtailq; · line 764

Quoted 6 lines
include/migration/vmstate.h +6 -0
+typedef struct GTreeInitData {
+    GCompareDataFunc key_compare_func;
+    gpointer key_compare_data;
+    GDestroyNotify key_destroy_func;
+    GDestroyNotify value_destroy_func;
+} GTreeInitData;

My understanding is that if you do this on the pre_load() function, you don't need this at all.

yep

Quoted 27 lines
Quoted 27 lines
migration/vmstate-types.c +11 -0
index bee658a1b2..06c4663de6 100644
@@ -17,6 +17,7 @@
 #include "qemu/error-report.h"
 #include "qemu/queue.h"
 #include "trace.h"
+#include <glib.h>
 
 /* bool */
 
@@ -691,3 +692,135 @@ const VMStateInfo vmstate_info_qtailq = {
     .get  = get_qtailq,
     .put  = put_qtailq,
 };
+
+struct put_gtree_data {
+    QEMUFile *f;
+    const VMStateField *field;
+    QJSON *vmdesc;
+};
+
+static gboolean put_gtree_elem(gpointer key, gpointer value, gpointer data)
+{
+    struct put_gtree_data *capsule = (struct put_gtree_data *)data;
Quoted 8 lines

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 710

Quoted 6 lines
migration/vmstate-types.c +6 -0
+    const VMStateField *field = capsule->field;
+    QEMUFile *f = capsule->f;
+    const VMStateDescription *key_vmsd = &field->vmsd[0];
+    const VMStateDescription *data_vmsd = &field->vmsd[1];
+
+    qemu_put_byte(f, true);

Ok. there is a marker O:-)

yep

Quoted 11 lines

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 719

Quoted 10 lines
migration/vmstate-types.c +9 -0
+
+    /* put the key */
+    if (!key_vmsd->fields) {
+        qemu_put_be32(f, GPOINTER_TO_UINT(key));
+    } else {
+        if (vmstate_save_state(f, key_vmsd, key, capsule->vmdesc)) {
+            return true;
+        }
+    }
Quoted 1 line

But it is magic to know if it is a simple or complex key.

key_vmsd->fields is used
this means you set

static const VMStateDescription vmstate_id_domain[2] = {
{}, VMSTATE_DOMAIN /* direct key, value */
};

Quoted 14 lines

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 793

Quoted 13 lines
migration/vmstate-types.c +11 -0
+    if (field->data) {
+        init_data = (GTreeInitData *)field->data;
+        tree = g_tree_new_full(init_data->key_compare_func,
+                               init_data->key_compare_data,
+                               init_data->key_destroy_func,
+                               init_data->value_destroy_func);
+        *pval = tree;
+    } else {
+        /* tree is externally allocated */
+        tree = *pval;
+    }
Quoted 1 line

This can be simplified while we are at it.

yep, only the else block remains

Quoted 5 lines

migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 795

Quoted 2 lines
migration/vmstate-types.c +1 -0
+    while (qemu_get_byte(f)) {

If we get out of sync, for any reason, we have no way to recover. The only way to recover is that we don't get a "false" in this position.

adding the number of nodes should do the job

Quoted 4 lines

Later, Juan.

So I think I will respin with the following modifications:
- use preload
- introduce 2 different macros
- encode/decode & test the number of nodes

Thank you for the quick feedbacks!

Eric