migration: Support gtree migration
PATCH v2 1/1 migration: Support gtree migration
Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.
When putting the items, the tree is traversed in sorted order by g_tree_foreach.
On the get() path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of functions must be stored within the VMStateField in a new opaque pointer.
Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.
Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).
Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.
---
v1 -> v2:
- fix compilation issues reported by robots
- fixed init of VMSD array
- direct key now dumped as a 32b
- removed useless cast from/to pointer
--- include/migration/vmstate.h | 31 +++ migration/trace-events | 6 + migration/vmstate-types.c | 133 +++++++++++ tests/test-vmstate.c | 427 ++++++++++++++++++++++++++++++++++++ 4 files changed, 597 insertions(+)
include/migration/vmstate.h +31 -0
@@ -171,6 +171,7 @@ struct VMStateField {int version_id;int struct_version_id;bool (*field_exists)(void *opaque,int version_id); +void *data; };struct VMStateDescription {
@@ -224,6 +225,7 @@ extern const VMStateInfo vmstate_info_unused_buffer;extern const VMStateInfo vmstate_info_tmp;extern const VMStateInfo vmstate_info_bitmap;extern const VMStateInfo vmstate_info_qtailq; +extern const VMStateInfo vmstate_info_gtree;#define type_check_2darray (t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)/*
@@ -754,6 +756,35 @@ extern const VMStateInfo vmstate_info_qtailq; .start = offsetof(_type, _next), \ } +typedef struct GTreeInitData { +GCompareDataFunc key_compare_func; +gpointer key_compare_data; +GDestroyNotify key_destroy_func; +GDestroyNotify value_destroy_func; +}GTreeInitData ; + +/* +* For migrating a GTree. +* _vmsd: Start address of the 2 element array containing the key vmsd +* and the data vmsd +* _key_type: type of the key +* _val_type: type of the value +* _data: pointer to a GTreeInitData struct +* if _data is NULL, the target tree must have been properly initialized +*/ +#define VMSTATE_GTREE_V (_field, _state, _version, _vmsd, \ + _key_type, _val_type, _data) \ +{ \ + .name = (stringify(_field)), \ + .version_id = (_version), \ + .vmsd = (_vmsd), \ + .info = &vmstate_info_gtree, \ + .start = sizeof(_key_type), \ + .size = sizeof(_val_type), \ + .offset = offsetof(_state, _field), \ + .data = (_data) \ +} +/* _f : field name _f_n : num of elements field_name _n : num of elements
migration/trace-events +6 -0
@@ -71,6 +71,12 @@ get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d" put_qtailq(const char *name, int version_id) "%s v%d" put_qtailq_end(const char *name, const char *reason) "%s %s" +get_gtree(const char *key_name, const char *val_name, int version) "%s/%s %d" +get_gtree_fail(const char *key_name, const char *reason, int val) "%s %s/%d" +get_gtree_succeed(const char *key_name, const char *val_name, const char *reason, int val) "%s/%s %s/%d" +put_gtree(const char *key_name, int key_version, const char *val_name, int val_version) "%s(%d)/%s(%d)" +put_gtree_end(const char *key_name, const char *val_name, const char *reason) "%s/%s %s" + # qemu-file.c qemu_file_fclose(void) ""
migration/vmstate-types.c +133 -0
142 lines not on this page · raw message
tests/test-vmstate.c +427 -0
445 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 32 lines · Eric Auger
Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.
When putting the items, the tree is traversed in sorted order by
g_tree_foreach.On the
get()path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of functions must be stored within the VMStateField in a new opaque pointer.Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.
Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).
Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.
Signed-off-by Eric Auger <[email protected]>---
migration/vmstate-types.c +0
-0
migration/vmstate-types.c line 20
Quoted 4 lines
migration/vmstate-types.c +1
-0@@ -17,6 +17,7 @@#include "qemu/error-report.h" #include "qemu/queue.h" #include "trace.h" +#include <glib.h>
Not required, everything in QEMU includes this already.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 10 lines · Eric Auger
Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.
When putting the items, the tree is traversed in sorted order by
g_tree_foreach.On the
get()path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of
^^
or.
Quoted 2 lines
functions must be stored within the VMStateField in a new opaque pointer.
I am not fully convinced that the automatic mode is needed. Especially the ->data field. I *fear* it being abused for other cases.
Quoted 3 lines
Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.
There is a pre_load() function for the VMState that creates this. it can be used to initualize the field value, no? That way the data part is not needed. I think this will make the code less complex, what do you think?
Quoted 2 lines
Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).
I am wondering if for this, it is better to add two VMSTATE (at least at the macro level). SIMPLE_TREE, and TREE, or whataver oyou want to call it. But I haven't fully looked into it.
The other general "consideration" that I have is that there is neither of:
- marker between elements
- number of elements
- total size/size of elements.
That makes completelly impractical to "walk" the migration stream without understanding exactyl what is there. (Now, to be fair, there are other parts of qemu that are like that. PCI cames to mind.)
Quoted 2 lines
Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.
Really nice to have the tests. Thanks.
include/migration/vmstate.h struct VMStateField { · line 175
Quoted 13 lines
Signed-off-by Eric Auger <[email protected]>include/migration/vmstate.h +1
-0@@ -171,6 +171,7 @@ struct VMStateField {int version_id;int struct_version_id;bool (*field_exists)(void *opaque,int version_id); +void *data; };
This is the bit that I don't really like :p
include/migration/vmstate.h extern const VMStateInfo vmstate_info_qtailq; · line 764
Quoted 7 lines
include/migration/vmstate.h +6
-0+typedef struct GTreeInitData { +GCompareDataFunc key_compare_func; +gpointer key_compare_data; +GDestroyNotify key_destroy_func; +GDestroyNotify value_destroy_func; +}GTreeInitData ;
My understanding is that if you do this on the pre_load() function, you don't need this at all.
Quoted 22 lines
migration/vmstate-types.c +7
-0@@ -17,6 +17,7 @@#include "qemu/error-report.h" #include "qemu/queue.h" #include "trace.h" +#include <glib.h> /* bool */ @@ -691,3 +692,135 @@ const VMStateInfo vmstate_info_qtailq = { .get = get_qtailq, .put = put_qtailq, }; + +struct put_gtree_data { +QEMUFile *f; +const VMStateField *field; +QJSON *vmdesc; +};
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 710
Quoted 10 lines
migration/vmstate-types.c +10
-0+ +static gboolean put_gtree_elem (gpointer key,gpointer value,gpointer data) +{ +struct put_gtree_data *capsule = (struct put_gtree_data *)data; +const VMStateField *field = capsule->field; +QEMUFile *f = capsule->f; +const VMStateDescription *key_vmsd = &field->vmsd[0 ]; +const VMStateDescription *data_vmsd = &field->vmsd[1 ]; + +qemu_put_byte (f, true);
Ok. there is a marker O:-)
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 719
Quoted 9 lines
migration/vmstate-types.c +9
-0+ +/* put the key */ +if (!key_vmsd->fields) { +qemu_put_be32 (f,GPOINTER_TO_UINT (key)); + }else { +if (vmstate_save_state (f, key_vmsd, key, capsule->vmdesc)) { +return true; + } + }
But it is magic to know if it is a simple or complex key.
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 793
Quoted 11 lines
migration/vmstate-types.c +11
-0+if (field->data) { + init_data = (GTreeInitData *)field->data; + tree =g_tree_new_full (init_data->key_compare_func, + init_data->key_compare_data, + init_data->key_destroy_func, + init_data->value_destroy_func); + *pval = tree; + }else { +/* tree is externally allocated */ + tree = *pval; + }
This can be simplified while we are at it.
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 795
Quoted 1 line
migration/vmstate-types.c +1
-0+while (qemu_get_byte (f)) {
If we get out of sync, for any reason, we have no way to recover. The only way to recover is that we don't get a "false" in this position.
Later, Juan.
Hi Juan,
Quoted 13 lines · Juan Quintela
Quoted 10 lines · Eric Auger
Introduce support for GTree migration. A custom save/restore is implemented. Each item is made of a key and a data. For that reason, 2 VMSD objects are passed into the GTree VMStateField.
When putting the items, the tree is traversed in sorted order by
g_tree_foreach.On the
get()path, gtrees must be allocated using the proper key compare, key destroy and value destroy. This can be done externally of automatically. If done automatically, the set of^^
Quoted 7 lines
or.
Quoted 2 lines
functions must be stored within the VMStateField in a new opaque pointer.
I am not fully convinced that the automatic mode is needed. Especially the ->data field. I *fear* it being abused for other cases.
OK. I implemented your suggestion, ie. using preload and it does the job. So I don't need that field anymore.
Quoted 9 lines
Quoted 4 lines
Automatic allocation is needed for complex state save/restore. For instance the virtio-iommu uses a gtree of domain and each domain has a gtree of mappings.
There is a
pre_load()function for the VMState that creates this. it can be used to initualize the field value, no? That way the data part is not needed. I think this will make the code less complex, what do you think?
agreed
Quoted 7 lines
Quoted 3 lines
Special care was taken about direct key (ie. when the key is not a pointer to an object but is directly a value).
I am wondering if for this, it is better to add two VMSTATE (at least at the macro level).
SIMPLE_TREE, and TREE, or whataver oyou want to call it. But I haven't fully looked into it.
I don't have a strong opinion here. At the moment I test the key_vmsd->field and if it is NULL this means the key is a direct one.
Otherwise we could have 2 macros, a single info, but 2 different field names. the name would allow to know the nature of the key.
Quoted 4 lines
The other general "consideration" that I have is that there is neither of:
- marker between elements
so we have one
Quoted 1 line
- number of elements
you don't have it
Quoted 1 line
- total size/size of elements.
you just have the size of the key and the size of the value at the moment. I could easily add the number of nodes.
Quoted 29 lines
That makes completelly impractical to "walk" the migration stream without understanding exactyl what is there. (Now, to be fair, there are other parts of qemu that are like that. PCI cames to mind.)
Quoted 2 lines
Tests are added to test save/dump of structs containing gtrees including the virtio-iommu domain/mappings scenario.
Really nice to have the tests. Thanks.
include/migration/vmstate.h struct VMStateField { · line 175
Quoted 13 lines
Signed-off-by Eric Auger <[email protected]>include/migration/vmstate.h +1
-0@@ -171,6 +171,7 @@ struct VMStateField {int version_id;int struct_version_id;bool (*field_exists)(void *opaque,int version_id); +void *data; };This is the bit that I don't really like :p
Quoted 1 line
Quoted 9 lines
include/migration/vmstate.h extern const VMStateInfo vmstate_info_qtailq; · line 764
Quoted 6 lines
include/migration/vmstate.h +6
-0+typedef struct GTreeInitData { +GCompareDataFunc key_compare_func; +gpointer key_compare_data; +GDestroyNotify key_destroy_func; +GDestroyNotify value_destroy_func; +}GTreeInitData ;My understanding is that if you do this on the
pre_load()function, you don't need this at all.
yep
Quoted 27 lines
Quoted 27 lines
migration/vmstate-types.c +11
-0@@ -17,6 +17,7 @@#include "qemu/error-report.h" #include "qemu/queue.h" #include "trace.h" +#include <glib.h> /* bool */ @@ -691,3 +692,135 @@ const VMStateInfo vmstate_info_qtailq = { .get = get_qtailq, .put = put_qtailq, }; + +struct put_gtree_data { +QEMUFile *f; +const VMStateField *field; +QJSON *vmdesc; +}; + +static gboolean put_gtree_elem (gpointer key,gpointer value,gpointer data) +{ +struct put_gtree_data *capsule = (struct put_gtree_data *)data;
Quoted 8 lines
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 710
Quoted 6 lines
migration/vmstate-types.c +6
-0+const VMStateField *field = capsule->field; +QEMUFile *f = capsule->f; +const VMStateDescription *key_vmsd = &field->vmsd[0 ]; +const VMStateDescription *data_vmsd = &field->vmsd[1 ]; + +qemu_put_byte (f, true);Ok. there is a marker O:-)
yep
Quoted 11 lines
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 719
Quoted 10 lines
migration/vmstate-types.c +9
-0+ +/* put the key */ +if (!key_vmsd->fields) { +qemu_put_be32 (f,GPOINTER_TO_UINT (key)); + }else { +if (vmstate_save_state (f, key_vmsd, key, capsule->vmdesc)) { +return true; + } + }
Quoted 1 line
But it is magic to know if it is a simple or complex key.
key_vmsd->fields is used
this means you set
static const VMStateDescription vmstate_id_domain[2] = {
{}, VMSTATE_DOMAIN /* direct key, value */
};
Quoted 14 lines
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 793
Quoted 13 lines
migration/vmstate-types.c +11
-0+if (field->data) { + init_data = (GTreeInitData *)field->data; + tree =g_tree_new_full (init_data->key_compare_func, + init_data->key_compare_data, + init_data->key_destroy_func, + init_data->value_destroy_func); + *pval = tree; + }else { +/* tree is externally allocated */ + tree = *pval; + }
Quoted 1 line
This can be simplified while we are at it.
yep, only the else block remains
Quoted 5 lines
migration/vmstate-types.c const VMStateInfo vmstate_info_qtailq = { · line 795
Quoted 2 lines
migration/vmstate-types.c +1
-0+while (qemu_get_byte (f)) {If we get out of sync, for any reason, we have no way to recover. The only way to recover is that we don't get a "false" in this position.
adding the number of nodes should do the job
Quoted 4 lines
Later, Juan.
So I think I will respin with the following modifications:
- use preload
- introduce 2 different macros
- encode/decode & test the number of nodes
Thank you for the quick feedbacks!
Eric