qom: Make "info qom-tree" show children sorted
cover qom: Make "info qom-tree" show children sorted
Extracted from my '[PATCH not-for-merge 0/5] Instrumentation for "Fixes around device realization"' on reviewer's advice.
Markus Armbruster (2):
qom: Constify object_get_canonical_path{,_component}()'s parameter
qom: Make "info qom-tree" show children sorted
include/qom/object.h | 4 ++-- qom/object.c | 4 ++-- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 3 files changed, 20 insertions(+), 12 deletions(-)
Signature · 1 line
2.21.3
PATCH 1/2 qom: Constify object_get_canonical_path{, _component}()'s parameter
--- include/qom/object.h | 4 ++-- qom/object.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
include/qom/object.h +2 -2
@@ -1406,7 +1406,7 @@ Object *object_get_internal_root(void);* path is the path within the composition tree starting from the root. * %NULL if the object doesn't have a parent (and thus a canonical path). */ -char *object_get_canonical_path_component (Object *obj); +char *object_get_canonical_path_component (const Object *obj);/** * object_get_canonical_path:
@@ -1414,7 +1414,7 @@ char *object_get_canonical_path_component(Object *obj);* Returns: The canonical path for a object. This is the path within the * composition tree starting from the root. */ -char *object_get_canonical_path (Object *obj); +char *object_get_canonical_path (const Object *obj);/** * object_resolve_path:
qom/object.c +2 -2
@@ -1883,7 +1883,7 @@ object_property_add_const_link(Object *obj, const char *name, NULL, OBJ_PROP_LINK_DIRECT); } -char *object_get_canonical_path_component (Object *obj) +char *object_get_canonical_path_component (const Object *obj) {ObjectProperty *prop =NULL ;GHashTableIter iter;
@@ -1908,7 +1908,7 @@ char *object_get_canonical_path_component(Object *obj)return NULL ; } -char *object_get_canonical_path (Object *obj) +char *object_get_canonical_path (const Object *obj) {Object *root =object_get_root ();char *newpath, *path =NULL ;
Signature · 1 line
2.21.3
Re: [PATCH 1/2] qom: Constify object_get_canonical_path{, _component}()'s parameter
Quoted 2 lines · Markus Armbruster
Suggested-by Eric Blake <[email protected]>Signed-off-by Markus Armbruster <[email protected]>
Quoted 50 lines
--- include/qom/object.h | 4 ++-- qom/object.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)include/qom/object.h +2
-2@@ -1406,7 +1406,7 @@ Object *object_get_internal_root(void);* path is the path within the composition tree starting from the root. * %NULL if the object doesn't have a parent (and thus a canonical path). */ -char *object_get_canonical_path_component (Object *obj); +char *object_get_canonical_path_component (const Object *obj);/** * object_get_canonical_path: @@ -1414,7 +1414,7 @@ char *object_get_canonical_path_component(Object *obj);* Returns: The canonical path for a object. This is the path within the * composition tree starting from the root. */ -char *object_get_canonical_path (Object *obj); +char *object_get_canonical_path (const Object *obj);/** * object_resolve_path: qom/object.c +2
-2@@ -1883,7 +1883,7 @@ object_property_add_const_link(Object *obj, const char *name, NULL, OBJ_PROP_LINK_DIRECT); } -char *object_get_canonical_path_component (Object *obj) +char *object_get_canonical_path_component (const Object *obj) {ObjectProperty *prop =NULL ;GHashTableIter iter;@@ -1908,7 +1908,7 @@ char *object_get_canonical_path_component(Object *obj)return NULL ; } -char *object_get_canonical_path (Object *obj) +char *object_get_canonical_path (const Object *obj) {Object *root =object_get_root ();char *newpath, *path =NULL ;
Re: [PATCH 1/2] qom: Constify object_get_canonical_path{,_component}()'s parameter
In reply to Markus ArmbrusterQuoted 44 lines · Markus Armbruster
Suggested-by Eric Blake <[email protected]>Signed-off-by Markus Armbruster <[email protected]>--- include/qom/object.h | 4 ++-- qom/object.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)include/qom/object.h +2
-2@@ -1406,7 +1406,7 @@ Object *object_get_internal_root(void);* path is the path within the composition tree starting from the root. * %NULL if the object doesn't have a parent (and thus a canonical path). */ -char *object_get_canonical_path_component (Object *obj); +char *object_get_canonical_path_component (const Object *obj);/** * object_get_canonical_path: @@ -1414,7 +1414,7 @@ char *object_get_canonical_path_component(Object *obj);* Returns: The canonical path for a object. This is the path within the * composition tree starting from the root. */ -char *object_get_canonical_path (Object *obj); +char *object_get_canonical_path (const Object *obj);/** * object_resolve_path: qom/object.c +1
-1@@ -1883,7 +1883,7 @@ object_property_add_const_link(Object *obj, const char *name, NULL, OBJ_PROP_LINK_DIRECT); } -char *object_get_canonical_path_component (Object *obj) +char *object_get_canonical_path_component (const Object *obj) {ObjectProperty *prop =NULL ;GHashTableIter iter;
qom/object.c char *object_get_canonical_path_component(Object *obj) · line 1914
Quoted 8 lines
qom/object.c +1
-1@@ -1908,7 +1908,7 @@ char *object_get_canonical_path_component(Object *obj)return NULL ; } -char *object_get_canonical_path (Object *obj) +char *object_get_canonical_path (const Object *obj) {Object *root =object_get_root ();char *newpath, *path =NULL ;
PATCH 2/2 qom: Make "info qom-tree" show children sorted
In reply to Markus Armbruster"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
qom/qom-hmp-cmds.c +16 -8
@@ -69,22 +69,25 @@ typedef struct QOMCompositionState {static void print_qom_composition (Monitor *mon,Object *obj,int indent); -static int print_qom_composition_child (Object *obj,void *opaque) +static int qom_composition_compare (const void *a,const void *b,void *ignore) { -QOMCompositionState *s = opaque; +return g_strcmp0 (a ?object_get_canonical_path_component (a) :NULL , + b ?object_get_canonical_path_component (b) :NULL ); +} -print_qom_composition (s->mon, obj, s->indent); +static int insert_qom_composition_child (Object *obj,void *opaque) +{ +GQueue *children = opaque; +g_queue_insert_sorted (children, obj, qom_composition_compare,NULL );return 0 ; }static void print_qom_composition (Monitor *mon,Object *obj,int indent) { -QOMCompositionState s = { - .mon = mon, - .indent = indent +2 , - };char *name; +GQueue children; +Object *child;if (obj ==object_get_root ()) { name =g_strdup ("" );
@@ -94,7 +97,12 @@ static void print_qom_composition(Monitor *mon, Object *obj, int indent)monitor_printf (mon,"%*s/%s (%s)\n" , indent,"" , name,object_get_typename (obj));g_free (name); -object_child_foreach (obj, print_qom_composition_child, &s); + +g_queue_init (&children); +object_child_foreach (obj, insert_qom_composition_child, &children); +while ((child =g_queue_pop_head (&children))) { +print_qom_composition (mon, child, indent +2 ); + } }void hmp_info_qom_tree (Monitor *mon,const QDict *dict)
Signature · 1 line
2.21.3
Quoted 3 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
yes. Thanks for fixing that.
Quoted 1 line
Signed-off-by Markus Armbruster <[email protected]>
I used the powernv9 machine.
Quoted 56 lines
--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)qom/qom-hmp-cmds.c +16
-8@@ -69,22 +69,25 @@ typedef struct QOMCompositionState {static void print_qom_composition (Monitor *mon,Object *obj,int indent); -static int print_qom_composition_child (Object *obj,void *opaque) +static int qom_composition_compare (const void *a,const void *b,void *ignore) { -QOMCompositionState *s = opaque; +return g_strcmp0 (a ?object_get_canonical_path_component (a) :NULL , + b ?object_get_canonical_path_component (b) :NULL ); +} -print_qom_composition (s->mon, obj, s->indent); +static int insert_qom_composition_child (Object *obj,void *opaque) +{ +GQueue *children = opaque; +g_queue_insert_sorted (children, obj, qom_composition_compare,NULL );return 0 ; }static void print_qom_composition (Monitor *mon,Object *obj,int indent) { -QOMCompositionState s = { - .mon = mon, - .indent = indent +2 , - };char *name; +GQueue children; +Object *child;if (obj ==object_get_root ()) { name =g_strdup ("" );@@ -94,7 +97,12 @@ static void print_qom_composition(Monitor *mon, Object *obj, int indent)monitor_printf (mon,"%*s/%s (%s)\n" , indent,"" , name,object_get_typename (obj));g_free (name); -object_child_foreach (obj, print_qom_composition_child, &s); + +g_queue_init (&children); +object_child_foreach (obj, insert_qom_composition_child, &children); +while ((child =g_queue_pop_head (&children))) { +print_qom_composition (mon, child, indent +2 ); + } }void hmp_info_qom_tree (Monitor *mon,const QDict *dict)
Quoted 51 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)qom/qom-hmp-cmds.c +10
-7@@ -69,22 +69,25 @@ typedef struct QOMCompositionState {static void print_qom_composition (Monitor *mon,Object *obj,int indent); -static int print_qom_composition_child (Object *obj,void *opaque) +static int qom_composition_compare (const void *a,const void *b,void *ignore) { -QOMCompositionState *s = opaque; +return g_strcmp0 (a ?object_get_canonical_path_component (a) :NULL , + b ?object_get_canonical_path_component (b) :NULL ); +} -print_qom_composition (s->mon, obj, s->indent); +static int insert_qom_composition_child (Object *obj,void *opaque) +{ +GQueue *children = opaque; +g_queue_insert_sorted (children, obj, qom_composition_compare,NULL );return 0 ; }static void print_qom_composition (Monitor *mon,Object *obj,int indent) { -QOMCompositionState s = { - .mon = mon, - .indent = indent +2 , - };char *name; +GQueue children; +Object *child;if (obj ==object_get_root ()) { name =g_strdup ("" );@@ -94,7 +97,12 @@ static void print_qom_composition(Monitor *mon, Object *obj, int indent)monitor_printf (mon,"%*s/%s (%s)\n" , indent,"" , name,object_get_typename (obj));g_free (name);
qom/qom-hmp-cmds.c static void print_qom_composition(Monitor *mon, Object *obj, int indent) · line 108
Quoted 10 lines
qom/qom-hmp-cmds.c +6
-1-object_child_foreach (obj, print_qom_composition_child, &s); + +g_queue_init (&children); +object_child_foreach (obj, insert_qom_composition_child, &children); +while ((child =g_queue_pop_head (&children))) { +print_qom_composition (mon, child, indent +2 ); + } }void hmp_info_qom_tree (Monitor *mon,const QDict *dict)
Slow down with: 'Make "info qom-tree" show children sorted'
In reply to Markus ArmbrusterQuoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10
the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
This causes our gitlab CI to constantly fail since the patch got merged, since the testing time now exceeds the 1h time limit:
https://gitlab.com/qemu-project/qemu/-/pipelines/156767175
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
Thanks,
Thomas
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 12 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
Quoted 8 lines
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
Argh, yesterday I reviewed again all the range except this patch... not sure why as looking at it now it is obvious.
Quoted 8 lines
This causes our gitlab CI to constantly fail since the patch got merged, since the testing time now exceeds the 1h time limit:
https://gitlab.com/qemu-project/qemu/-/pipelines/156767175
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
Suggestion: add a '-u' option for unsorted mode, to use in qtests.
Humans want the new behavior (default: sorted).
Regards,
Phil.
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 12 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
Quoted 8 lines
Quoted 8 lines
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
That's surprising.
Quoted 8 lines
Argh, yesterday I reviewed again all the range except this patch... not sure why as looking at it now it is obvious.
Quoted 5 lines
This causes our gitlab CI to constantly fail since the patch got merged, since the testing time now exceeds the 1h time limit:
Quoted 7 lines
Quoted 3 lines
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
Suggestion: add a '-u' option for unsorted mode, to use in qtests.
Humans want the new behavior (default: sorted).
Last resort.
I'll look into speeding up the sort first.
Work-around: drop -m slow until we get this sorted (pardon the pun).
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 18 lines · Markus Armbruster
Quoted 16 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
That's surprising.
It's a bit surprising indeed, but on the other hand using g_queue_insert_sorted results in a quadratic loop. It should probably be fixed by using g_queue_sort, or switching to g_list_prepend+g_list_sort.
Paolo
Quoted 22 lines
Quoted 8 lines
Argh, yesterday I reviewed again all the range except this patch... not sure why as looking at it now it is obvious.
Quoted 5 lines
This causes our gitlab CI to constantly fail since the patch got merged, since the testing time now exceeds the 1h time limit:
Quoted 7 lines
Quoted 3 lines
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
Suggestion: add a '-u' option for unsorted mode, to use in qtests.
Humans want the new behavior (default: sorted).
Last resort.
I'll look into speeding up the sort first.
Work-around: drop -m slow until we get this sorted (pardon the pun).
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 22 lines · Paolo Bonzini
Quoted 21 lines · Markus Armbruster
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
That's surprising.
It's a bit surprising indeed, but on the other hand using
g_queue_insert_sortedresults in a quadratic loop.
The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
Quoted 2 lines
It should probably
be fixed by usingg_queue_sort, or switching to g_list_prepend+g_list_sort.
Yes.
Additional ideas on how to make the test less wasteful elsewhere in the thread.
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 27 lines · Markus Armbruster
Quoted 25 lines · Paolo Bonzini
Quoted 24 lines · Markus Armbruster
Quoted 21 lines · Philippe Mathieu-Daudé
Quoted 20 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
That's surprising.
It's a bit surprising indeed, but on the other hand using
g_queue_insert_sortedresults in a quadratic loop.The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
The test suite as a whole is more like n^3 because it is running sooooo many repeats of "info qom-tree"
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: Slow down with: 'Make "info qom-tree" show children sorted'
In reply to Markus ArmbrusterQuoted 27 lines · On Tue, 07 Jul 2020 14:00:06 +0200 Markus Armbruster <[email protected]> wrote:
Quoted 25 lines · Paolo Bonzini
Quoted 24 lines · Markus Armbruster
Quoted 21 lines · Philippe Mathieu-Daudé
Quoted 20 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
That's surprising.
It's a bit surprising indeed, but on the other hand using
g_queue_insert_sortedresults in a quadratic loop.The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
Is this another consequence of the ludicrous number of QOM objects we create for LMB DRCs (one for every 256MiB of guest RAM)? Avoiding that is on my list. Though avoiding a n^2 behaviour here is probably a good idea anyway.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 35 lines · David Gibson
Quoted 32 lines · On Tue, 07 Jul 2020 14:00:06 +0200 Markus Armbruster <[email protected]> wrote:
Quoted 29 lines · Paolo Bonzini
Quoted 18 lines · Markus Armbruster
Quoted 16 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
That's surprising.
It's a bit surprising indeed, but on the other hand using
g_queue_insert_sortedresults in a quadratic loop.
Quoted 6 lines
Quoted 2 lines
The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
Is this another consequence of the ludicrous number of QOM objects we create for LMB DRCs (one for every 256MiB of guest RAM)? Avoiding that is on my list.
You're talking about machine pseries, I presume. With print_qom_composition() patched to print the number of children, I get
$ echo -e 'info qom-tree\nq' | ../qemu/bld/ppc64-softmmu/qemu-system-ppc64 -S -display none -M pseries -accel qtest -monitor stdio | grep '###' | sort | uniq -c | sort -k 3n
360 ### 0 children
5 ### 1 children
5 ### 2 children
2 ### 3 children
1 ### 4 children
1 ### 15 children
1 ### 16 children
1 ### 18 children
1 ### 37 children
1 ### 266 childrenThe outlier is
/device[5] (spapr-pci-host-bridge)
due to its 256 spapr-drc-pci children.
I found quite a few machines with similar outliers. ARM machines nuri and smdkc210 together take the cake: they each have a node with 513 children.
My stupid n^2 sort is unnoticable in normal, human usage even for n=513.
Quoted 2 lines
Though avoiding a n^2 behaviour here is probably a good
idea anyway.
Agreed.
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 13 lines · On Mon, 13 Jul 2020 18:13:42 +0200 Markus Armbruster <[email protected]> wrote:
David Gibson <[email protected]> writes:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
Is this another consequence of the ludicrous number of QOM objects we create for LMB DRCs (one for every 256MiB of guest RAM)? Avoiding that is on my list.
You're talking about machine pseries, I presume.
Yes.
Quoted 15 lines
With
print_qom_composition()patched to print the number of children, I get$ echo -e 'info qom-tree\nq' | ../qemu/bld/ppc64-softmmu/qemu-system-ppc64 -S -display none -M pseries -accel qtest -monitor stdio | grep '###' | sort | uniq -c | sort -k 3n 360 ### 0 children 5 ### 1 children 5 ### 2 children 2 ### 3 children 1 ### 4 children 1 ### 15 children 1 ### 16 children 1 ### 18 children 1 ### 37 children 1 ### 266 children
Quoted 5 lines
The outlier is
/device[5] (spapr-pci-host-bridge)due to its 256 spapr-drc-pci children.
Right, that's one for each possible PCI slot on the bus. That will be reduced by the idea I have in mind for this, but...
Quoted 5 lines
I found quite a few machines with similar outliers. ARM machines nuri and smdkc210 together take the cake: they each have a node with 513 children.
My stupid n^2 sort is unnoticable in normal, human usage even for n=513.
... as you say, 256 shouldn't really be a problem. I was concerned about LMB DRCs rather than PCI DRCs. To have that show up, you might need to create a machine with a large difference between initial memory and maxmem - I think you'll get a DRC object for every 256MiB in there, which can easily get into the thousands for large (potential) memory VMs.
I don't know what the config was that showed up this problem in the first place, and whether that could be the case there.
Quoted 4 lines
Quoted 2 lines
Though avoiding a n^2 behaviour here is probably a good
idea anyway.Agreed.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 55 lines · David Gibson
Quoted 13 lines · On Mon, 13 Jul 2020 18:13:42 +0200 Markus Armbruster <[email protected]> wrote:
David Gibson <[email protected]> writes:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
The surprising part is that n turns out to be large enough for n^2 to matter *that* much.
Is this another consequence of the ludicrous number of QOM objects we create for LMB DRCs (one for every 256MiB of guest RAM)? Avoiding that is on my list.
You're talking about machine pseries, I presume.
Yes.
Quoted 15 lines
With
print_qom_composition()patched to print the number of children, I get$ echo -e 'info qom-tree\nq' | ../qemu/bld/ppc64-softmmu/qemu-system-ppc64 -S -display none -M pseries -accel qtest -monitor stdio | grep '###' | sort | uniq -c | sort -k 3n 360 ### 0 children 5 ### 1 children 5 ### 2 children 2 ### 3 children 1 ### 4 children 1 ### 15 children 1 ### 16 children 1 ### 18 children 1 ### 37 children 1 ### 266 childrenQuoted 5 lines
The outlier is
/device[5] (spapr-pci-host-bridge)due to its 256 spapr-drc-pci children.
Right, that's one for each possible PCI slot on the bus. That will be reduced by the idea I have in mind for this, but...
Quoted 4 lines
I found quite a few machines with similar outliers. ARM machines nuri and smdkc210 together take the cake: they each have a node with 513 children.
Quoted 8 lines
Quoted 1 line
My stupid n^2 sort is unnoticable in normal, human usage even for n=513.
... as you say, 256 shouldn't really be a problem. I was concerned about LMB DRCs rather than PCI DRCs. To have that show up, you might need to create a machine with a large difference between initial memory and maxmem - I think you'll get a DRC object for every 256MiB in there, which can easily get into the thousands for large (potential) memory VMs.
Okay, I can reproduce: with -m 256,128G, /machine has 549 children, of which 511 are spapr-drc-lmb.
Quoted 2 lines
I don't know what the config was that showed up this problem in the first place, and whether that could be the case there.
Thomas reported device-introspect-test -m slow has become much slower for ppc64. Bisection traced it to my commit e8c9e65816 "qom: Make "info qom-tree" show children sorted". Uses default memory size, no spapr-drc-lmb as far as I remember.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Though avoiding a n^2 behaviour here is probably a good
idea anyway.Agreed.
Patch posted:
Subject: [PATCH for-5.1 5/5] qom: Make info qom-tree sort children more efficiently
Message-Id: <[email protected]>
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 19 lines · On Thu, 16 Jul 2020 07:37:17 +0200 Markus Armbruster <[email protected]> wrote:
David Gibson <[email protected]> writes:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]Quoted 7 lines
... as you say, 256 shouldn't really be a problem. I was concerned about LMB DRCs rather than PCI DRCs. To have that show up, you might need to create a machine with a large difference between initial memory and maxmem - I think you'll get a DRC object for every 256MiB in there, which can easily get into the thousands for large (potential) memory VMs.
Quoted 2 lines
Okay, I can reproduce: with -m 256,128G, /machine has 549 children, of which 511 are spapr-drc-lmb.
Ok.
Quoted 7 lines
Quoted 2 lines
I don't know what the config was that showed up this problem in the first place, and whether that could be the case there.
Thomas reported device-introspect-test -m slow has become much slower for ppc64. Bisection traced it to my commit
e8c9e65816"qom: Make "info qom-tree" show children sorted". Uses default memory size, no spapr-drc-lmb as far as I remember.
Ok, I think that nixes my theory.
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Though avoiding a n^2 behaviour here is probably a good
idea anyway.Agreed.
Patch posted:
Subject: [PATCH for-5.1 5/5] qom: Make info qom-tree sort children more efficiently
Message-Id: <[email protected]>
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Re: Slow down with: 'Make "info qom-tree" show children sorted'
In reply to Thomas HuthQuoted 21 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
Quoted 8 lines
This causes our gitlab CI to constantly fail since the patch got merged, since the testing time now exceeds the 1h time limit:
https://gitlab.com/qemu-project/qemu/-/pipelines/156767175
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
I think the answer is to stop using q_queue_insert_sorted(). The impl of it looks like it is quadratic in complexity. Instead store the objects in a plain array and then use qsort() at the end.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 8 lines · Daniel P. Berrangé
Quoted 3 lines · Thomas Huth
Sure, we can work around that problem in the CI (Alex has already a patch queued), but still, is there something you could do about this massive slowdown?
I think the answer is to stop using
q_queue_insert_sorted(). The impl of it looks like it is quadratic in complexity. Instead store the objects in a plain array and then useqsort()at the end.
g_list_sort uses a mergesort, so it's a good choice as well. But yeah GPtrArray is another possibility.
Paolo
Re: Slow down with: 'Make "info qom-tree" show children sorted'
In reply to Thomas HuthQuoted 12 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
Quoted 8 lines
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
I think the test case itself could be optimized. Currently it does approx
for each device type
info qom-tree
device_addr type,help
info qom-tree
it compares the before/after qom-tree to look for stray objects or to try to trigger crashes.
The info qom-tree calls could be pushed outside the loop
info qom-tree
for each device type
device_addr type,help
info qom-tree
Taking /x86_64/device/introspect/concrete/defaults/pc-q35-5.1 as a example, this change is the difference between 20 seconds running and 3 seconds running.
Reverting Markus' change actually didn't make much difference, only reducing the 20 seconds to 17 seconds.
The downside is that if there is a stray object/crash, it would not immediately associate with the device type. I'm not sure that's a real problem though. Especially if we are running this as pre-merge CI we'll only need to look at the patch series to find the broken device. If this is quick enough that we can run it as standard, instead of only with -m slow, then its a net win I think.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: Slow down with: 'Make "info qom-tree" show children sorted'
Quoted 11 lines · Daniel P. Berrangé
The info qom-tree calls could be pushed outside the loop
info qom-tree
for each device type
device_addrtype,help
info qom-treeTaking /x86_64/device/introspect/concrete/defaults/pc-q35-5.1 as a example, this change is the difference between 20 seconds running and 3 seconds running.
Or even, saving half the runtime:
info qom-tree
for each device type
device_addr type,help
info qom-tree
Paolo
Quoted 9 lines
Reverting Markus' change actually didn't make much difference, only reducing the 20 seconds to 17 seconds.
The downside is that if there is a stray object/crash, it would not immediately associate with the device type. I'm not sure that's a real problem though. Especially if we are running this as pre-merge CI we'll only need to look at the patch series to find the broken device. If this is quick enough that we can run it as standard, instead of only with -m slow, then its a net win I think.
Re: Slow down with: 'Make "info qom-tree" show children sorted'
In reply to Daniel P. BerrangéQuoted 35 lines · Daniel P. Berrangé
Quoted 12 lines · Thomas Huth
Quoted 8 lines · Markus Armbruster
"info qom-tree" prints children in unstable order. This is a pain when diffing output for different versions to find change. Print it sorted.
Signed-off-by Markus Armbruster <[email protected]>--- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)Hi Markus,
Quoted 8 lines
this patch causes a slow down of the qtests which becomes quite massive when e.g. using the ppc64 and thourough testing. When I'm running
QTEST_QEMU_BINARY="ppc64-softmmu/qemu-system-ppc64" time \
./tests/qtest/device-introspect-test -m slow | tail -n 10the test runs for ca. 6m40s here before the patch got applied, and for mor than 20 minutes after the patch got applied!
I think the test case itself could be optimized. Currently it does approx
for each device type
info qom-tree
device_addrtype,help
info qom-treeit compares the before/after qom-tree to look for stray objects or to try to trigger crashes.
The info qom-tree calls could be pushed outside the loop
Quoted 8 lines
info qom-tree
for each device type
device_addrtype,help
info qom-treeTaking /x86_64/device/introspect/concrete/defaults/pc-q35-5.1 as a example, this change is the difference between 20 seconds running and 3 seconds running.
Patch?
Quoted 2 lines
Reverting Markus' change actually didn't make much difference, only reducing the 20 seconds to 17 seconds.
I found and plugged a memory leak. I haven't checked its impact on test performance.
Quoted 6 lines
The downside is that if there is a stray object/crash, it would not immediately associate with the device type. I'm not sure that's a real problem though. Especially if we are running this as pre-merge CI we'll only need to look at the patch series to find the broken device. If this is quick enough that we can run it as standard, instead of only with -m slow, then its a net win I think.
Easier investigation of failures is a valid argument, but not at this price. Our "make check" is painfully slow.
Quoted 11 lines · Markus Armbruster
Extracted from my '[PATCH not-for-merge 0/5] Instrumentation for "Fixes around device realization"' on reviewer's advice.
Markus Armbruster (2):
qom: Constify object_get_canonical_path{,_component}()'s parameter
qom: Make "info qom-tree" show children sortedinclude/qom/object.h | 4 ++-- qom/object.c | 4 ++-- qom/qom-hmp-cmds.c | 24 ++++++++++++++++-------- 3 files changed, 20 insertions(+), 12 deletions(-)
I've just given this a quick spin on a couple of my local branches here and it looks good:
ATB,
Mark.