numa: deprecate -numa node, mem and default memory distribution
cover numa: deprecate -numa node, mem and default memory distribution
1) "I'm considering to deprecating -mem-path/prealloc CLI options and replacing
them with a single memdev Machine property to allow interested users to pick used backend for initial RAM (fixes mixed -mem-path+hostmem backends issues) and as a transition step to modeling initial as a Device instead of (ab)using MemoryRegion APIs."
(for more details see: https://www.mail-archive.com/[email protected]/msg596314.html)
However there is a couple of roadblocks on the way (s390x and numa memory handling). I think I finally thought out a way to hack s390x in migration compatible manner, but I don't see a way to do it for -numa node,mem and default RAM assignement to nodes. Considering both numa usecases aren't meaningfully using NUMA (aside guest side testing), and could be replaced with explicitly used memdev parameter, I'd like to propose removing these fake NUMA friends, hence this deprecation.
As result of removing deprecated options and replacing initial RAM allocation with 'memdev's (1), QEMU will allocate guest RAM in consistent way, fixing mixed use-case and allowing boards to move towards modelling initial RAM as Device(s). Which in its own turn should allow to cleanup NUMA/HMP/memory accounting code more by dropping ad-hoc node_mem tracking and reusing device enumeration instead.
Igor Mammedov (2):
numa: deprecate 'mem' parameter of '-numa node' option
numa: deprecate implict memory distribution between nodes
numa.c | 5 +++++ qemu-deprecated.texi | 21 +++++++++++++++++++++ 2 files changed, 26 insertions(+)
Signature · 1 line
2.7.4
PATCH 1/2 numa: deprecate 'mem' parameter of '-numa node' option
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use node_mem to generate FDT/ACPI description from it.
--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)
numa.c +2 -0
@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;
qemu-deprecated.texi +14 -0
@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)
Signature · 1 line
2.7.4
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.
Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 52 lines
Signed-off-by Igor Mammedov <[email protected]>--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)numa.c +2
-0@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;qemu-deprecated.texi +14
-0@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)Signature · 6 lines
2.7.4 -- libvir-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/libvir-list
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Mixed memory issue[1] first came from libvirt side RHBZ1624223, back then it was resolved on libvirt side in favor of migration compatibility vs correctness (i.e. bind policy doesn't work as expected). What worse that it was made default and affects all new machines, as I understood it.
In case of -mem-path + -mem-prealloc (with 1 numa node or numa less) it's possible on QEMU side to make conversion to memdev in migration compatible way (that's what stopped Michal from memdev approach). But it's hard to do so in multi-nodes case as amount of MemoryRegions is different.
Point is to consider 'mem' as mis-configuration error, as the user in the first place using broken numa configuration
(i.e. fake numa configuration doesn't actually improve performance).
CCed David, maybe he could offer a way to do 1:n migration and other way around.
Quoted 54 lines
Quoted 51 lines
Signed-off-by Igor Mammedov <[email protected]>--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)numa.c +2
-0@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;qemu-deprecated.texi +14
-0@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)Signature · 6 lines
2.7.4 -- libvir-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/libvir-listRegards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
So, how could this formal notice be served constructively?
If we reject outdated NUMA configurations starting with machine type T, we can remove the means to create those configurations along with machine type T-1. Won't happen anytime soon, will happen eventually, because in the long run, all machine types are dead (apologies to Keynes).
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 52 lines · Markus Armbruster
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
Quoted 6 lines
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
If upstream deletes the feature, then that in turn breaks the downstream unless downstream reverts the upstream change. When we have large overlap between downstream & upstream maintainer, it is not beneficial to delete the feature upstream as any effort saved upstream usually expands into larger effort downstream.
Quoted 9 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
So, how could this formal notice be served constructively?
Quoted 8 lines
If we reject outdated NUMA configurations starting with machine type T, we can remove the means to create those configurations along with machine type T-1. Won't happen anytime soon, will happen eventually, because in the long run, all machine types are dead (apologies to Keynes).
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 60 lines · Daniel P. Berrangé
Quoted 52 lines · Markus Armbruster
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
Quoted 6 lines
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 5 lines
If upstream deletes the feature, then that in turn breaks the downstream unless downstream reverts the upstream change. When we have large overlap between downstream & upstream maintainer, it is not beneficial to delete the feature upstream as any effort saved upstream usually expands into larger effort downstream.
It can't "break" existing downstreams, only future ones forked off after the deletion. Such a fork cares only if it has backward compatibility requirements to satisfy that require the feature. My point is: it's not a simple absolute, it's a complex tradeoff.
Quoted 15 lines
Quoted 15 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
So, how could this formal notice be served constructively?
If we reject outdated NUMA configurations starting with machine type T, we can remove the means to create those configurations along with machine type T-1. Won't happen anytime soon, will happen eventually, because in the long run, all machine types are dead (apologies to Keynes).
Quoted 7 lines
Quoted 2 lines
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Fair point. A practical way for management applications to decide which of the two interfaces they can use with which machine type may be required for deprecating one of the interfaces with new machine types.
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Daniel P. Berrangé
Quoted 2 lines · Markus Armbruster
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
Quoted 8 lines
Quoted 4 lines
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Fair point. A practical way for management applications to decide which of the two interfaces they can use with which machine type may be required for deprecating one of the interfaces with new machine types.
We currently have "qom-list-properties" which can report on the existance of properties registered against object types. What it can't do though is report on the default values of these properties.
What's interesting though is that qmp_qom_list_properties will actually instantiate objects in order to query properties, if the type isn't an abstract type.
IOW, even if you are running "$QEMU -machine none", then if at the qmp-shell you do
(QEMU) qom-list-properties typename=pc-q35-2.6-machine
it will have actually instantiate the pc-q35-2.6-machine machine type. Since it has instantiated the machine, the object initializer function will have run and initialized the default values for various properties.
IOW, it is possible for qom-list-properties to report on default values for non-abstract types.
I did a quick hack to PoC the theory:
qapi/misc.json +2 -1
@@ -1368,7 +1368,8 @@ # Since: 1.2 ## { 'struct': 'ObjectPropertyInfo', - 'data': { 'name': 'str', 'type': 'str', '*description': 'str' } } + 'data': { 'name': 'str', 'type': 'str', '*description': 'str', + '*default': 'str'} } ## # @qom-list:
qmp.c +5 -0
@@ -594,6 +594,11 @@ ObjectPropertyInfoList *qmp_qom_list_properties(const char *typename, info->has_description = !!prop->description; info->description =g_strdup (prop->description); +if (obj &&g_str_equal (info->type,"string" )) { + info->q_default =g_strdup (object_property_get_str (obj, info->name,NULL )); + info->has_q_default = info->q_default !=NULL ; + } + entry =g_malloc0 (sizeof (*entry)); entry->value = info; entry->next = prop_list;
If we could make this hack less of a hack, then perhaps this is good enough to cope reporting machine types which forbid use of "mem" in favour of "memdev" ? They would need to have a property registered against them of course to identify the "memdev" requirement.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 70 lines · On Mon, 4 Mar 2019 15:28:11 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Daniel P. Berrangé
Quoted 2 lines · Markus Armbruster
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
Quoted 8 lines
Quoted 4 lines
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Fair point. A practical way for management applications to decide which of the two interfaces they can use with which machine type may be required for deprecating one of the interfaces with new machine types.
We currently have "qom-list-properties" which can report on the existance of properties registered against object types. What it can't do though is report on the default values of these properties.
What's interesting though is that
qmp_qom_list_propertieswill actually instantiate objects in order to query properties, if the type isn't an abstract type.IOW, even if you are running "$QEMU -machine none", then if at the qmp-shell you do
(QEMU) qom-list-properties typename=pc-q35-2.6-machine
it will have actually instantiate the pc-q35-2.6-machine machine type. Since it has instantiated the machine, the object initializer function will have run and initialized the default values for various properties.
IOW, it is possible for qom-list-properties to report on default values for non-abstract types.
I did a quick hack to PoC the theory:
qapi/misc.json +2
-1@@ -1368,7 +1368,8 @@ # Since: 1.2 ## { 'struct': 'ObjectPropertyInfo', - 'data': { 'name': 'str', 'type': 'str', '*description': 'str' } } + 'data': { 'name': 'str', 'type': 'str', '*description': 'str', + '*default': 'str'} } ## # @qom-list:qmp.c +5
-0@@ -594,6 +594,11 @@ ObjectPropertyInfoList *qmp_qom_list_properties(const char *typename, info->has_description = !!prop->description; info->description =g_strdup (prop->description); +if (obj &&g_str_equal (info->type,"string" )) { + info->q_default =g_strdup (object_property_get_str (obj, info->name,NULL )); + info->has_q_default = info->q_default !=NULL ; + } + entry =g_malloc0 (sizeof (*entry)); entry->value = info; entry->next = prop_list;
Quoted 4 lines
If we could make this hack less of a hack, then perhaps this is good enough to cope reporting machine types which forbid use of "mem" in favour of "memdev" ? They would need to have a property registered against them of course to identify the "memdev" requirement.
Thanks, I'll look into it and try to come up with patches.
Quoted 2 lines
Regards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Daniel P. BerrangéQuoted 12 lines · Daniel P. Berrangé
Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Daniel P. Berrangé
Quoted 2 lines · Markus Armbruster
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Quoted 7 lines
Quoted 3 lines
Fair point. A practical way for management applications to decide which of the two interfaces they can use with which machine type may be required for deprecating one of the interfaces with new machine types.
We currently have "qom-list-properties" which can report on the existance of properties registered against object types. What it can't do though is report on the default values of these properties.
Yes.
Quoted 3 lines
What's interesting though is that
qmp_qom_list_propertieswill actually instantiate objects in order to query properties, if the type isn't an abstract type.
If it's an abstract type, qom-list-properties returns the properties created with object_class_property_add() & friends, typically by the class_init method. This is possible without instantiating the type.
If it's a concrete type, qom-list-properties additionally returns the properties created with object_property_add(), typically by the instance_init() method. This requires instantiating the type.
Both kinds of properties can be added or deleted at any time. For instance, setting a property value with object_property_set() or similar could create additional properties.
For historical reasons, we use often use object_property_add() where object_class_property_add() would do. Sad.
Quoted 11 lines
IOW, even if you are running "$QEMU -machine none", then if at the qmp-shell you do
(QEMU) qom-list-properties typename=pc-q35-2.6-machine
it will have actually instantiate the pc-q35-2.6-machine machine type. Since it has instantiated the machine, the object initializer function will have run and initialized the default values for various properties.
IOW, it is possible for qom-list-properties to report on default values for non-abstract types.
instance_init() also initializes the properties' values. qom-list-properties could show these initial values (I hesitate calling them default values).
Setting a property's value can change other properties' values by side effect.
My point is: the properties qom-list-properties shows and the initial values it could show are not necessarily final. QOM is designed to be maximally flexible, and flexibility brings along its bosom-buddy complexity.
If you keep that in mind, qom-list-properties can be put to good use all the same.
A way to report "default values" (really: whatever the values are after object_new()) feels like a fair feature request to me, if backed by an actual use case.
[...]
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 67 lines · On Sun, 10 Mar 2019 11:14:08 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 12 lines · Daniel P. Berrangé
Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Daniel P. Berrangé
Quoted 2 lines · Markus Armbruster
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
How is libvirt going to know what machines it can use with the feature ? We don't have any way to introspect machine type specific logic, since we run all probing with "-machine none", and QEMU can't report anything about machines without instantiating them.
Quoted 7 lines
Quoted 3 lines
Fair point. A practical way for management applications to decide which of the two interfaces they can use with which machine type may be required for deprecating one of the interfaces with new machine types.
We currently have "qom-list-properties" which can report on the existance of properties registered against object types. What it can't do though is report on the default values of these properties.
Yes.
Quoted 3 lines
What's interesting though is that
qmp_qom_list_propertieswill actually instantiate objects in order to query properties, if the type isn't an abstract type.If it's an abstract type, qom-list-properties returns the properties created with
object_class_property_add()& friends, typically by theclass_initmethod. This is possible without instantiating the type.If it's a concrete type, qom-list-properties additionally returns the properties created with
object_property_add(), typically by theinstance_init()method. This requires instantiating the type.Both kinds of properties can be added or deleted at any time. For instance, setting a property value with
object_property_set()or similar could create additional properties.For historical reasons, we use often use
object_property_add()whereobject_class_property_add()would do. Sad.Quoted 11 lines
IOW, even if you are running "$QEMU -machine none", then if at the qmp-shell you do
(QEMU) qom-list-properties typename=pc-q35-2.6-machine
it will have actually instantiate the pc-q35-2.6-machine machine type. Since it has instantiated the machine, the object initializer function will have run and initialized the default values for various properties.
IOW, it is possible for qom-list-properties to report on default values for non-abstract types.
instance_init()also initializes the properties' values. qom-list-properties could show these initial values (I hesitate calling them default values).Setting a property's value can change other properties' values by side effect.
My point is: the properties qom-list-properties shows and the initial values it could show are not necessarily final. QOM is designed to be maximally flexible, and flexibility brings along its bosom-buddy complexity.
Quoted 6 lines
If you keep that in mind, qom-list-properties can be put to good use all the same.
A way to report "default values" (really: whatever the values are after
object_new()) feels like a fair feature request to me, if backed by an actual use case.
Looks like trying to migrate from 'mem' to 'memdev', just creates another train-wreck (where libvirt would have to hunt for 'right' backend configuration to make migration work and even that would be best effort attempt). If that would work reliably, I'd go for it since it allows to drop 'mem' codepath altogether but it doesn't look possible. So I'll look into adding machine level introspection and deprecating 'mem' option for new machine types.
Quoted 2 lines
[...]
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Daniel P. Berrangé[Thanks Igor for bringing this onto my radar. I don't follow qemu-devel that close]
Quoted 32 lines · Daniel P. Berrangé
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrangé
Quoted 27 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
I'm with Daniel on this. The reason why libvirt still defaults to '-numa node,mem=' is exactly because of backward compatibility. Since a machine can't be migrated from '-numa node,mem=' to '-numa node,memdev= + -object memory-backend-*' libvirt hast to play it safe and chose a combination that is acessible the widest.
If you remove this, how would you expect older machines to migrate to newer cmd line?
I'm all for deprecating old stuff. In fact, I've suggested that in libvirt(!) here and there, but I'm afraid we can't just remove functionatlity unless we give users a way to migrate to the one we prefer now.
And if libvirt doesn't follow qemu's warnings then it definitely should. It's a libvirt bug if it doesn't follow the best practicies (well, if can).
Michal
Re: [Qemu-arm] [libvirt] [Qemu-devel] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 49 lines · On Mon, 4 Mar 2019 15:24:28 +0100 Michal Privoznik <[email protected]> wrote:
[Thanks Igor for bringing this onto my radar. I don't follow qemu-devel that close]
Quoted 32 lines · Daniel P. Berrangé
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrangé
Quoted 27 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
I'm with Daniel on this. The reason why libvirt still defaults to '-numa node,mem=' is exactly because of backward compatibility. Since a machine can't be migrated from '-numa node,mem=' to '-numa node,memdev= + -object memory-backend-*' libvirt hast to play it safe and chose a combination that is acessible the widest.
Quoted 7 lines
If you remove this, how would you expect older machines to migrate to newer cmd line?
I'm all for deprecating old stuff. In fact, I've suggested that in libvirt(!) here and there, but I'm afraid we can't just remove functionatlity unless we give users a way to migrate to the one we prefer now.
Agreed, it's clear now that I can't remove just 'mem' for OLD machine types (even if this safe variant is broken and doesn't actually do what it should). Libvirt should use 'memdev' for new VMs for them to actually benefit from NUMA configuration.
Currently we are talking about disabling 'mem' for new machine types only (pity that I have to keep around legacy code but at least we would be able to move on to normal device modeling for initial memory on new machines).
Quoted 4 lines
And if libvirt doesn't follow qemu's warnings then it definitely should. It's a libvirt bug if it doesn't follow the best practicies (well, if can).
Michal
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Markus ArmbrusterQuoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
*) https://www.redhat.com/archives/libvir-list/2018-November/msg00159.html
Quoted 11 lines
So, how could this formal notice be served constructively?
If we reject outdated NUMA configurations starting with machine type T, we can remove the means to create those configurations along with machine type T-1. Won't happen anytime soon, will happen eventually, because in the long run, all machine types are dead (apologies to Keynes).
If we deprecate outdated NUMA configurations now, we can start rejecting them with new machine types after a suitable grace period.
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
More exact question specific to this series usecase,
how libvirt decides when to use -numa node,memdev or not currently?
Quoted 3 lines
Regards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 96 lines · Igor Mammedov
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
Quoted 2 lines
More exact question specific to this series usecase,
how libvirt decides when to use -numa node,memdev or not currently?
It is pretty hard to follow the code, but IIUC we only use memdev when stting up NVDIMMs, or for guests configured to have the "shared" flag on the memory region.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 104 lines · On Mon, 4 Mar 2019 14:24:32 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 96 lines · Igor Mammedov
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
Any suggestions what direction we should proceed?
(I'm really not keen to develop a new introspection feature but if that the only way to move forward ...)
Quoted 6 lines
Quoted 2 lines
More exact question specific to this series usecase,
how libvirt decides when to use -numa node,memdev or not currently?It is pretty hard to follow the code, but IIUC we only use memdev when stting up NVDIMMs, or for guests configured to have the "shared" flag on the memory region.
Then I'd guess that most VMs end up with default '-numa node,mem' which by design can produce only fake NUMA without ability to manage guest RAM on host side. So such VMs aren't getting performance benefits or worse run with performance regression (due to wrong sched/mm decisions as guest kernel assumes NUMA topology is valid one).
Quoted 2 lines
Regards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 5 lines · Igor Mammedov
Then I'd guess that most VMs end up with default '-numa node,mem' which by design can produce only fake NUMA without ability to manage guest RAM on host side. So such VMs aren't getting performance benefits or worse run with performance regression (due to wrong sched/mm decisions as guest kernel assumes NUMA topology is valid one).
Specifying NUMA distances in libvirt XML makes it generate the modern cmd line.
Michal
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 10 lines · Michal Privoznik
Quoted 5 lines · Igor Mammedov
Then I'd guess that most VMs end up with default '-numa node,mem' which by design can produce only fake NUMA without ability to manage guest RAM on host side. So such VMs aren't getting performance benefits or worse run with performance regression (due to wrong sched/mm decisions as guest kernel assumes NUMA topology is valid one).
Specifying NUMA distances in libvirt XML makes it generate the modern cmd line.
AFAIK, specifying any guest NUMA -> Host NUMA affinity makes it use the modern cmd line. eg I just modified a plain 8 CPU / 2 GB RAM guest with this:
<numatune>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
<cpu mode='host-model'>
<numa>
<cell id='0' cpus='0-3' memory='1024000' unit='KiB'/>
<cell id='1' cpus='4-7' memory='1024000' unit='KiB'/>
</numa></cpu>
and I can see libvirt decided to use memdev
-object memory-backend-ram,id=ram-node0,size=1048576000,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-3,memdev=ram-node0
-object memory-backend-ram,id=ram-node1,size=1048576000,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=4-7,memdev=ram-node1
So unless I'm missing something, we aren't suffering from the problem described by Igor above even today.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Daniel P. BerrangéQuoted 104 lines · Daniel P. Berrangé
Quoted 96 lines · Igor Mammedov
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit f309db1f4d51009bad0d32e12efc75530b66836b
Author: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100
qemu: Create memory-backend-{ram,file} iff needed
Or this 7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.
Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd line
But I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Michal
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 137 lines · * Michal Privoznik ([email protected]) wrote:
Quoted 101 lines · Daniel P. Berrangé
Quoted 93 lines · Igor Mammedov
Quoted 81 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 71 lines · Igor Mammedov
Quoted 57 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 35 lines · Daniel P. Berrangé
Quoted 21 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
Quoted 7 lines
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd lineBut I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
What's interesting here is that this problem isn't really machine-type related; so providing introspection on the machine type doesn't immediately help. What we're actually trying to do here is (mis)use a machine type as a proxy for knowing that both sides are new enough to handle the new command line.
That's an OK thing to do, and if we did have introspection we could add a fudge flag to say it's allowed now.
Dave
Quoted 1 line
Michal
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Michal PrivoznikQuoted 137 lines · Michal Privoznik
Quoted 101 lines · Daniel P. Berrangé
Quoted 93 lines · Igor Mammedov
Quoted 81 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 71 lines · Igor Mammedov
Quoted 57 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 35 lines · Daniel P. Berrangé
Quoted 21 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
Quoted 7 lines
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd lineBut I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Yeah supporting backwards migration is a non-optional requirement from at least one of the mgmt apps using libvirt, so breaking the new to old case is something we always aim to avoid.
These incompabilities are reminding me why we haven't tied these kind of changes to machine type versions in the past. New machine type != new libvirt, so we can't tie usage of a feature in livirt to a new machine type.
I'm wondering exactly which cases libvirt will still use the "mem" option in as opposed to "memdev". If none of the cases using "mem" actually suffer from the ill-effects of "mem", then there's not a compelling reason to stop using it. It can be discouraged in QEMU documentation but otherwise left alone.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 143 lines · On Mon, 4 Mar 2019 16:35:16 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 142 lines · Michal Privoznik
Quoted 101 lines · Daniel P. Berrangé
Quoted 93 lines · Igor Mammedov
Quoted 81 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 71 lines · Igor Mammedov
Quoted 57 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 35 lines · Daniel P. Berrangé
Quoted 21 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd line
Quoted 6 lines
Quoted 2 lines
But I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Yeah supporting backwards migration is a non-optional requirement from at least one of the mgmt apps using libvirt, so breaking the new to old case is something we always aim to avoid.
Aiming for support of
"new QEMU + new machine type" => "old QEMU + non-existing machine type" seems a bit difficult.
Note old machine types will continue to work with old CLI.
Quoted 13 lines
These incompabilities are reminding me why we haven't tied these kind of changes to machine type versions in the past. New machine type != new libvirt, so we can't tie usage of a feature in livirt to a new machine type.
I'm wondering exactly which cases libvirt will still use the "mem" option in as opposed to "memdev". If none of the cases using "mem" actually suffer from the ill-effects of "mem", then there's not a compelling reason to stop using it. It can be discouraged in QEMU documentation but otherwise left alone.
Regards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 40 lines · Igor Mammedov
Quoted 37 lines · On Mon, 4 Mar 2019 16:35:16 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 35 lines · Michal Privoznik
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd lineBut I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Quoted 6 lines
Quoted 3 lines
Yeah supporting backwards migration is a non-optional requirement from at least one of the mgmt apps using libvirt, so breaking the new to old case is something we always aim to avoid.
Aiming for support of
"new QEMU + new machine type" => "old QEMU + non-existing machine type" seems a bit difficult.
That's not the scenario that's the problem. The problem is
new QEMU + new machine type + new libvirt -> new QEMU + new machine type + old libvirt
Previously released versions of libvirt will happily use any new machine type that QEMU introduces. So we can't make new libvirt use a different options, only for new machine types, as old libvirt supports those machine types too.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 48 lines · Daniel P. Berrangé
Quoted 40 lines · Igor Mammedov
Quoted 37 lines · On Mon, 4 Mar 2019 16:35:16 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 35 lines · Michal Privoznik
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd lineBut I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Quoted 6 lines
Quoted 3 lines
Yeah supporting backwards migration is a non-optional requirement from at least one of the mgmt apps using libvirt, so breaking the new to old case is something we always aim to avoid.
Aiming for support of
"new QEMU + new machine type" => "old QEMU + non-existing machine type" seems a bit difficult.
Quoted 8 lines
That's not the scenario that's the problem. The problem is
new QEMU + new machine type + new libvirt -> new QEMU + new machine type + old libvirt
Previously released versions of libvirt will happily use any new machine type that QEMU introduces. So we can't make new libvirt use a different options, only for new machine types, as old libvirt supports those machine types too.
Avoiding tight coupling between QEMU und libvirt versions makes sense, because having to upgrade stuff in lock-step is such a pain.
Does not imply we must support arbitrary combinations of QEMU and libvirt versions.
Unless upstream libvirt's test matrix covers all versions of libvirt against all released versions of QEMU, "previously released versions of libvirt will continue to work with new QEMU" is largely an empty promise anyway. The real promise is more like "we won't break it intentionally; good luck".
Mind, I'm not criticizing that real promise. I'm criticizing cutting yourself off from large areas of the solution space so you can continue to pretend to yourself you actually deliver on the empty promise.
Now, if you limited what you promise to something more realistic, ideally to something you actually test, we could talk about deprecation schedules constructively.
For instance, if you promised
QEMU as of time T + its latest machine type + libvirt as of time T
-> QEMU as of time T + its latest machine type + libvirt as of time T - d
will work for a certain value of d, then once all released versions of libvirt since T - d support a new way of doing things, flipping to that new way becomes a whole lot easier.
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 59 lines · On Sun, 10 Mar 2019 11:16:33 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 48 lines · Daniel P. Berrangé
Quoted 40 lines · Igor Mammedov
Quoted 37 lines · On Mon, 4 Mar 2019 16:35:16 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 35 lines · Michal Privoznik
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd line
b) migrating from one host to another won't modernize the cmd lineBut I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
Quoted 6 lines
Quoted 3 lines
Yeah supporting backwards migration is a non-optional requirement from at least one of the mgmt apps using libvirt, so breaking the new to old case is something we always aim to avoid.
Aiming for support of
"new QEMU + new machine type" => "old QEMU + non-existing machine type" seems a bit difficult.Quoted 8 lines
That's not the scenario that's the problem. The problem is
new QEMU + new machine type + new libvirt -> new QEMU + new machine type + old libvirt
Previously released versions of libvirt will happily use any new machine type that QEMU introduces. So we can't make new libvirt use a different options, only for new machine types, as old libvirt supports those machine types too.
Quoted 5 lines
Avoiding tight coupling between QEMU und libvirt versions makes sense, because having to upgrade stuff in lock-step is such a pain.
Does not imply we must support arbitrary combinations of QEMU and libvirt versions.
Isn't it typically a job of downstream to ship a bundle that works together and it is rather limited set.
e.g
System 1 libvirt 0 QEMU 0 (machine 0.1 (latest)) could be migrated to 2 ways to System 2 libvirt 1 QEMU 1 (machine 0.1 (still the same old machine))
while installing QEMU 1 on System 1 might work (if it doesn't break due to dependencies) and even be able to start machine 1.0, wouldn't it really fall in unsupported category?
Quoted 14 lines
Unless upstream libvirt's test matrix covers all versions of libvirt against all released versions of QEMU, "previously released versions of libvirt will continue to work with new QEMU" is largely an empty promise anyway. The real promise is more like "we won't break it intentionally; good luck".
Mind, I'm not criticizing that real promise. I'm criticizing cutting yourself off from large areas of the solution space so you can continue to pretend to yourself you actually deliver on the empty promise.
Now, if you limited what you promise to something more realistic, ideally to something you actually test, we could talk about deprecation schedules constructively.
Quoted 8 lines
For instance, if you promised
QEMU as of time T + its latest machine type + libvirt as of time T
-> QEMU as of time T + its latest machine type + libvirt as of time T - dwill work for a certain value of d, then once all released versions of libvirt since T - d support a new way of doing things, flipping to that new way becomes a whole lot easier.
On top of that libvirt should be willing to adapt to the new way for above combination.
I'm experimenting with making 'mem' migration compatible with 'memdev' (I have a working hack currently) to get a feeling of a toll it would take on QEMU, so far outlook isn't good. It just adds more compat code and new CLI options to deal with it (it just increases amount of maintenance nightmare for indefinite period of time). And everyone would hate adding such new CLI options which as it turns out we aren't able to deprecate and remove in practice.
We really need to figure out how to introduce breaking change on management (CLI) side* in QEMU and make it digestible for libvirt and others. (* at least for new machine types).
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Michal PrivoznikQuoted 124 lines · On Mon, 4 Mar 2019 17:20:13 +0100 Michal Privoznik <[email protected]> wrote:
Quoted 104 lines · Daniel P. Berrangé
Quoted 96 lines · Igor Mammedov
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Quoted 6 lines
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
For example I don't see anything introspection related when we have been removing deprecated options recently.
Quoted 6 lines
Right, with other stuff we deprecate we've had a simpler time, as it either didn't affect migration at all, or the new replacement stuff was fully compatible with the migration data stream. IOW, libvirt could unconditionally use the new feature as soon as it saw that it exists in QEMU. We didn't have any machine type dependancy to deal with before now.
We couldn't have done that. How we would migrate from older qemu?
Anyway, now that I look into this (esp. git log) I came accross:
commit
f309db1f4d51009bad0d32e12efc75530b66836bAuthor: Michal Privoznik <[email protected]> AuthorDate: Thu Dec 18 12:36:48 2014 +0100 Commit: Michal Privoznik <[email protected]> CommitDate: Fri Dec 19 07:44:44 2014 +0100qemu: Create memory-backend-{ram,file} iff needed
Quoted 8 lines
Or this
7832fac84741d65e851dbdbfaf474785cbfdcf3c. We did try to generated newer cmd line but then for various reasong (e.g. avoiding triggering a qemu bug) we turned it off and make libvirt default to older (now deprecated) cmd line.Frankly, I don't know how to proceed. Unless qemu is fixed to allow migration from deprecated to new cmd line (unlikely, if not impossible, right?) then I guess the only approach we can have is that:
In non numa case (or 1 node case) there is migration compatible solution, I'm working on it.
The only problematic case is where -mem-path + -mem-prealloc + pc-dimm are used with muti-node variant + '-numa node,mem=xx' option or plain '-numa node' with implicit RAM distribution (the same as 'mem' just done by QEMU), here I can't translate options due to different memory layout, so these VMs would still continue using old 'mem' with old machine types.
Quoted 2 lines
1) whenever so called cold booting a new machine (fresh, brand new start of a new domain) libvirt would default to modern cmd line,
that's what I'd prefer and complained that this route wasn't followed. (it would even work with currently broken QEMU, since modern CLI variant works)
It would wean down 'mem' userbase that configures NUMA incorrectly (i.e. non pinned mem&cpus). Users still can do that with 'memdev' by leaving out bind policy but
when they ask for pinning with bind policy they will really get it since under-hood "-mem-path + -mem-prealloc" will be the same 'memdev,prealloc=on', without surprise due to quite non obvious call ordering.
Quoted 3 lines
2) on migration, libvirt would record in the migration stream (or status XML or wherever) that modern cmd line was generated and thus it'll make the destination generate modern cmd line too.
since legacy option is not removed, it's not necessary for converting legacy SOURCE to new CLI (new QEMU would still accept legacy CLI for old machine types), i.e.e business as usual.
Quoted 5 lines
This solution still suffers a couple of problems:
a) migration to older libvirt will fail as older libvirt won't recognize the flag set in 2) and therefore would default to deprecated cmd lineb) migrating from one host to another won't modernize the cmd line
I didn't knew that there was 'modernizing' feature in libvirt but modernizing CLI only necessary when we remove option completely.
Quoted 2 lines
But I guess we have to draw a line somewhere (if we are not willing to write those migration patches).
I wasn't able to make migration from 1 to N MemoryRegions and back work. So the only choice left is to start deprecation of numa 'mem' option with new machine types, so that at least there memory pinning would not be broken and it wouldn't be possible to mis-configure it.
Quoted 1 line
Michal
Re: [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Igor MammedovQuoted 92 lines · Igor Mammedov
Quoted 84 lines · On Mon, 4 Mar 2019 12:39:08 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 74 lines · Igor Mammedov
Quoted 59 lines · On Mon, 04 Mar 2019 08:13:53 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 36 lines · Daniel P. Berrangé
Quoted 22 lines · Igor Mammedov
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Quoted 5 lines
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
If we can't migration from one to the other, then we can not deprecate the existing 'mem' syntax. Even if libvirt were to provide a config option to let apps opt-in to the new syntax, we need to be able to support live migration of existing running VMs indefinitely. Effectively this means we need the to keep 'mem' support forever, or at least such a long time that it effectively means forever.
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
We have this habit of postulating absolutes like "can not deprecate" instead of engaging with the tradeoffs. We need to kick it.
So let's have an actual look at the tradeoffs.
We don't actually "support live migration of existing running VMs indefinitely".
We support live migration to any newer version of QEMU that still supports the machine type.
We support live migration to any older version of QEMU that already supports the machine type and all the devices the machine uses.
Aside: "support" is really an honest best effort here. If you rely on it, use a downstream that puts in the (substantial!) QA work real support takes.
Quoted 6 lines
Feature deprecation is not a contract to drop the feature after two releases, or even five. It's a formal notice that users of the feature should transition to its replacement in an orderly manner.
If I understand Igor correctly, all users should transition away from outdated NUMA configurations at least for new VMs in an orderly manner.
Yes, we can postpone removing options until there are machines type versions that were capable to use it (unfortunate but probably unavoidable unless there is a migration trick to make transition transparent) but that should not stop us from disabling broken options on new machine types at least.
Quoted 8 lines
This series can serve as formal notice with follow up disabling of deprecated options for new machine types. (As Thomas noted, just warnings do not work and users continue to use broken features regardless whether they are don't know about issues or aware of it [*])
Hence suggested deprecation approach and enforced rejection of legacy numa options for new machine types in 2 releases so users would stop using them eventually.
Quoted 5 lines
When we deprecate something, we need to have a way for apps to use the new alternative approach *at the same time*. So even if we only want to deprecate for new machine types, we still have to first solve the problem of how mgmt apps will introspect QEMU to learn which machine types expect the new options.
Quoted 7 lines
I'm not aware any mechanism to introspect machine type options (existing or something being developed). Are/were there any ideas about it that were discussed in the past?
Aside from developing a new mechanism what are alternative approaches? I mean when we delete deprecated CLI option, how it's solved on libvirt side currently?
Libvirt queries qemu capabilites via QMP. And in all places it can it preferes the latest recommended cmd line options (well, those known to a libvirt developer at the time he/she is writing the code). So as long as you remove only old stuff and libvirt refreshes itself when following best practicies we're okay.
Quoted 6 lines
For example I don't see anything introspection related when we have been removing deprecated options recently.
More exact question specific to this series usecase,
how libvirt decides when to use -numa node,memdev or not currently?
It has a mechanism to tell if '-numa node,memdev=' is needed (i.e. there is no other way to satisfy user requested configuration) and only then it uses ,memdev. For all other cases it defaults to -numa node,mem= simply to keep backwards compatibility (as I'm explaining in another e-mail I've just sent to this list).
Anyway, in the libvirt code you want to be looking at:
src/qemu/qemu_command.c: qemuBuildNumaArgStr src/qemu/qemu_command.c: qemuBuildMemoryCellBackendStr
Michal
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Daniel P. BerrangéOn 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]
Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Thomas
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 11 lines · On Mon, 4 Mar 2019 09:11:19 +0100 Thomas Huth <[email protected]> wrote:
On 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
Quoted 6 lines
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Is it acceptable to silently disable CLI options (even if they are broken like in this case) for new machine types?
I was under impression that it should go through deprecation first.
Quoted 3 lines
Thomas
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 20 lines · Igor Mammedov
Quoted 11 lines · On Mon, 4 Mar 2019 09:11:19 +0100 Thomas Huth <[email protected]> wrote:
On 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
Quoted 6 lines
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Quoted 3 lines
Is it acceptable to silently disable CLI options (even if they are broken like in this case) for new machine types?
I was under impression that it should go through deprecation first.
Yes, it must go through deprecation. I was saying we can't disable the CLI options at all, until there is a way for libvirt to correctly use the new options.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 21 lines · On Mon, 4 Mar 2019 13:59:09 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 20 lines · Igor Mammedov
Quoted 17 lines · On Mon, 4 Mar 2019 09:11:19 +0100 Thomas Huth <[email protected]> wrote:
On 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Quoted 7 lines
Quoted 3 lines
Is it acceptable to silently disable CLI options (even if they are broken like in this case) for new machine types?
I was under impression that it should go through deprecation first.Yes, it must go through deprecation. I was saying we can't disable the CLI options at all, until there is a way for libvirt to correctly use the new options.
I'm not adding new options (nor plan to for numa case (yet)), -numa node,memdev is around several years by now and should be used as default for creating new configs.
In light of keeping 'mem' option around for old machines, Deprecation should have served for notifying users that legacy options will be disabled later on (for new machines at least if no way found for migration compatible transition for older ones).
What I'm mainly aiming here is to prevent using broken legacy options for new VMs (like in RHBZ1624223 case) and deprecation is the only way we have now to notify users about CLI breaking changes.
In the -mem-path/prealloc case, there will be a new machine option 'memdev' to replace them but that's migration compatible, so libvirt could use new CLI syntax to replace even old configs.
(If there is a mechanism for introducing/introspecting a new options per-machine or whole QEMU, I'd happy to add a new option there on top of just deprecation notice that we have now).
Quoted 2 lines
Regards,
Daniel
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 36 lines · Igor Mammedov
Quoted 21 lines · On Mon, 4 Mar 2019 13:59:09 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 20 lines · Igor Mammedov
Quoted 17 lines · On Mon, 4 Mar 2019 09:11:19 +0100 Thomas Huth <[email protected]> wrote:
On 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Quoted 7 lines
Quoted 3 lines
Is it acceptable to silently disable CLI options (even if they are broken like in this case) for new machine types?
I was under impression that it should go through deprecation first.Yes, it must go through deprecation. I was saying we can't disable the CLI options at all, until there is a way for libvirt to correctly use the new options.
I'm not adding new options (nor plan to for numa case (yet)), -numa node,memdev is around several years by now and should be used as default for creating new configs.
Quoted 8 lines
In light of keeping 'mem' option around for old machines, Deprecation should have served for notifying users that legacy options will be disabled later on (for new machines at least if no way found for migration compatible transition for older ones).
What I'm mainly aiming here is to prevent using broken legacy options for new VMs (like in RHBZ1624223 case) and deprecation is the only way we have now to notify users about CLI breaking changes.
The idea of doing advance warnings via deprecations is that applications have time to adapt to the new mechanism several releases before the old mechanism is removed/disabled. Since the new mechanism isn't fully usable yet, applications can't adapt to use it. So we can't start the deprecation process yet, as it would be telling apps to do a switch that isn't possible for many to actually do.
In the meantime, qemu-options.hx could be updated. It documents both "mem" and "memdev" currently but doesn't tell people that "memdev" is the preferred syntax for future usage / warn against using "mem".
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-arm] [Qemu-ppc] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 46 lines · On Mon, 4 Mar 2019 15:02:18 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 36 lines · Igor Mammedov
Quoted 21 lines · On Mon, 4 Mar 2019 13:59:09 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 20 lines · Igor Mammedov
Quoted 17 lines · On Mon, 4 Mar 2019 09:11:19 +0100 Thomas Huth <[email protected]> wrote:
On 01/03/2019 18.48, Daniel P. Berrangé wrote:
[...]Quoted 2 lines
So I think this patch has to be dropped & replaced with one that simply documents that memdev syntax is preferred.
That's definitely not enough. I've had a couple of cases already where we documented that certain options should not be used anymore, and people simply ignored it (aka. if it ain't broken, don't do any change). Then they just started to complain when I really tried to remove the option after the deprecation period.
So Igor, if you can not officially deprecate these things here yet, you should at least make sure that they can not be used with new machine types anymore. Then, after a couple of years, when we feel sure that there are only some few or no people left who still use it with the old machine types, we can start to discuss the deprecation process again, I think.
Quoted 7 lines
Quoted 3 lines
Is it acceptable to silently disable CLI options (even if they are broken like in this case) for new machine types?
I was under impression that it should go through deprecation first.Yes, it must go through deprecation. I was saying we can't disable the CLI options at all, until there is a way for libvirt to correctly use the new options.
I'm not adding new options (nor plan to for numa case (yet)), -numa node,memdev is around several years by now and should be used as default for creating new configs.
Quoted 8 lines
In light of keeping 'mem' option around for old machines, Deprecation should have served for notifying users that legacy options will be disabled later on (for new machines at least if no way found for migration compatible transition for older ones).
What I'm mainly aiming here is to prevent using broken legacy options for new VMs (like in RHBZ1624223 case) and deprecation is the only way we have now to notify users about CLI breaking changes.
Quoted 6 lines
The idea of doing advance warnings via deprecations is that applications have time to adapt to the new mechanism several releases before the old mechanism is removed/disabled. Since the new mechanism isn't fully usable yet, applications can't adapt to use it. So we can't start the deprecation process yet, as it would be telling apps to do a switch that isn't possible for many to actually do.
At least a hope attempt to deprecate 'mem', served its purpose somewhat. Now it's clear that libvirt defaults to wrong legacy mode and should use 'memdev' instead. I hope that it will be addressed on libvirt side regardless of the fate of 'mem' deprecation process. Basically new VM should default to 'memdev' if it's available.
Quoted 3 lines
In the meantime, qemu-options.hx could be updated. It documents both "mem" and "memdev" currently but doesn't tell people that "memdev" is the preferred syntax for future usage / warn against using "mem".
Will do so.
Quoted 3 lines
Regards,
Daniel
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
In reply to Igor MammedovQuoted 40 lines · * Igor Mammedov ([email protected]) wrote:
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Mixed memory issue[1] first came from libvirt side RHBZ1624223, back then it was resolved on libvirt side in favor of migration compatibility vs correctness (i.e. bind policy doesn't work as expected). What worse that it was made default and affects all new machines, as I understood it.
In case of -mem-path + -mem-prealloc (with 1 numa node or numa less) it's possible on QEMU side to make conversion to memdev in migration compatible way (that's what stopped Michal from memdev approach). But it's hard to do so in multi-nodes case as amount of MemoryRegions is different.
Quoted 6 lines
Point is to consider 'mem' as mis-configuration error, as the user in the first place using broken numa configuration
(i.e. fake numa configuration doesn't actually improve performance).CCed David, maybe he could offer a way to do 1:n migration and other way around.
I can't see a trivial way.
About the easiest I can think of is if you had a way to create a memdev that was an alias to pc.ram (of a particular size and offset).
Dave
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Signed-off-by Igor Mammedov <[email protected]>--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)numa.c +2
-0@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;qemu-deprecated.texi +14
-0@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)Signature · 1 line
2.7.4
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
--
libvir-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/libvir-listRegards,
Daniel
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 45 lines · On Fri, 1 Mar 2019 18:01:52 +0000 "Dr. David Alan Gilbert"…
Quoted 44 lines · * Igor Mammedov ([email protected]) wrote:
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Mixed memory issue[1] first came from libvirt side RHBZ1624223, back then it was resolved on libvirt side in favor of migration compatibility vs correctness (i.e. bind policy doesn't work as expected). What worse that it was made default and affects all new machines, as I understood it.
In case of -mem-path + -mem-prealloc (with 1 numa node or numa less) it's possible on QEMU side to make conversion to memdev in migration compatible way (that's what stopped Michal from memdev approach). But it's hard to do so in multi-nodes case as amount of MemoryRegions is different.
Point is to consider 'mem' as mis-configuration error, as the user in the first place using broken numa configuration
(i.e. fake numa configuration doesn't actually improve performance).
Quoted 6 lines
Quoted 2 lines
CCed David, maybe he could offer a way to do 1:n migration and other way around.
I can't see a trivial way.
About the easiest I can think of is if you had a way to create a memdev that was an alias to pc.ram (of a particular size and offset).
If I get you right that's what I was planning to do for numa-less machines that use -mem-path/prealloc options, where it's possible to replace an initial RAM MemoryRegion with a correspondingly named memdev and its backing MemoryRegion.
But I don't see how it could work in case of legacy NUMA 'mem' options where initial RAM is 1 MemoryRegion (it's a fake numa after all) and how to translate that into several MemoryRegions (one per node/memdev).
Quoted 60 lines
Dave
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Signed-off-by Igor Mammedov <[email protected]>--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)numa.c +2
-0@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;qemu-deprecated.texi +14
-0@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)Signature · 1 line
2.7.4Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
--
libvir-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/libvir-listRegards,
Daniel--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-arm] [Qemu-devel] [libvirt] [PATCH 1/2] numa: deprecate 'mem' parameter of '-numa node' option
Quoted 54 lines · On Mon, 4 Mar 2019 14:52:30 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 45 lines · On Fri, 1 Mar 2019 18:01:52 +0000 "Dr. David Alan Gilbert"…
Quoted 44 lines · * Igor Mammedov ([email protected]) wrote:
Quoted 11 lines · On Fri, 1 Mar 2019 15:49:47 +0000 Daniel P. Berrangé <[email protected]>…
Quoted 10 lines · Igor Mammedov
The parameter allows to configure fake NUMA topology where guest VM simulates NUMA topology but not actually getting a performance benefits from it. The same or better results could be achieved using 'memdev' parameter. In light of that any VM that uses NUMA to get its benefits should use 'memdev' and to allow transition initial RAM to device based model, deprecate 'mem' parameter as its ad-hoc partitioning of initial RAM MemoryRegion can't be translated to memdev based backend transparently to users and in compatible manner (migration wise).
Quoted 8 lines
Quoted 3 lines
That will also allow to clean up a bit our numa code, leaving only 'memdev' impl. in place and several boards that use
node_memto generate FDT/ACPI description from it.Can you confirm that the 'mem' and 'memdev' parameters to -numa are 100% live migration compatible in both directions ? Libvirt would need this to be the case in order to use the 'memdev' syntax instead.
Unfortunately they are not migration compatible in any direction, if it where possible to translate them to each other I'd alias 'mem' to 'memdev' without deprecation. The former sends over only one MemoryRegion to target, while the later sends over several (one per memdev).
Mixed memory issue[1] first came from libvirt side RHBZ1624223, back then it was resolved on libvirt side in favor of migration compatibility vs correctness (i.e. bind policy doesn't work as expected). What worse that it was made default and affects all new machines, as I understood it.
In case of -mem-path + -mem-prealloc (with 1 numa node or numa less) it's possible on QEMU side to make conversion to memdev in migration compatible way (that's what stopped Michal from memdev approach). But it's hard to do so in multi-nodes case as amount of MemoryRegions is different.
Point is to consider 'mem' as mis-configuration error, as the user in the first place using broken numa configuration
(i.e. fake numa configuration doesn't actually improve performance).Quoted 6 lines
Quoted 2 lines
CCed David, maybe he could offer a way to do 1:n migration and other way around.
I can't see a trivial way.
About the easiest I can think of is if you had a way to create a memdev that was an alias to pc.ram (of a particular size and offset).
Quoted 8 lines
If I get you right that's what I was planning to do for numa-less machines that use -mem-path/prealloc options, where it's possible to replace an initial RAM MemoryRegion with a correspondingly named memdev and its backing MemoryRegion.
But I don't see how it could work in case of legacy NUMA 'mem' options where initial RAM is 1 MemoryRegion (it's a fake numa after all) and how to translate that into several MemoryRegions (one per node/memdev).
Limiting it to x86 for demo purposes.
What would work (if*) is to create special MemoryRegion container, i.e.
1. make memory_region_allocate_system_memory():memory_region_init() that special which already has id pc.ram and size that matches the single RAMBlock with the same id in incoming migration stream from OLD qemu ( started with -numa node,mem=x ... options)
2. register "1" with vmstate_register_ram_global()/or other API which undercover will make migration code, split the single incoming RAM block into several smaller consecutive RAMBlocks represented by memdev backends that are mapped as subregions within container 'pc.ram'
3. in case of backward migration container MemoryRegion 'pc.ram' will serve other way around stitching back memdev subregions into the single 'pc.ram' migration stream.
(if*) - but above describes an ideal use-case where -numa node,mem are properly sized. In practice though QEMU doesn't have any checks on numa's 'mem' value option. So users were able to 'split' RAM in arbitrary chunks which memdev based backends might not be able to recreate due to used backing storage limitations (alignment/page size). To make it worse we don't really know what source (old QEMU) uses for backend for real as it might fallback to anonymous RAM if mem-path fails (there is no fallback in memdev case as there user gets what he/she asked for or hard error).
There might be other issues on migration side of things as well, but I just don't know about it enough to see them.
Quoted 62 lines
Quoted 60 lines
Dave
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Signed-off-by Igor Mammedov <[email protected]>--- numa.c | 2 ++ qemu-deprecated.texi | 14 ++++++++++++++ 2 files changed, 16 insertions(+)numa.c +2
-0@@ -121,6 +121,8 @@ static void parse_numa_node(MachineState *ms, NumaNodeOptions *node,if (node->has_mem) { numa_info[nodenr].node_mem = node->mem; +warn_report ("Parameter -numa node,mem is deprecated," +" use -numa node,memdev instead" ); }if (node->has_memdev) {Object *o;qemu-deprecated.texi +14
-0@@ -60,6 +60,20 @@ Support for invalid topologies will be removed, the user must ensure topologies described with -smp include all possible cpus, i.e. @math{@var{sockets} * @var{cores} * @var{threads} = @var{maxcpus}}. +@subsection -numa node,mem=@var{size} (since 4.0) + +The parameter @option{mem} of @option{-numa node} is used to assign a part of +guest RAM to a NUMA node. But when using it, it's impossible to manage specified +size on the host side (like bind it to a host node, setting bind policy, ...), +so guest end-ups with the fake NUMA configuration with suboptiomal performance. +However since 2014 there is an alternative way to assign RAM to a NUMA node +using parameter @option{memdev}, which does the same as @option{mem} and has +an ability to actualy manage node RAM on the host side. Use parameter +@option{memdev} with @var{memory-backend-ram} backend as an replacement for +parameter @option{mem} to achieve the same fake NUMA effect or a properly +configured @var{memory-backend-file} backend to actually benefit from NUMA +configuration. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)Signature · 1 line
2.7.4Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
--
libvir-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/libvir-listRegards,
Daniel--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 2/2 numa: deprecate implict memory distribution between nodes
In reply to Igor MammedovImplict RAM distribution between nodes has exactly the same issues as:
"numa: deprecate 'mem' parameter of '-numa node' option"
only with QEMU being the user that's 'adding' 'mem' parameter.
Depricate it, to get it out of the way so that we could switch to consistent guest RAM allocation using memory backends and possibly memory devices later on top of that.
--- numa.c | 3 +++ qemu-deprecated.texi | 7 +++++++ 2 files changed, 10 insertions(+)
numa.c +3 -0
@@ -409,6 +409,9 @@ void numa_complete_configuration(MachineState *ms)if (i == nb_numa_nodes) {assert (mc->numa_auto_assign_ram); mc->numa_auto_assign_ram (mc, numa_info, nb_numa_nodes, ram_size); +warn_report ("Default splitting of RAM between nodes is deprecated," +" Use '-numa node,memdev' to explictly define RAM" +" allocation per node" ); } numa_total =0 ;
qemu-deprecated.texi +7 -0
@@ -74,6 +74,13 @@ parameter @option{mem} to achieve the same fake NUMA effect or a properly configured @var{memory-backend-file} backend to actually benefit from NUMA configuration. +@subsection -numa node (without memory specified) (since 4.0) + +Splitting RAM by default between NUMA nodes has the same issues as @option{mem} +parameter described above with a difference that role of the user plays QEMU +using generic splitting rule or a board specific one. Use @option{memdev} with +@var{memory-backend-ram} backend to define mapping explictly instead. + @section QEMU Machine Protocol (QMP) commands @subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)
Signature · 1 line
2.7.4