Skip to contentSkip to conversation

numa: Introduce MachineClass::auto_enable_numa for implicit NUMA node

PATCH Superseded Tao Xu 4 participants 11 messages Aug 5, 2019 – Sep 5, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Tao Xu

author
#0 lore raw

PATCH 1/1 numa: Introduce MachineClass::auto_enable_numa for implicit NUMA node

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

---

This patch has a dependency on
https://patchwork.kernel.org/cover/11063235/

---
 hw/core/numa.c      | 9 +++++++--
 hw/ppc/spapr.c      | 9 +--------
 include/hw/boards.h | 1 +
 3 files changed, 9 insertions(+), 10 deletions(-)
hw/core/numa.c +7 -2
index 75db35ac19..756d243d3f 100644
@@ -580,9 +580,14 @@ void numa_complete_configuration(MachineState *ms)
      *   guest tries to use it with that drivers.
      *
      * Enable NUMA implicitly by adding a new NUMA node automatically.
+     *
+     * Or if MachineClass::auto_enable_numa is true and no NUMA nodes,
+     * assume there is just one node with whole RAM.
      */
-    if (ms->ram_slots > 0 && ms->numa_state->num_nodes == 0 &&
-        mc->auto_enable_numa_with_memhp) {
+    if (ms->numa_state->num_nodes == 0 &&
+        ((ms->ram_slots > 0 &&
+        mc->auto_enable_numa_with_memhp) ||
+        mc->auto_enable_numa)) {
             NumaNodeOptions node = { };
             parse_numa_node(ms, &node, &error_abort);
     }
hw/ppc/spapr.c +1 -8
index f607ca567b..e50343f326 100644
@@ -400,14 +400,6 @@ static int spapr_populate_memory(SpaprMachineState *spapr, void *fdt)
     hwaddr mem_start, node_size;
     int i, nb_nodes = machine->numa_state->num_nodes;
     NodeInfo *nodes = machine->numa_state->nodes;
-    NodeInfo ramnode;
-
-    /* No NUMA nodes, assume there is just one node with whole RAM */
-    if (!nb_nodes) {
-        nb_nodes = 1;
-        ramnode.node_mem = machine->ram_size;
-        nodes = &ramnode;
-    }
 
     for (i = 0, mem_start = 0; i < nb_nodes; ++i) {
         if (!nodes[i].node_mem) {
@@ -4369,6 +4361,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
      */
     mc->numa_mem_align_shift = 28;
     mc->numa_mem_supported = true;
+    mc->auto_enable_numa = true;
 
     smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
     smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
     bool smbus_no_migration_support;
     bool nvdimm_supported;
     bool numa_mem_supported;
+    bool auto_enable_numa;
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);
Signature · 1 line
2.20.1
#1 lore raw
Quoted 58 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

---

This patch has a dependency on
https://patchwork.kernel.org/cover/11063235/

---
 hw/core/numa.c      | 9 +++++++--
 hw/ppc/spapr.c      | 9 +--------
 include/hw/boards.h | 1 +
 3 files changed, 9 insertions(+), 10 deletions(-)
hw/core/numa.c +7 -2
index 75db35ac19..756d243d3f 100644
@@ -580,9 +580,14 @@ void numa_complete_configuration(MachineState *ms)
      *   guest tries to use it with that drivers.
      *
      * Enable NUMA implicitly by adding a new NUMA node automatically.
+     *
+     * Or if MachineClass::auto_enable_numa is true and no NUMA nodes,
+     * assume there is just one node with whole RAM.
      */
-    if (ms->ram_slots > 0 && ms->numa_state->num_nodes == 0 &&
-        mc->auto_enable_numa_with_memhp) {
+    if (ms->numa_state->num_nodes == 0 &&
+        ((ms->ram_slots > 0 &&
+        mc->auto_enable_numa_with_memhp) ||
+        mc->auto_enable_numa)) {
             NumaNodeOptions node = { };
             parse_numa_node(ms, &node, &error_abort);
     }
hw/ppc/spapr.c +0 -8
index f607ca567b..e50343f326 100644
@@ -400,14 +400,6 @@ static int spapr_populate_memory(SpaprMachineState *spapr, void *fdt)
     hwaddr mem_start, node_size;
     int i, nb_nodes = machine->numa_state->num_nodes;
     NodeInfo *nodes = machine->numa_state->nodes;
-    NodeInfo ramnode;
-
-    /* No NUMA nodes, assume there is just one node with whole RAM */
-    if (!nb_nodes) {
-        nb_nodes = 1;
-        ramnode.node_mem = machine->ram_size;
-        nodes = &ramnode;
-    }
 
     for (i = 0, mem_start = 0; i < nb_nodes; ++i) {
         if (!nodes[i].node_mem) {

hw/ppc/spapr.c static void spapr_machine_class_init(ObjectClass *oc, void *data) · line 4364

Quoted 4 lines
hw/ppc/spapr.c +1 -0
@@ -4369,6 +4361,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
      */
     mc->numa_mem_align_shift = 28;
     mc->numa_mem_supported = true;
+    mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

Quoted 14 lines
hw/ppc/spapr.c +0 -0
     smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
     smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
     bool smbus_no_migration_support;
     bool nvdimm_supported;
     bool numa_mem_supported;
+    bool auto_enable_numa;
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);
#2 lore raw
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

[...]

Quoted 9 lines
Quoted 1 line

+ mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

After a quick look, all spapr code seems to have the same behavior when nb_numa_nodes==0 and nb_numa_nodes==1, but I'd like to be sure.

David and/or Tao Xu: do you confirm there's no ABI change at all on spapr after implicitly creating a NUMA node?

Quoted 16 lines
Quoted 15 lines
     smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
     smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
     bool smbus_no_migration_support;
     bool nvdimm_supported;
     bool numa_mem_supported;
+    bool auto_enable_numa;
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);
Signature · 1 line
Eduardo
author
#3 lore raw
Quoted 22 lines · Eduardo Habkost
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

[...]

Quoted 9 lines
Quoted 1 line

+ mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

Quoted 7 lines

After a quick look, all spapr code seems to have the same behavior when nb_numa_nodes==0 and nb_numa_nodes==1, but I'd like to be sure.

David and/or Tao Xu: do you confirm there's no ABI change at all on spapr after implicitly creating a NUMA node?

Even without this patch and HMAT patch, if without numa configuration, global nb_numa_nodes is always existing and default is 0, so nb_nodes will be auto set to 1, so from my point of view, this patch will not change ABI.

And I would also want to listen David's opinion.

Quoted 17 lines
Quoted 16 lines
Quoted 15 lines
      smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
      smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
      bool smbus_no_migration_support;
      bool nvdimm_supported;
      bool numa_mem_supported;
+    bool auto_enable_numa;
 
      HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                             DeviceState *dev);
#4 lore raw
In reply to Eduardo Habkost
Quoted 22 lines · Eduardo Habkost
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

[...]

Quoted 9 lines
Quoted 1 line

+ mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

Quoted 3 lines

After a quick look, all spapr code seems to have the same behavior when nb_numa_nodes==0 and nb_numa_nodes==1, but I'd like to be sure.

That's certainly the intention. If there are cases where it doesn't behave that way, it's a bug - although possible one we have to maintainer for machine compatibility.

Quoted 2 lines

David and/or Tao Xu: do you confirm there's no ABI change at all on spapr after implicitly creating a NUMA node?

I don't believe there is, no.

Quoted 18 lines
Quoted 17 lines
Quoted 16 lines

     smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
     smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
     bool smbus_no_migration_support;
     bool nvdimm_supported;
     bool numa_mem_supported;
+    bool auto_enable_numa;
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#5 lore raw
Quoted 27 lines · David Gibson
Quoted 22 lines · Eduardo Habkost
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

[...]

Quoted 9 lines
Quoted 1 line

+ mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

Quoted 3 lines

After a quick look, all spapr code seems to have the same behavior when nb_numa_nodes==0 and nb_numa_nodes==1, but I'd like to be sure.

Quoted 8 lines

That's certainly the intention. If there are cases where it doesn't behave that way, it's a bug - although possible one we have to maintainer for machine compatibility.

Quoted 2 lines

David and/or Tao Xu: do you confirm there's no ABI change at all on spapr after implicitly creating a NUMA node?

I don't believe there is, no.

Oh, FWIW, the PAPR interface which is what defines the guest environment has no notion of "non NUMA" except in the sense of a system with exactly one NUMA node.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
Reviewed-by
#6 lore raw
In reply to David Gibson
Quoted 23 lines · On Thu, 8 Aug 2019 16:35:00 +1000 David Gibson <[email protected]>…
Quoted 22 lines · Eduardo Habkost
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Mon, 5 Aug 2019 15:13:02 +0800 Tao Xu <[email protected]> wrote:

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

[...]

Quoted 9 lines
Quoted 1 line

+ mc->auto_enable_numa = true;

this will always create a numa node (that will affect not only RAM but also all other components that depends on numa state (like CPUs)), where as spapr_populate_memory() was only faking numa node in DT for RAM. It makes non-numa configuration impossible.
Seeing David's ACK on the patch it might be fine, but I believe commit message should capture that and explain why the change in behavior is fine.

Quoted 7 lines
Quoted 3 lines

After a quick look, all spapr code seems to have the same behavior when nb_numa_nodes==0 and nb_numa_nodes==1, but I'd like to be sure.

That's certainly the intention. If there are cases where it doesn't behave that way, it's a bug - although possible one we have to maintainer for machine compatibility.

considering DT is firmware we typically do not add any compat code for the later.

Quoted 5 lines
Quoted 3 lines

David and/or Tao Xu: do you confirm there's no ABI change at all on spapr after implicitly creating a NUMA node?

I don't believe there is, no.

Also seeing your next reply, it seems there is no non-numa usecase is spec so it would be a bug to begin with, hence:

Quoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 16 lines
  
  
     smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
     smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
include/hw/boards.h +1 -0
index 2eb9a0b4e0..4a350b87d2 100644
@@ -220,6 +220,7 @@ struct MachineClass {
     bool smbus_no_migration_support;
     bool nvdimm_supported;
     bool numa_mem_supported;
+    bool auto_enable_numa;
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);  
#7 lore raw
In reply to Tao Xu
Quoted 7 lines · Tao Xu

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

This introduces spurious warnings when running qemu-system-ppc64.
See: https://lore.kernel.org/qemu-devel/CAFEAcA-AvFS2cbDH-t5SxgY9hA=LGL81_8dn-vh193vtV9W1Lg@mail.gmail.com/

To reproduce it, just run 'qemu-system-ppc64 -machine pseries' without any -numa arguments.

I have removed this patch from machine-next so it won't block the existing pull request.

Signature · 1 line
Eduardo
author
#8 lore raw
Quoted 9 lines · Eduardo Habkost
Quoted 7 lines · Tao Xu

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

Quoted 9 lines

This introduces spurious warnings when running qemu-system-ppc64.
See: https://lore.kernel.org/qemu-devel/CAFEAcA-AvFS2cbDH-t5SxgY9hA=LGL81_8dn-vh193vtV9W1Lg@mail.gmail.com/

To reproduce it, just run 'qemu-system-ppc64 -machine pseries' without any -numa arguments.

I have removed this patch from machine-next so it won't block the existing pull request.

I got it. If default splitting of RAM between nodes is
deprecated, this patch can't reuse the splitting code. I agree with droping this patch.

#9 lore raw
Quoted 16 lines · Tao Xu
Quoted 15 lines · Eduardo Habkost
Quoted 7 lines · Tao Xu

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

This introduces spurious warnings when running qemu-system-ppc64.
See: https://lore.kernel.org/qemu-devel/CAFEAcA-AvFS2cbDH-t5SxgY9hA=LGL81_8dn-vh193vtV9W1Lg@mail.gmail.com/

To reproduce it, just run 'qemu-system-ppc64 -machine pseries' without any -numa arguments.

Quoted 6 lines
Quoted 3 lines

I have removed this patch from machine-next so it won't block the existing pull request.

I got it. If default splitting of RAM between nodes is deprecated, this patch can't reuse the splitting code. I agree with droping this patch.

Probably all we need to fix this issue is to replace
NumaNodeOptions node = { };
with
NumaNodeOptions node = { .size = ram_size };
in the auto_enable_numa block.

Do you plan to send v2?

Signature · 1 line
Eduardo
author
#10 lore raw
Quoted 24 lines · Eduardo Habkost
Quoted 16 lines · Tao Xu
Quoted 15 lines · Eduardo Habkost
Quoted 7 lines · Tao Xu

Add MachineClass::auto_enable_numa field. When it is true, a NUMA node is expected to be created implicitly.

This introduces spurious warnings when running qemu-system-ppc64.
See: https://lore.kernel.org/qemu-devel/CAFEAcA-AvFS2cbDH-t5SxgY9hA=LGL81_8dn-vh193vtV9W1Lg@mail.gmail.com/

To reproduce it, just run 'qemu-system-ppc64 -machine pseries' without any -numa arguments.

Quoted 6 lines
Quoted 3 lines

I have removed this patch from machine-next so it won't block the existing pull request.

I got it. If default splitting of RAM between nodes is deprecated, this patch can't reuse the splitting code. I agree with droping this patch.

Quoted 8 lines

Probably all we need to fix this issue is to replace
NumaNodeOptions node = { };
with
NumaNodeOptions node = { .size = ram_size };
in the auto_enable_numa block.

Do you plan to send v2?

OK, thank you for your suggestion. I will fix it and send v2.