Skip to contentSkip to conversation

cpu hotplug: Extend data provided by query-hotpluggable-cpus

RFC PATCH In review Peter Krempa 4 participants 18 messages Jul 7, 2016 – Jul 12, 2016

lore
Share & tools
lore
#0 lore raw

cover cpu hotplug: Extend data provided by query-hotpluggable-cpus

Few entries that would help me in adding the stuff to libvirt.

Peter Krempa (2):
qapi: Add vcpu id to query-hotpluggable-cpus output
numa: Add node_id data in query-hotpluggable-cpus

 hmp.c                 |  1 +
 hw/i386/pc.c          |  8 ++++++++
 hw/ppc/spapr.c        |  9 +++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 qapi-schema.json      |  2 ++
 6 files changed, 32 insertions(+), 2 deletions(-)
Signature · 1 line
2.9.0
#1 lore raw

RFC PATCH 1/2 qapi: Add vcpu id to query-hotpluggable-cpus output

Add 'vcpu index' to the output of query hotpluggable cpus. This output is identical to the linear cpu index taken by the 'cpus' attribute passed to -numa.

This will allow to reliably map the cpu number to a given topology element without making mgmt apps to reimplement the mapping.

---
 hmp.c            | 1 +
 hw/i386/pc.c     | 1 +
 hw/ppc/spapr.c   | 1 +
 qapi-schema.json | 2 ++
 4 files changed, 5 insertions(+)
hmp.c +1 -0
index 0cf5baa..613601e 100644
@@ -2450,6 +2450,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  type: \"%s\"\n", l->value->type);
         monitor_printf(mon, "  vcpus_count: \"%" PRIu64 "\"\n",
                        l->value->vcpus_count);
+        monitor_printf(mon, "  vcpu_id: \"%" PRIu64 "\"\n", l->value->vcpu_id);
         if (l->value->has_qom_path) {
             monitor_printf(mon, "  qom_path: \"%s\"\n", l->value->qom_path);
         }
hw/i386/pc.c +1 -0
index f293a0c..4ba02c4 100644
@@ -2131,6 +2131,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = g_strdup(cpu_type);
         cpu_item->vcpus_count = 1;
+        cpu_item->vcpu_id = i;
         cpu_props->has_socket_id = true;
         cpu_props->socket_id = topo.pkg_id;
         cpu_props->has_core_id = true;
hw/ppc/spapr.c +1 -0
index 7f33a1b..d1f5195 100644
@@ -2378,6 +2378,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = spapr_get_cpu_core_type(machine->cpu_model);
         cpu_item->vcpus_count = smp_threads;
+        cpu_item->vcpu_id = i;
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
         /* TODO: add 'has_node/node' here to describe
qapi-schema.json +2 -0
index ba3bf14..6db9294 100644
@@ -4292,6 +4292,7 @@
 # @type: CPU object type for usage with device_add command
 # @props: list of properties to be used for hotplugging CPU
 # @vcpus-count: number of logical VCPU threads @HotpluggableCPU provides
+# @vcpu-id: linear index of the vcpu
 # @qom-path: #optional link to existing CPU object if CPU is present or
 #            omitted if CPU is not present.
 #
@@ -4300,6 +4301,7 @@
 { 'struct': 'HotpluggableCPU',
   'data': { 'type': 'str',
             'vcpus-count': 'int',
+            'vcpu-id': 'int',
             'props': 'CpuInstanceProperties',
             '*qom-path': 'str'
           }
Signature · 1 line
2.9.0
#2 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] qapi: Add vcpu id to query-hotpluggable-cpus output

Quoted 3 lines · On Thu, 7 Jul 2016 17:17:13 +0200 Peter Krempa <[email protected]> wrote:

Add 'vcpu index' to the output of query hotpluggable cpus. This output is identical to the linear cpu index taken by the 'cpus' attribute passed to -numa.

The problem is, the vcpu index of what? Each entry in the hotpluggable cpus table could represent more than one vcpu.

Quoted 40 lines

This will allow to reliably map the cpu number to a given topology element without making mgmt apps to reimplement the mapping.

---
 hmp.c            | 1 +
 hw/i386/pc.c     | 1 +
 hw/ppc/spapr.c   | 1 +
 qapi-schema.json | 2 ++
 4 files changed, 5 insertions(+)
hmp.c +1 -0
index 0cf5baa..613601e 100644
@@ -2450,6 +2450,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  type: \"%s\"\n", l->value->type);
         monitor_printf(mon, "  vcpus_count: \"%" PRIu64 "\"\n",
                        l->value->vcpus_count);
+        monitor_printf(mon, "  vcpu_id: \"%" PRIu64 "\"\n", l->value->vcpu_id);
         if (l->value->has_qom_path) {
             monitor_printf(mon, "  qom_path: \"%s\"\n", l->value->qom_path);
         }
hw/i386/pc.c +1 -0
index f293a0c..4ba02c4 100644
@@ -2131,6 +2131,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = g_strdup(cpu_type);
         cpu_item->vcpus_count = 1;
+        cpu_item->vcpu_id = i;
         cpu_props->has_socket_id = true;
         cpu_props->socket_id = topo.pkg_id;
         cpu_props->has_core_id = true;
hw/ppc/spapr.c +0 -0
index 7f33a1b..d1f5195 100644

hw/ppc/spapr.c static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) · line 2381

Quoted 4 lines
hw/ppc/spapr.c +1 -0
@@ -2378,6 +2378,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = spapr_get_cpu_core_type(machine->cpu_model);
         cpu_item->vcpus_count = smp_threads;
+        cpu_item->vcpu_id = i;

This is wrong. This is the index of the core. The individual vcpus within the core will have ids starting at core_id and working up.

Quoted 26 lines
hw/ppc/spapr.c +0 -0
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
         /* TODO: add 'has_node/node' here to describe
qapi-schema.json +2 -0
index ba3bf14..6db9294 100644
@@ -4292,6 +4292,7 @@
 # @type: CPU object type for usage with device_add command
 # @props: list of properties to be used for hotplugging CPU
 # @vcpus-count: number of logical VCPU threads @HotpluggableCPU provides
+# @vcpu-id: linear index of the vcpu
 # @qom-path: #optional link to existing CPU object if CPU is present or
 #            omitted if CPU is not present.
 #
@@ -4300,6 +4301,7 @@
 { 'struct': 'HotpluggableCPU',
   'data': { 'type': 'str',
             'vcpus-count': 'int',
+            'vcpu-id': 'int',
             'props': 'CpuInstanceProperties',
             '*qom-path': 'str'
           }
Signature · 1 line
2.9.0
Signature · 2 lines
David Gibson <[email protected]>
Senior Software Engineer, Virtualization, Red Hat
#3 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] qapi: Add vcpu id to query-hotpluggable-cpus output

Quoted 10 lines · On Fri, 8 Jul 2016 12:18:55 +1000 David Gibson <[email protected]> wrote:
Quoted 3 lines · On Thu, 7 Jul 2016 17:17:13 +0200 Peter Krempa <[email protected]> wrote:

Add 'vcpu index' to the output of query hotpluggable cpus. This output is identical to the linear cpu index taken by the 'cpus' attribute passed to -numa.

The problem is, the vcpu index of what? Each entry in the hotpluggable cpus table could represent more than one vcpu.

agreed,
-numa cpus should take socket/core/thread-ids instead so that mgmt could do layout at start-up time

for example if mgmt specifies
-smp cpus=1,sockets=2,cores=2,maxcpus=4
it knows socket/core layout and can assign them as desired
-numa nodeid=0,cpu=[socket-id=0,core-id=0] \
-numa nodeid=1,cpu=[socket-id=0,core-id=1] \
-numa nodeid=2,cpu=[socket-id=1]

that of cause assuming that QEMU would guarantee IDs are are ranges [0..sockets), ... it's so for x86, can we guarantee it for spapr?

Quoted 77 lines
Quoted 41 lines

This will allow to reliably map the cpu number to a given topology element without making mgmt apps to reimplement the mapping.

---
 hmp.c            | 1 +
 hw/i386/pc.c     | 1 +
 hw/ppc/spapr.c   | 1 +
 qapi-schema.json | 2 ++
 4 files changed, 5 insertions(+)
hmp.c +1 -0
index 0cf5baa..613601e 100644
@@ -2450,6 +2450,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  type: \"%s\"\n", l->value->type);
         monitor_printf(mon, "  vcpus_count: \"%" PRIu64 "\"\n",
                        l->value->vcpus_count);
+        monitor_printf(mon, "  vcpu_id: \"%" PRIu64 "\"\n", l->value->vcpu_id);
         if (l->value->has_qom_path) {
             monitor_printf(mon, "  qom_path: \"%s\"\n", l->value->qom_path);
         }
hw/i386/pc.c +1 -0
index f293a0c..4ba02c4 100644
@@ -2131,6 +2131,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = g_strdup(cpu_type);
         cpu_item->vcpus_count = 1;
+        cpu_item->vcpu_id = i;
         cpu_props->has_socket_id = true;
         cpu_props->socket_id = topo.pkg_id;
         cpu_props->has_core_id = true;
hw/ppc/spapr.c +0 -0
index 7f33a1b..d1f5195 100644

hw/ppc/spapr.c static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) · line 2381

Quoted 4 lines
hw/ppc/spapr.c +1 -0
@@ -2378,6 +2378,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = spapr_get_cpu_core_type(machine->cpu_model);
         cpu_item->vcpus_count = smp_threads;
+        cpu_item->vcpu_id = i;  

This is wrong. This is the index of the core. The individual vcpus within the core will have ids starting at core_id and working up.

Quoted 26 lines
hw/ppc/spapr.c +0 -0
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
         /* TODO: add 'has_node/node' here to describe
qapi-schema.json +2 -0
index ba3bf14..6db9294 100644
@@ -4292,6 +4292,7 @@
 # @type: CPU object type for usage with device_add command
 # @props: list of properties to be used for hotplugging CPU
 # @vcpus-count: number of logical VCPU threads @HotpluggableCPU provides
+# @vcpu-id: linear index of the vcpu
 # @qom-path: #optional link to existing CPU object if CPU is present or
 #            omitted if CPU is not present.
 #
@@ -4300,6 +4301,7 @@
 { 'struct': 'HotpluggableCPU',
   'data': { 'type': 'str',
             'vcpus-count': 'int',
+            'vcpu-id': 'int',
             'props': 'CpuInstanceProperties',
             '*qom-path': 'str'
           }
Signature · 1 line
2.9.0
#4 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] qapi: Add vcpu id to query-hotpluggable-cpus output

Quoted 24 lines · On Fri, 8 Jul 2016 13:40:38 +0200 Igor Mammedov <[email protected]> wrote:
Quoted 10 lines · On Fri, 8 Jul 2016 12:18:55 +1000 David Gibson <[email protected]> wrote:
Quoted 3 lines · On Thu, 7 Jul 2016 17:17:13 +0200 Peter Krempa <[email protected]> wrote:

Add 'vcpu index' to the output of query hotpluggable cpus. This output is identical to the linear cpu index taken by the 'cpus' attribute passed to -numa.

The problem is, the vcpu index of what? Each entry in the hotpluggable cpus table could represent more than one vcpu.

agreed,
-numa cpus should take socket/core/thread-ids instead so that mgmt could do layout at start-up time

for example if mgmt specifies
-smp cpus=1,sockets=2,cores=2,maxcpus=4
it knows socket/core layout and can assign them as desired
-numa nodeid=0,cpu=[socket-id=0,core-id=0] \
-numa nodeid=1,cpu=[socket-id=0,core-id=1] \
-numa nodeid=2,cpu=[socket-id=1]

Quoted 2 lines

that of cause assuming that QEMU would guarantee IDs are are ranges [0..sockets), ... it's so for x86, can we guarantee it for spapr?

Urgh.. we could for spapr, but I think it's bad idea to do that in general. For powernv (or others) we might want to use socket ids with a physical meaning (e.g. actual values of chip select lines) and those might not be contiguous.

Quoted 79 lines
Quoted 78 lines
Quoted 42 lines

This will allow to reliably map the cpu number to a given topology element without making mgmt apps to reimplement the mapping.

---
 hmp.c            | 1 +
 hw/i386/pc.c     | 1 +
 hw/ppc/spapr.c   | 1 +
 qapi-schema.json | 2 ++
 4 files changed, 5 insertions(+)
hmp.c +1 -0
index 0cf5baa..613601e 100644
@@ -2450,6 +2450,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  type: \"%s\"\n", l->value->type);
         monitor_printf(mon, "  vcpus_count: \"%" PRIu64 "\"\n",
                        l->value->vcpus_count);
+        monitor_printf(mon, "  vcpu_id: \"%" PRIu64 "\"\n", l->value->vcpu_id);
         if (l->value->has_qom_path) {
             monitor_printf(mon, "  qom_path: \"%s\"\n", l->value->qom_path);
         }
hw/i386/pc.c +1 -0
index f293a0c..4ba02c4 100644
@@ -2131,6 +2131,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = g_strdup(cpu_type);
         cpu_item->vcpus_count = 1;
+        cpu_item->vcpu_id = i;
         cpu_props->has_socket_id = true;
         cpu_props->socket_id = topo.pkg_id;
         cpu_props->has_core_id = true;
hw/ppc/spapr.c +0 -0
index 7f33a1b..d1f5195 100644

hw/ppc/spapr.c static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) · line 2381

Quoted 4 lines
hw/ppc/spapr.c +1 -0
@@ -2378,6 +2378,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = spapr_get_cpu_core_type(machine->cpu_model);
         cpu_item->vcpus_count = smp_threads;
+        cpu_item->vcpu_id = i;    

This is wrong. This is the index of the core. The individual vcpus within the core will have ids starting at core_id and working up.

Quoted 26 lines
hw/ppc/spapr.c +0 -0
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
         /* TODO: add 'has_node/node' here to describe
qapi-schema.json +2 -0
index ba3bf14..6db9294 100644
@@ -4292,6 +4292,7 @@
 # @type: CPU object type for usage with device_add command
 # @props: list of properties to be used for hotplugging CPU
 # @vcpus-count: number of logical VCPU threads @HotpluggableCPU provides
+# @vcpu-id: linear index of the vcpu
 # @qom-path: #optional link to existing CPU object if CPU is present or
 #            omitted if CPU is not present.
 #
@@ -4300,6 +4301,7 @@
 { 'struct': 'HotpluggableCPU',
   'data': { 'type': 'str',
             'vcpus-count': 'int',
+            'vcpu-id': 'int',
             'props': 'CpuInstanceProperties',
             '*qom-path': 'str'
           }
Signature · 1 line
2.9.0
Signature · 2 lines
David Gibson <[email protected]>
Senior Software Engineer, Virtualization, Red Hat
#5 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] qapi: Add vcpu id to query-hotpluggable-cpus output

Quoted 27 lines · On Mon, 11 Jul 2016 13:30:38 +1000 David Gibson <[email protected]> wrote:
Quoted 24 lines · On Fri, 8 Jul 2016 13:40:38 +0200 Igor Mammedov <[email protected]> wrote:
Quoted 10 lines · On Fri, 8 Jul 2016 12:18:55 +1000 David Gibson <[email protected]> wrote:
Quoted 3 lines · On Thu, 7 Jul 2016 17:17:13 +0200 Peter Krempa <[email protected]> wrote:

Add 'vcpu index' to the output of query hotpluggable cpus. This output is identical to the linear cpu index taken by the 'cpus' attribute passed to -numa.

The problem is, the vcpu index of what? Each entry in the hotpluggable cpus table could represent more than one vcpu.

agreed,
-numa cpus should take socket/core/thread-ids instead so that mgmt could do layout at start-up time

for example if mgmt specifies
-smp cpus=1,sockets=2,cores=2,maxcpus=4
it knows socket/core layout and can assign them as desired
-numa nodeid=0,cpu=[socket-id=0,core-id=0] \
-numa nodeid=1,cpu=[socket-id=0,core-id=1] \
-numa nodeid=2,cpu=[socket-id=1]

Quoted 7 lines
Quoted 2 lines

that of cause assuming that QEMU would guarantee IDs are are ranges [0..sockets), ... it's so for x86, can we guarantee it for spapr?

Urgh.. we could for spapr, but I think it's bad idea to do that in general. For powernv (or others) we might want to use socket ids with a physical meaning (e.g. actual values of chip select lines) and those might not be contiguous.

Are there any other ideas how we can design numa mapping interface, then?

Quoted 83 lines
Quoted 81 lines
Quoted 80 lines
Quoted 44 lines

This will allow to reliably map the cpu number to a given topology element without making mgmt apps to reimplement the mapping.

---
 hmp.c            | 1 +
 hw/i386/pc.c     | 1 +
 hw/ppc/spapr.c   | 1 +
 qapi-schema.json | 2 ++
 4 files changed, 5 insertions(+)
hmp.c +1 -0
index 0cf5baa..613601e 100644
@@ -2450,6 +2450,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  type: \"%s\"\n", l->value->type);
         monitor_printf(mon, "  vcpus_count: \"%" PRIu64 "\"\n",
                        l->value->vcpus_count);
+        monitor_printf(mon, "  vcpu_id: \"%" PRIu64 "\"\n", l->value->vcpu_id);
         if (l->value->has_qom_path) {
             monitor_printf(mon, "  qom_path: \"%s\"\n", l->value->qom_path);
         }
hw/i386/pc.c +1 -0
index f293a0c..4ba02c4 100644
@@ -2131,6 +2131,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = g_strdup(cpu_type);
         cpu_item->vcpus_count = 1;
+        cpu_item->vcpu_id = i;
         cpu_props->has_socket_id = true;
         cpu_props->socket_id = topo.pkg_id;
         cpu_props->has_core_id = true;
hw/ppc/spapr.c +0 -0
index 7f33a1b..d1f5195 100644

hw/ppc/spapr.c static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) · line 2381

Quoted 4 lines
hw/ppc/spapr.c +1 -0
@@ -2378,6 +2378,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
 
         cpu_item->type = spapr_get_cpu_core_type(machine->cpu_model);
         cpu_item->vcpus_count = smp_threads;
+        cpu_item->vcpu_id = i;      

This is wrong. This is the index of the core. The individual vcpus within the core will have ids starting at core_id and working up.

Quoted 26 lines
hw/ppc/spapr.c +0 -0
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
         /* TODO: add 'has_node/node' here to describe
qapi-schema.json +2 -0
index ba3bf14..6db9294 100644
@@ -4292,6 +4292,7 @@
 # @type: CPU object type for usage with device_add command
 # @props: list of properties to be used for hotplugging CPU
 # @vcpus-count: number of logical VCPU threads @HotpluggableCPU provides
+# @vcpu-id: linear index of the vcpu
 # @qom-path: #optional link to existing CPU object if CPU is present or
 #            omitted if CPU is not present.
 #
@@ -4300,6 +4301,7 @@
 { 'struct': 'HotpluggableCPU',
   'data': { 'type': 'str',
             'vcpus-count': 'int',
+            'vcpu-id': 'int',
             'props': 'CpuInstanceProperties',
             '*qom-path': 'str'
           }
Signature · 1 line
2.9.0
#6 lore raw

RFC PATCH 2/2 numa: Add node_id data in query-hotpluggable-cpus

In reply to Peter Krempa

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)
hw/i386/pc.c +7 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
+
         cpu_item->props = cpu_props;
 
         cpu = pcms->possible_cpus->cpus[i].cpu;
hw/ppc/spapr.c +6 -2
index d1f5195..06ba7fc 100644
@@ -2370,6 +2370,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
     sPAPRMachineState *spapr = SPAPR_MACHINE(machine);
     int spapr_max_cores = max_cpus / smp_threads;
     int smt = kvmppc_smt_threads();
+    int node_id;
 
     for (i = 0; i < spapr_max_cores; i++) {
         HotpluggableCPUList *list_item = g_new0(typeof(*list_item), 1);
@@ -2381,8 +2382,11 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
         cpu_item->vcpu_id = i;
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
 
         cpu_item->props = cpu_props;
         if (spapr->cores[i]) {
include/sysemu/numa.h +1 -0
index bb184c9..04d7097 100644
@@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
+int numa_node_get_by_cpu_index(int cpu_index);
 
 #endif
numa.c +13 -0
index cbae430..365738a 100644
@@ -506,6 +506,19 @@ void query_numa_node_mem(uint64_t node_mem[])
     }
 }
 
+int numa_node_get_by_cpu_index(int cpu_index)
+{
+    int i;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cpu_index, numa_info[i].node_cpu)) {
+            return i;
+        }
+    }
+
+    return -1;
+}
+
 static int query_memdev(Object *obj, void *opaque)
 {
     MemdevList **list = opaque;
Signature · 1 line
2.9.0
#7 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 10 lines · Peter Krempa

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)

The helper function should be a separate patch, and Igor beat you to it, see

https://www.mail-archive.com/[email protected]/msg383461.html

drew

Quoted 91 lines
hw/i386/pc.c +7 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
+
         cpu_item->props = cpu_props;
 
         cpu = pcms->possible_cpus->cpus[i].cpu;
hw/ppc/spapr.c +6 -2
index d1f5195..06ba7fc 100644
@@ -2370,6 +2370,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
     sPAPRMachineState *spapr = SPAPR_MACHINE(machine);
     int spapr_max_cores = max_cpus / smp_threads;
     int smt = kvmppc_smt_threads();
+    int node_id;
 
     for (i = 0; i < spapr_max_cores; i++) {
         HotpluggableCPUList *list_item = g_new0(typeof(*list_item), 1);
@@ -2381,8 +2382,11 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
         cpu_item->vcpu_id = i;
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
 
         cpu_item->props = cpu_props;
         if (spapr->cores[i]) {
include/sysemu/numa.h +1 -0
index bb184c9..04d7097 100644
@@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
+int numa_node_get_by_cpu_index(int cpu_index);
 
 #endif
numa.c +13 -0
index cbae430..365738a 100644
@@ -506,6 +506,19 @@ void query_numa_node_mem(uint64_t node_mem[])
     }
 }
 
+int numa_node_get_by_cpu_index(int cpu_index)
+{
+    int i;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cpu_index, numa_info[i].node_cpu)) {
+            return i;
+        }
+    }
+
+    return -1;
+}
+
 static int query_memdev(Object *obj, void *opaque)
 {
     MemdevList **list = opaque;
Signature · 1 line
2.9.0
#8 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

In reply to Peter Krempa
Quoted 2 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

IIUC how the query thing works this means that the node id issued by query-hotpluggable-cpus will be echoed back to device add by libvirt. I'm not sure we actually process that information in the core at present, so I don't know that that's right.

We need to be clear on which direction information is flowing here. Does query-hotpluggable-cpus *define* the NUMA node allocation which is then passed to the core device which implements it. Or is the NUMA allocation defined elsewhere, and query-hotpluggable-cpus just reports it.

Quoted 46 lines
---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)
hw/i386/pc.c +7 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
+
         cpu_item->props = cpu_props;
 
         cpu = pcms->possible_cpus->cpus[i].cpu;
hw/ppc/spapr.c +1 -0
index d1f5195..06ba7fc 100644
@@ -2370,6 +2370,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
     sPAPRMachineState *spapr = SPAPR_MACHINE(machine);
     int spapr_max_cores = max_cpus / smp_threads;
     int smt = kvmppc_smt_threads();
+    int node_id;
 
     for (i = 0; i < spapr_max_cores; i++) {
         HotpluggableCPUList *list_item = g_new0(typeof(*list_item), 1);

hw/ppc/spapr.c static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) · line 2386

Quoted 7 lines
hw/ppc/spapr.c +2 -2
@@ -2381,8 +2382,11 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
         cpu_item->vcpu_id = i;
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {

As with the previous patch this is incorrect, becauyse numa_node_get_by_cpu_index() is working from a vcpu (i.e. thread) index, but you're passing a core index.

Quoted 44 lines
hw/ppc/spapr.c +3 -0
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
 
         cpu_item->props = cpu_props;
         if (spapr->cores[i]) {
include/sysemu/numa.h +1 -0
index bb184c9..04d7097 100644
@@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
+int numa_node_get_by_cpu_index(int cpu_index);
 
 #endif
numa.c +13 -0
index cbae430..365738a 100644
@@ -506,6 +506,19 @@ void query_numa_node_mem(uint64_t node_mem[])
     }
 }
 
+int numa_node_get_by_cpu_index(int cpu_index)
+{
+    int i;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cpu_index, numa_info[i].node_cpu)) {
+            return i;
+        }
+    }
+
+    return -1;
+}
+
 static int query_memdev(Object *obj, void *opaque)
 {
     MemdevList **list = opaque;
Signature · 1 line
2.9.0
Signature · 2 lines
David Gibson <[email protected]>
Senior Software Engineer, Virtualization, Red Hat
#9 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 9 lines · David Gibson
Quoted 2 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

IIUC how the query thing works this means that the node id issued by query-hotpluggable-cpus will be echoed back to device add by libvirt.

It will be echoed back, but the problem is that it's configured separately ...

Quoted 8 lines

I'm not sure we actually process that information in the core at present, so I don't know that that's right.

We need to be clear on which direction information is flowing here. Does query-hotpluggable-cpus *define* the NUMA node allocation which is then passed to the core device which implements it. Or is the NUMA allocation defined elsewhere, and query-hotpluggable-cpus just reports it.

Currently (in the pre-hotplug era) the NUMA topology is defined by specifying a CPU numbers (see previous patch) on the commandline:

-numa node=1,cpus=1-5,cpus=8,cpus=11...

This is then reported to the guest.

For a machine started with:

-smp 5,maxcpus=8,sockets=2,cores=2,threads=2 -numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,mem=500 -numa node,nodeid=1,cpus=1,cpus=3,cpus=5,cpus=7,mem=500

you get the following topology that is not really possible with hardware:

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                5
On-line CPU(s) list:   0-4
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 6
Model name:            QEMU Virtual CPU version 2.5+
Stepping:              3
CPU MHz:               3504.318
BogoMIPS:              7008.63
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0,2,4
NUMA node1 CPU(s):     1,3

Note that the count of cpus per numa node does not need to be identical.

As of the above 'query-hotpluggable-cpus' will need to report the data that was configured above even if it doesn't make much sense in a real world.

I did not try the above on a PPC host and thus I'm not sure whether the config above is allowed or not.

While for the hotplug cpus it would be possible to plug in the correct one according to the requested use the queried data but with the current approach it's impossible to set the initial vcpus differently.

Peter

Note: For libvirt it's a no-go to start a throwaway qemu process just to query the information and thus it's desired to have a way to configure all this without the need to query with a specific machine type/topology.

#10 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

On Fri, 8 Jul 2016 09:46:00 +0200
Peter Krempa <[email protected]> wrote:

[...]

Quoted 4 lines

Note: For libvirt it's a no-go to start a throwaway qemu process just to query the information and thus it's desired to have a way to configure all this without the need to query with a specific machine type/topology.

Is it no-go to start throwaway qemu on the new domain creation time? i.e. user create a new domain, with specific -machine/-smp layout libvirt stores results of query-hotpluggable-cpus and then allow user in (eg)virt-manager to pick which CPUs are enabled and optionally to which numa nodes they are mapped.

#11 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 9 lines · Igor Mammedov

On Fri, 8 Jul 2016 09:46:00 +0200
Peter Krempa <[email protected]> wrote:

[...]

Quoted 4 lines

Note: For libvirt it's a no-go to start a throwaway qemu process just to query the information and thus it's desired to have a way to configure all this without the need to query with a specific machine type/topology.

Is it no-go to start throwaway qemu on the new domain creation time?

Yes. The policy is that once we are starting the VM the qemu process that we create is the one running the VM later. We can tweak stuff using QMP and/or kill it if the configuration requested by the user can't be achieved, but starting a different qemu process would not be accepted upstream.

Capability querying is done prior to that with a qemu process with -M none and the results are cached for the given combination of qemu binary and libvirtd binary (even across restarts of the libvirtd binary).

Quoted 4 lines

i.e. user create a new domain, with specific -machine/-smp layout libvirt stores results of query-hotpluggable-cpus and then allow user in (eg)virt-manager to pick which CPUs are enabled and optionally to which numa nodes they are mapped.

We can update certain stuff that was not explicitly configured by the user to the state that will then be used by qemu.

#12 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

In reply to Peter Krempa
Quoted 64 lines · On Fri, 8 Jul 2016 09:46:00 +0200 Peter Krempa <[email protected]> wrote:
Quoted 9 lines · David Gibson
Quoted 2 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

IIUC how the query thing works this means that the node id issued by query-hotpluggable-cpus will be echoed back to device add by libvirt.

It will be echoed back, but the problem is that it's configured separately ...

Quoted 8 lines

I'm not sure we actually process that information in the core at present, so I don't know that that's right.

We need to be clear on which direction information is flowing here. Does query-hotpluggable-cpus *define* the NUMA node allocation which is then passed to the core device which implements it. Or is the NUMA allocation defined elsewhere, and query-hotpluggable-cpus just reports it.

Currently (in the pre-hotplug era) the NUMA topology is defined by specifying a CPU numbers (see previous patch) on the commandline:

-numa node=1,cpus=1-5,cpus=8,cpus=11...

This is then reported to the guest.

For a machine started with:

(quoted diff) +0 -3
-smp 5,maxcpus=8,sockets=2,cores=2,threads=2
-numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,mem=500
-numa node,nodeid=1,cpus=1,cpus=3,cpus=5,cpus=7,mem=500

you get the following topology that is not really possible with hardware:

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                5
On-line CPU(s) list:   0-4
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 6
Model name:            QEMU Virtual CPU version 2.5+
Stepping:              3
CPU MHz:               3504.318
BogoMIPS:              7008.63
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0,2,4
NUMA node1 CPU(s):     1,3
Quoted 8 lines

Note that the count of cpus per numa node does not need to be identical.

As of the above 'query-hotpluggable-cpus' will need to report the data that was configured above even if it doesn't make much sense in a real world.

I did not try the above on a PPC host and thus I'm not sure whether the config above is allowed or not.

It's not - although I'm not sure that we actually have something enforcing this.

However, single cores *must* be in the same NUMA node - there's no way of reporting to the guest anything finer grained.

Quoted 10 lines

While for the hotplug cpus it would be possible to plug in the correct one according to the requested use the queried data but with the current approach it's impossible to set the initial vcpus differently.

Peter

Note: For libvirt it's a no-go to start a throwaway qemu process just to query the information and thus it's desired to have a way to configure all this without the need to query with a specific machine type/topology.

Signature · 2 lines
David Gibson <[email protected]>
Senior Software Engineer, Virtualization, Red Hat
#13 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

In reply to Peter Krempa
Quoted 24 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)
hw/i386/pc.c +1 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */

hw/i386/pc.c static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine) · line 2146

Quoted 8 lines
hw/i386/pc.c +5 -0
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }

I've not included node_id for a reason,
"-numa cpus=1,2,3..." looks to me hopelessly broken now but
I've not came up with an idea how to redo it in nice and clean way yet.

Alternative could be CLI-less numa configuration, where QEMU is started without "-numa cpus" but with "-S" then mgmt could call query_hotpluggable_cpus() to get possible CPUs and then
map them to numa nodes with a new QMP command using attributes it got from query_hotpluggable_cpus().

it's along the way start QEMU -smp 1,maxcpus=X and then add remaining CPUs with device_add after getting properties from query_hotpluggable_cpus().

then at machine_done time we can adjust DT/ACPI data to reflect configured mapping.

Quoted 64 lines
hw/i386/pc.c +0 -0
         cpu_item->props = cpu_props;
 
         cpu = pcms->possible_cpus->cpus[i].cpu;
hw/ppc/spapr.c +6 -2
index d1f5195..06ba7fc 100644
@@ -2370,6 +2370,7 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
     sPAPRMachineState *spapr = SPAPR_MACHINE(machine);
     int spapr_max_cores = max_cpus / smp_threads;
     int smt = kvmppc_smt_threads();
+    int node_id;
 
     for (i = 0; i < spapr_max_cores; i++) {
         HotpluggableCPUList *list_item = g_new0(typeof(*list_item), 1);
@@ -2381,8 +2382,11 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine)
         cpu_item->vcpu_id = i;
         cpu_props->has_core_id = true;
         cpu_props->core_id = i * smt;
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }
 
         cpu_item->props = cpu_props;
         if (spapr->cores[i]) {
include/sysemu/numa.h +1 -0
index bb184c9..04d7097 100644
@@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
+int numa_node_get_by_cpu_index(int cpu_index);
 
 #endif
numa.c +13 -0
index cbae430..365738a 100644
@@ -506,6 +506,19 @@ void query_numa_node_mem(uint64_t node_mem[])
     }
 }
 
+int numa_node_get_by_cpu_index(int cpu_index)
+{
+    int i;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cpu_index, numa_info[i].node_cpu)) {
+            return i;
+        }
+    }
+
+    return -1;
+}
+
 static int query_memdev(Object *obj, void *opaque)
 {
     MemdevList **list = opaque;
#14 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 39 lines · Igor Mammedov
Quoted 24 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)
hw/i386/pc.c +1 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */

hw/i386/pc.c static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine) · line 2146

Quoted 8 lines
hw/i386/pc.c +5 -0
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }

I've not included node_id for a reason,
"-numa cpus=1,2,3..." looks to me hopelessly broken now but
I've not came up with an idea how to redo it in nice and clean way yet.

Quoted 5 lines

Alternative could be CLI-less numa configuration, where QEMU is started without "-numa cpus" but with "-S" then mgmt could call query_hotpluggable_cpus() to get possible CPUs and then
map them to numa nodes with a new QMP command using attributes it got from query_hotpluggable_cpus().

I think this could work for libvirt. The CPU index we currently expose in the XML would become just a libvirt internal detail and the new QMP command would be used to do the setup. Adding some QMP calls during VM startup is okay.

Quoted 3 lines

it's along the way start QEMU -smp 1,maxcpus=X and then add remaining CPUs with device_add after getting properties from query_hotpluggable_cpus().

I'm going to use a similar approach even for the hotpluggable cpus so I can query the data for a new VM. On the other hand I can't make libvirt use the approach with -smp 1,... all the time since we guarantee that a XML that worked on a older version will be migratable back to the older version.

Quoted 2 lines

then at machine_done time we can adjust DT/ACPI data to reflect configured mapping.

In such case this series can be dropped since it provides what I need differently.

Thanks,

Peter

#15 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 55 lines · On Fri, 8 Jul 2016 14:04:23 +0200 Peter Krempa <[email protected]> wrote:
Quoted 39 lines · Igor Mammedov
Quoted 24 lines · On Thu, 7 Jul 2016 17:17:14 +0200 Peter Krempa <[email protected]> wrote:

Add a helper that looks up the NUMA node for a given CPU and use it to fill the node_id in the PPC and X86 impls of query-hotpluggable-cpus.

---
 hw/i386/pc.c          |  7 +++++++
 hw/ppc/spapr.c        |  8 ++++++--
 include/sysemu/numa.h |  1 +
 numa.c                | 13 +++++++++++++
 4 files changed, 27 insertions(+), 2 deletions(-)
hw/i386/pc.c +1 -0
index 4ba02c4..a0b9507 100644
@@ -2115,6 +2115,7 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
     HotpluggableCPUList *head = NULL;
     PCMachineState *pcms = PC_MACHINE(machine);
     const char *cpu_type;
+    int node_id;
 
     cpu = pcms->possible_cpus->cpus[0].cpu;
     assert(cpu); /* BSP is always present */

hw/i386/pc.c static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine) · line 2146

Quoted 8 lines
hw/i386/pc.c +5 -0
@@ -2138,6 +2139,12 @@ static HotpluggableCPUList *pc_query_hotpluggable_cpus(MachineState *machine)
         cpu_props->core_id = topo.core_id;
         cpu_props->has_thread_id = true;
         cpu_props->thread_id = topo.smt_id;
+
+        if ((node_id = numa_node_get_by_cpu_index(i)) >= 0) {
+            cpu_props->has_node_id = true;
+            cpu_props->node_id = node_id;
+        }  

I've not included node_id for a reason,
"-numa cpus=1,2,3..." looks to me hopelessly broken now but
I've not came up with an idea how to redo it in nice and clean way yet.

Quoted 5 lines

Alternative could be CLI-less numa configuration, where QEMU is started without "-numa cpus" but with "-S" then mgmt could call query_hotpluggable_cpus() to get possible CPUs and then
map them to numa nodes with a new QMP command using attributes it got from query_hotpluggable_cpus().

I think this could work for libvirt. The CPU index we currently expose in the XML would become just a libvirt internal detail and the new QMP command would be used to do the setup. Adding some QMP calls during VM startup is okay.

Quoted 3 lines

it's along the way start QEMU -smp 1,maxcpus=X and then add remaining CPUs with device_add after getting properties from query_hotpluggable_cpus().

Quoted 5 lines

I'm going to use a similar approach even for the hotpluggable cpus so I can query the data for a new VM. On the other hand I can't make libvirt use the approach with -smp 1,... all the time since we guarantee that a XML that worked on a older version will be migratable back to the older version.

Could you explain a little bit more about issue?

Quoted 11 lines
Quoted 3 lines

then at machine_done time we can adjust DT/ACPI data to reflect configured mapping.

In such case this series can be dropped since it provides what I need differently.

Thanks,

Peter

#16 lore raw

Re: [Qemu-devel] [RFC PATCH 2/2] numa: Add node_id data in query-hotpluggable-cpus

Quoted 6 lines · Igor Mammedov
Quoted 3 lines · On Fri, 8 Jul 2016 14:04:23 +0200 Peter Krempa <[email protected]> wrote:
Quoted 2 lines · Igor Mammedov

On Thu, 7 Jul 2016 17:17:14 +0200
Peter Krempa <[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

it's along the way start QEMU -smp 1,maxcpus=X and then add remaining CPUs with device_add after getting properties from query_hotpluggable_cpus().

I'm going to use a similar approach even for the hotpluggable cpus so I can query the data for a new VM. On the other hand I can't make libvirt use the approach with -smp 1,... all the time since we guarantee that a XML that worked on a older version will be migratable back to the older version.

Could you explain a little bit more about issue?

Libvirt attempts to maintain compatibility of the XML definition file even for migrations to older versions.

If you are able to start a VM with a XML on libvirt version 'A' then when you use the same XML to start it on a newer version 'B' we still need to be able to migrate the VM back to 'A'. This means that vCPUs added via -device/device_add can be used only for configurations that will explicitly have configuration enabled.

This shouldn't be a problem generally, but this means that we still either the 'old' way to work properly or a approach that is compatible with migration.

For the specific case of vCPU hotplug this means that if you don't hotplug any CPU it needs to work with older libvirt including the numa topology and all other possible stuff.

#17 lore raw
In reply to Peter Krempa
Quoted 1 line · Peter Krempa

Few entries that would help me in adding the stuff to libvirt.

I forgot to mention that I didn't really test the PPC64 stuff as I currently don't have a box at hand.