Skip to contentSkip to conversation

hw/arm/virt: Fix CPU's default NUMA node ID

PATCH Superseded Gavin Shan 4 participants 15 messages Mar 3, 2022 – Mar 23, 2022

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Gavin Shan

author
#0 lore raw

v2 cover hw/arm/virt: Fix CPU's default NUMA node ID

When the CPU-to-NUMA association isn't provided by user, the default NUMA node ID for the specific CPU is returned from virt_get_default_cpu_node_id(). Unfortunately, the default NUMA node ID breaks socket boundary and leads to the broken CPU topology warning message in Linux guest. This series intends to fix the issue.

PATCH[1/3]: Fixes the broken CPU topology by considering the socket boundary
when the default NUMA node ID is calculated.
PATCH[2/3]: Use the existing CPU topology to build PPTT table. However, the
cluster ID has to be calculated dynamically because there is no corresponding information in CPU instance properties.
PATCH[3/3]: Take thread ID as the ACPI processor ID in MDAT and SRAT tables.

Changelog
=========
v2:
* Populate the CPU topology in virt_possible_cpu_arch_ids() so that it
can be reused in virt_get_default_cpu_node_id() (Igor)
* Added PATCH[2/3] to use the existing CPU topology when PPTT table
is built (Igor)
* Added PATCH[3/3] to take thread ID as ACPI processor ID in MADT and
SRAT table (Gavin)

Gavin Shan (3):
hw/arm/virt: Fix CPU's default NUMA node ID hw/acpi/aml-build: Use existing CPU topology to build PPTT table hw/arm/virt: Unify ACPI processor ID in MADT and SRAT table

 hw/acpi/aml-build.c      | 106 ++++++++++++++++++++++++++++++---------
 hw/arm/virt-acpi-build.c |  12 +++--
 hw/arm/virt.c            |  17 ++++++-
 3 files changed, 107 insertions(+), 28 deletions(-)
Signature · 1 line
2.23.0
author
#1 lore raw

PATCH v2 1/3 hw/arm/virt: Fix CPU's default NUMA node ID

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

  /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
  -accel kvm -machine virt,gic-version=host               \
  -cpu host                                               \
  -smp 6,sockets=2,cores=3,threads=1                      \
  -m 1024M,slots=16,maxmem=64G                            \
  -object memory-backend-ram,id=mem0,size=128M            \
  -object memory-backend-ram,id=mem1,size=128M            \
  -object memory-backend-ram,id=mem2,size=128M            \
  -object memory-backend-ram,id=mem3,size=128M            \
  -object memory-backend-ram,id=mem4,size=128M            \
  -object memory-backend-ram,id=mem4,size=384M            \
  -numa node,nodeid=0,memdev=mem0                         \
  -numa node,nodeid=1,memdev=mem1                         \
  -numa node,nodeid=2,memdev=mem2                         \
  -numa node,nodeid=3,memdev=mem3                         \
  -numa node,nodeid=4,memdev=mem4                         \
  -numa node,nodeid=5,memdev=mem5
         :
  alternatives: patching kernel code
  BUG: arch topology borken
  the CLS domain not a subset of the MC domain
  <the above error log repeats>
  BUG: arch topology borken
  the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

---
 hw/arm/virt.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +16 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
 static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
 {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
 }
 
 static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
     int n;
     unsigned int max_cpus = ms->smp.max_cpus;
     VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
     if (ms->possible_cpus) {
         assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
         ms->possible_cpus->cpus[n].type = ms->cpu_type;
         ms->possible_cpus->cpus[n].arch_id =
             virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
         ms->possible_cpus->cpus[n].props.has_thread_id = true;
         ms->possible_cpus->cpus[n].props.thread_id = n;
     }
Signature · 1 line
2.23.0
#2 lore raw

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

   /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
   -accel kvm -machine virt,gic-version=host               \
   -cpu host                                               \
   -smp 6,sockets=2,cores=3,threads=1                      \
   -m 1024M,slots=16,maxmem=64G                            \
   -object memory-backend-ram,id=mem0,size=128M            \
   -object memory-backend-ram,id=mem1,size=128M            \
   -object memory-backend-ram,id=mem2,size=128M            \
   -object memory-backend-ram,id=mem3,size=128M            \
   -object memory-backend-ram,id=mem4,size=128M            \
   -object memory-backend-ram,id=mem4,size=384M            \
   -numa node,nodeid=0,memdev=mem0                         \
   -numa node,nodeid=1,memdev=mem1                         \
   -numa node,nodeid=2,memdev=mem2                         \
   -numa node,nodeid=3,memdev=mem3                         \
   -numa node,nodeid=4,memdev=mem4                         \
   -numa node,nodeid=5,memdev=mem5
          :
   alternatives: patching kernel code
   BUG: arch topology borken
   the CLS domain not a subset of the MC domain
   <the above error log repeats>
   BUG: arch topology borken
   the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids, and the other fixes the numa node ID issue.

Quoted 33 lines
---
  hw/arm/virt.c | 17 ++++++++++++++++-
  1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +4 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
  static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
  {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
  }
 
  static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
      int n;
      unsigned int max_cpus = ms->smp.max_cpus;
      VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
      if (ms->possible_cpus) {
          assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
          ms->possible_cpus->cpus[n].type = ms->cpu_type;
          ms->possible_cpus->cpus[n].arch_id =
              virt_cpu_mp_affinity(vms, n);

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2524

Quoted 10 lines
hw/arm/virt.c +10 -0
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2529

Quoted 5 lines
hw/arm/virt.c +2 -0
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
          ms->possible_cpus->cpus[n].props.has_thread_id = true;
          ms->possible_cpus->cpus[n].props.thread_id = n;
      }

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

#3 lore raw
Quoted 90 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

   /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
   -accel kvm -machine virt,gic-version=host               \
   -cpu host                                               \
   -smp 6,sockets=2,cores=3,threads=1                      \
   -m 1024M,slots=16,maxmem=64G                            \
   -object memory-backend-ram,id=mem0,size=128M            \
   -object memory-backend-ram,id=mem1,size=128M            \
   -object memory-backend-ram,id=mem2,size=128M            \
   -object memory-backend-ram,id=mem3,size=128M            \
   -object memory-backend-ram,id=mem4,size=128M            \
   -object memory-backend-ram,id=mem4,size=384M            \
   -numa node,nodeid=0,memdev=mem0                         \
   -numa node,nodeid=1,memdev=mem1                         \
   -numa node,nodeid=2,memdev=mem2                         \
   -numa node,nodeid=3,memdev=mem3                         \
   -numa node,nodeid=4,memdev=mem4                         \
   -numa node,nodeid=5,memdev=mem5
          :
   alternatives: patching kernel code
   BUG: arch topology borken
   the CLS domain not a subset of the MC domain
   <the above error log repeats>
   BUG: arch topology borken
   the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,
and the other fixes the numa node ID issue.

Quoted 41 lines
---
  hw/arm/virt.c | 17 ++++++++++++++++-
  1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +12 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
  static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
  {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
  }
 
  static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
      int n;
      unsigned int max_cpus = ms->smp.max_cpus;
      VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
      if (ms->possible_cpus) {
          assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
          ms->possible_cpus->cpus[n].type = ms->cpu_type;
          ms->possible_cpus->cpus[n].arch_id =
              virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
Quoted 8 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2524

Quoted 2 lines
hw/arm/virt.c +2 -0
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }  

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Quoted 17 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2529

Quoted 5 lines
hw/arm/virt.c +2 -0
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
          ms->possible_cpus->cpus[n].props.has_thread_id = true;
          ms->possible_cpus->cpus[n].props.thread_id = n;
      }  

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

#4 lore raw
Quoted 100 lines · Igor Mammedov
Quoted 89 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

    /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
    -accel kvm -machine virt,gic-version=host               \
    -cpu host                                               \
    -smp 6,sockets=2,cores=3,threads=1                      \
    -m 1024M,slots=16,maxmem=64G                            \
    -object memory-backend-ram,id=mem0,size=128M            \
    -object memory-backend-ram,id=mem1,size=128M            \
    -object memory-backend-ram,id=mem2,size=128M            \
    -object memory-backend-ram,id=mem3,size=128M            \
    -object memory-backend-ram,id=mem4,size=128M            \
    -object memory-backend-ram,id=mem4,size=384M            \
    -numa node,nodeid=0,memdev=mem0                         \
    -numa node,nodeid=1,memdev=mem1                         \
    -numa node,nodeid=2,memdev=mem2                         \
    -numa node,nodeid=3,memdev=mem3                         \
    -numa node,nodeid=4,memdev=mem4                         \
    -numa node,nodeid=5,memdev=mem5
           :
    alternatives: patching kernel code
    BUG: arch topology borken
    the CLS domain not a subset of the MC domain
    <the above error log repeats>
    BUG: arch topology borken
    the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,
and the other fixes the numa node ID issue.

Quoted 40 lines
---
   hw/arm/virt.c | 17 ++++++++++++++++-
   1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +12 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
   static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
   {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
   }
 
   static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
       int n;
       unsigned int max_cpus = ms->smp.max_cpus;
       VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
       if (ms->possible_cpus) {
           assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
           ms->possible_cpus->cpus[n].type = ms->cpu_type;
           ms->possible_cpus->cpus[n].arch_id =
               virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
Quoted 8 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2524

Quoted 2 lines
hw/arm/virt.c +2 -0
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

Quoted 2 lines

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Sorry, I actually meant that we currently don't support dies for arm, so that we will always have "mc->smp_props.dies_supported == False" here, which makes the code a bit unnecessary.  dies are only supported for x86 for now. :)

Thanks,
Yanan

Quoted 18 lines
Quoted 10 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2529

Quoted 5 lines
hw/arm/virt.c +2 -0
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
           ms->possible_cpus->cpus[n].props.has_thread_id = true;
           ms->possible_cpus->cpus[n].props.thread_id = n;
       }

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Quoted 7 lines

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

.

#5 lore raw
Quoted 101 lines · On Fri, 18 Mar 2022 21:00:35 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 100 lines · Igor Mammedov
Quoted 97 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

    /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
    -accel kvm -machine virt,gic-version=host               \
    -cpu host                                               \
    -smp 6,sockets=2,cores=3,threads=1                      \
    -m 1024M,slots=16,maxmem=64G                            \
    -object memory-backend-ram,id=mem0,size=128M            \
    -object memory-backend-ram,id=mem1,size=128M            \
    -object memory-backend-ram,id=mem2,size=128M            \
    -object memory-backend-ram,id=mem3,size=128M            \
    -object memory-backend-ram,id=mem4,size=128M            \
    -object memory-backend-ram,id=mem4,size=384M            \
    -numa node,nodeid=0,memdev=mem0                         \
    -numa node,nodeid=1,memdev=mem1                         \
    -numa node,nodeid=2,memdev=mem2                         \
    -numa node,nodeid=3,memdev=mem3                         \
    -numa node,nodeid=4,memdev=mem4                         \
    -numa node,nodeid=5,memdev=mem5
           :
    alternatives: patching kernel code
    BUG: arch topology borken
    the CLS domain not a subset of the MC domain
    <the above error log repeats>
    BUG: arch topology borken
    the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,
and the other fixes the numa node ID issue.

Quoted 32 lines
---
   hw/arm/virt.c | 17 ++++++++++++++++-
   1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +4 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
   static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
   {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
   }
 
   static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
       int n;
       unsigned int max_cpus = ms->smp.max_cpus;
       VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
       if (ms->possible_cpus) {
           assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
           ms->possible_cpus->cpus[n].type = ms->cpu_type;
           ms->possible_cpus->cpus[n].arch_id =
               virt_cpu_mp_affinity(vms, n);

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2524

Quoted 10 lines
hw/arm/virt.c +10 -0
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }  

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

Quoted 8 lines
Quoted 2 lines

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Sorry, I actually meant that we currently don't support dies for arm, so that
we will always have "mc->smp_props.dies_supported == False" here, which makes the code a bit unnecessary.  dies are only supported for x86 for now. :)

then perhaps add an assert() here, so that we would notice and fix this place when dies_supported becomes true.

Quoted 21 lines

Thanks,
Yanan

Quoted 18 lines
Quoted 10 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2529

Quoted 5 lines
hw/arm/virt.c +2 -0
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
           ms->possible_cpus->cpus[n].props.has_thread_id = true;
           ms->possible_cpus->cpus[n].props.thread_id = n;
       }  

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Quoted 7 lines

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

.

#6 lore raw
Quoted 106 lines · Igor Mammedov
Quoted 103 lines · On Fri, 18 Mar 2022 21:00:35 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 102 lines · Igor Mammedov
Quoted 89 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

     /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
     -accel kvm -machine virt,gic-version=host               \
     -cpu host                                               \
     -smp 6,sockets=2,cores=3,threads=1                      \
     -m 1024M,slots=16,maxmem=64G                            \
     -object memory-backend-ram,id=mem0,size=128M            \
     -object memory-backend-ram,id=mem1,size=128M            \
     -object memory-backend-ram,id=mem2,size=128M            \
     -object memory-backend-ram,id=mem3,size=128M            \
     -object memory-backend-ram,id=mem4,size=128M            \
     -object memory-backend-ram,id=mem4,size=384M            \
     -numa node,nodeid=0,memdev=mem0                         \
     -numa node,nodeid=1,memdev=mem1                         \
     -numa node,nodeid=2,memdev=mem2                         \
     -numa node,nodeid=3,memdev=mem3                         \
     -numa node,nodeid=4,memdev=mem4                         \
     -numa node,nodeid=5,memdev=mem5
            :
     alternatives: patching kernel code
     BUG: arch topology borken
     the CLS domain not a subset of the MC domain
     <the above error log repeats>
     BUG: arch topology borken
     the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,
and the other fixes the numa node ID issue.

Quoted 40 lines
---
    hw/arm/virt.c | 17 ++++++++++++++++-
    1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +12 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 
    static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
    {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
    }
 
    static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
        int n;
        unsigned int max_cpus = ms->smp.max_cpus;
        VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
 
        if (ms->possible_cpus) {
            assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
            ms->possible_cpus->cpus[n].type = ms->cpu_type;
            ms->possible_cpus->cpus[n].arch_id =
                virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true;
+        ms->possible_cpus->cpus[n].props.socket_id =
+            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true;
+            ms->possible_cpus->cpus[n].props.die_id =
Quoted 8 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2524

Quoted 2 lines
hw/arm/virt.c +2 -0
+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads);
+        }

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Quoted 8 lines
Quoted 6 lines

Sorry, I actually meant that we currently don't support dies for arm, so that
we will always have "mc->smp_props.dies_supported == False" here, which makes the code a bit unnecessary.  dies are only supported for x86 for now. :)

then perhaps add an assert() here, so that we would notice and fix this place when dies_supported becomes true.

A simple assert() works here, I think.

Thanks,
Yanan

Quoted 21 lines
Quoted 20 lines

Thanks,
Yanan

Quoted 18 lines
Quoted 10 lines

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 2529

Quoted 5 lines
hw/arm/virt.c +2 -0
+        ms->possible_cpus->cpus[n].props.has_core_id = true;
+        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads;
            ms->possible_cpus->cpus[n].props.has_thread_id = true;
            ms->possible_cpus->cpus[n].props.thread_id = n;
        }

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Quoted 7 lines

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

.

.

author
#7 lore raw

Hi Yanan and Igor,

Quoted 106 lines · wangyanan (Y)
Quoted 105 lines · Igor Mammedov
Quoted 103 lines · On Fri, 18 Mar 2022 21:00:35 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 94 lines · Igor Mammedov
Quoted 84 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

     /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
     -accel kvm -machine virt,gic-version=host               \
     -cpu host                                               \
     -smp 6,sockets=2,cores=3,threads=1                      \
     -m 1024M,slots=16,maxmem=64G                            \
     -object memory-backend-ram,id=mem0,size=128M            \
     -object memory-backend-ram,id=mem1,size=128M            \
     -object memory-backend-ram,id=mem2,size=128M            \
     -object memory-backend-ram,id=mem3,size=128M            \
     -object memory-backend-ram,id=mem4,size=128M            \
     -object memory-backend-ram,id=mem4,size=384M            \
     -numa node,nodeid=0,memdev=mem0                         \
     -numa node,nodeid=1,memdev=mem1                         \
     -numa node,nodeid=2,memdev=mem2                         \
     -numa node,nodeid=3,memdev=mem3                         \
     -numa node,nodeid=4,memdev=mem4                         \
     -numa node,nodeid=5,memdev=mem5
            :
     alternatives: patching kernel code

     BUG: arch topology borken
     the CLS domain not a subset of the MC domain
     <the above error log repeats>
     BUG: arch topology borken

     the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,
and the other fixes the numa node ID issue.

Quoted 37 lines

---
    hw/arm/virt.c | 17 ++++++++++++++++-

    1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +4 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
     static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
     {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
     }
     static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
         int n;
         unsigned int max_cpus = ms->smp.max_cpus;
         VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
         if (ms->possible_cpus) {
             assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
             ms->possible_cpus->cpus[n].type = ms->cpu_type;

            ms->possible_cpus->cpus[n].arch_id = virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true; +        ms->possible_cpus->cpus[n].props.socket_id = +            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true; +            ms->possible_cpus->cpus[n].props.die_id =

Quoted 8 lines
Quoted 2 lines

+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads); +        }

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

Quoted 2 lines

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Sorry, I actually meant that we currently don't support dies for arm, so that
we will always have "mc->smp_props.dies_supported == False" here, which makes the code a bit unnecessary.  dies are only supported for x86 for now. :)

Quoted 4 lines
Quoted 2 lines

then perhaps add an assert() here, so that we would notice and fix this place when dies_supported becomes true.

A simple assert() works here, I think.

Ok. I will have the changes in v3. ms->smp.dies won't be included in the calculation and assert(!mc->smp_props.clusters_supported) will be added.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

+        ms->possible_cpus->cpus[n].props.has_core_id = true; +        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads; ms->possible_cpus->cpus[n].props.has_thread_id = true; ms->possible_cpus->cpus[n].props.thread_id = n; }

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86 TopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

It's a good idea, but I think it's something for future. Once the die is supported, we may have generic mechanism to generate the CPU topology based on its index or thread ID. It would be nice if the mechanism can be shared by various architectures.

In the guest, which is booted with the given command lines in the commit log, CPUs are associated with NUMA node#0/1 and no CPUs are associated with node#2/3/4/5 after the patch is applied on arm/virt machine. x86 has same behavior.

Thanks,
Gavin

author
#8 lore raw
In reply to wangyanan (Y) via

Hi Yanan,

Quoted 48 lines · wangyanan (Y)
Quoted 47 lines · Igor Mammedov
Quoted 44 lines · On Fri, 18 Mar 2022 21:00:35 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 43 lines · Igor Mammedov
Quoted 41 lines · On Fri, 18 Mar 2022 14:23:34 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 38 lines · Gavin Shan

The default CPU-to-NUMA association is given by mc->get_default_cpu_node_id() when it isn't provided explicitly. However, the CPU topology isn't fully considered in the default association and it causes CPU topology broken warnings on booting Linux guest.

For example, the following warning messages are observed when the Linux guest is booted with the following command lines.

     /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
     -accel kvm -machine virt,gic-version=host               \
     -cpu host                                               \
     -smp 6,sockets=2,cores=3,threads=1                      \
     -m 1024M,slots=16,maxmem=64G                            \
     -object memory-backend-ram,id=mem0,size=128M            \
     -object memory-backend-ram,id=mem1,size=128M            \
     -object memory-backend-ram,id=mem2,size=128M            \
     -object memory-backend-ram,id=mem3,size=128M            \
     -object memory-backend-ram,id=mem4,size=128M            \
     -object memory-backend-ram,id=mem4,size=384M            \
     -numa node,nodeid=0,memdev=mem0                         \
     -numa node,nodeid=1,memdev=mem1                         \
     -numa node,nodeid=2,memdev=mem2                         \
     -numa node,nodeid=3,memdev=mem3                         \
     -numa node,nodeid=4,memdev=mem4                         \
     -numa node,nodeid=5,memdev=mem5
            :
     alternatives: patching kernel code

     BUG: arch topology borken
     the CLS domain not a subset of the MC domain
     <the above error log repeats>
     BUG: arch topology borken

     the DIE domain not a subset of the NODE domain

With current implementation of mc->get_default_cpu_node_id(), CPU#0 to CPU#5 are associated with NODE#0 to NODE#5 separately. That's incorrect because CPU#0/1/2 should be associated with same NUMA node because they're seated in same socket.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

This fixes the issue by populating the CPU topology in virt_possible_cpu_arch_ids() and considering the socket index when default CPU-to-NUMA association is given in virt_possible_cpu_arch_ids(). With this applied, no more CPU topology broken warnings are seen from the Linux guest. The 6 CPUs are associated with NODE#0/1, but there are no CPUs associated with NODE#2/3/4/5.

It may be better to split this patch into two. One extends virt_possible_cpu_arch_ids,

Agreed, I will do in v3. Sorry that I forgot to mention it in last reply.

Thanks,
Gavin

Quoted 81 lines
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Quoted 38 lines

and the other fixes the numa node ID issue.

Quoted 37 lines

---
    hw/arm/virt.c | 17 ++++++++++++++++-

    1 file changed, 16 insertions(+), 1 deletion(-)
hw/arm/virt.c +4 -1
index 46bf7ceddf..dee02b60fc 100644
@@ -2488,7 +2488,9 @@ virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
     static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx)
     {
-    return idx % ms->numa_state->num_nodes;
+    int64_t socket_id = ms->possible_cpus->cpus[idx].props.socket_id;
+
+    return socket_id % ms->numa_state->num_nodes;
     }
     static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
@@ -2496,6 +2498,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
         int n;
         unsigned int max_cpus = ms->smp.max_cpus;
         VirtMachineState *vms = VIRT_MACHINE(ms);
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
         if (ms->possible_cpus) {
             assert(ms->possible_cpus->len == max_cpus);
@@ -2509,6 +2512,18 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
             ms->possible_cpus->cpus[n].type = ms->cpu_type;

            ms->possible_cpus->cpus[n].arch_id = virt_cpu_mp_affinity(vms, n);
+
+        ms->possible_cpus->cpus[n].props.has_socket_id = true; +        ms->possible_cpus->cpus[n].props.socket_id = +            n / (ms->smp.dies * ms->smp.clusters *
+                ms->smp.cores * ms->smp.threads);
+        if (mc->smp_props.dies_supported) {
+            ms->possible_cpus->cpus[n].props.has_die_id = true; +            ms->possible_cpus->cpus[n].props.die_id =

Quoted 8 lines
Quoted 2 lines

+                n / (ms->smp.clusters * ms->smp.cores * ms->smp.threads); +        }

I still don't think we need to consider dies if it's certainly not supported yet, IOW, we will never come into the if-branch. We are populating arm-specific topo info instead of the generic, we can probably uniformly update this part together with other necessary places when we decide to support dies for arm virt machine in the future. :)

it seems we do support dies and they are supposed to be numa boundary too, so perhaps we should account for it when generating node-id.

Quoted 8 lines
Quoted 6 lines

Sorry, I actually meant that we currently don't support dies for arm, so that
we will always have "mc->smp_props.dies_supported == False" here, which makes the code a bit unnecessary.  dies are only supported for x86 for now. :)

then perhaps add an assert() here, so that we would notice and fix this place when dies_supported becomes true.

A simple assert() works here, I think.

Thanks,
Yanan

Quoted 20 lines
Quoted 19 lines

Thanks,
Yanan

Quoted 17 lines
Quoted 10 lines
Quoted 5 lines

+        ms->possible_cpus->cpus[n].props.has_core_id = true; +        ms->possible_cpus->cpus[n].props.core_id = n / ms->smp.threads; ms->possible_cpus->cpus[n].props.has_thread_id = true; ms->possible_cpus->cpus[n].props.thread_id = n; }

Maybe we should use the same algorithm in x86_topo_ids_from_idx to populate the IDs, so that scope of socket-id will be [0, total_sockets), scope of thread-id is [0, threads_per_core), and so on. Then with a group of socket/cluster/core/thread-id, we determine a CPU.

Quoted 6 lines

Suggestion: For the long term, is it necessary now to add similar topo info infrastructure for ARM, such as X86CPUTopoInfo, X86CPUTopoIDs, x86_topo_ids_from_idx?

Thanks,
Yanan

.

.

author
#9 lore raw

PATCH v2 2/3 hw/acpi/aml-build: Use existing CPU topology to build PPTT table

In reply to Gavin Shan

When the PPTT table is built, the CPU topology is re-calculated, but it's unecessary because the CPU topology, except the cluster IDs, has been populated in virt_possible_cpu_arch_ids() on arm/virt machine.

This avoids to re-calculate the CPU topology by reusing the existing one in ms->possible_cpus. However, the cluster ID for the CPU instance has to be calculated dynamically because there is no corresponding field in struct CpuInstanceProperties. Currently, the only user of build_pptt() is arm/virt machine.

---
 hw/acpi/aml-build.c | 106 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 82 insertions(+), 24 deletions(-)
hw/acpi/aml-build.c +82 -24
index 8966e16320..572cf5fc00 100644
@@ -2002,18 +2002,27 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                 const char *oem_id, const char *oem_table_id)
 {
     MachineClass *mc = MACHINE_GET_CLASS(ms);
+    CPUArchIdList *cpus = ms->possible_cpus;
+    GQueue *socket_list = g_queue_new();
+    GQueue *cluster_list = g_queue_new();
+    GQueue *core_list = g_queue_new();
     GQueue *list = g_queue_new();
     guint pptt_start = table_data->len;
     guint parent_offset;
     guint length, i;
-    int uid = 0;
-    int socket;
+    int n, id, socket_id, cluster_id, core_id, thread_id;
     AcpiTable table = { .sig = "PPTT", .rev = 2,
                         .oem_id = oem_id, .oem_table_id = oem_table_id };
 
     acpi_table_begin(&table, table_data);
 
-    for (socket = 0; socket < ms->smp.sockets; socket++) {
+    for (n = 0; n < cpus->len; n++) {
+        socket_id = cpus->cpus[n].props.socket_id;
+        if (g_queue_find(socket_list, GUINT_TO_POINTER(socket_id))) {
+            continue;
+        }
+
+        g_queue_push_tail(socket_list, GUINT_TO_POINTER(socket_id));
         g_queue_push_tail(list,
             GUINT_TO_POINTER(table_data->len - pptt_start));
         build_processor_hierarchy_node(
@@ -2023,65 +2032,114 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
              * of a physical package
              */
             (1 << 0),
-            0, socket, NULL, 0);
+            0, socket_id, NULL, 0);
     }
 
     if (mc->smp_props.clusters_supported) {
         length = g_queue_get_length(list);
         for (i = 0; i < length; i++) {
-            int cluster;
-
             parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-            for (cluster = 0; cluster < ms->smp.clusters; cluster++) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+
+            for (n = 0; n < cpus->len; n++) {
+                if (cpus->cpus[n].props.socket_id != socket_id) {
+                    continue;
+                }
+
+                /*
+                 * We have to calculate the cluster ID because it isn't
+                 * available in the CPU instance properties.
+                 */
+                cluster_id = cpus->cpus[n].props.thread_id /
+                             (ms->smp.cores * ms->smp.threads);
+                if (g_queue_find(cluster_list, GUINT_TO_POINTER(cluster_id))) {
+                    continue;
+                }
+
+                g_queue_push_tail(cluster_list, GUINT_TO_POINTER(cluster_id));
                 g_queue_push_tail(list,
                     GUINT_TO_POINTER(table_data->len - pptt_start));
                 build_processor_hierarchy_node(
                     table_data,
                     (0 << 0), /* not a physical package */
-                    parent_offset, cluster, NULL, 0);
+                    parent_offset, cluster_id, NULL, 0);
             }
         }
     }
 
     length = g_queue_get_length(list);
     for (i = 0; i < length; i++) {
-        int core;
-
         parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (core = 0; core < ms->smp.cores; core++) {
-            if (ms->smp.threads > 1) {
-                g_queue_push_tail(list,
-                    GUINT_TO_POINTER(table_data->len - pptt_start));
-                build_processor_hierarchy_node(
-                    table_data,
-                    (0 << 0), /* not a physical package */
-                    parent_offset, core, NULL, 0);
-            } else {
+        if (!mc->smp_props.clusters_supported) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+        } else {
+            cluster_id = GPOINTER_TO_UINT(g_queue_pop_head(cluster_list));
+        }
+
+        for (n = 0; n < cpus->len; n++) {
+            if (!mc->smp_props.clusters_supported &&
+                cpus->cpus[n].props.socket_id != socket_id) {
+                continue;
+            }
+
+            /*
+             * We have to calculate the cluster ID because it isn't
+             * available in the CPU instance properties.
+             */
+            id = cpus->cpus[n].props.thread_id /
+                (ms->smp.cores * ms->smp.threads);
+            if (mc->smp_props.clusters_supported && id != cluster_id) {
+                continue;
+            }
+
+            core_id = cpus->cpus[n].props.core_id;
+            if (ms->smp.threads <= 1) {
                 build_processor_hierarchy_node(
                     table_data,
                     (1 << 1) | /* ACPI Processor ID valid */
                     (1 << 3),  /* Node is a Leaf */
-                    parent_offset, uid++, NULL, 0);
+                    parent_offset, core_id, NULL, 0);
+                continue;
             }
+
+            if (g_queue_find(core_list, GUINT_TO_POINTER(core_id))) {
+                continue;
+            }
+
+            g_queue_push_tail(core_list, GUINT_TO_POINTER(core_id));
+            g_queue_push_tail(list,
+                GUINT_TO_POINTER(table_data->len - pptt_start));
+            build_processor_hierarchy_node(
+                table_data,
+                (0 << 0), /* not a physical package */
+                parent_offset, core_id, NULL, 0);
         }
     }
 
     length = g_queue_get_length(list);
     for (i = 0; i < length; i++) {
-        int thread;
-
         parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (thread = 0; thread < ms->smp.threads; thread++) {
+        core_id = GPOINTER_TO_UINT(g_queue_pop_head(core_list));
+
+        for (n = 0; n < cpus->len; n++) {
+            if (cpus->cpus[n].props.core_id != core_id) {
+                continue;
+            }
+
+            thread_id = cpus->cpus[n].props.thread_id;
             build_processor_hierarchy_node(
                 table_data,
                 (1 << 1) | /* ACPI Processor ID valid */
                 (1 << 2) | /* Processor is a Thread */
                 (1 << 3),  /* Node is a Leaf */
-                parent_offset, uid++, NULL, 0);
+                parent_offset, thread_id, NULL, 0);
         }
     }
 
     g_queue_free(list);
+    g_queue_free(core_list);
+    g_queue_free(cluster_list);
+    g_queue_free(socket_list);
     acpi_table_end(linker, &table);
 }
 
Signature · 1 line
2.23.0
#10 lore raw

Re: [PATCH v2 2/3] hw/acpi/aml-build: Use existing CPU topology to build PPTT table

Hi Gavin,

Quoted 66 lines · Gavin Shan

When the PPTT table is built, the CPU topology is re-calculated, but it's unecessary because the CPU topology, except the cluster IDs, has been populated in virt_possible_cpu_arch_ids() on arm/virt machine.

This avoids to re-calculate the CPU topology by reusing the existing one in ms->possible_cpus. However, the cluster ID for the CPU instance has to be calculated dynamically because there is no corresponding field in struct CpuInstanceProperties. Currently, the only user of build_pptt() is arm/virt machine.

---
  hw/acpi/aml-build.c | 106 ++++++++++++++++++++++++++++++++++----------
  1 file changed, 82 insertions(+), 24 deletions(-)
hw/acpi/aml-build.c +14 -7
index 8966e16320..572cf5fc00 100644
@@ -2002,18 +2002,27 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                  const char *oem_id, const char *oem_table_id)
  {
      MachineClass *mc = MACHINE_GET_CLASS(ms);
+    CPUArchIdList *cpus = ms->possible_cpus;
+    GQueue *socket_list = g_queue_new();
+    GQueue *cluster_list = g_queue_new();
+    GQueue *core_list = g_queue_new();
      GQueue *list = g_queue_new();
      guint pptt_start = table_data->len;
      guint parent_offset;
      guint length, i;
-    int uid = 0;
-    int socket;
+    int n, id, socket_id, cluster_id, core_id, thread_id;
      AcpiTable table = { .sig = "PPTT", .rev = 2,
                          .oem_id = oem_id, .oem_table_id = oem_table_id };
 
      acpi_table_begin(&table, table_data);
 
-    for (socket = 0; socket < ms->smp.sockets; socket++) {
+    for (n = 0; n < cpus->len; n++) {
+        socket_id = cpus->cpus[n].props.socket_id;
+        if (g_queue_find(socket_list, GUINT_TO_POINTER(socket_id))) {
+            continue;
+        }
+
+        g_queue_push_tail(socket_list, GUINT_TO_POINTER(socket_id));
          g_queue_push_tail(list,
              GUINT_TO_POINTER(table_data->len - pptt_start));
          build_processor_hierarchy_node(
@@ -2023,65 +2032,114 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
               * of a physical package
               */
              (1 << 0),
-            0, socket, NULL, 0);
+            0, socket_id, NULL, 0);
      }
 
      if (mc->smp_props.clusters_supported) {
          length = g_queue_get_length(list);
          for (i = 0; i < length; i++) {
-            int cluster;
-
              parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-            for (cluster = 0; cluster < ms->smp.clusters; cluster++) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));

hw/acpi/aml-build.c void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms, · line 2052

Quoted 10 lines
hw/acpi/aml-build.c +10 -0
+
+            for (n = 0; n < cpus->len; n++) {
+                if (cpus->cpus[n].props.socket_id != socket_id) {
+                    continue;
+                }
+
+                /*
+                 * We have to calculate the cluster ID because it isn't
+                 * available in the CPU instance properties.
+                 */

Since we need cluster ID now, maybe we can simply make it supported in the CPU instance properties.

Thanks,
Yanan

Quoted 110 lines
hw/acpi/aml-build.c +58 -17
+                cluster_id = cpus->cpus[n].props.thread_id /
+                             (ms->smp.cores * ms->smp.threads);
+                if (g_queue_find(cluster_list, GUINT_TO_POINTER(cluster_id))) {
+                    continue;
+                }
+
+                g_queue_push_tail(cluster_list, GUINT_TO_POINTER(cluster_id));
                  g_queue_push_tail(list,
                      GUINT_TO_POINTER(table_data->len - pptt_start));
                  build_processor_hierarchy_node(
                      table_data,
                      (0 << 0), /* not a physical package */
-                    parent_offset, cluster, NULL, 0);
+                    parent_offset, cluster_id, NULL, 0);
              }
          }
      }
 
      length = g_queue_get_length(list);
      for (i = 0; i < length; i++) {
-        int core;
-
          parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (core = 0; core < ms->smp.cores; core++) {
-            if (ms->smp.threads > 1) {
-                g_queue_push_tail(list,
-                    GUINT_TO_POINTER(table_data->len - pptt_start));
-                build_processor_hierarchy_node(
-                    table_data,
-                    (0 << 0), /* not a physical package */
-                    parent_offset, core, NULL, 0);
-            } else {
+        if (!mc->smp_props.clusters_supported) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+        } else {
+            cluster_id = GPOINTER_TO_UINT(g_queue_pop_head(cluster_list));
+        }
+
+        for (n = 0; n < cpus->len; n++) {
+            if (!mc->smp_props.clusters_supported &&
+                cpus->cpus[n].props.socket_id != socket_id) {
+                continue;
+            }
+
+            /*
+             * We have to calculate the cluster ID because it isn't
+             * available in the CPU instance properties.
+             */
+            id = cpus->cpus[n].props.thread_id /
+                (ms->smp.cores * ms->smp.threads);
+            if (mc->smp_props.clusters_supported && id != cluster_id) {
+                continue;
+            }
+
+            core_id = cpus->cpus[n].props.core_id;
+            if (ms->smp.threads <= 1) {
                  build_processor_hierarchy_node(
                      table_data,
                      (1 << 1) | /* ACPI Processor ID valid */
                      (1 << 3),  /* Node is a Leaf */
-                    parent_offset, uid++, NULL, 0);
+                    parent_offset, core_id, NULL, 0);
+                continue;
              }
+
+            if (g_queue_find(core_list, GUINT_TO_POINTER(core_id))) {
+                continue;
+            }
+
+            g_queue_push_tail(core_list, GUINT_TO_POINTER(core_id));
+            g_queue_push_tail(list,
+                GUINT_TO_POINTER(table_data->len - pptt_start));
+            build_processor_hierarchy_node(
+                table_data,
+                (0 << 0), /* not a physical package */
+                parent_offset, core_id, NULL, 0);
          }
      }
 
      length = g_queue_get_length(list);
      for (i = 0; i < length; i++) {
-        int thread;
-
          parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (thread = 0; thread < ms->smp.threads; thread++) {
+        core_id = GPOINTER_TO_UINT(g_queue_pop_head(core_list));
+
+        for (n = 0; n < cpus->len; n++) {
+            if (cpus->cpus[n].props.core_id != core_id) {
+                continue;
+            }
+
+            thread_id = cpus->cpus[n].props.thread_id;
              build_processor_hierarchy_node(
                  table_data,
                  (1 << 1) | /* ACPI Processor ID valid */
                  (1 << 2) | /* Processor is a Thread */
                  (1 << 3),  /* Node is a Leaf */
-                parent_offset, uid++, NULL, 0);
+                parent_offset, thread_id, NULL, 0);
          }
      }
 
      g_queue_free(list);
+    g_queue_free(core_list);
+    g_queue_free(cluster_list);
+    g_queue_free(socket_list);
      acpi_table_end(linker, &table);
  }
 
#11 lore raw

Re: [PATCH v2 2/3] hw/acpi/aml-build: Use existing CPU topology to build PPTT table

Quoted 73 lines · On Fri, 18 Mar 2022 14:34:12 +0800 "wangyanan (Y)" <[email protected]>…

Hi Gavin,

Quoted 70 lines · Gavin Shan

When the PPTT table is built, the CPU topology is re-calculated, but it's unecessary because the CPU topology, except the cluster IDs, has been populated in virt_possible_cpu_arch_ids() on arm/virt machine.

This avoids to re-calculate the CPU topology by reusing the existing one in ms->possible_cpus. However, the cluster ID for the CPU instance has to be calculated dynamically because there is no corresponding field in struct CpuInstanceProperties. Currently, the only user of build_pptt() is arm/virt machine.

---
  hw/acpi/aml-build.c | 106 ++++++++++++++++++++++++++++++++++----------
  1 file changed, 82 insertions(+), 24 deletions(-)
hw/acpi/aml-build.c +18 -7
index 8966e16320..572cf5fc00 100644
@@ -2002,18 +2002,27 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                  const char *oem_id, const char *oem_table_id)
  {
      MachineClass *mc = MACHINE_GET_CLASS(ms);
+    CPUArchIdList *cpus = ms->possible_cpus;
+    GQueue *socket_list = g_queue_new();
+    GQueue *cluster_list = g_queue_new();
+    GQueue *core_list = g_queue_new();
      GQueue *list = g_queue_new();
      guint pptt_start = table_data->len;
      guint parent_offset;
      guint length, i;
-    int uid = 0;
-    int socket;
+    int n, id, socket_id, cluster_id, core_id, thread_id;
      AcpiTable table = { .sig = "PPTT", .rev = 2,
                          .oem_id = oem_id, .oem_table_id = oem_table_id };
 
      acpi_table_begin(&table, table_data);
 
-    for (socket = 0; socket < ms->smp.sockets; socket++) {
+    for (n = 0; n < cpus->len; n++) {
+        socket_id = cpus->cpus[n].props.socket_id;
+        if (g_queue_find(socket_list, GUINT_TO_POINTER(socket_id))) {
+            continue;
+        }
+
+        g_queue_push_tail(socket_list, GUINT_TO_POINTER(socket_id));
          g_queue_push_tail(list,
              GUINT_TO_POINTER(table_data->len - pptt_start));
          build_processor_hierarchy_node(
@@ -2023,65 +2032,114 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
               * of a physical package
               */
              (1 << 0),
-            0, socket, NULL, 0);
+            0, socket_id, NULL, 0);
      }
 
      if (mc->smp_props.clusters_supported) {
          length = g_queue_get_length(list);
          for (i = 0; i < length; i++) {
-            int cluster;
-
              parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-            for (cluster = 0; cluster < ms->smp.clusters; cluster++) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+
+            for (n = 0; n < cpus->len; n++) {
+                if (cpus->cpus[n].props.socket_id != socket_id) {
+                    continue;
Quoted 8 lines

hw/acpi/aml-build.c void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms, · line 2052

Quoted 6 lines
hw/acpi/aml-build.c +6 -0
+                }
+
+                /*
+                 * We have to calculate the cluster ID because it isn't
+                 * available in the CPU instance properties.
+                 */  

Since we need cluster ID now, maybe we can simply make it supported in the CPU instance properties.

agreed

Quoted 114 lines

Thanks,
Yanan

Quoted 110 lines
hw/acpi/aml-build.c +58 -17
+                cluster_id = cpus->cpus[n].props.thread_id /
+                             (ms->smp.cores * ms->smp.threads);
+                if (g_queue_find(cluster_list, GUINT_TO_POINTER(cluster_id))) {
+                    continue;
+                }
+
+                g_queue_push_tail(cluster_list, GUINT_TO_POINTER(cluster_id));
                  g_queue_push_tail(list,
                      GUINT_TO_POINTER(table_data->len - pptt_start));
                  build_processor_hierarchy_node(
                      table_data,
                      (0 << 0), /* not a physical package */
-                    parent_offset, cluster, NULL, 0);
+                    parent_offset, cluster_id, NULL, 0);
              }
          }
      }
 
      length = g_queue_get_length(list);
      for (i = 0; i < length; i++) {
-        int core;
-
          parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (core = 0; core < ms->smp.cores; core++) {
-            if (ms->smp.threads > 1) {
-                g_queue_push_tail(list,
-                    GUINT_TO_POINTER(table_data->len - pptt_start));
-                build_processor_hierarchy_node(
-                    table_data,
-                    (0 << 0), /* not a physical package */
-                    parent_offset, core, NULL, 0);
-            } else {
+        if (!mc->smp_props.clusters_supported) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+        } else {
+            cluster_id = GPOINTER_TO_UINT(g_queue_pop_head(cluster_list));
+        }
+
+        for (n = 0; n < cpus->len; n++) {
+            if (!mc->smp_props.clusters_supported &&
+                cpus->cpus[n].props.socket_id != socket_id) {
+                continue;
+            }
+
+            /*
+             * We have to calculate the cluster ID because it isn't
+             * available in the CPU instance properties.
+             */
+            id = cpus->cpus[n].props.thread_id /
+                (ms->smp.cores * ms->smp.threads);
+            if (mc->smp_props.clusters_supported && id != cluster_id) {
+                continue;
+            }
+
+            core_id = cpus->cpus[n].props.core_id;
+            if (ms->smp.threads <= 1) {
                  build_processor_hierarchy_node(
                      table_data,
                      (1 << 1) | /* ACPI Processor ID valid */
                      (1 << 3),  /* Node is a Leaf */
-                    parent_offset, uid++, NULL, 0);
+                    parent_offset, core_id, NULL, 0);
+                continue;
              }
+
+            if (g_queue_find(core_list, GUINT_TO_POINTER(core_id))) {
+                continue;
+            }
+
+            g_queue_push_tail(core_list, GUINT_TO_POINTER(core_id));
+            g_queue_push_tail(list,
+                GUINT_TO_POINTER(table_data->len - pptt_start));
+            build_processor_hierarchy_node(
+                table_data,
+                (0 << 0), /* not a physical package */
+                parent_offset, core_id, NULL, 0);
          }
      }
 
      length = g_queue_get_length(list);
      for (i = 0; i < length; i++) {
-        int thread;
-
          parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (thread = 0; thread < ms->smp.threads; thread++) {
+        core_id = GPOINTER_TO_UINT(g_queue_pop_head(core_list));
+
+        for (n = 0; n < cpus->len; n++) {
+            if (cpus->cpus[n].props.core_id != core_id) {
+                continue;
+            }
+
+            thread_id = cpus->cpus[n].props.thread_id;
              build_processor_hierarchy_node(
                  table_data,
                  (1 << 1) | /* ACPI Processor ID valid */
                  (1 << 2) | /* Processor is a Thread */
                  (1 << 3),  /* Node is a Leaf */
-                parent_offset, uid++, NULL, 0);
+                parent_offset, thread_id, NULL, 0);
          }
      }
 
      g_queue_free(list);
+    g_queue_free(core_list);
+    g_queue_free(cluster_list);
+    g_queue_free(socket_list);
      acpi_table_end(linker, &table);
  }
 
author
#12 lore raw

Re: [PATCH v2 2/3] hw/acpi/aml-build: Use existing CPU topology to build PPTT table

Hi Igor and Yanan,

Quoted 82 lines · Igor Mammedov
Quoted 71 lines · On Fri, 18 Mar 2022 14:34:12 +0800 "wangyanan (Y)" <[email protected]>…
Quoted 70 lines · Gavin Shan

When the PPTT table is built, the CPU topology is re-calculated, but it's unecessary because the CPU topology, except the cluster IDs, has been populated in virt_possible_cpu_arch_ids() on arm/virt machine.

This avoids to re-calculate the CPU topology by reusing the existing one in ms->possible_cpus. However, the cluster ID for the CPU instance has to be calculated dynamically because there is no corresponding field in struct CpuInstanceProperties. Currently, the only user of build_pptt() is arm/virt machine.

---
   hw/acpi/aml-build.c | 106 ++++++++++++++++++++++++++++++++++----------
   1 file changed, 82 insertions(+), 24 deletions(-)
hw/acpi/aml-build.c +18 -7
index 8966e16320..572cf5fc00 100644
@@ -2002,18 +2002,27 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                   const char *oem_id, const char *oem_table_id)
   {
       MachineClass *mc = MACHINE_GET_CLASS(ms);
+    CPUArchIdList *cpus = ms->possible_cpus;
+    GQueue *socket_list = g_queue_new();
+    GQueue *cluster_list = g_queue_new();
+    GQueue *core_list = g_queue_new();
       GQueue *list = g_queue_new();
       guint pptt_start = table_data->len;
       guint parent_offset;
       guint length, i;
-    int uid = 0;
-    int socket;
+    int n, id, socket_id, cluster_id, core_id, thread_id;
       AcpiTable table = { .sig = "PPTT", .rev = 2,
                           .oem_id = oem_id, .oem_table_id = oem_table_id };
 
       acpi_table_begin(&table, table_data);
 
-    for (socket = 0; socket < ms->smp.sockets; socket++) {
+    for (n = 0; n < cpus->len; n++) {
+        socket_id = cpus->cpus[n].props.socket_id;
+        if (g_queue_find(socket_list, GUINT_TO_POINTER(socket_id))) {
+            continue;
+        }
+
+        g_queue_push_tail(socket_list, GUINT_TO_POINTER(socket_id));
           g_queue_push_tail(list,
               GUINT_TO_POINTER(table_data->len - pptt_start));
           build_processor_hierarchy_node(
@@ -2023,65 +2032,114 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                * of a physical package
                */
               (1 << 0),
-            0, socket, NULL, 0);
+            0, socket_id, NULL, 0);
       }
 
       if (mc->smp_props.clusters_supported) {
           length = g_queue_get_length(list);
           for (i = 0; i < length; i++) {
-            int cluster;
-
               parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-            for (cluster = 0; cluster < ms->smp.clusters; cluster++) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+
+            for (n = 0; n < cpus->len; n++) {
+                if (cpus->cpus[n].props.socket_id != socket_id) {
+                    continue;
Quoted 8 lines

hw/acpi/aml-build.c void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms, · line 2052

Quoted 6 lines
hw/acpi/aml-build.c +6 -0
+                }
+
+                /*
+                 * We have to calculate the cluster ID because it isn't
+                 * available in the CPU instance properties.
+                 */

Since we need cluster ID now, maybe we can simply make it supported in the CPU instance properties.

Quoted 2 lines

agreed

Thanks for your review. I will add it in v3. FYI, the addition needs to be done in the PATCH[v3 01/04] where the CPU topology is populated :)

Thanks,
Gavin

Quoted 112 lines
Quoted 111 lines
Quoted 110 lines
hw/acpi/aml-build.c +58 -17
+                cluster_id = cpus->cpus[n].props.thread_id /
+                             (ms->smp.cores * ms->smp.threads);
+                if (g_queue_find(cluster_list, GUINT_TO_POINTER(cluster_id))) {
+                    continue;
+                }
+
+                g_queue_push_tail(cluster_list, GUINT_TO_POINTER(cluster_id));
                   g_queue_push_tail(list,
                       GUINT_TO_POINTER(table_data->len - pptt_start));
                   build_processor_hierarchy_node(
                       table_data,
                       (0 << 0), /* not a physical package */
-                    parent_offset, cluster, NULL, 0);
+                    parent_offset, cluster_id, NULL, 0);
               }
           }
       }
 
       length = g_queue_get_length(list);
       for (i = 0; i < length; i++) {
-        int core;
-
           parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (core = 0; core < ms->smp.cores; core++) {
-            if (ms->smp.threads > 1) {
-                g_queue_push_tail(list,
-                    GUINT_TO_POINTER(table_data->len - pptt_start));
-                build_processor_hierarchy_node(
-                    table_data,
-                    (0 << 0), /* not a physical package */
-                    parent_offset, core, NULL, 0);
-            } else {
+        if (!mc->smp_props.clusters_supported) {
+            socket_id = GPOINTER_TO_UINT(g_queue_pop_head(socket_list));
+        } else {
+            cluster_id = GPOINTER_TO_UINT(g_queue_pop_head(cluster_list));
+        }
+
+        for (n = 0; n < cpus->len; n++) {
+            if (!mc->smp_props.clusters_supported &&
+                cpus->cpus[n].props.socket_id != socket_id) {
+                continue;
+            }
+
+            /*
+             * We have to calculate the cluster ID because it isn't
+             * available in the CPU instance properties.
+             */
+            id = cpus->cpus[n].props.thread_id /
+                (ms->smp.cores * ms->smp.threads);
+            if (mc->smp_props.clusters_supported && id != cluster_id) {
+                continue;
+            }
+
+            core_id = cpus->cpus[n].props.core_id;
+            if (ms->smp.threads <= 1) {
                   build_processor_hierarchy_node(
                       table_data,
                       (1 << 1) | /* ACPI Processor ID valid */
                       (1 << 3),  /* Node is a Leaf */
-                    parent_offset, uid++, NULL, 0);
+                    parent_offset, core_id, NULL, 0);
+                continue;
               }
+
+            if (g_queue_find(core_list, GUINT_TO_POINTER(core_id))) {
+                continue;
+            }
+
+            g_queue_push_tail(core_list, GUINT_TO_POINTER(core_id));
+            g_queue_push_tail(list,
+                GUINT_TO_POINTER(table_data->len - pptt_start));
+            build_processor_hierarchy_node(
+                table_data,
+                (0 << 0), /* not a physical package */
+                parent_offset, core_id, NULL, 0);
           }
       }
 
       length = g_queue_get_length(list);
       for (i = 0; i < length; i++) {
-        int thread;
-
           parent_offset = GPOINTER_TO_UINT(g_queue_pop_head(list));
-        for (thread = 0; thread < ms->smp.threads; thread++) {
+        core_id = GPOINTER_TO_UINT(g_queue_pop_head(core_list));
+
+        for (n = 0; n < cpus->len; n++) {
+            if (cpus->cpus[n].props.core_id != core_id) {
+                continue;
+            }
+
+            thread_id = cpus->cpus[n].props.thread_id;
               build_processor_hierarchy_node(
                   table_data,
                   (1 << 1) | /* ACPI Processor ID valid */
                   (1 << 2) | /* Processor is a Thread */
                   (1 << 3),  /* Node is a Leaf */
-                parent_offset, uid++, NULL, 0);
+                parent_offset, thread_id, NULL, 0);
           }
       }
 
       g_queue_free(list);
+    g_queue_free(core_list);
+    g_queue_free(cluster_list);
+    g_queue_free(socket_list);
       acpi_table_end(linker, &table);
   }
 
author
#13 lore raw

PATCH v2 3/3 hw/arm/virt: Unify ACPI processor ID in MADT and SRAT table

In reply to Gavin Shan

The value of the following field has been used in ACPI PPTT table to identify the corresponding processor. This takes the same field as the ACPI processor ID in MADT and SRAT tables.

ms->possible_cpus->cpus[i].props.thread_id

---
 hw/arm/virt-acpi-build.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
hw/arm/virt-acpi-build.c +9 -3
index 449fab0080..7fedb56eea 100644
@@ -534,13 +534,16 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
 
     for (i = 0; i < cpu_list->len; ++i) {
         uint32_t nodeid = cpu_list->cpus[i].props.node_id;
+        uint32_t thread_id = cpu_list->cpus[i].props.thread_id;
+
         /*
          * 5.2.16.4 GICC Affinity Structure
          */
         build_append_int_noprefix(table_data, 3, 1);      /* Type */
         build_append_int_noprefix(table_data, 18, 1);     /* Length */
         build_append_int_noprefix(table_data, nodeid, 4); /* Proximity Domain */
-        build_append_int_noprefix(table_data, i, 4); /* ACPI Processor UID */
+        build_append_int_noprefix(table_data,
+                                  thread_id, 4); /* ACPI Processor UID */
         /* Flags, Table 5-76 */
         build_append_int_noprefix(table_data, 1 /* Enabled */, 4);
         build_append_int_noprefix(table_data, 0, 4); /* Clock Domain */
@@ -704,6 +707,7 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
 {
     int i;
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     const MemMapEntry *memmap = vms->memmap;
     AcpiTable table = { .sig = "APIC", .rev = 3, .oem_id = vms->oem_id,
                         .oem_table_id = vms->oem_table_id };
@@ -725,8 +729,9 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     build_append_int_noprefix(table_data, vms->gic_version, 1);
     build_append_int_noprefix(table_data, 0, 3);   /* Reserved */
 
-    for (i = 0; i < MACHINE(vms)->smp.cpus; i++) {
+    for (i = 0; i < ms->smp.cpus; i++) {
         ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(i));
+        uint32_t thread_id = ms->possible_cpus->cpus[i].props.thread_id;
         uint64_t physical_base_address = 0, gich = 0, gicv = 0;
         uint32_t vgic_interrupt = vms->virt ? PPI(ARCH_GIC_MAINT_IRQ) : 0;
         uint32_t pmu_interrupt = arm_feature(&armcpu->env, ARM_FEATURE_PMU) ?
@@ -743,7 +748,8 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         build_append_int_noprefix(table_data, 76, 1);   /* Length */
         build_append_int_noprefix(table_data, 0, 2);    /* Reserved */
         build_append_int_noprefix(table_data, i, 4);    /* GIC ID */
-        build_append_int_noprefix(table_data, i, 4);    /* ACPI Processor UID */
+        build_append_int_noprefix(table_data,
+                                  thread_id, 4);        /* ACPI Processor UID */
         /* Flags */
         build_append_int_noprefix(table_data, 1, 4);    /* Enabled */
         /* Parking Protocol Version */
Signature · 1 line
2.23.0
author
#14 lore raw
In reply to Gavin Shan

Hi Igor,

Quoted 13 lines · Gavin Shan

When the CPU-to-NUMA association isn't provided by user, the default NUMA node ID for the specific CPU is returned from virt_get_default_cpu_node_id(). Unfortunately, the default NUMA node ID breaks socket boundary and leads to the broken CPU topology warning message in Linux guest. This series intends to fix the issue.

PATCH[1/3]: Fixes the broken CPU topology by considering the socket boundary
when the default NUMA node ID is calculated.
PATCH[2/3]: Use the existing CPU topology to build PPTT table. However, the
cluster ID has to be calculated dynamically because there is no corresponding information in CPU instance properties.
PATCH[3/3]: Take thread ID as the ACPI processor ID in MDAT and SRAT tables.

Quoted 10 lines

Changelog
=========
v2:
* Populate the CPU topology in virt_possible_cpu_arch_ids() so that it
can be reused in virt_get_default_cpu_node_id() (Igor)
* Added PATCH[2/3] to use the existing CPU topology when PPTT table
is built (Igor)
* Added PATCH[3/3] to take thread ID as ACPI processor ID in MADT and
SRAT table (Gavin)

Kindly ping. Could you help to review when you have free cycles? :)

Thanks,
Gavin