Skip to contentSkip to conversation

i386: fixup number of logical CPUs when host-cache-info=on

PATCH In review Igor Mammedov 4 participants 12 messages May 24, 2022 – Jun 1, 2022

lore
Share & tools
lore
#0 lore raw

cover i386: fixup number of logical CPUs when host-cache-info=on

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/2 x86: cpu: make sure number of addressable IDs for processor cores meets the spec

Accourding Intel's CPUID[EAX=04H] resulting bits 31 - 26 in EAX
should be:
"
**** The nearest power-of-2 integer that is not smaller than (1 + EAX[31:26]) is the number of unique
Core_IDs reserved for addressing different processor cores in a physical package. Core ID is a subset of bits of the initial APIC ID.
"

ensure that values stored in EAX[31-26] always meets this condition.

---
 target/i386/cpu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
target/i386/cpu.c +1 -1
index 35c3475e6c..bbe37dce2e 100644
@@ -5279,7 +5279,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
             /* QEMU gives out its own APIC IDs, never pass down bits 31..26.  */
             *eax &= ~0xFC000000;
             if ((*eax & 31) && cs->nr_cores > 1) {
-                *eax |= (cs->nr_cores - 1) << 26;
+                *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
             }
         } else if (cpu->vendor_cpuid_only && IS_AMD_CPU(env)) {
             *eax = *ebx = *ecx = *edx = 0;
Signature · 1 line
2.31.1
#2 lore raw

PATCH 2/2 x86: cpu: fixup number of addressable IDs for logical processors sharing cache

In reply to Igor Mammedov

When QEMU is started with '-cpu host,host-cache-info=on', it will passthrough host's number of logical processors sharing cache and number of processor cores in the physical package. QEMU already fixes up the later to correctly reflect number of configured cores for VM, however number of logical processors sharing cache is still comes from host CPU, which confuses guest started with:

       -machine q35,accel=kvm \
       -cpu host,host-cache-info=on,l3-cache=off \
       -smp 20,sockets=2,dies=1,cores=10,threads=1  \
       -numa node,nodeid=0,memdev=ram-node0 \
       -numa node,nodeid=1,memdev=ram-node1 \
       -numa cpu,socket-id=0,node-id=0 \
       -numa cpu,socket-id=1,node-id=1

on 2 socket Xeon 4210R host with 10 cores per socket with CPUID[04H]:

      ...
        --- cache 3 ---
      cache type                           = unified cache (3)
      cache level                          = 0x3 (3)
      self-initializing cache level        = true
      fully associative cache              = false
      maximum IDs for CPUs sharing cache   = 0x1f (31)
      maximum IDs for cores in pkg         = 0xf (15)
      ...

that doesn't match number of logical processors VM was configured with and as result RHEL 9.0 guest complains:

sched: CPU #10's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.

   WARNING: CPU: 10 PID: 0 at arch/x86/kernel/smpboot.c:421 topology_sane.isra.0+0x67/0x80
   ...
   Call Trace:
     set_cpu_sibling_map+0x176/0x590
     start_secondary+0x5b/0x150
     secondary_startup_64_no_verify+0xc2/0xcb

Fix it by capping max number of logical processors to vcpus/socket as it was configured, which fixes the issue.

---
PS:
capping to logical cpus/socket was arbitrarily chosen (maybe it should be per die or something else but don't see that in spec)

---
 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
target/i386/cpu.c +16 -4
index bbe37dce2e..ffb274dcf6 100644
@@ -5276,10 +5276,22 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         /* cache info: needed for Core compatibility */
         if (cpu->cache_info_passthrough) {
             x86_cpu_get_cache_cpuid(index, count, eax, ebx, ecx, edx);
-            /* QEMU gives out its own APIC IDs, never pass down bits 31..26.  */
-            *eax &= ~0xFC000000;
-            if ((*eax & 31) && cs->nr_cores > 1) {
-                *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
+            /*
+             * QEMU has its own number of cores/logical cpus,
+             * set 24..14, 31..26 bit to configured values
+             */
+            if (*eax & 31) {
+                int host_vcpus_per_cache = 1 + ((*eax & 0x3FFC000) >> 14);
+                int vcpus_per_socket = env->nr_dies * cs->nr_cores *
+                                       cs->nr_threads;
+                if (cs->nr_cores > 1) {
+                    *eax &= ~0xFC000000;
+                    *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
+                }
+                if (host_vcpus_per_cache > vcpus_per_socket) {
+                    *eax &= ~0x3FFC000;
+                    *eax |= (pow2ceil(vcpus_per_socket) - 1) << 14;
+                }
             }
         } else if (cpu->vendor_cpuid_only && IS_AMD_CPU(env)) {
             *eax = *ebx = *ecx = *edx = 0;
Signature · 1 line
2.31.1
#3 lore raw
In reply to Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

Quoted 9 lines

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
#4 lore raw
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Am I missing something?

Quoted 10 lines
Quoted 10 lines

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 2 lines
Thanks
Babu Moger
#5 lore raw
Quoted 14 lines · Babu
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

Quoted 8 lines

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Am I missing something?

Hi Babu,

Hopefully this will help you reproduce the issue if you are testing on Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year, while I was working on patches for encoding the topology CPUID leaf in different Zen platforms.

What I found from my experiments on Milan, is that the warning will appear whenever the NUMA topology requested in QEMU cmdline assigns a number of CPUs to each node that is smaller than the default # of CPUs sharing a LLC on the host platform. In short, on a Milan host where we have 16 CPUs sharing a CCX:

# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7,128-135

If a guest is launched with the following arguments:

-cpu host,+topoext \
-smp cpus=64,cores=32,threads=2,sockets=1 \
-numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
-numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
-numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
-numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \

it assigns 8 cpus to each NUMA node, causing the error above to be displayed.

Note that ultimately the guest topology is built based on the NUMA information, so the LLC domains on the guest only end up spanning a single NUMA node. e.g.:

# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7

Hope that helps,
Alejandro

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

  target/i386/cpu.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
#6 lore raw
Quoted 31 lines · Alejandro Jimenez
Quoted 15 lines · Babu
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

Quoted 8 lines

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Am I missing something?

Hi Babu,

Hopefully this will help you reproduce the issue if you are testing on Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year, while I was working on patches for encoding the topology CPUID leaf in different Zen platforms.

Quoted 5 lines

What I found from my experiments on Milan, is that the warning will appear whenever the NUMA topology requested in QEMU cmdline assigns a number of CPUs to each node that is smaller than the default # of CPUs sharing a LLC on the host platform. In short, on a Milan host where we have 16 CPUs sharing a CCX:

Yes. I recreated the issue with this following command line.

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,+topoext -smp 16,sockets=1,dies=1,cores=16,threads=1 -object memory-backend-ram,id=ram-node0,size=2G -object memory-backend-ram,id=ram-node1,size=2G  -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa node,nodeid=1,cpus=8-15,memdev=ram-node1

But solving this will be bit complicated. For AMD, this information comes from CPUID 0x8000001d. But, when this cpuid is being populated we don't have all the information about numa nodes etc..

But you can work-around it by modifying the command line by including dies(dies=2 in this case) information.  Something like this.

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,+topoext,host-cache-info=on -smp 16,sockets=1,dies=2,cores=8,threads=1 -object memory-backend-ram,id=ram-node0,size=2G -object memory-backend-ram,id=ram-node1,size=2G  -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa node,nodeid=1,cpus=8-15,memdev=ram-node1

But this may not be acceptable solution in all the cases.

Quoted 38 lines
# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7,128-135

If a guest is launched with the following arguments:

(quoted diff) +0 -6
-cpu host,+topoext \
-smp cpus=64,cores=32,threads=2,sockets=1 \
-numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
-numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
-numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
-numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \

it assigns 8 cpus to each NUMA node, causing the error above to be displayed.

Note that ultimately the guest topology is built based on the NUMA information, so the LLC domains on the guest only end up spanning a single NUMA node. e.g.:

# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7

Hope that helps,
Alejandro

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Igor Mammedov (2):
   x86: cpu: make sure number of addressable IDs for processor cores meets the spec
   x86: cpu: fixup number of addressable IDs for logical processors sharing cache

  target/i386/cpu.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 2 lines
Thanks
Babu Moger
#7 lore raw
Quoted 49 lines · Babu
Quoted 31 lines · Alejandro Jimenez
Quoted 15 lines · Babu
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

Quoted 8 lines

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Am I missing something?

Hi Babu,

Hopefully this will help you reproduce the issue if you are testing on Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year, while I was working on patches for encoding the topology CPUID leaf in different Zen platforms.

Quoted 5 lines

What I found from my experiments on Milan, is that the warning will appear whenever the NUMA topology requested in QEMU cmdline assigns a number of CPUs to each node that is smaller than the default # of CPUs sharing a LLC on the host platform. In short, on a Milan host where we have 16 CPUs sharing a CCX:

Yes. I recreated the issue with this following command line.

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,+topoext -smp 16,sockets=1,dies=1,cores=16,threads=1 -object memory-backend-ram,id=ram-node0,size=2G -object memory-backend-ram,id=ram-node1,size=2G  -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa node,nodeid=1,cpus=8-15,memdev=ram-node1

Quoted 6 lines

But solving this will be bit complicated. For AMD, this information comes from CPUID 0x8000001d. But, when this cpuid is being populated we don't have all the information about numa nodes etc..

But you can work-around it by modifying the command line by including dies(dies=2 in this case) information.  Something like this.

Makes sense; using dies=2 makes it so the cache topology leaf is built with 8cores/CCX, matching the # of NUMA nodes so all is well.

Quoted 11 lines

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,+topoext,host-cache-info=on -smp 16,sockets=1,dies=2,cores=8,threads=1 -object memory-backend-ram,id=ram-node0,size=2G -object memory-backend-ram,id=ram-node1,size=2G  -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa node,nodeid=1,cpus=8-15,memdev=ram-node1

But this may not be acceptable solution in all the cases.

This is not specific to host-cache-info behavior so it is probably better to discuss it separately. With that being said...

The idea that I considered was to automatically calculate a value of 'dies' iff a explicit value was not requested via the '-smp' options, instead of just using the current default of dies=1. i.e. automatically mimic the host cache topology in the guest so that if we are running on Rome, the guest OS sees 4cores/CCX, but when running on Milan it sees 8cores/CCX. This can be done by querying the host CPUID and using that info to build the guest CPUID leaf in QEMU, similar to what Igor is doing here but also adjusting the number of dies that is encoded.

I built prototype code that seemed to work correctly, but did not consider the complication added by '-numa' options.

I think there is a much larger debate involved about what defaults are "sane", so rather than derailing this thread more, I'll send a follow up message in the future when I can take another look at the prototype patches I have.

Thank you,
Alejandro

Quoted 39 lines
Quoted 39 lines

# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7,128-135

If a guest is launched with the following arguments:

(quoted diff) +0 -6
-cpu host,+topoext \
-smp cpus=64,cores=32,threads=2,sockets=1 \
-numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
-numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
-numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
-numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \

it assigns 8 cpus to each NUMA node, causing the error above to be displayed.

Note that ultimately the guest topology is built based on the NUMA information, so the LLC domains on the guest only end up spanning a single NUMA node. e.g.:

# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7

Hope that helps,
Alejandro

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Igor Mammedov (2):
   x86: cpu: make sure number of addressable IDs for processor cores meets the spec
   x86: cpu: fixup number of addressable IDs for logical processors sharing cache

   target/i386/cpu.c | 20 ++++++++++++++++----
   1 file changed, 16 insertions(+), 4 deletions(-)
#8 lore raw
In reply to Moger, Babu
Quoted 13 lines · Babu
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

Quoted 8 lines

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Am I missing something?

Yep, sorry I've omitted -object memory-backend-foo definitions for ram-node0 and ram-node1

one can use any memory backend, it doesn't really matter in this case, for example following should do:
-object memory-backend-ram,id=ram-node0,size=2G \
-object memory-backend-ram,id=ram-node1,size=2G

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
#9 lore raw
Quoted 22 lines · Igor Mammedov
Quoted 19 lines · Babu
Quoted 4 lines · Igor Mammedov

On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <[email protected]> wrote:

CCing AMD folks as that might be of interest to them

I am trying to recreate the bug on my AMD system here.. Seeing this message..

qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0 is ambiguous

Here is my command line..

#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net nic  -nographic -machine q35,accel=kvm -cpu host,host-cache-info=on,l3-cache=off -smp 20,sockets=2,dies=1,cores=10,threads=1 -numa node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1

Quoted 8 lines
Quoted 1 line

Am I missing something?

Yep, sorry I've omitted -object memory-backend-foo definitions for ram-node0 and ram-node1

one can use any memory backend, it doesn't really matter in this case, for example following should do:
-object memory-backend-ram,id=ram-node0,size=2G \
-object memory-backend-ram,id=ram-node1,size=2G

Thanks Igor. However these changes(patch 1 and 2) does not affect AMD systems as far i can see.

Thanks

Babu

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 2 lines
Thanks
Babu Moger
#10 lore raw
In reply to Igor Mammedov

Paolo,
can you pick this up if it looks fine, please?

Quoted 9 lines · On Tue, 24 May 2022 11:10:18 -0400 Igor Mammedov <[email protected]> wrote:

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

 target/i386/cpu.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
#11 lore raw
Quoted 16 lines · Igor Mammedov

Paolo,
can you pick this up if it looks fine, please?

Quoted 9 lines · On Tue, 24 May 2022 11:10:18 -0400 Igor Mammedov <[email protected]> wrote:

Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache

  target/i386/cpu.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)

Yup, queued now. Thanks,

Paolo