Windows2012 as Guest 64+cores on KVM Halts

Dear all:
I try to boot windows2012R2 (more than 64 cores)on kvm platform with hyper-v on*<relaxed state='on'/>*. the guest halts on the starting time.
Also tested in these cases:

<topology sockets='64' cores='1' threads='1'/>  - boot sucess
<topology sockets='1' cores='64' threads='1'/>  - boot sucess
<topology sockets='1' cores='1' threads='64'/>  - boot sucess
<topology sockets='1' cores='68' threads='1'/>  - boot failed
<topology sockets='68' cores='1' threads='1'/>  - boot sucess(in Guest we can found 64)
<topology sockets='1' cores='1' threads='68'/>  - boot failed

Environment:
GUEST:
windows2012R2
HOST:

   kernel version:4.10.0-rc4
   QEMU version:QEMU emulator version 2.8.50 (v2.8.0-664-gd1c82f7-dirty)

part of GuestXML:
<vcpu placement='static'>68</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> <boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>

     <relaxed state='on'/>

</hyperv>
</features>
<cpu>
<topology sockets='1' cores='68' threads='1'/>
</cpu>

//////////////////////////////////
debug info:
virsh qemu-monitor-command win2012 --hmp "info cpus"
CPU #61: pc=0xfffff8031296a21c thread_id=21410 CPU #62: pc=0xfffff8031296a21c thread_id=21411 CPU #63: pc=0xfffff8031296a21c thread_id=21414 CPU #64: pc=0x00000000000fd31c (halted) thread_id=21415 CPU #65: pc=0x00000000000fd31c (halted) thread_id=21416 CPU #66: pc=0x00000000000fd31c (halted) thread_id=21417 CPU #67: pc=0x00000000000fd31c (halted) thread_id=21418

Any suggestions?

Thanks a lot.

Gaohuai Han

Behalf Of Yan Vugenfirer

Hi,

First off all to use more than 64 CPUs on Windows you should use CPU grouping. Each group can contain maximum 64 CPUs. Here is a good article explaining possible configurations:
https://msdn.microsoft.com/en-us/windows/hardware/drivers/devtest/boot-p
arameters-to-test-drivers-for-multiple-processor-group-support
<https://msdn.microsoft.com/en-us/windows/hardware/drivers/devtest/boot-
parameters-to-test-drivers-for-multiple-processor-group-support> and another one on the grouping subject:
https://blogs.technet.microsoft.com/mlucas/2012/03/15/windows-server-and-
processor-cores/
<https://blogs.technet.microsoft.com/mlucas/2012/03/15/windows-server-an
d-processor-cores/>

Regarding the boot failure - it is quite possible that Windows cannot handle more that 64 logical CPUs on one physical CPU.

Best regards,
Yan.

Quoted 4 lines · hangaohuai

Dear all:
I try to boot windows2012R2 (more than 64 cores)on kvm platform with

hyper-v on*<relaxed state='on'/>*.

Quoted 7 lines

the guest halts on the starting time.
Also tested in these cases:

<topology sockets='64' cores='1' threads='1'/>  - boot sucess
<topology sockets='1' cores='64' threads='1'/>  - boot sucess
<topology sockets='1' cores='1' threads='64'/>  - boot sucess
<topology sockets='1' cores='68' threads='1'/>  - boot failed
<topology sockets='68' cores='1' threads='1'/>  - boot sucess(in Guest we

can found 64)

Quoted 8 lines

<topology sockets='1' cores='1' threads='68'/> - boot failed

Environment:
GUEST:
windows2012R2
HOST:
kernel version:4.10.0-rc4
QEMU version:QEMU emulator version 2.8.50

(v2.8.0-664-gd1c82f7-dirty)

Quoted 40 lines

part of GuestXML:
<vcpu placement='static'>68</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> <boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>

     <relaxed state='on'/>

</hyperv>
</features>
<cpu>
<topology sockets='1' cores='68' threads='1'/>
</cpu>

//////////////////////////////////
debug info:
virsh qemu-monitor-command win2012 --hmp "info cpus"
CPU #61: pc=0xfffff8031296a21c thread_id=21410 CPU #62: pc=0xfffff8031296a21c thread_id=21411 CPU #63: pc=0xfffff8031296a21c thread_id=21414 CPU #64: pc=0x00000000000fd31c (halted) thread_id=21415 CPU #65: pc=0x00000000000fd31c (halted) thread_id=21416 CPU #66: pc=0x00000000000fd31c (halted) thread_id=21417 CPU #67: pc=0x00000000000fd31c (halted) thread_id=21418

Any suggestions?

Thanks a lot.

Gaohuai Han

But We tested the same cases on Xen platform and VMware, and the guest booted successfully.

Today I also tested on tcg, the guest also boots successfully.

BTW, if we disable hyper-v, there isn't problem on boot for the case.

QEMU command line (enable hyper-v with hv_relaxed):

# ./qemu-system-x86_64 -name win2012 -machine accel=kvm,usb=off -cpu qemu64,hv_relaxed -m 4001 -realtime mlock=off -smp 68,sockets=1,cores=68,threads=1 \

(quoted diff) +0 -3

-drive file=/mnt/sdb/gonglei/win2012_r2_dc_64_raw,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-boot menu=on -device piix3-usb-uhci,id=uhci -chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios \
-monitor stdio -device cirrus-vga,id=video0,vgamem_mb=16 -vnc 0.0.0.0:10

So, I think it's possible a KVM bug on Hyper-V enlightenments support.

Any help will be appreciated!

CC'ing Paolo, Radim and Vadim.

Quoted 2 lines

But We tested the same cases on Xen platform and VMware, and the guest booted successfully.

Were these two also tested with enlightenments enabled? TCG surely isn't.

Paolo

Quoted 7 lines

Today I also tested on tcg, the guest also boots successfully.

BTW, if we disable hyper-v, there isn't problem on boot for the case.

QEMU command line (enable hyper-v with hv_relaxed):

# ./qemu-system-x86_64 -name win2012 -machine accel=kvm,usb=off -cpu

qemu64,hv_relaxed -m 4001 -realtime mlock=off -smp 68,sockets=1,cores=68,threads=1 \

Quoted 1 line

-drive

file=/mnt/sdb/gonglei/win2012_r2_dc_64_raw,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \

Quoted 1 line

-boot menu=on -device piix3-usb-uhci,id=uhci -chardev

file,id=seabios,path=/home/seabios.log -device
isa-debugcon,iobase=0x402,chardev=seabios \

Quoted 7 lines

-monitor stdio -device cirrus-vga,id=video0,vgamem_mb=16 -vnc 0.0.0.0:10

So, I think it's possible a KVM bug on Hyper-V enlightenments support.

Any help will be appreciated!

CC'ing Paolo, Radim and Vadim.

Quoted 5 lines · Gonglei (Arei)

Quoted 2 lines

But We tested the same cases on Xen platform and VMware, and the guest booted successfully.

Quoted 1 line

Were these two also tested with enlightenments enabled? TCG surely isn't.

About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from the below QEMU Command line, I thought the hyper-V enabled then. Sorry about that.

But for Xen, we set 'viridian=1' which be thought the Hyper-V is enabled.

For VMWare we also enabled the Hyper-V enlightenments.

Quoted 11 lines

Quoted 10 lines · Gonglei (Arei)

Quoted 3 lines

But We tested the same cases on Xen platform and VMware, and the guest booted successfully.

Were these two also tested with enlightenments enabled?  TCG surely isn't.

Quoted 8 lines

About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from the below QEMU
Command line, I thought the hyper-V enabled then. Sorry about that.

But for Xen, we set 'viridian=1' which be thought the Hyper-V is enabled.

For VMWare we also enabled the Hyper-V enlightenments.

Ok, thanks for the clarifications!

Paolo

Quoted 15 lines

Quoted 15 lines

Quoted 10 lines · Gonglei (Arei)

Quoted 3 lines

But We tested the same cases on Xen platform and VMware, and the guest booted successfully.

Were these two also tested with enlightenments enabled?  TCG surely isn't.

About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from the below QEMU
Command line, I thought the hyper-V enabled then. Sorry about that.

Quoted 4 lines

Quoted 4 lines

But for Xen, we set 'viridian=1' which be thought the Hyper-V is enabled.

For VMWare we also enabled the Hyper-V enlightenments.

If I'm not mistaken, even Hyper-V server doesn't allow specify more than 64 vCPUs for Generation 1 VMs.

In any case, if you are only interested in hv_relaxed, you can drop it off for WS2012 as long as you have cpu hypervisor flag (CPUID.1:ECX [bit 31]=1) turned on.

Hi,

Quoted 25 lines · Paolo Bonzini

Quoted 22 lines

Quoted 22 lines

Quoted 15 lines · Gonglei (Arei)

Quoted 5 lines

But We tested the same cases on Xen platform and VMware, and
the guest booted successfully.

Were these two also tested with enlightenments enabled?  TCG surely isn't.

About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from the
below QEMU
Command line, I thought the hyper-V enabled then. Sorry about that.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

But for Xen, we set 'viridian=1' which be thought the Hyper-V is
enabled.

For VMWare we also enabled the Hyper-V enlightenments.

If I'm not mistaken, even Hyper-V server doesn't allow specify more than 64 vCPUs for Generation 1 VMs.

Normally yes, but I found the explanation from Microsoft document about it:

Maximum Supported Virtual Processors

On Windows operating systems versions through Windows Server 2008 R2,
reporting the HV#1 hypervisor interface limits the Windows virtual machine
to a maximum of 64 VPs, regardless of what is reported via CPUID.40000005.EAX.
Starting with Windows Server 2012 and Windows 8, if CPUID.40000005.EAX
contains a value of -1, Windows assumes that the hypervisor imposes no specific
limit to the number of VPs. In this case, Windows Server 2012 guest VMs may
use more than 64 VPs, up to the maximum supported number of processors
applicable to the specific Windows version being used.

Link https://docs.microsoft.com/en-us/virtualization/hyper-v-on-wind

ows/reference/tlfs

"Requirements for Implementing the Microsoft Hypervisor Interface"

And the below patch works for me, I can support max 255 vcpus for WS2012
with hyper-v enlightenments.

target/i386/kvm.c +0 -0

index 27fd050..efe3cbc 100644

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

In any case, if you are only interested in hv_relaxed, you can drop it
off for WS2012 as long as you have cpu hypervisor flag (CPUID.1:ECX [bit 31]=1) turned on.

hy_relaxed is just a example of enabling hyperv-v enlightenments.

Thanks,
-Gonglei

And the below patch works for me, I can support max 255 vcpus for WS2012 with hyper-v enlightenments.

target/i386/kvm.c +0 -0

index 27fd050..efe3cbc 100644

Quoted 15 lines

And the below patch works for me, I can support max 255 vcpus for WS2012 with hyper-v enlightenments.

target/i386/kvm.c +1 -1

index 27fd050..efe3cbc 100644
@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;
 

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 1 line

target/i386/kvm.c +0 -0

         kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

you should submit the patch for 2.9.

Hi Paolo,

Quoted 18 lines · Gonglei (Arei)

Quoted 15 lines

And the below patch works for me, I can support max 255 vcpus for WS2012 with hyper-v enlightenments.

target/i386/kvm.c +1 -1

index 27fd050..efe3cbc 100644
@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;
 

Quoted 3 lines

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 1 line

target/i386/kvm.c +0 -0

         kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

I don't know why. Because the negative effects for this change don't exist on current QEMU IIUC, and we don't have compatible problems for live migration.

Quoted 2 lines

you should submit the patch for 2.9.

Thanks,
-Gonglei

Quoted 6 lines

Hi Paolo,

Quoted 4 lines · Gonglei (Arei)

Quoted 1 line

And the below patch works for me, I can support max 255 vcpus for

WS2012

Quoted 18 lines

Quoted 17 lines

Quoted 7 lines

with hyper-v enlightenments.

target/i386/kvm.c +0 -0

index 27fd050..efe3cbc 100644

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 8 lines

target/i386/kvm.c +1 -1

@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;
 
         kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

Quoted 2 lines

I don't know why. Because the negative effects for this change don't exist on current QEMU IIUC, and we don't have compatible problems for live

migration.

CPUID should never change with the same machine type and command line.

Paolo

Quoted 6 lines

Quoted 2 lines

you should submit the patch for 2.9.

Thanks,
-Gonglei

Quoted 27 lines · Gonglei (Arei)

Quoted 6 lines

Hi Paolo,

Quoted 4 lines · Gonglei (Arei)

Quoted 1 line

And the below patch works for me, I can support max 255 vcpus for

WS2012

Quoted 18 lines

Quoted 17 lines

Quoted 7 lines

with hyper-v enlightenments.

target/i386/kvm.c +0 -0

index 27fd050..efe3cbc 100644

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 8 lines

target/i386/kvm.c +1 -1

@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;
 
         kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

Quoted 6 lines

Quoted 2 lines

I don't know why. Because the negative effects for this change don't exist on current QEMU IIUC, and we don't have compatible problems for live

migration.

CPUID should never change with the same machine type and command line.

OK, then how do we add compat code for the CPUID change? It's not a visible property.

Quoted 29 lines

Quoted 29 lines · Gonglei (Arei)

Quoted 6 lines

Hi Paolo,

Quoted 4 lines · Gonglei (Arei)

Quoted 1 line

And the below patch works for me, I can support max 255 vcpus for

WS2012

Quoted 20 lines

Quoted 14 lines

Quoted 14 lines

with hyper-v enlightenments.

target/i386/kvm.c +1 -1

index 27fd050..efe3cbc 100644
@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;
 

Quoted 3 lines

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 1 line

target/i386/kvm.c +0 -0

         kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

I don't know why. Because the negative effects for this change don't exist on current QEMU IIUC, and we don't have compatible problems for live

Quoted 5 lines

Quoted 4 lines

migration.

CPUID should never change with the same machine type and command line.

OK, then how do we add compat code for the CPUID change? It's not a visible

property.

You can add a new property, something like hv-cpuid-limits-eax.

Quoted 32 lines · Gonglei (Arei)

Quoted 30 lines

Quoted 21 lines · Gonglei (Arei)

Quoted 5 lines

Hi Paolo,

Quoted 3 lines · Gonglei (Arei)

Quoted 1 line

And the below patch works for me, I can support max 255 vcpus for

WS2012

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

with hyper-v enlightenments.

target/i386/kvm.c +1 -1

index 27fd050..efe3cbc 100644
@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
          c = &cpuid_data.entries[cpuid_i++];
          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
          c->ebx = 0x40;
 

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 1 line

target/i386/kvm.c +0 -0

          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

I don't know why. Because the negative effects for this change don't exist on current QEMU IIUC, and we don't have compatible problems for live

migration.

CPUID should never change with the same machine type and command line.

OK, then how do we add compat code for the CPUID change? It's not a visible

Quoted 4 lines

property.

You can add a new property, something like hv-cpuid-limits-eax.

Sounds good. Let me try. :)

Thanks,
-Gonglei

-----Original Message-----
From: Vadim Rozenfeld [mailto:[email protected]]
Sent: Tuesday, September 05, 2017 2:33 PM
To: Gonglei (Arei); Paolo Bonzini
Cc: Hangaohuai; Yan Vugenfirer; [email protected]; [email protected]; Radim Krcmar
Subject: Re: [Qemu-devel] [Help] Windows2012 as Guest 64+cores on KVM Halts

Quoted 23 lines · Gonglei (Arei)

Quoted 23 lines · Gonglei (Arei)

Quoted 21 lines

Quoted 21 lines · Gonglei (Arei)

Quoted 5 lines

Hi Paolo,

Quoted 3 lines · Gonglei (Arei)

Quoted 1 line

And the below patch works for me, I can support max 255 vcpus for

WS2012

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

with hyper-v enlightenments.

target/i386/kvm.c +1 -1

index 27fd050..efe3cbc 100644
@@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
          c = &cpuid_data.entries[cpuid_i++];
          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
          c->ebx = 0x40;
 

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 2 lines

target/i386/kvm.c int kvm_arch_init_vcpu(CPUState *cs) · line 778

Quoted 1 line

target/i386/kvm.c +0 -0

          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think

I don't know why. Because the negative effects for this change don't

exist

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 1 line

on current QEMU IIUC, and we don't have compatible problems for live

migration.

CPUID should never change with the same machine type and command

line.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

OK, then how do we add compat code for the CPUID change? It's not a

visible

Quoted 8 lines

Quoted 4 lines

property.

You can add a new property, something like hv-cpuid-limits-eax.

Sounds good. Let me try. :)

Thanks,
-Gonglei

Hi Gonglei,
Any update on this issue?

We've found that SVVP on WS2016 is passing successfully with the above change
and more than 64 vCPUs defined.

Vadim.