vcpu handle MWAIT different

help Longpeng (Mike) 2 participants 4 messages Jun 9, 2017 KVM

Share & tools

Longpeng (Mike)

author

2017-06-09 03:49 UTC #0 lore raw

Hi guys,

I wrote a testcase which repeat execute MWAIT insn in kvm-unit-test and I tested it on different hardware, but got different results.

1) cpu: E5620 (hardware don't support MONITOR/MWAIT)

This works well because I saw the following info in trace:

       CPU 0/KVM-7301  [015] .... 1534719.099573: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
       CPU 0/KVM-7301  [015] d... 1534719.099573: kvm_entry: vcpu 0
       CPU 0/KVM-7301  [015] .... 1534719.099574: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
       CPU 0/KVM-7301  [015] d... 1534719.099574: kvm_entry: vcpu 0

2) cpu: E5-2620 v3 and E5-2690 v4 (both hardwares support MONITOR/MWAIT)

This...

       CPU 0/KVM-1683  [017] .... 4844125.856164: kvm_exit: reason EXCEPTION_NMI
rip 0x400382 info 0 80000306
       CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_emulate_insn: 0:400382:0f
01 c9 (prot64)
       CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_inj_exception: #UD (0x0)
       CPU 0/KVM-1683  [017] d... 4844125.856167: kvm_entry: vcpu 0

and the kvm-unit-test guest reported:

Unhandled exception 6 #UD at ip 0000000000400387
error_code=0000 rflags=00010002 cs=00000008
rax=0000000000000000 rcx=0000000000000000 rdx=0000000200000000 rbx=0000000000000000 rbp=000000000044db60 rsi=0000000000000001 rdi=0000000000004020
r8=000000000000000a r9=00000000000003f8 r10=000000000044d1e2 r11=000000000044d1e1
r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 cr0=0000000080010011 cr2=0000000000000000 cr3=0000000007fff000 cr4=0000000000000020 cr8=0000000000000000

	STACK: @400387 40028f
0x0000000000400387: mwait at x86/perf_exit.c:17
        	/* "mwait %eax, %ecx;" */
      > 	asm volatile(".byte 0x0f, 0x01, 0xc9;"
        		:: "a" (eax), "c" (ecx));
 (inlined by) main at x86/perf_exit.c:93
        		mb();
      > 		mwait(0, 0);
        		mb();

0x000000000040028e: start64 at x86/cstart64.S:206
lea __argv(%rip), %rsi
> call main
mov %eax, %edi

In result 2, the trace shows that vcpu is in 64bit protect-mode (the testcase didn't jump to ring3 manually, so it's in ring0), as intel vol2 said the MWAIT will #UD in this situation when:

#UD If the current privilege level is not 0.

	If CPUID.01H:ECX.MONITOR[bit 3] = 0.

I think CPUID.01H:ECX.MONITOR does not matter, so I'm so curious about the result 2 .

Do you have any idea ? Thanks.

PS: Wanpeng tested this on his laptop (maybe the cpu is Core-i7) and it worked well as result 1.

kvm version : 4.4.11
host kernel : 3.10

The testcase is:

#include "libcflat.h"
#include "smp.h"
#include "vm.h"
#include "processor.h"
#include "desc.h"
#include "apic.h"
#include "apic-defs.h"
#include "asm/barrier.h"

[...]

static inline void mwait(unsigned long eax, unsigned long ecx)

{
        /* "mwait %eax, %ecx;" */
        asm volatile(".byte 0x0f, 0x01, 0xc9;"
                :: "a" (eax), "c" (ecx));
}

[...]

int main(int argc, char **argv)

{
        setup_vm();
        smp_init();

	while (test_cnt--) {
                [...]
                mwait(0, 0);
                [...]
        }
	[...]
}

Signature · 2 lines

Regards,
Longpeng(Mike)

Longpeng (Mike)

author

2017-06-09 03:55 UTC #1 lore raw

Quoted 20 lines · Longpeng (Mike)

Hi guys,
I wrote a testcase which repeat execute MWAIT insn in kvm-unit-test and I tested it on different hardware, but got different results.
1) cpu: E5620 (hardware don't support MONITOR/MWAIT)
This works well because I saw the following info in trace:
       CPU 0/KVM-7301  [015] .... 1534719.099573: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
       CPU 0/KVM-7301  [015] d... 1534719.099573: kvm_entry: vcpu 0
       CPU 0/KVM-7301  [015] .... 1534719.099574: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
       CPU 0/KVM-7301  [015] d... 1534719.099574: kvm_entry: vcpu 0
2) cpu: E5-2620 v3 and E5-2690 v4 (both hardwares support MONITOR/MWAIT)

Quoted 9 lines

This...

       CPU 0/KVM-1683  [017] .... 4844125.856164: kvm_exit: reason EXCEPTION_NMI
rip 0x400382 info 0 80000306
       CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_emulate_insn: 0:400382:0f
01 c9 (prot64)
       CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_inj_exception: #UD (0x0)
       CPU 0/KVM-1683  [017] d... 4844125.856167: kvm_entry: vcpu 0

Sorry, I made a mistake, the trace of result 2 is:

       CPU 0/KVM-14333 [017] .... 4842658.749717: kvm_exit: reason EXCEPTION_NMI
rip 0x400387 info 0 80000306
       CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_emulate_insn: 0:400387:0f
01 c9 (prot64)
       CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_inj_exception: #UD (0x0)
       CPU 0/KVM-14333 [017] d... 4842658.749720: kvm_entry: vcpu 0

Quoted 79 lines

and the kvm-unit-test guest reported:
Unhandled exception 6 #UD at ip 0000000000400387
error_code=0000 rflags=00010002 cs=00000008
rax=0000000000000000 rcx=0000000000000000 rdx=0000000200000000 rbx=0000000000000000 rbp=000000000044db60 rsi=0000000000000001 rdi=0000000000004020
r8=000000000000000a r9=00000000000003f8 r10=000000000044d1e2 r11=000000000044d1e1
r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 cr0=0000000080010011 cr2=0000000000000000 cr3=0000000007fff000 cr4=0000000000000020 cr8=0000000000000000
	STACK: @400387 40028f
0x0000000000400387: mwait at x86/perf_exit.c:17
        	/* "mwait %eax, %ecx;" */
      > 	asm volatile(".byte 0x0f, 0x01, 0xc9;"
        		:: "a" (eax), "c" (ecx));
 (inlined by) main at x86/perf_exit.c:93
        		mb();
      > 		mwait(0, 0);
        		mb();
0x000000000040028e: start64 at x86/cstart64.S:206
lea __argv(%rip), %rsi
> call main
mov %eax, %edi
In result 2, the trace shows that vcpu is in 64bit protect-mode (the testcase didn't jump to ring3 manually, so it's in ring0), as intel vol2 said the MWAIT will #UD in this situation when:
#UD If the current privilege level is not 0.
	If CPUID.01H:ECX.MONITOR[bit 3] = 0.
I think CPUID.01H:ECX.MONITOR does not matter, so I'm so curious about the result 2 .
Do you have any idea ? Thanks.
PS: Wanpeng tested this on his laptop (maybe the cpu is Core-i7) and it worked well as result 1.
kvm version : 4.4.11
host kernel : 3.10
The testcase is:
#include "libcflat.h"
#include "smp.h"
#include "vm.h"
#include "processor.h"
#include "desc.h"
#include "apic.h"
#include "apic-defs.h"
#include "asm/barrier.h"
[...]
static inline void mwait(unsigned long eax, unsigned long ecx)
{
        /* "mwait %eax, %ecx;" */
        asm volatile(".byte 0x0f, 0x01, 0xc9;"
                :: "a" (eax), "c" (ecx));
}
[...]
int main(int argc, char **argv)
{
        setup_vm();
        smp_init();

	while (test_cnt--) {
                [...]
                mwait(0, 0);
                [...]
        }
	[...]
}

Signature · 2 lines

Regards,
Longpeng(Mike)

Nadav Amit

2017-06-09 04:51 UTC #2 lore raw

Maybe you disabled MWAIT it in the BIOS. Try to read IA32_MISC_ENABLE[18] (MSR 0x1a0). If it is clear, MWAIT/MONITOR is disabled.

Quoted 130 lines · Longpeng (Mike)

Quoted 20 lines · Longpeng (Mike)
Hi guys,
I wrote a testcase which repeat execute MWAIT insn in kvm-unit-test and I tested it on different hardware, but got different results.
1) cpu: E5620 (hardware don't support MONITOR/MWAIT)
This works well because I saw the following info in trace:
      CPU 0/KVM-7301  [015] .... 1534719.099573: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
      CPU 0/KVM-7301  [015] d... 1534719.099573: kvm_entry: vcpu 0
      CPU 0/KVM-7301  [015] .... 1534719.099574: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
      CPU 0/KVM-7301  [015] d... 1534719.099574: kvm_entry: vcpu 0
2) cpu: E5-2620 v3 and E5-2690 v4 (both hardwares support MONITOR/MWAIT)
Quoted 9 lines
This...
      CPU 0/KVM-1683  [017] .... 4844125.856164: kvm_exit: reason EXCEPTION_NMI
rip 0x400382 info 0 80000306
      CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_emulate_insn: 0:400382:0f
01 c9 (prot64)
      CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_inj_exception: #UD (0x0)
      CPU 0/KVM-1683  [017] d... 4844125.856167: kvm_entry: vcpu 0
Sorry, I made a mistake, the trace of result 2 is:
      CPU 0/KVM-14333 [017] .... 4842658.749717: kvm_exit: reason EXCEPTION_NMI
rip 0x400387 info 0 80000306
      CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_emulate_insn: 0:400387:0f
01 c9 (prot64)
      CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_inj_exception: #UD (0x0)
      CPU 0/KVM-14333 [017] d... 4842658.749720: kvm_entry: vcpu 0
Quoted 79 lines
and the kvm-unit-test guest reported:
Unhandled exception 6 #UD at ip 0000000000400387
error_code=0000 rflags=00010002 cs=00000008
rax=0000000000000000 rcx=0000000000000000 rdx=0000000200000000 rbx=0000000000000000 rbp=000000000044db60 rsi=0000000000000001 rdi=0000000000004020 r8=000000000000000a r9=00000000000003f8 r10=000000000044d1e2 r11=000000000044d1e1 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 cr0=0000000080010011 cr2=0000000000000000 cr3=0000000007fff000 cr4=0000000000000020 cr8=0000000000000000
	STACK: @400387 40028f
0x0000000000400387: mwait at x86/perf_exit.c:17
       	/* "mwait %eax, %ecx;" */
Quoted 1 line
	asm volatile(".byte 0x0f, 0x01, 0xc9;"
       		:: "a" (eax), "c" (ecx));
(inlined by) main at x86/perf_exit.c:93
       		mb();
Quoted 1 line
		mwait(0, 0);
       		mb();
0x000000000040028e: start64 at x86/cstart64.S:206
       	lea __argv(%rip), %rsi
Quoted 1 line
call main
mov %eax, %edi
In result 2, the trace shows that vcpu is in 64bit protect-mode (the testcase didn't jump to ring3 manually, so it's in ring0), as intel vol2 said the MWAIT will #UD in this situation when:
#UD If the current privilege level is not 0.
	If CPUID.01H:ECX.MONITOR[bit 3] = 0.
I think CPUID.01H:ECX.MONITOR does not matter, so I'm so curious about the result 2 .
Do you have any idea ? Thanks.
PS: Wanpeng tested this on his laptop (maybe the cpu is Core-i7) and it worked well as result 1.
kvm version : 4.4.11
host kernel : 3.10
The testcase is:
#include "libcflat.h"
#include "smp.h"
#include "vm.h"
#include "processor.h"
#include "desc.h"
#include "apic.h"
#include "apic-defs.h"
#include "asm/barrier.h"
[...]
static inline void mwait(unsigned long eax, unsigned long ecx)
{
       /* "mwait %eax, %ecx;" */
       asm volatile(".byte 0x0f, 0x01, 0xc9;"
               :: "a" (eax), "c" (ecx));
}
[...]
int main(int argc, char **argv)
{
       setup_vm();
       smp_init();

	while (test_cnt--) {
               [...]
               mwait(0, 0);
               [...]
       }
	[...]
}
Signature · 2 lines
Regards,
Longpeng(Mike)

Longpeng (Mike)

author

2017-06-09 06:00 UTC #3 lore raw

Hi Nadav,

Thanks for your suggestion, you're right.

I checked my hardwares, the E5620/E5-2620 v3/E5-2690 v4 all support MONITOR/MWAIT, but my E5-2620 server and E5-2690 server disable MWAIT in the BIOS, and E5620 server enable. So I made a mistake, sorry guys.

Signature · 2 lines

Regards,
Longpeng(Mike)

Quoted 140 lines · Nadav Amit

Maybe you disabled MWAIT it in the BIOS. Try to read IA32_MISC_ENABLE[18] (MSR 0x1a0). If it is clear, MWAIT/MONITOR is disabled.
Quoted 116 lines · Longpeng (Mike)
Quoted 20 lines · Longpeng (Mike)
Hi guys,
I wrote a testcase which repeat execute MWAIT insn in kvm-unit-test and I tested it on different hardware, but got different results.
1) cpu: E5620 (hardware don't support MONITOR/MWAIT)
This works well because I saw the following info in trace:
      CPU 0/KVM-7301  [015] .... 1534719.099573: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
      CPU 0/KVM-7301  [015] d... 1534719.099573: kvm_entry: vcpu 0
      CPU 0/KVM-7301  [015] .... 1534719.099574: kvm_exit: reason
MWAIT_INSTRUCTION rip 0x400387 info 0 0
      CPU 0/KVM-7301  [015] d... 1534719.099574: kvm_entry: vcpu 0
2) cpu: E5-2620 v3 and E5-2690 v4 (both hardwares support MONITOR/MWAIT)
Quoted 9 lines
This...
      CPU 0/KVM-1683  [017] .... 4844125.856164: kvm_exit: reason EXCEPTION_NMI
rip 0x400382 info 0 80000306
      CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_emulate_insn: 0:400382:0f
01 c9 (prot64)
      CPU 0/KVM-1683  [017] .... 4844125.856166: kvm_inj_exception: #UD (0x0)
      CPU 0/KVM-1683  [017] d... 4844125.856167: kvm_entry: vcpu 0
Sorry, I made a mistake, the trace of result 2 is:
      CPU 0/KVM-14333 [017] .... 4842658.749717: kvm_exit: reason EXCEPTION_NMI
rip 0x400387 info 0 80000306
      CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_emulate_insn: 0:400387:0f
01 c9 (prot64)
      CPU 0/KVM-14333 [017] .... 4842658.749719: kvm_inj_exception: #UD (0x0)
      CPU 0/KVM-14333 [017] d... 4842658.749720: kvm_entry: vcpu 0
Quoted 70 lines
and the kvm-unit-test guest reported:
Unhandled exception 6 #UD at ip 0000000000400387
error_code=0000 rflags=00010002 cs=00000008
rax=0000000000000000 rcx=0000000000000000 rdx=0000000200000000 rbx=0000000000000000 rbp=000000000044db60 rsi=0000000000000001 rdi=0000000000004020 r8=000000000000000a r9=00000000000003f8 r10=000000000044d1e2 r11=000000000044d1e1 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 cr0=0000000080010011 cr2=0000000000000000 cr3=0000000007fff000 cr4=0000000000000020 cr8=0000000000000000
	STACK: @400387 40028f
0x0000000000400387: mwait at x86/perf_exit.c:17
       	/* "mwait %eax, %ecx;" */
Quoted 1 line
	asm volatile(".byte 0x0f, 0x01, 0xc9;"
       		:: "a" (eax), "c" (ecx));
(inlined by) main at x86/perf_exit.c:93
       		mb();
Quoted 1 line
		mwait(0, 0);
       		mb();
0x000000000040028e: start64 at x86/cstart64.S:206
       	lea __argv(%rip), %rsi
Quoted 1 line
call main
mov %eax, %edi
In result 2, the trace shows that vcpu is in 64bit protect-mode (the testcase didn't jump to ring3 manually, so it's in ring0), as intel vol2 said the MWAIT will #UD in this situation when:
#UD If the current privilege level is not 0.
	If CPUID.01H:ECX.MONITOR[bit 3] = 0.
I think CPUID.01H:ECX.MONITOR does not matter, so I'm so curious about the result 2 .
Do you have any idea ? Thanks.
PS: Wanpeng tested this on his laptop (maybe the cpu is Core-i7) and it worked well as result 1.
kvm version : 4.4.11
host kernel : 3.10
The testcase is:
#include "libcflat.h"
#include "smp.h"
#include "vm.h"
#include "processor.h"
#include "desc.h"
#include "apic.h"
#include "apic-defs.h"
#include "asm/barrier.h"
[...]
static inline void mwait(unsigned long eax, unsigned long ecx)
{
       /* "mwait %eax, %ecx;" */
       asm volatile(".byte 0x0f, 0x01, 0xc9;"
               :: "a" (eax), "c" (ecx));
}
[...]
int main(int argc, char **argv)
{
       setup_vm();
       smp_init();
Quoted 14 lines
Quoted 9 lines
	while (test_cnt--) {
               [...]
               mwait(0, 0);
               [...]
       }
	[...]
}
Signature · 2 lines
Regards,
Longpeng(Mike)
.

Signature · 2 lines

Regards,
Longpeng(Mike)