regression in v3.9? a guest stuck in BIOS if emulate_invalid_guest_state=Y

DISCUSSION Jun'ichi Nomura 3 participants 11 messages May 7, 2013 – May 10, 2013 KVM

Share & tools

Jun'ichi Nomura

author

2013-05-07 10:02 UTC #0 lore raw

Hello,

With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)

The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.

Is this a kernel bug? Or is the userspace just too old to run on v3.9?

# virsh start vm12; virsh console vm12
Domain vm12 started

Connected to domain vm12
Escape character is ^]

Google, Inc.
Serial Graphics Adapter 07/26/11
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ ([email protected]) Tue Jul 26 15:05:08 UTC 2011 Term:
<Ctrl-]>

# virsh qemu-monitor-command vm12 --hmp info registers
EAX=f000e850 EBX=0000ffff ECX=00000000 EDX=0000ffff
ESI=00000104 EDI=00000060 EBP=00000000 ESP=00006eb0
EIP=000006ca EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0040 00000400 ffffffff 00809300
CS =c000 000c0000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0040 00000400 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000fc558 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

# cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state 
Y
# cat /sys/module/kvm_intel/parameters/unrestricted_guest
N

Signature · 1 line

Jun'ichi Nomura, NEC Corporation

Gleb Natapov

2013-05-07 10:35 UTC #1 lore raw

Quoted 7 lines · Jun'ichi Nomura

Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)

Quoted 6 lines

The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?

It should work. What seabios version are you using?

Quoted 39 lines

# virsh start vm12; virsh console vm12
Domain vm12 started

Connected to domain vm12
Escape character is ^]

Google, Inc.
Serial Graphics Adapter 07/26/11
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ ([email protected]) Tue Jul 26 15:05:08 UTC 2011 Term:
<Ctrl-]>

# virsh qemu-monitor-command vm12 --hmp info registers
EAX=f000e850 EBX=0000ffff ECX=00000000 EDX=0000ffff
ESI=00000104 EDI=00000060 EBP=00000000 ESP=00006eb0
EIP=000006ca EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0040 00000400 ffffffff 00809300
CS =c000 000c0000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0040 00000400 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000fc558 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

Quoted 11 lines

# cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state 
Y
# cat /sys/module/kvm_intel/parameters/unrestricted_guest
N

Signature · 5 lines

Jun'ichi Nomura, NEC Corporation
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Gleb.

Jun'ichi Nomura

author

2013-05-07 11:05 UTC #2 lore raw

Quoted 8 lines · Gleb Natapov

Quoted 7 lines · Jun'ichi Nomura
Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)

Quoted 7 lines

Quoted 6 lines
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?
It should work. What seabios version are you using?

It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)

Quoted 44 lines

# virsh start vm12; virsh console vm12
Domain vm12 started

Connected to domain vm12
Escape character is ^]

Google, Inc.
Serial Graphics Adapter 07/26/11
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ ([email protected]) Tue Jul 26 15:05:08 UTC 2011 Term:
<Ctrl-]>

# virsh qemu-monitor-command vm12 --hmp info registers
EAX=f000e850 EBX=0000ffff ECX=00000000 EDX=0000ffff
ESI=00000104 EDI=00000060 EBP=00000000 ESP=00006eb0
EIP=000006ca EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0040 00000400 ffffffff 00809300
CS =c000 000c0000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0040 00000400 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000fc558 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

# cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state 
Y
# cat /sys/module/kvm_intel/parameters/unrestricted_guest
N

Signature · 1 line

Jun'ichi Nomura, NEC Corporation

Gleb Natapov

2013-05-07 12:06 UTC #3 lore raw

Quoted 12 lines · Jun'ichi Nomura

Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jun'ichi Nomura
Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.

Quoted 8 lines

Quoted 4 lines
Quoted 3 lines
Is this a kernel bug? Or is the userspace just too old to run on v3.9?
It should work. What seabios version are you using?
It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)

Works for me, but it looks like in your case the bios is in option rom. What are you booting from? What's the qemu command line?

What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?

--
Gleb.

Paolo Bonzini

maintainer

2013-05-07 14:46 UTC #4 lore raw

Il 07/05/2013 14:06, Gleb Natapov ha scritto:

Quoted 16 lines · Jun'ichi Nomura

Quoted 15 lines · Gleb Natapov
Quoted 14 lines · Jun'ichi Nomura
Quoted 13 lines
Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?

Quoted 6 lines

Quoted 5 lines
Quoted 1 line
It should work. What seabios version are you using?
It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)
Works for me, but it looks like in your case the bios is in option rom.

What version of vgabios, too?

Quoted 4 lines

What are you booting from? What's the qemu command line?
What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?

Paolo

Gleb Natapov

2013-05-07 15:04 UTC #5 lore raw

Quoted 17 lines · Paolo Bonzini

Il 07/05/2013 14:06, Gleb Natapov ha scritto:
Quoted 16 lines · Jun'ichi Nomura
Quoted 15 lines · Gleb Natapov
Quoted 14 lines · Jun'ichi Nomura
Quoted 13 lines
Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?

Quoted 9 lines

Quoted 6 lines
Quoted 5 lines
Quoted 1 line
It should work. What seabios version are you using?
It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)
Works for me, but it looks like in your case the bios is in option rom.
What version of vgabios, too?

Heh, good call, actually he uses sgabios. It is right there in the output :). Need to try that.

Quoted 6 lines

Quoted 4 lines
What are you booting from? What's the qemu command line?
What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?
Paolo

--
Gleb.

Paolo Bonzini

maintainer

2013-05-07 16:11 UTC #6 lore raw

----- Messaggio originale -----

Quoted 29 lines

Da: "Gleb Natapov" <[email protected]>
A: "Paolo Bonzini" <[email protected]>
Cc: "Jun'ichi Nomura" <[email protected]>, [email protected]
Inviato: Martedì, 7 maggio 2013 17:04:15
Oggetto: Re: regression in v3.9? a guest stuck in BIOS if emulate_invalid_guest_state=Y
Quoted 22 lines · Paolo Bonzini
Il 07/05/2013 14:06, Gleb Natapov ha scritto:
Quoted 21 lines · Jun'ichi Nomura
Quoted 20 lines · Gleb Natapov
Quoted 8 lines · Jun'ichi Nomura
Quoted 7 lines
Hello,
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
Quoted 7 lines
Quoted 6 lines
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?
It should work. What seabios version are you using?
It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)

Quoted 6 lines

Quoted 4 lines
Quoted 1 line
Works for me, but it looks like in your case the bios is in option rom.
What version of vgabios, too?
Heh, good call, actually he uses sgabios. It is right there in the output :). Need to try that.

sgabios is in addition to vgabios, not instead of it.

Paolo

Jun'ichi Nomura

author

2013-05-08 03:22 UTC #7 lore raw

In reply to Paolo Bonzini

Quoted 15 lines · Paolo Bonzini

Il 07/05/2013 14:06, Gleb Natapov ha scritto:
Quoted 14 lines · Jun'ichi Nomura
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jun'ichi Nomura
Quoted 11 lines
With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?

Quoted 8 lines

Quoted 6 lines
Quoted 5 lines
Quoted 1 line
It should work. What seabios version are you using?
It's seabios-0.6.1.2-26.el6.
(and qemu-kvm-0.12.1.2-2.355.el6)
Works for me, but it looks like in your case the bios is in option rom.
What version of vgabios, too?

vgabios-0.6b-3.7.el6
sgabios-bin-0-0.3.20110621svn.el6

Quoted 3 lines

Quoted 1 line
What are you booting from? What's the qemu command line?
From disk image.

Command line is like this:
/usr/libexec/qemu-kvm -name vm12 -S -M rhel6.3.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -uuid 539b7cdb-7afc-4237-bdbc-8ddbdf5e0204 -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm12.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/kvmvg/vm12,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

When I edit VM definition so that qemu-kvm is invoked without '-nographic', the guest boots fine.

Quoted 2 lines

Quoted 2 lines
What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?

# virsh qemu-monitor-command vm12 --hmp x/4i \$pc
0x00000000000c06ca:  aam    $0xa
0x00000000000c06cc:  mov    %ax,%bx
0x00000000000c06ce:  mov    %bh,%al
0x00000000000c06d0:  aam    $0xa

# virsh qemu-monitor-command vm12 --hmp x/8b \$pc
00000000000c06ca: 0xd4 0x0a 0x89 0xc3 0x88 0xf8 0xd4 0x0a

Signature · 1 line

Jun'ichi Nomura, NEC Corporation

Jun'ichi Nomura

author

2013-05-08 07:34 UTC #8 lore raw

Quoted 4 lines · Jun'ichi Nomura

Quoted 3 lines
Il 07/05/2013 14:06, Gleb Natapov ha scritto:
Quoted 2 lines
What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?

Quoted 8 lines

# virsh qemu-monitor-command vm12 --hmp x/4i \$pc
0x00000000000c06ca:  aam    $0xa
0x00000000000c06cc:  mov    %ax,%bx
0x00000000000c06ce:  mov    %bh,%al
0x00000000000c06d0:  aam    $0xa

# virsh qemu-monitor-command vm12 --hmp x/8b \$pc
00000000000c06ca: 0xd4 0x0a 0x89 0xc3 0x88 0xf8 0xd4 0x0a

I could also reproduce the problem with following:

# dd if=/dev/zero of=/root/empty.img bs=1M count=1
# /usr/libexec/qemu-kvm -enable-kvm -nographic -nodefconfig -nodefaults -chardev socket,id=cmon,host=localhost,port=8888,server,nowait -mon chardev=cmon,mode=readline -drive file=/root/empty.img -chardev stdio,id=ser0 -device isa-serial,chardev=ser0

With v3.8 kernel, it reaches to the point showing "No bootable device" (as expected).
With v3.9 kernel, no visible characters appear on console.

EIP of the stalled guest points to other instruction than the previously reported case though:

(qemu) info registers
info registers
EAX=f000e81b EBX=00000130 ECX=0000fa2b EDX=0000031b
ESI=000000ed EDI=00000050 EBP=00000000 ESP=00006eaa
EIP=00000564 EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0040 00000400 ffffffff 00809300
CS =c000 000c0000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =c000 000c0000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000fc558 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000 (qemu)
(qemu) x/8b $pc
x/8b $pc
00000000000c0564: 0xd7 0x1f 0x24 0x7f 0x88 0xc4 0x88 0xd0
(qemu)
(qemu) x/i $pc
x/i $pc
0x00000000000c0564: xlat %ds:(%bx)

Signature · 1 line

Jun'ichi Nomura, NEC Corporation

Paolo Bonzini

maintainer

2013-05-08 08:30 UTC #9 lore raw

Il 08/05/2013 09:34, Jun'ichi Nomura ha scritto:

Quoted 14 lines · Jun'ichi Nomura

Quoted 13 lines
Quoted 3 lines
Il 07/05/2013 14:06, Gleb Natapov ha scritto:
Quoted 2 lines
What is the output of "virsh qemu-monitor-command vm12 --hmp x/i $pc" when it hangs?
# virsh qemu-monitor-command vm12 --hmp x/4i \$pc
0x00000000000c06ca:  aam    $0xa
0x00000000000c06cc:  mov    %ax,%bx
0x00000000000c06ce:  mov    %bh,%al
0x00000000000c06d0:  aam    $0xa
# virsh qemu-monitor-command vm12 --hmp x/8b \$pc
00000000000c06ca: 0xd4 0x0a 0x89 0xc3 0x88 0xf8 0xd4 0x0a

Quoted 7 lines

Quoted 7 lines
(qemu) x/8b $pc
x/8b $pc
00000000000c0564: 0xd7 0x1f 0x24 0x7f 0x88 0xc4 0x88 0xd0 (qemu)
(qemu) x/i $pc
x/i $pc
0x00000000000c0564: xlat %ds:(%bx)

Both of these sequences are found in sgabios. The second goes on as follows:

popw %ds
andb $0x7f, %al
movb %al, %ah
movb %dl, %al

Thanks for the report!

Paolo

Jun'ichi Nomura

author

2013-05-10 02:52 UTC #10 lore raw

In reply to Jun'ichi Nomura

Quoted 9 lines · Jun'ichi Nomura

With v3.9 kernel and Nehalem CPU (i.e. unrestricted_guest=N), a guest stuck during boot (seemingly in BIOS).
When setting emulate_invalid_guest_state=0, it does boot. (With v3.8 kernel and older, the guest used to boot fine by default.)
The userspace is qemu-kvm of RHEL6.
It seems the guest boots up fine with upstream qemu 1.4.1.
Is this a kernel bug? Or is the userspace just too old to run on v3.9?

With the following patches on top of v3.9, the problem goes away. Thank you!

Gleb Natapov (1):
KVM: VMX: fix halt emulation while emulating invalid guest sate

Paolo Bonzini (3):
KVM: emulator: emulate AAM
KVM: emulator: emulate XLAT
KVM: emulator: emulate SALC

Signature · 1 line

Jun'ichi Nomura, NEC Corporation