kvm problems on new hardware

Hello,
I have a KVM virtualization problem.
I've put together new hardware (supermicro) server with 2 E5530 cpu's and memory & disk to start experimenting with virtualization.

I intend to use the www.proxmox.com system/setup.
I installed proxmox and started stress testing the hardware:
parallel kernel compiles in a loop (concurrency_level=32) & memtest86+ during the night etc.
The hardware/os performs rocksolid when i stress test it, but the moment i start a virtual guest (eg debian netinstall) i get the first screen of the installation procedure in a vnc screen. I choose either normal install or expert install , the guest screen goes blank with only a cursor and the kvm process prints an error on the console and starts to eat cpu cycles. So the host OS is not barfing, only the kvm process is giving problems and the guest is frozen.

To see if it was/is related to the "older" proxmox kernel setup i installed ubuntu karmic with libvirt on another harddrive for comparison:
Same error happens when i start a guest.

Oct 23 09:34:14 ubuntu kernel: [  416.226550] device vnet0 left promiscuous mode
Oct 23 09:34:14 ubuntu kernel: [  416.226554] br0: port 2(vnet0) entering disabled state
Oct 23 09:34:57 ubuntu kernel: [  459.544150] type=1505 audit(1256290497.414:17): operation="profile_load" pid=1676 name=libvirt-2ae923e6-f06d-9f0d-d072-c2067b7cbee4
Oct 23 09:34:57 ubuntu kernel: [  459.550725] device vnet0 entered promiscuous mode
Oct 23 09:34:57 ubuntu kernel: [  459.551888] br0: port 2(vnet0) entering learning state
Oct 23 09:34:57 ubuntu kernel: [  459.557989] type=1503 audit(1256290497.429:18): operation="open" pid=1679 parent=1 profile="libvirt-2ae923e6-f06d-9f0d-d072-c2067b7cbee4" requested_mask="rw::" denied_mask="w::" fsuid=0 ouid=0 name="/var/tmp/debian-503-amd64-netinst.iso"
Oct 23 09:35:05 ubuntu kernel: [  468.066681] handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
Oct 23 09:35:05 ubuntu kernel: [  468.066760] handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
Oct 23 09:35:05 ubuntu kernel: [  468.066836] handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d

In the bios there are settings for vt-d etc, i tried (imho) all combinations but i am not able to start a guest. I spoke to supermicro support and even got a more recent (yet unpublished) bios. All without success.

No i am up to the point where i dont know if i have a hardware or a software problem. I installed the intel-microcode package to see if that maybe fixed
something: It didn't

References:
chassis:
http://supermicro.com/products/system/2U/6026/SYS-6026TT-BIBQRF.cfm?INF=
motherboard:
http://supermicro.com/products/motherboard/QPI/5500/X8DTT-F.cfm
7 hourmemtest:
http://dth.net/supermicro/memtest86_completed_7hr.jpg

More output (dmesg and other stuff) in the same dir:
http://dth.net/supermicro/

Am i trying to run this on hardware that is to recent and not yet tested ?

I hope somebody has some ideas/hints about this.

Danny

Does this happen for all guests (different OSes), or just this one?

Please provide a link to the install media.

error compiling committee.c: too many arguments to function

I tried an iso of both X86_64 and i386 of debian.
I even burned the iso image to a real cd to try if it would boot (it did)
Then i tried a rescuecd image: same behaviour.

It's really a standard cd image i got from

http://cdimage.debian.org/debian-cd/5.0.3/amd64/iso-cd/debian-503-amd64-netinst.iso

I opened a question at the proxmox forum (first):

http://www.proxmox.com/forum/showthread.php?t=2384

I just followed up with this posting:

http://www.proxmox.com/forum/showthread.php?p=13192#post13192

which may give more detailed/useful information.

I just tried a windows7 cd image
Same error:

vmbr0: port 2(vmtab105i0) entering learning state
vmbr0: topology change detected, propagating
vmbr0: port 2(vmtab105i0) entering forwarding state
handle_exception: unexpected, vectoring info 0x80000410 intr info 0x80000b0d

I am thinking it's either a bios setting (i tried (afaik) all possible combinations). I even have a not-yet-released version of the bios. All still won't let me use virtualization.
Could it be i have faulty cpu's ?
Is there a way to debug what happens when ik start the kvm guest process ?

I'm not able to reproduce this on a similar processor. Can you post your qemu command line?

error compiling committee.c: too many arguments to function

vhost1:/var/tmp# ps axuw |grep kvm
root      5843 42.7  0.0 621048 13560 ?        Sl   11:35   0:06 /usr/bin/kvm -monitor unix:/var/run/qemu-server/105.mon,server,nowait -vnc unix:/var/run/qemu-server/105.vnc,password -pidfile /var/run/qemu-server/105.pid -daemonize -usbdevice tablet -name win7 -smp sockets=1,cores=1 -vga cirrus -tdf -localtime -rtc-td-hack -k en-us -drive file=/var/lib/vz/images/105/vm-105-disk-1.raw,if=ide,index=0,boot=on -drive file=/var/lib/vz/template/iso/windows7.iso,if=ide,index=2,media=cdrom -m 512 -net tap,vlan=0,ifname=vmtab105i0,script=/var/lib/qemu-server/bridge-vlan -net nic,vlan=0,model=e1000,macaddr=BA:29:E3:CC:95:F6 -id 105 -cpuunits 1000

vhost1:/etc/qemu-server# cat 105.conf 

name: win7
ide2: local:iso/windows.iso,media=cdrom
vlan0: e1000=BA:29:E3:CC:95:F6
bootdisk: ide0
ostype: w2k8
ide0: local:105/vm-105-disk-1.raw
memory: 512
sockets: 1

I started the debian install again,
attached a strace and then hit enter in the vnc screen. The output of the strace can be found here:
http://dth.net/supermicro/strace_debian_install

Don't know if it helps, _i_ don't know what i'm looking at ;-)

Update:
I compiled/installed 2.6.32-rc5-git3 on this machine

I manually start kvm:

vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso

The bootscreen comes up, i hit enter to install and i get these messages (copied from dmesg)

handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d [this line is repeated many times:
dmesg |grep 0x80000b0d | wc -l
570 ]
and finally before ending the kvm session i get:
vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021

Can somebody make a slightly related guess what this means ?

Bad hardware ?
cpu ? mem ?

Bueller, anyone ?

I've seen that in a couple places. I don't think we have root cause, but in at least one situation (running win2k3 with > 4G of ram) the work around was to use:

-cpu pentium3

This relates to what cpu features are exported in the virtual processor, so if this helps it's masking the real issue, but hopefully can help narrow down what we have to look at.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
[email protected]

Thanks for your response.

When i add this option it segv's on bootup, it doesn't even make it to the bootscreen of debian:

vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso -cpu pentium3

kvm[4829]: segfault at c ip 00000000004674ac sp 0000000042323fd0 error 4 in kvm[400000+21b000]

Does this provide more hints where to look ?

Actually, I'm not 100% sure, but I might have root-caused this. At least, the exit reason is the same exit reason I fixed. That exit reason essentially means "vmenter failed because of invalid guest fields". In the case I tracked down, it was because we were wrongly sign-extending the segment fields (instead of zero-extending them). Can either you or Danny:

1) Get the output from stderr of qemu when this happens? I believe qemu dumps the state of all of the guest fields when it's going to crash like this, and that can tell us if the GUEST_STATE is wrong.

2) Try the patch posted here: http://lkml.org/lkml/2009/10/28/201, and see if it helps?

Chris Lalancette

<blush>
Any pointer how to produce/where to find the dump ? </blush>

[oneliner patch manually applied]

vhost1:/usr/src/linux-2.6.32-rc5-git4# time make-kpkg kernel-image --initrd

[SNIP]

dpkg-deb: building package linux-image-2.6.32-rc5-git4' in ../linux-image-2.6.32-rc5-git4_2.6.32-rc5-git4-10.00.Custom_amd64.deb'. make[2]: Leaving directory `/usr/src/linux-2.6.32-rc5-git4'
make[1]: Leaving directory `/usr/src/linux-2.6.32-rc5-git4'

real    4m50.866s
user    30m3.549s
sys     2m41.270s

[REBOOT]

vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso -cpu pentium3
Segmentation fault
vhost1:~# dmesg| tail
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021
kvm[5138]: segfault at c ip 00000000004674ac sp 0000000041cebfd0 error 4 in kvm[400000+21b000]

In my case it doesn't seem to help.

I straced the process in case that should provide some more help:
http://dth.net/supermicro/strace_2.6.32-rc5-git4_p3_cpu

Thanks for the feedback!

Can't guess, so a debugging patch.

Please install a kvm.git kernel and qemu.git qemu, both from the simultaenous-exceptions branch. kvm.git is very similar to 2.5.32-rc5, except for kvm changes. Please enable ftrace in the kernel.

For starters, run your guest; instead of dmesg errors it should print something on stderr and freeze. Please report those messages, as well as the output of the following monitor commands:

(qemu) info registers
(qemu) x/30i $eip-20

You can find kvm.git and qemu-kvm.git in

git://git.kernel.org/pub/scm/virt/kvm/kvm.git
git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git

remember, the simultaneous-exceptions branch, or we don't get any output.

error compiling committee.c: too many arguments to function