Skip to contentSkip to conversation

WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

DISCUSSION Mark Cave-Ayland 5 participants 34 messages Jan 5, 2010 – Jan 26, 2010 KVM

lore
Share & tools
lore
#0 lore raw

Hi all,

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error: DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

Many thanks,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#1 lore raw
Quoted 7 lines · Mark Cave-Ayland

Hi all,

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error: DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Signature · 1 line
error compiling committee.c: too many arguments to function
#2 lore raw
Quoted 3 lines · Avi Kivity
Quoted 1 line

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Hi Avi,

The command line is created using libvirt 0.7.5 (compiled by me) and looks like this:

/usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 512 -smp 1 -name winxp -uuid f99c79b6-2706-dff0-af5c-cea53559f89c -chardev socket,id=monitor,path=/usr/local/var/lib/libvirt/qemu/winxp.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/home/images/winxp.raw,if=ide,index=0,boot=on -net nic,macaddr=52:54:00:c4:90:94,vlan=0,model=rtl8139,name=rtl8139.0 -net tap,fd=41,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc :5 -vga cirrus

While the host /proc/cpuinfo contents look like this:

demo-laptop:/home/images# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T6670  @ 2.20GHz
stepping        : 10
cpu MHz         : 1200.000
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4388.31
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

I may also be able to arrange for remote SSH access into the box - mail me off-list if you are interested.

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#3 lore raw
In reply to Avi Kivity
Quoted 7 lines · Avi Kivity
Quoted 5 lines

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error: DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Hi Avi,

Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:

4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1

Author: Avi Kivity <[email protected]>
Date:   Mon Dec 28 10:48:00 2009 +0200

Reinstate cpuid vendor override when kvm is enabled

Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.

Change the default to override when kvm is enabled.

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#4 lore raw
Quoted 39 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

Quoted 7 lines
Quoted 4 lines

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Hi Avi,

Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:

4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1

Author: Avi Kivity <[email protected]>
Date:   Mon Dec 28 10:48:00 2009 +0200

Reinstate cpuid vendor override when kvm is enabled

Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.

Change the default to override when kvm is enabled.

Quoted 7 lines

HTH,

Mark.

Mark,

Thanks for tracking it down. Is there any difference with "-cpu host" option?

#5 lore raw
Quoted 44 lines · Marcelo Tosatti
Quoted 43 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

Quoted 7 lines
Quoted 4 lines

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Hi Avi,

Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:

4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1

Author: Avi Kivity <[email protected]>
Date:   Mon Dec 28 10:48:00 2009 +0200

Reinstate cpuid vendor override when kvm is enabled

Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.

Change the default to override when kvm is enabled.

Quoted 9 lines
Quoted 3 lines

HTH,

Mark.

Mark,

Thanks for tracking it down. Is there any difference with "-cpu host" option?

And what output of "cat /proc/cpuinfo" on the host looks like?

--
Gleb.

#6 lore raw
Quoted 47 lines · Gleb Natapov
Quoted 46 lines · Marcelo Tosatti
Quoted 45 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

Quoted 7 lines
Quoted 4 lines

I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue. What's your command line? What's your host cpu type?

Hi Avi,

Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:

4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1

Author: Avi Kivity <[email protected]>
Date:   Mon Dec 28 10:48:00 2009 +0200

Reinstate cpuid vendor override when kvm is enabled

Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.

Change the default to override when kvm is enabled.

HTH,

Quoted 9 lines
Quoted 7 lines
Quoted 1 line

Mark.

Mark,

Thanks for tracking it down. Is there any difference with "-cpu host" option?

And what output of "cat /proc/cpuinfo" on the host looks like?

Ah sorry, you already did that.

--
Gleb.

#8 lore raw
In reply to Mark Cave-Ayland
Quoted 16 lines · Mark Cave-Ayland

Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:

4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1

Author: Avi Kivity <[email protected]>
Date:   Mon Dec 28 10:48:00 2009 +0200

Reinstate cpuid vendor override when kvm is enabled

Due to upstream qemu changes we no longer expose the host cpu
vendor id
to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.

Quoted 5 lines

Change the default to override when kvm is enabled.

Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.

Signature · 1 line
error compiling committee.c: too many arguments to function
#7 lore raw
In reply to Marcelo Tosatti
Quoted 4 lines · Marcelo Tosatti

Mark,

Thanks for tracking it down. Is there any difference with "-cpu host" option?

Yeah; if I launch 0.12.1.1 which normally works with an additional "-cpu host" option then kvm crashes in exactly the same way.

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#9 lore raw
In reply to Avi Kivity
Quoted 1 line · Avi Kivity

Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.

No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here (http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#10 lore raw
Quoted 11 lines · Mark Cave-Ayland
Quoted 2 lines · Avi Kivity

Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.

No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here
(http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.

Can you start is in safe mode and see which driver fails?

--
Gleb.

#13 lore raw
In reply to Mark Cave-Ayland
Quoted 11 lines · Mark Cave-Ayland
Quoted 2 lines · Avi Kivity

Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.

No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here
(http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.

Which version of kvm you've used before 0.12.1.1? Does this VM work on 0.11?

--
Gleb.

#14 lore raw
In reply to Mark Cave-Ayland
Quoted 6 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity

Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.

No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here (http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.

It probably did make some kind of difference. Please try a clean install.

Signature · 1 line
error compiling committee.c: too many arguments to function
#11 lore raw
In reply to Gleb Natapov
Quoted 1 line · Gleb Natapov

Can you start is in safe mode and see which driver fails?

Hmmm. Booting in Safe Mode seems to work fine, although the next attempt to boot into Normal Mode returns a BSOD with "INTERNAL_POWER_ERROR" which is new to me. Subsequent reboots then return to the "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD.

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#12 lore raw
Quoted 11 lines · Mark Cave-Ayland
Quoted 1 line · Gleb Natapov

Can you start is in safe mode and see which driver fails?

Hmmm. Booting in Safe Mode seems to work fine, although the next attempt to boot into Normal Mode returns a BSOD with "INTERNAL_POWER_ERROR" which is new to me. Subsequent reboots then return to the "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD.

ATB,

Mark.

Try logging the boot (possible via the F8 menu as well). Y.

#15 lore raw
In reply to Avi Kivity
Quoted 1 line · Avi Kivity

It probably did make some kind of difference. Please try a clean install.

After several hours of testing, I've finally found out what the problem is.

I tried a clean WinXP guest install and that worked, so it was obviously a driver issue. After disabling various drivers in the WinXP guest, I didn't get anywhere so I decided to take a break and test Marcelo's VNC patch. With this applied, I could actually see all of the information in the BSOD which showed the error was in intelppm.sys.

A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)

I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#16 lore raw
Quoted 1 line · Mark Cave-Ayland

A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)

I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#20 lore raw
In reply to Mark Cave-Ayland
Quoted 4 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity

It probably did make some kind of difference. Please try a clean install.

Quoted 8 lines

After several hours of testing, I've finally found out what the problem is.

I tried a clean WinXP guest install and that worked, so it was obviously a driver issue. After disabling various drivers in the WinXP guest, I didn't get anywhere so I decided to take a break and test Marcelo's VNC patch. With this applied, I could actually see all of the information in the BSOD which showed the error was in intelppm.sys.

A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)

I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?

Can you try loading kvm.ko with the ignore_msrs module parameter set?

Signature · 1 line
error compiling committee.c: too many arguments to function
#17 lore raw
In reply to Mark Cave-Ayland
Quoted 6 lines · Mark Cave-Ayland
Quoted 1 line · Mark Cave-Ayland

A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)

I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?

That was actually a fix for a regression relative to 0.11.

Signature · 1 line
error compiling committee.c: too many arguments to function
#21 lore raw
In reply to Avi Kivity
Quoted 4 lines · Avi Kivity
Quoted 2 lines

I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?

Can you try loading kvm.ko with the ignore_msrs module parameter set?

Hi Avi,

I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:

virsh stop winxp
rmmod kvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobe kvm_intel
virsh start winxp

Unfortunately it still crashes with the same "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#18 lore raw
In reply to Avi Kivity
Quoted 4 lines · Avi Kivity
Quoted 2 lines

I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?

That was actually a fix for a regression relative to 0.11.

Really? Damn :( Any pointers towards the relevant bug in the bug tracker?

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#19 lore raw
Quoted 10 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity
Quoted 2 lines

I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?

That was actually a fix for a regression relative to 0.11.

Really? Damn :( Any pointers towards the relevant bug in the bug tracker?

No, it was reported on list.

Signature · 1 line
error compiling committee.c: too many arguments to function
#22 lore raw
In reply to Mark Cave-Ayland
Quoted 11 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity
Quoted 2 lines

I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?

Can you try loading kvm.ko with the ignore_msrs module parameter set?

Hi Avi,

I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:

Quoted 8 lines

virsh stop winxp
rmmod kvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobe kvm_intel
virsh start winxp

Unfortunately it still crashes with the same "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(

Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#23 lore raw
Quoted 11 lines · Avi Kivity
Quoted 11 lines

Hi Avi,

I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:

virsh stop winxp
rmmod kvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobe kvm_intel
virsh start winxp

Quoted 3 lines
Quoted 1 line

Unfortunately it still crashes with the same "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(

Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?

No, not at all. My only concern was that the VM had been running absolutely fine under older KVM and VirtualBox until the upgrade from 0.12.1.1 to 0.12.1.2 which made me think there had been a regression somewhere along the line.

I appreciate from tracking both qemu and kvm mailing lists that there is currently a lot of rapid development occuring across both QEMU and KVM, and hence sometimes things can break. It would be interesting to find out exactly *why* this doesn't work in KVM and so I can provide debugging assistance if you can point me in the right direction.

At the moment, I'm just happy that I can run the VM under KVM even with the processor.sys driver disabled. At least by bringing up the problem and solution on this mailing list thread then the solution is documented for other people who find themselves in the same situation.

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#24 lore raw
Quoted 6 lines · Mark Cave-Ayland
Quoted 3 lines
Quoted 1 line

Unfortunately it still crashes with the same "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(

Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?

No, not at all. My only concern was that the VM had been running absolutely fine under older KVM and VirtualBox until the upgrade from 0.12.1.1 to 0.12.1.2 which made me think there had been a regression somewhere along the line.

Well, there was a regression, but it was in 0.12.1.1.

There were two bugs involved, a serious one (that caused the cpuid to show up as AMD) hiding the less serious one (that causes processor.sys to BSOD).

Quoted 4 lines

I appreciate from tracking both qemu and kvm mailing lists that there is currently a lot of rapid development occuring across both QEMU and KVM, and hence sometimes things can break. It would be interesting to find out exactly *why* this doesn't work in KVM and so I can provide debugging assistance if you can point me in the right direction.

At the moment, I'm just happy that I can run the VM under KVM even with the processor.sys driver disabled. At least by bringing up the problem and solution on this mailing list thread then the solution is documented for other people who find themselves in the same situation.

I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.

Signature · 1 line
error compiling committee.c: too many arguments to function
#25 lore raw
Quoted 3 lines · Avi Kivity

Well, there was a regression, but it was in 0.12.1.1.

There were two bugs involved, a serious one (that caused the cpuid to show up as AMD) hiding the less serious one (that causes processor.sys to BSOD).

Okay, understood. I must have just got lucky when upgrading from older KVM to 0.12.1.1.

Quoted 1 line

I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.

I've had a quick look at this, and I can't work out how to enable kvm trace in Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#26 lore raw
Quoted 5 lines · Mark Cave-Ayland
Quoted 2 lines

I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.

I've had a quick look at this, and I can't work out how to enable kvm trace in Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?

CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think. 2.6.32 or later IIRC.

Signature · 1 line
error compiling committee.c: too many arguments to function
#27 lore raw
Quoted 3 lines · Avi Kivity
Quoted 1 line

I've had a quick look at this, and I can't work out how to enable kvm trace in Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?

CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think. 2.6.32 or later IIRC.

Hmmm these already look as if they have been enabled. Does the output go to syslog or does it appear on a device somewhere in /proc?

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#28 lore raw
Quoted 8 lines · Mark Cave-Ayland
Quoted 3 lines · Avi Kivity
Quoted 1 line

I've had a quick look at this, and I can't work out how to enable kvm trace in Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?

CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think. 2.6.32 or later IIRC.

Hmmm these already look as if they have been enabled. Does the output go to syslog or does it appear on a device somewhere in /proc?

echo kvm > /sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see /sys/kernel/debug/tracing/events)

cat /sys/kernel/debug/trace

Signature · 1 line
error compiling committee.c: too many arguments to function
#29 lore raw
Quoted 4 lines · Avi Kivity

echo kvm > /sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see /sys/kernel/debug/tracing/events)

cat /sys/kernel/debug/trace

(goes on a kernel debugging crash course)

Okay I think I've got the information you need for msr and cpuid:

http://pastebin.com/m39e26e6e

This is from a fresh start of the VM up to just after the point where it reaches the BSOD.

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#30 lore raw
Quoted 7 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity

echo kvm > /sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see /sys/kernel/debug/tracing/events)

cat /sys/kernel/debug/trace

Quoted 9 lines

(goes on a kernel debugging crash course)

Okay I think I've got the information you need for msr and cpuid:

http://pastebin.com/m39e26e6e

This is from a fresh start of the VM up to just after the point where it reaches the BSOD.

Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.

Signature · 1 line
error compiling committee.c: too many arguments to function
#31 lore raw
Quoted 1 line · Avi Kivity

Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.

Done.

http://pastebin.com/m209b1f13

Hopefully this trace should give a better indication of when the problem occurs since I also found the <on_reboot> option in libvirt - hence on this trace the VM shuts down immediately instead of rebooting like the earlier trace.

HTH,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs
#32 lore raw
Quoted 11 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity

Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.

Done.

http://pastebin.com/m209b1f13

Hopefully this trace should give a better indication of when the problem occurs since I also found the <on_reboot> option in libvirt - hence on this trace the VM shuts down immediately instead of rebooting like the earlier trace.

Unfortunately, no such luck. Apparently this is not msr/cpuid related - perhaps power management. Can you enable the kvm_mmio and kvm_pio events? Perhaps they will provide a clue.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#33 lore raw
Quoted 1 line · Avi Kivity

Unfortunately, no such luck. Apparently this is not msr/cpuid related - perhaps power management. Can you enable the kvm_mmio and kvm_pio events? Perhaps they will provide a clue.

Yeah, I did take a quick glance at the trace and figured that there would be some "(#GP)" strings in there if this were the cause. At least that probably explains why the ignore_msrs=1 parameter didn't have an effect.

I've just created the kvm_mmio and kvm_pio trace here for you to look at: http://www.siriusit.co.uk/tmp/kvm1.trace.

ATB,

Mark.

Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs