WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1
Hi all,
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error: DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian Lenny host with a 2.6.32.2 kernel on Intel.
Many thanks,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 7 lines · Mark Cave-Ayland
Hi all,
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.
It's not a known issue. What's your command line? What's your host cpu type?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Quoted 1 line
I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.It's not a known issue. What's your command line? What's your host cpu type?
Hi Avi,
The command line is created using libvirt 0.7.5 (compiled by me) and looks like this:
/usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 512 -smp 1 -name winxp -uuid f99c79b6-2706-dff0-af5c-cea53559f89c -chardev socket,id=monitor,path=/usr/local/var/lib/libvirt/qemu/winxp.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/home/images/winxp.raw,if=ide,index=0,boot=on -net nic,macaddr=52:54:00:c4:90:94,vlan=0,model=rtl8139,name=rtl8139.0 -net tap,fd=41,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc :5 -vga cirrus
While the host /proc/cpuinfo contents look like this:
demo-laptop:/home/images# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Duo CPU T6670 @ 2.20GHz stepping : 10 cpu MHz : 1200.000 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm ida tpr_shadow vnmi flexpriority bogomips : 4388.31 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
I may also be able to arrange for remote SSH access into the box - mail me off-list if you are interested.
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 7 lines · Avi Kivity
Quoted 5 lines
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.It's not a known issue. What's your command line? What's your host cpu type?
Hi Avi,
Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:
4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
Author: Avi Kivity <[email protected]> Date: Mon Dec 28 10:48:00 2009 +0200
Reinstate cpuid vendor override when kvm is enabled
Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.
Change the default to override when kvm is enabled.
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 39 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.Quoted 7 lines
Quoted 4 lines
I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.It's not a known issue. What's your command line? What's your host cpu type?
Hi Avi,
Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:
4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1is first bad commit commit4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1Author: Avi Kivity <[email protected]> Date: Mon Dec 28 10:48:00 2009 +0200Reinstate cpuid vendor override when kvm is enabled
Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.Change the default to override when kvm is enabled.
Quoted 7 lines
Signed-off-by Avi Kivity <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>HTH,
Mark.
Mark,
Thanks for tracking it down. Is there any difference with "-cpu host" option?
Quoted 44 lines · Marcelo Tosatti
Quoted 43 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.Quoted 7 lines
Quoted 4 lines
I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.It's not a known issue. What's your command line? What's your host cpu type?
Hi Avi,
Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:
4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1is first bad commit commit4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1Author: Avi Kivity <[email protected]> Date: Mon Dec 28 10:48:00 2009 +0200Reinstate cpuid vendor override when kvm is enabled
Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.Change the default to override when kvm is enabled.
Signed-off-by Avi Kivity <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>
Quoted 9 lines
Quoted 3 lines
HTH,
Mark.
Mark,
Thanks for tracking it down. Is there any difference with "-cpu host" option?
And what output of "cat /proc/cpuinfo" on the host looks like?
--
Gleb.
Quoted 47 lines · Gleb Natapov
Quoted 46 lines · Marcelo Tosatti
Quoted 45 lines · Mark Cave-Ayland
Quoted 8 lines · Avi Kivity
Quoted 8 lines
I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the login screen, the WinXP guest goes BSOD with the following error:
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.Quoted 7 lines
Quoted 4 lines
I've confirmed by switching between the two installations several times that the error consistently occurs with qemu-kvm-0.12.1.2 but not qemu-kvm-0.12.1.1. Is this a known issue? This is on an
x86_64Debian Lenny host with a 2.6.32.2 kernel on Intel.It's not a known issue. What's your command line? What's your host cpu type?
Hi Avi,
Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:
4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1is first bad commit commit4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1Author: Avi Kivity <[email protected]> Date: Mon Dec 28 10:48:00 2009 +0200Reinstate cpuid vendor override when kvm is enabled
Due to upstream qemu changes we no longer expose the host cpu vendor id to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.Change the default to override when kvm is enabled.
Signed-off-by Avi Kivity <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>HTH,
Quoted 9 lines
Quoted 7 lines
Quoted 1 line
Mark.
Mark,
Thanks for tracking it down. Is there any difference with "-cpu host" option?
And what output of "cat
/proc/cpuinfo" on the host looks like?
Ah sorry, you already did that.
--
Gleb.
Quoted 4 lines · Marcelo Tosatti
Mark,
Thanks for tracking it down. Is there any difference with "-cpu host" option?
Yeah; if I launch 0.12.1.1 which normally works with an additional "-cpu host" option then kvm crashes in exactly the same way.
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 16 lines · Mark Cave-Ayland
Good news - I downloaded the userspace git repository and managed to identify the offending commit between 0.12.1.1 and 0.12.1.2 using git bisect:
4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1is first bad commit commit4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1Author: Avi Kivity <[email protected]> Date: Mon Dec 28 10:48:00 2009 +0200Reinstate cpuid vendor override when kvm is enabled
Due to upstream qemu changes we no longer expose the host cpu
vendor id
to the guest. This leads to failures when the syscall/sysenter
instructions
are used in compatibility mode.
Quoted 5 lines
Change the default to override when kvm is enabled.
Signed-off-by Avi Kivity <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>
Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.
No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here (http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 11 lines · Mark Cave-Ayland
Quoted 2 lines · Avi Kivity
Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.
No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here
(http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.
Can you start is in safe mode and see which driver fails?
--
Gleb.
Quoted 1 line · Gleb Natapov
Can you start is in safe mode and see which driver fails?
Hmmm. Booting in Safe Mode seems to work fine, although the next attempt to boot into Normal Mode returns a BSOD with "INTERNAL_POWER_ERROR" which is new to me. Subsequent reboots then return to the "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD.
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 11 lines · Mark Cave-Ayland
Quoted 1 line · Gleb Natapov
Can you start is in safe mode and see which driver fails?
Hmmm. Booting in Safe Mode seems to work fine, although the next attempt to boot into Normal Mode returns a BSOD with "
INTERNAL_POWER_ERROR" which is new to me. Subsequent reboots then return to the "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD.ATB,
Mark.
Try logging the boot (possible via the F8 menu as well). Y.
Quoted 11 lines · Mark Cave-Ayland
Quoted 2 lines · Avi Kivity
Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.
No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here
(http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.
Which version of kvm you've used before 0.12.1.1? Does this VM work on 0.11?
--
Gleb.
Quoted 6 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity
Did you install using 0.12.1.1 and then run using 0.12.1.2? If so, it seems Windows XP is not able to move from AMD to Intel.
No, although the original VM was built in VirtualBox if that makes a difference? I had to manually implement the "merge ide" fix here (http://support.microsoft.com/kb/314082) in order to bring up the VM under KVM but it has worked fine until the upgrade to 0.12.1.2.
It probably did make some kind of difference. Please try a clean install.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
It probably did make some kind of difference. Please try a clean install.
After several hours of testing, I've finally found out what the problem is.
I tried a clean WinXP guest install and that worked, so it was obviously a driver issue. After disabling various drivers in the WinXP guest, I didn't get anywhere so I decided to take a break and test Marcelo's VNC patch. With this applied, I could actually see all of the information in the BSOD which showed the error was in intelppm.sys.
A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)
I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 1 line · Mark Cave-Ayland
A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)
I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 6 lines · Mark Cave-Ayland
Quoted 1 line · Mark Cave-Ayland
A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)
I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?
That was actually a fix for a regression relative to 0.11.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?
That was actually a fix for a regression relative to 0.11.
Really? Damn :( Any pointers towards the relevant bug in the bug tracker?
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 10 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I've just re-created the KVM image fresh from the VDI image once again and can confirm that disabling just the Processor driver is enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the default for "-cpu host" should not be changed in a micro release as there is a risk of breaking existing VMs?
That was actually a fix for a regression relative to 0.11.
Really? Damn :( Any pointers towards the relevant bug in the bug tracker?
No, it was reported on list.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity
It probably did make some kind of difference. Please try a clean install.
Quoted 8 lines
After several hours of testing, I've finally found out what the problem is.
I tried a clean WinXP guest install and that worked, so it was obviously a driver issue. After disabling various drivers in the WinXP guest, I didn't get anywhere so I decided to take a break and test Marcelo's VNC patch. With this applied, I could actually see all of the information in the BSOD which showed the error was in intelppm.sys.
A quick search took me to this page here: http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx which explains the issue in more detail. I first tried disabling the intelppm driver and rebooting, but that didn't make a difference; however disabling the Processor driver worked and my guest VM booted in Normal Mode :)
I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?
Can you try loading kvm.ko with the ignore_msrs module parameter set?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?
Can you try loading kvm.ko with the
ignore_msrsmodule parameter set?
Hi Avi,
I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:
virsh stop winxp
rmmod kvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobe kvm_intel
virsh start winxp
Unfortunately it still crashes with the same "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 11 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I think the issue is probably similar to that explained in the article above; with a new processor reported to the guest, the internal processor driver tries to upload some kind of microcode to the new device which fails and causes the guest to fall over. Can we teach KVM to silently discard these kinds of updates?
Can you try loading kvm.ko with the
ignore_msrsmodule parameter set?Hi Avi,
I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:
Quoted 8 lines
virsh stop winxp
rmmodkvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobekvm_intel
virsh start winxpUnfortunately it still crashes with the same "
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(
Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 11 lines · Avi Kivity
Quoted 11 lines
Hi Avi,
I've just done a quick test re-enabling processor.sys on my WinXP guest and then did the following:
virsh stop winxp
rmmodkvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobekvm_intel
virsh start winxp
Quoted 3 lines
Quoted 1 line
Unfortunately it still crashes with the same "
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?
No, not at all. My only concern was that the VM had been running absolutely fine under older KVM and VirtualBox until the upgrade from 0.12.1.1 to 0.12.1.2 which made me think there had been a regression somewhere along the line.
I appreciate from tracking both qemu and kvm mailing lists that there is currently a lot of rapid development occuring across both QEMU and KVM, and hence sometimes things can break. It would be interesting to find out exactly *why* this doesn't work in KVM and so I can provide debugging assistance if you can point me in the right direction.
At the moment, I'm just happy that I can run the VM under KVM even with the processor.sys driver disabled. At least by bringing up the problem and solution on this mailing list thread then the solution is documented for other people who find themselves in the same situation.
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 6 lines · Mark Cave-Ayland
Quoted 3 lines
Quoted 1 line
Unfortunately it still crashes with the same "
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(Well, don't do that then. Is there any specific functionality in processor.sys that you're missing?
No, not at all. My only concern was that the VM had been running absolutely fine under older KVM and VirtualBox until the upgrade from 0.12.1.1 to 0.12.1.2 which made me think there had been a regression somewhere along the line.
Well, there was a regression, but it was in 0.12.1.1.
There were two bugs involved, a serious one (that caused the cpuid to show up as AMD) hiding the less serious one (that causes processor.sys to BSOD).
Quoted 4 lines
I appreciate from tracking both qemu and kvm mailing lists that there is currently a lot of rapid development occuring across both QEMU and KVM, and hence sometimes things can break. It would be interesting to find out exactly *why* this doesn't work in KVM and so I can provide debugging assistance if you can point me in the right direction.
At the moment, I'm just happy that I can run the VM under KVM even with the processor.sys driver disabled. At least by bringing up the problem and solution on this mailing list thread then the solution is documented for other people who find themselves in the same situation.
I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Well, there was a regression, but it was in 0.12.1.1.
There were two bugs involved, a serious one (that caused the cpuid to show up as AMD) hiding the less serious one (that causes processor.sys to BSOD).
Okay, understood. I must have just got lucky when upgrading from older KVM to 0.12.1.1.
Quoted 1 line
I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to
UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.
I've had a quick look at this, and I can't work out how to enable kvm trace in Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 5 lines · Mark Cave-Ayland
Quoted 2 lines
I'd like to find out why processor.sys fails, but the .1->.2 change isn't any help unfortunately. It looks like here too there are two bugs involved: one in kvm which doesn't act like processor.sys expects it, and one in processor.sys which causes it to
UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS. You might try running with kvm trace enabled and look at msr and cpuid accesses just prior to the crash.I've had a quick look at this, and I can't work out how to enable kvm trace in
Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?
CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think. 2.6.32 or later IIRC.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Quoted 1 line
I've had a quick look at this, and I can't work out how to enable kvm trace in
Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?
CONFIG_FTRACE,CONFIG_TRACEPOINTSshould be sufficient, I think. 2.6.32 or later IIRC.
Hmmm these already look as if they have been enabled. Does the output go to syslog or does it appear on a device somewhere in /proc?
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 8 lines · Mark Cave-Ayland
Quoted 3 lines · Avi Kivity
Quoted 1 line
I've had a quick look at this, and I can't work out how to enable kvm trace in
Kconfig. Is there any documentation on how to build with kvm trace enabled for debugging?
CONFIG_FTRACE,CONFIG_TRACEPOINTSshould be sufficient, I think. 2.6.32 or later IIRC.Hmmm these already look as if they have been enabled. Does the output go to syslog or does it appear on a device somewhere in /proc?
echo kvm > /sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see /sys/kernel/debug/tracing/events)
cat /sys/kernel/debug/trace
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
echo kvm >
/sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see/sys/kernel/debug/tracing/events)cat
/sys/kernel/debug/trace
(goes on a kernel debugging crash course)
Okay I think I've got the information you need for msr and cpuid:
This is from a fresh start of the VM up to just after the point where it reaches the BSOD.
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 7 lines · Mark Cave-Ayland
Quoted 4 lines · Avi Kivity
echo kvm >
/sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see/sys/kernel/debug/tracing/events)cat
/sys/kernel/debug/trace
Quoted 9 lines
(goes on a kernel debugging crash course)
Okay I think I've got the information you need for msr and cpuid:
This is from a fresh start of the VM up to just after the point where it reaches the BSOD.
Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.
Done.
Hopefully this trace should give a better indication of when the problem occurs since I also found the <on_reboot> option in libvirt - hence on this trace the VM shuts down immediately instead of rebooting like the earlier trace.
HTH,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
Quoted 11 lines · Mark Cave-Ayland
Quoted 1 line · Avi Kivity
Unfortunately msr tracing fails to record some important information. I just posted a patch to fix this. Can you rerun from kvm.git branch msr-trace? That contains the fix.
Done.
Hopefully this trace should give a better indication of when the problem occurs since I also found the <
on_reboot> option in libvirt - hence on this trace the VM shuts down immediately instead of rebooting like the earlier trace.
Unfortunately, no such luck. Apparently this is not msr/cpuid related - perhaps power management. Can you enable the kvm_mmio and kvm_pio events? Perhaps they will provide a clue.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 1 line · Avi Kivity
Unfortunately, no such luck. Apparently this is not msr/cpuid related - perhaps power management. Can you enable the
kvm_mmioandkvm_pioevents? Perhaps they will provide a clue.
Yeah, I did take a quick glance at the trace and figured that there would be some "(#GP)" strings in there if this were the cause. At least that probably explains why the ignore_msrs=1 parameter didn't have an effect.
I've just created the kvm_mmio and kvm_pio trace here for you to look at: http://www.siriusit.co.uk/tmp/kvm1.trace.
ATB,
Mark.
Signature · 7 lines
Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs