lsi_scsi: error: Bad Status move errors with kvm-79
Hi,
I have kvm environment with linux-2.6.28-rc7 x86_64 (Xeon), kvm-79 host and bunch of Win2K3 guests. Sometimes I get 'lsi_scsi: error: Bad Status move' from kvm (qemu) and in Windows event log "The device, \Device\Scsi\sym_hi1, did not respond within the timeout period." These errors come somewhat at random, usually with 10-30 second intervals when there is enough disk usage in the guest (seems that installing Bea Weblogic or Oracle database is pretty nice for causing these errors).
Usually windows is able to recover from these but sometimes (=too often) I get random delays and hangups. Also I have gotten BSOD 0x77 (0x02, 0x00, 0x00, 0x5f4000) about once a day.
Any ideas how to debug / fix this problem?
KVM startup command:/usr/local/bin/qemu-system-x86_64 -name vm1
-smp 1 -m 1024 -vnc :4 -k fi -serial mon:telnet::10004,server,nowait -daemonize -localtime -vga std -usb -usbdevice tablet
-net nic,macaddr=00:16:3e:00:00:4,model=e1000 -net tap,ifname=tap-vm1 -pidfile /var/run/kvm/vm1.pid -boot c
-drive index=0,media=disk,if=scsi,boot=on,file=/dev/mapper/vg0-vm
Signature · 3 lines
Anssi Kolehmainen [email protected] 040-5085390
Quoted 10 lines · Anssi Kolehmainen
Hi,
I have kvm environment with linux-2.6.28-rc7
x86_64(Xeon), kvm-79 host and bunch of Win2K3 guests. Sometimes I get 'lsi_scsi: error: Bad Status move' from kvm (qemu) and in Windows event log "The device, \Device\Scsi\sym_hi1, did not respond within the timeout period." These errors come somewhat at random, usually with 10-30 second intervals when there is enough disk usage in the guest (seems that installing Bea Weblogic or Oracle database is pretty nice for causing these errors).
Quoted 5 lines
Usually windows is able to recover from these but sometimes (=too often) I get random delays and hangups. Also I have gotten BSOD
0x77(0x02,0x00,0x00,0x5f4000) about once a day.Any ideas how to debug / fix this problem?
Current KVM userspace has a bogus line in the scsi code relating to the DBC register which looks like is what is tripping up the Bad Status, or could be anyhow. Try out with this patch applied to your qemu dir:
http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg00043.html
You can also try older KVM releases, kvm 76 at least doesn't have that line present. That might be easier than applying the patch.
You can also enable debugging in qemu/hw/lsi53c895a.c and inqemu/hw/scsi-disk.c Sending that output here would be helpful if we're
still tracking it.
If you can recreate with the patch applied or on an older KVM that doesn't have that line in there, I'll try to reproduce. Are there free/downloadable copies of Bae or Oracle that I can use to recreate?
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 15 lines · Ryan Harper
Quoted 14 lines · Anssi Kolehmainen
Hi,
I have kvm environment with linux-2.6.28-rc7
x86_64(Xeon), kvm-79 host and bunch of Win2K3 guests. Sometimes I get 'lsi_scsi: error: Bad Status move' from kvm (qemu) and in Windows event log "The device, \Device\Scsi\sym_hi1, did not respond within the timeout period." These errors come somewhat at random, usually with 10-30 second intervals when there is enough disk usage in the guest (seems that installing Bea Weblogic or Oracle database is pretty nice for causing these errors).Usually windows is able to recover from these but sometimes (=too often) I get random delays and hangups. Also I have gotten BSOD
0x77(0x02,0x00,0x00,0x5f4000) about once a day.
Quoted 7 lines
Quoted 1 line
Any ideas how to debug / fix this problem?
Current KVM userspace has a bogus line in the scsi code relating to the DBC register which looks like is what is tripping up the Bad Status, or could be anyhow. Try out with this patch applied to your qemu dir:
http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg00043.html
That line is not in kvm-79... And with that line added Windows doesn't even seem to start.
Quoted 3 lines
You can also enable debugging in
qemu/hw/lsi53c895a.cand inqemu/hw/scsi-disk.cSending that output here would be helpful if we're
still tracking it.
Nice 160mb log from just starting windows and running the installer until BSOD. Only 1.7mb with bzip2 compression:
http://kelvin.aketzu.net/kvm-qemu.log.bz2
(The "error:" line isn't there since it went to stderr. This time Windows crashed so the error should be somewhere in the end.)
Quoted 2 lines
If you can recreate with the patch applied or on an older KVM that doesn't have that line in there, I'll try to reproduce.
kvm-79 doesn't seem to have it.
Quoted 2 lines
Are there free/downloadable copies of Bae or Oracle that I can use to recreate?
Yeah, they are both available for free. You can get the Bea (now Oracle) Weblogic (application server) from:
http://www.oracle.com/technology/software/products/ias/htdocs/wls_main.html
The version we used is "Oracle WebLogic Server 10.0 MP1"
(I guess any big application which uncompresses to HD / installs lots of files might work.)
Signature · 3 lines
Anssi Kolehmainen [email protected] 040-5085390
Quoted 18 lines · Anssi Kolehmainen
Quoted 17 lines · Ryan Harper
Quoted 15 lines · Anssi Kolehmainen
Hi,
I have kvm environment with linux-2.6.28-rc7
x86_64(Xeon), kvm-79 host and bunch of Win2K3 guests. Sometimes I get 'lsi_scsi: error: Bad Status move' from kvm (qemu) and in Windows event log "The device, \Device\Scsi\sym_hi1, did not respond within the timeout period." These errors come somewhat at random, usually with 10-30 second intervals when there is enough disk usage in the guest (seems that installing Bea Weblogic or Oracle database is pretty nice for causing these errors).Usually windows is able to recover from these but sometimes (=too often) I get random delays and hangups. Also I have gotten BSOD
0x77(0x02,0x00,0x00,0x5f4000) about once a day.Any ideas how to debug / fix this problem?
Quoted 8 lines
Quoted 5 lines
Current KVM userspace has a bogus line in the scsi code relating to the DBC register which looks like is what is tripping up the Bad Status, or could be anyhow. Try out with this patch applied to your qemu dir:
http://lists.gnu.org/archive/html/qemu-devel/2008-12/msg00043.html
That line is not in kvm-79... And with that line added Windows doesn't even seem to start.
Yeah, I was hoping it was something simple, but you're right, 79 was released before that bogus line made it into qemu. It's now been removed from qemu cvs, and kvm-userspace just sync'ed.
Quoted 8 lines
Quoted 4 lines
You can also enable debugging in
qemu/hw/lsi53c895a.cand inqemu/hw/scsi-disk.cSending that output here would be helpful if we're
still tracking it.Nice 160mb log from just starting windows and running the installer until BSOD. Only 1.7mb with bzip2 compression:
http://kelvin.aketzu.net/kvm-qemu.log.bz2
I scanned through that, but didn't see anything that jumped out at me. Typically, I need a good run and a failing run to tell what's going on.
Quoted 10 lines
Quoted 2 lines
Are there free/downloadable copies of Bae or Oracle that I can use to recreate?
Yeah, they are both available for free. You can get the Bea (now Oracle) Weblogic (application server) from:
http://www.oracle.com/technology/software/products/ias/htdocs/wls_main.html
The version we used is "Oracle WebLogic Server 10.0 MP1"(I guess any big application which uncompresses to HD / installs lots of files might work.)
Using kvm-userspace.git (kvm-80 is equivalent w.r.t scsi level) I installed win2k3 sp2 with scsi as the main device, qcow2 backed file. Downloaded, unzipped and installed the Weblogic server. All with no issues at all. I'll give it a run against raw devices, that seems to be what you were using. Any more details on how to reliably reproduce the issue will help me track down the bug.
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 9 lines · Ryan Harper
Quoted 8 lines
Quoted 8 lines · Anssi Kolehmainen
Quoted 7 lines
I have kvm environment with linux-2.6.28-rc7
x86_64(Xeon), kvm-79 host and bunch of Win2K3 guests. Sometimes I get 'lsi_scsi: error: Bad Status move' from kvm (qemu) and in Windows event log "The device, \Device\Scsi\sym_hi1, did not respond within the timeout period." These errors come somewhat at random, usually with 10-30 second intervals when there is enough disk usage in the guest (seems that installing Bea Weblogic or Oracle database is pretty nice for causing these errors).
Quoted 6 lines
Using kvm-userspace.git (kvm-80 is equivalent w.r.t scsi level) I installed win2k3 sp2 with scsi as the main device, qcow2 backed file. Downloaded, unzipped and installed the Weblogic server. All with no issues at all. I'll give it a run against raw devices, that seems to be what you were using. Any more details on how to reliably reproduce the issue will help me track down the bug.
Tried kvm-82 (2.6.28-rc7 kernel, kvm-82 modules) today and the same bug still exists. I thought everything was working somewhat fine but then Windows decided to shoot with shotgun at its own system files and I got to spend the day recovering and then rebuilding the vm :)
I'll setup another computer and see whether I can reproduce that on another system.
Signature · 3 lines
Anssi Kolehmainen [email protected] 040-5085390
Quoted 7 lines · Anssi Kolehmainen
Tried kvm-82 (2.6.28-rc7 kernel, kvm-82 modules) today and the same bug still exists. I thought everything was working somewhat fine but then Windows decided to shoot with shotgun at its own system files and I got to spend the day recovering and then rebuilding the vm :)
I'll setup another computer and see whether I can reproduce that on another system.
Well, this time on a brand new PC with 2.6.28, KVM-82. Everything started out working fine and I thought I couldn't get any errors. First Oracle DB installation resulted only in few "windows detected controller error" messages in event log (but those don't cause any problems). After that Weblogic installation caused windows to BSOD (but no Bad Status move error). After reboot I tried copying the installation file to guest drive and that caused 4 Bad Status moves and BSOD...
Tried without kvm modules and everything worked fine (except being rather slow). Did the same thing with kvm modules loaded and it hit the error. Could this be some kind of timing issue?
http://kelvin.aketzu.net/kvm-2sec.log.bz2 contains about two last seconds of debugging output (with DEBUG_LSI and DEBUG_LSI_REG). Added second.microsecond timestamps to keep better track of what is going on. At the moment of crash windows was writing to disk about 10mb/s (copying file over 100mbps lan).
Signature · 3 lines
Anssi Kolehmainen [email protected] 040-5085390