severe bug in 2.6.23+ kvm.git

Hi list,

we've experienced a severe bug in current kvm.git, that may have been introduced to the git tree quite recently around last weekend. 2.6.23 is broken, 2.6.23-rc8 works for us. The symptom is, that our operon kvm test machine shredders its hard disk content to a state that is not correctably by the file system checker. We use raid1 md mirrored ext3 file systems on 4 sata hard disks on it, and we've verified correct operation of the hardware via badblocks and memtest86. The problem occurs even without kvm modules loaded, so the cause seems to be something that Avi pulled elsewhere. Did anyone else experience similar problems with the 2.6.23 based kvm tree? Does anyone have an idea about a possible cause, which would help us debugging it?

thanks,
Carsten

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Could you please precise what is corrupted? The guest disk image?

If that's the case, I experienced the same problem since kvm-userspace.git has been updated to the latest qemu CVS, and I can reproduce it with plain QEMU. I am able to reproduce it easily by booting FreeBSD.

The problem is actually in QEMU, it has been broken by this commit:
http://cvs.savannah.nongnu.org/viewvc/qemu/hw/ide.c?root=qemu&r1=1.64&r2=1.65

You can try to apply the attached patch, it reverts this commit and can be applied to the latest QEMU CVS and to the latest KVM versions.

  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   [email protected]         | [email protected]
   `-    people.debian.org/~aurel32 | www.aurel32.net

Index: hw/ide.c
===================================================================
RCS file: /sources/qemu/qemu/hw/ide.c,v
retrieving revision 1.69

diff -u -r1.69 ide.c
@@ -865,44 +865,10 @@
     ide_set_irq(s);
 }
 
-static void ide_sector_write_aio_cb(void *opaque, int ret)
-{
-    BMDMAState *bm = opaque;
-    IDEState *s = bm->ide_if;
-
-#ifdef TARGET_I386
-    if (win2k_install_hack && ((++s->irq_count % 16) == 0)) {
-	/* It seems there is a bug in the Windows 2000 installer HDD
-	   IDE driver which fills the disk with empty logs when the
-	   IDE write IRQ comes too early. This hack tries to correct
-	   that at the expense of slower write performances. Use this
-	   option _only_ to install Windows 2000. You must disable it
-	   for normal use. */
-	qemu_mod_timer(s->sector_write_timer,
-		       qemu_get_clock(vm_clock) + (ticks_per_sec / 1000));
-    } else
-#endif
-    {
-	ide_set_irq(s);
-    }
-    bm->aiocb = NULL;
-}
-
 static void ide_sector_write(IDEState *s)
 {
-    BMDMAState *bm;
     int64_t sector_num;
-    int n, n1;
-
-    s->io_buffer_index = 0;
-    s->io_buffer_size = 0;
-    bm = s->bmdma;
-    if(bm == NULL) {
-	bm = qemu_mallocz(sizeof(BMDMAState));
-	s->bmdma = bm;
-    }
-    bm->ide_if = s;
-    bm->dma_cb = ide_sector_write_aio_cb;
+    int ret, n, n1;
 
     s->status = READY_STAT | SEEK_STAT;
     sector_num = ide_get_sector(s);

@@ -912,6 +878,7 @@
     n = s->nsector;
     if (n > s->req_nb_sectors)
         n = s->req_nb_sectors;
+    ret = bdrv_write(s->bs, sector_num, s->io_buffer, n);
     s->nsector -= n;
     if (s->nsector == 0) {
         /* no more sectors to write */

@@ -924,8 +891,21 @@
     }
     ide_set_sector(s, sector_num + n);
 
-    bm->aiocb = bdrv_aio_write(s->bs, sector_num, s->io_buffer, n,
-			       ide_sector_write_aio_cb, bm);
+#ifdef TARGET_I386
+    if (win2k_install_hack && ((++s->irq_count % 16) == 0)) {
+        /* It seems there is a bug in the Windows 2000 installer HDD
+           IDE driver which fills the disk with empty logs when the
+           IDE write IRQ comes too early. This hack tries to correct
+           that at the expense of slower write performances. Use this
+           option _only_ to install Windows 2000. You must disable it
+           for normal use. */
+        qemu_mod_timer(s->sector_write_timer, 
+                       qemu_get_clock(vm_clock) + (ticks_per_sec / 1000));
+    } else 
+#endif
+    {
+        ide_set_irq(s);
+    }
 }
 
 /* XXX: handle errors */

As stated, we actually did not run any guests and did not load the kvm kernel modules.
The host root file system gets corrupted to an extend not correctable by the file system checker (we gave it 24h to repair, then interrupted it), and it's very easy to reproduce: a simple kernel make on the hosts lets us reinstall the entire host operating system.

How do you know the problem has been introduced by kvm ?

Laurent

---------------- Laurent.Vivier-6ktuUTfB/[email protected]  -----------------
"Given enough eyeballs, all bugs are shallow" E. S. Raymond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

I don't. In fact I think it has not been introduced by kvm. All I stated, is that we experienced the problem when running the kvm.git kernel after the 2.6.23 update that has not been present in the kvm.git -rc8 as of last thursday.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Perhaps 2.6.23.1 corrects this ?

http://lkml.org/lkml/2007/10/12/302

Laurent

---------------- Laurent.Vivier-6ktuUTfB/[email protected]  -----------------
"Given enough eyeballs, all bugs are shallow" E. S. Raymond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Am Freitag, 19. Oktober 2007 schrieb Laurent Vivier:

No, we dont have an marvel chipset.

kvm:~# lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a4)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev b1)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a4)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f3)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)]
01:00.1 Display controller: ATI Technologies Inc RV370 [Radeon X300SE]
02:00.0 PCI bridge: Intel Corporation 6702PXH PCI Express-to-PCI Bridge A (rev 09)
04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 21)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 21)

Christian

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

There was a commit ab9c232286c2b77be78441c2d8396500b045777e regarding libata on linus's master tree that happened on Friday, that was pulled into kvm git over the weekend.. I dont know if that may be affecting you.. there is/was also chatter on LKML regarding some problems with s/g, you may want to check there.

Cheers
Mike

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Oh, that's a couple of patches in question. Git-bisect seems to be a loong way once you loose your installation every time you try. First thing we do, is figure whether or not 2.6.23.1 as released breaks our system too. This way, we can either focus on differences between Linus and Avi, or turn on the big red warning sign saying "regression".

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Looks like 2.6.23.1 works fine on that box. We'll leave it running over the weekend with "while true; do make; make clean; done".

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Well, do you happen to use sata_mv?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Am Freitag, 19. Oktober 2007 schrieb Jan Engelhardt:

no, we have nvidia, so its sata_nv.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Did you patch kvm.git with patch-2.6.23.1.bz2 or did you download linux-2.6.23.1.tar.bz2 ?

2.6.23.1 corrects nothing except sata_mv...

Laurent

---------------- [email protected]  -----------------
"Given enough eyeballs, all bugs are shallow" E. S. Raymond

Am Freitag, 19. Oktober 2007 schrieb Laurent Vivier:

Yes I know. The question we tried to answer was: is the bug in 2.6.23 or was it introduced after 2.6.23, as kvm.git already contains lots of post 2. 6.23 stuff.
Current state is:

kvm.git with tag 2.6.23-rc6 works for days without a problem. 2.6.23.1 vanilla has survived and is currently still under test. kvm.git tag master killed our filesystem at least three times.since monday.

I will continue to bang on 2.6.23.1 to see if its really fine. After that, maybe I will try to bisect on kvm.git, but this will take quite a long time, given that we had to reinstall the system due to this error.

Christian

linus-git has at least one bug with SG chaining, but usually it just hangs the machine. Patch is here:

http://lkml.org/lkml/2007/10/17/269

Luca

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Am Freitag, 19. Oktober 2007 schrieb Luca Tettamanti:

Looks promising.
I pulled this fix by pulling the latest Linus-git into the kvm.git. I also enabled some debug options in the kernel hacking section. This resulting kernel seems to be stable so far. We will see in the next days if the problem is really gone.

Thanks to all for your ideas.

Christian

Actually, the working version was 2.6.23-rc6, git-head of kvm.git as of October 11.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

kvm.git is actually 2.6.24-rc, pulled from -linus at a random point in time, so it's not at all surprising if something is broken.

One option is for you to pull -linus to get the latest and hopefully greatest and see if the bug is fixed.

Another is to use the external module capability to build kvm.git against 2.6.23.1.

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Right. We have a backup now, so next time we'll be ok ;-). Would you please pull from Linus again to get the fix into kvm.git so that we can use your tree on that machine again?

thanks,
Carsten

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

I pulled yesterday so it should be all right (and you don't need me for that; you can pull from Linus on top of kvm.git).

error compiling committee.c: too many arguments to function

Thanks :-).

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

The machine runs kvm.git for a while now, seems to work ok.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/