Skip to contentSkip to conversation

: fix QEMU SCSI lock up

PATCH In review Marcelo Tosatti 4 participants 5 messages Sep 24, 2008 – Oct 21, 2008

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 : fix QEMU SCSI lock up

From: Matteo Frigo <[email protected]>
Date: Wed, 02 Apr 2008 20:41:24 -0400
To: [email protected]
Subject: [Qemu-devel] QEMU/KVM SCSI lock up
X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4)
X-Mailman-Approved-At: Thu, 03 Apr 2008 00:59:59 -0400
Reply-To: [email protected]

kvm-64 hangs under heavy disk I/O with scsi disks. To reproduce, create a fresh qcow2 disk, boot linux, and execute

dd if=/dev/sdX of=/dev/null bs=1M

on the fresh disk. See also https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1895893&group_id=180599

I have attached a patch that appears to fix the problem. The bug seems to be the following. scsi_read_data() does the following

    bdrv_aio_read()
    r->sector += n;
    r->sector_count -= n;

For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively. Since ``r->sector += n;'' has not been executed yet, the re-entrant call triggers a read of the same sector, which breaks the producer-consumer lockstep. The fix is to swap the operations as follows:

    r->sector += n;
    r->sector_count -= n;
    bdrv_aio_read()

A similar fix applies to scsi_write_data().

Thanks for developing kvm, it is truly an amazing piece of software.

Regards,
Matteo Frigo

qemu/hw/scsi-disk.c +6 -6
diff -aur kvm-64.old/qemu/hw/scsi-disk.c kvm-64.new/qemu/hw/scsi-disk.c
@@ -196,12 +196,12 @@
         n = SCSI_DMA_BUF_SIZE / 512;
 
     r->buf_len = n * 512;
-    r->aiocb = bdrv_aio_read(s->bdrv, r->sector, r->dma_buf, n,
+    r->sector += n;
+    r->sector_count -= n;
+    r->aiocb = bdrv_aio_read(s->bdrv, r->sector - n, r->dma_buf, n,
                              scsi_read_complete, r);
     if (r->aiocb == NULL)
         scsi_command_complete(r, SENSE_HARDWARE_ERROR);
-    r->sector += n;
-    r->sector_count -= n;
 }
 
 static void scsi_write_complete(void * opaque, int ret)
@@ -248,12 +248,12 @@
         BADF("Data transfer already in progress\n");
     n = r->buf_len / 512;
     if (n) {
-        r->aiocb = bdrv_aio_write(s->bdrv, r->sector, r->dma_buf, n,
+        r->sector += n;
+        r->sector_count -= n;
+        r->aiocb = bdrv_aio_write(s->bdrv, r->sector - n, r->dma_buf, n,
                                   scsi_write_complete, r);
         if (r->aiocb == NULL)
             scsi_command_complete(r, SENSE_HARDWARE_ERROR);
-        r->sector += n;
-        r->sector_count -= n;
     } else {
         /* Invoke completion routine to fetch data from host.  */
         scsi_write_complete(r, 0);
#1 lore raw
Quoted 18 lines · Marcelo Tosatti

From: Matteo Frigo <[email protected]>
Date: Wed, 02 Apr 2008 20:41:24 -0400
To: [email protected]
Subject: [Qemu-devel] QEMU/KVM SCSI lock up
X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4)
X-Mailman-Approved-At: Thu, 03 Apr 2008 00:59:59 -0400
Reply-To: [email protected]

kvm-64 hangs under heavy disk I/O with scsi disks. To reproduce, create a fresh qcow2 disk, boot linux, and execute

dd if=/dev/sdX of=/dev/null bs=1M

on the fresh disk. See also https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1895893&group_id=180599

I have attached a patch that appears to fix the problem. The bug seems to be the following. scsi_read_data() does the following

Quoted 7 lines
    bdrv_aio_read()
    r->sector += n;
    r->sector_count -= n;

For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively.

This bothers me. bdrv_aio_read() should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?

Regards,

Anthony Liguori

#2 lore raw
Quoted 8 lines · Anthony Liguori
Quoted 4 lines

For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively.

This bothers me. bdrv_aio_read() should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?

qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so bdrv_aio_read() will never be invoked.

Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#3 lore raw
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 4 lines

For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively.

This bothers me. bdrv_aio_read() should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?

Quoted 6 lines

qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so bdrv_aio_read() will never be invoked.

Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.

FWIW, I was told this reproduces on kvm-77 (which has the latest qemu scsi bits).

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 4 lines

For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively.

This bothers me. bdrv_aio_read() should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?

Quoted 8 lines
Quoted 6 lines

qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so bdrv_aio_read() will never be invoked.

Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.

FWIW, I was told this reproduces on kvm-77 (which has the latest qemu scsi bits).

qemu_aio_wait() will run bottom halves when emulating synchronous IO. I don't think this is exploitable practically speaking but it seems to me like a major flaw. I think the proper fix is what you describe, modifying qcow2 to schedule a bottom half to read metadata. Better yet, a full conversion to make the meta data reading/writing asynchronous.

Regards,

Anthony Liguori