: fix QEMU SCSI lock up
PATCH 1/1 : fix QEMU SCSI lock up
From: Matteo Frigo <[email protected]>
Date: Wed, 02 Apr 2008 20:41:24 -0400
To: [email protected]
Subject: [Qemu-devel] QEMU/KVM SCSI lock up
X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4)
X-Mailman-Approved-At: Thu, 03 Apr 2008 00:59:59 -0400
Reply-To: [email protected]
kvm-64 hangs under heavy disk I/O with scsi disks. To reproduce, create a fresh qcow2 disk, boot linux, and execute
dd if=/dev/sdX of=/dev/null bs=1M
on the fresh disk. See also https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1895893&group_id=180599
I have attached a patch that appears to fix the problem. The bug seems to be the following. scsi_read_data() does the following
bdrv_aio_read()
r->sector += n;
r->sector_count -= n;For reasons that I do not fully understand, bdrv_aio_read() does not return immediately, but instead it calls scsi_read_data() recursively. Since ``r->sector += n;'' has not been executed yet, the re-entrant call triggers a read of the same sector, which breaks the producer-consumer lockstep. The fix is to swap the operations as follows:
r->sector += n;
r->sector_count -= n;
bdrv_aio_read()A similar fix applies to scsi_write_data().
Thanks for developing kvm, it is truly an amazing piece of software.
Regards,
Matteo Frigo
qemu/hw/scsi-disk.c +6 -6
@@ -196,12 +196,12 @@ n =SCSI_DMA_BUF_SIZE /512 ; r->buf_len = n *512 ; - r->aiocb =bdrv_aio_read (s->bdrv, r->sector, r->dma_buf, n, + r->sector += n; + r->sector_count -= n; + r->aiocb =bdrv_aio_read (s->bdrv, r->sector - n, r->dma_buf, n, scsi_read_complete, r);if (r->aiocb ==NULL )scsi_command_complete (r,SENSE_HARDWARE_ERROR ); - r->sector += n; - r->sector_count -= n; }static void scsi_write_complete (void * opaque,int ret)
@@ -248,12 +248,12 @@BADF ("Data transfer already in progress\n" ); n = r->buf_len /512 ;if (n) { - r->aiocb =bdrv_aio_write (s->bdrv, r->sector, r->dma_buf, n, + r->sector += n; + r->sector_count -= n; + r->aiocb =bdrv_aio_write (s->bdrv, r->sector - n, r->dma_buf, n, scsi_write_complete, r);if (r->aiocb ==NULL )scsi_command_complete (r,SENSE_HARDWARE_ERROR ); - r->sector += n; - r->sector_count -= n; }else {/* Invoke completion routine to fetch data from host. */ scsi_write_complete (r,0 );
Quoted 18 lines · Marcelo Tosatti
From: Matteo Frigo <[email protected]>
Date: Wed, 02 Apr 2008 20:41:24 -0400
To: [email protected]
Subject: [Qemu-devel] QEMU/KVM SCSI lock up
X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4)
X-Mailman-Approved-At: Thu, 03 Apr 2008 00:59:59 -0400
Reply-To: [email protected]kvm-64 hangs under heavy disk I/O with scsi disks. To reproduce, create a fresh qcow2 disk, boot linux, and execute
dd if=/dev/sdX of=/dev/null bs=1M
on the fresh disk. See also https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1895893&group_id=180599
I have attached a patch that appears to fix the problem. The bug seems to be the following.
scsi_read_data()does the following
Quoted 7 lines
bdrv_aio_read() r->sector += n; r->sector_count -= n;For reasons that I do not fully understand,
bdrv_aio_read()does not return immediately, but instead it callsscsi_read_data()recursively.
This bothers me. bdrv_aio_read() should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 4 lines
For reasons that I do not fully understand,
bdrv_aio_read()does not return immediately, but instead it callsscsi_read_data()recursively.This bothers me.
bdrv_aio_read()should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?
qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so bdrv_aio_read() will never be invoked.
Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 4 lines
For reasons that I do not fully understand,
bdrv_aio_read()does not return immediately, but instead it callsscsi_read_data()recursively.This bothers me.
bdrv_aio_read()should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?
Quoted 6 lines
qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so
bdrv_aio_read()will never be invoked.Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.
FWIW, I was told this reproduces on kvm-77 (which has the latest qemu scsi bits).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 4 lines
For reasons that I do not fully understand,
bdrv_aio_read()does not return immediately, but instead it callsscsi_read_data()recursively.This bothers me.
bdrv_aio_read()should never immediately invoke the callback to prevent exactly this sort of problem. Perhaps this was a bug that has since been fixed? Is this still reproducible?
Quoted 8 lines
Quoted 6 lines
qcow2 metadata is synchronous, and if the disk is empty, there will be no data I/O, so
bdrv_aio_read()will never be invoked.Maybe we should fix this in qcow2 (and the other block formats) by scheduling a BH.
FWIW, I was told this reproduces on kvm-77 (which has the latest qemu scsi bits).
qemu_aio_wait() will run bottom halves when emulating synchronous IO. I don't think this is exploitable practically speaking but it seems to me like a major flaw. I think the proper fix is what you describe, modifying qcow2 to schedule a bottom half to read metadata. Better yet, a full conversion to make the meta data reading/writing asynchronous.
Regards,
Anthony Liguori