Skip to contentSkip to conversation

hw/ide: fix ATAPI CD reads racing port reset and drain

PATCH In review Denis V. Lunev via qemu development 3 participants 20 messages Jun 19 – Jul 20

lore
Share & tools
lore
#0 lore raw

cover hw/ide: fix ATAPI CD reads racing port reset and drain

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

 hw/ide/ahci.c             |   3 +
 hw/ide/atapi.c            | 180 +++++++++++++++-----------------
 tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
 tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
 tests/qtest/libqos/ahci.c |   8 ++
 tests/qtest/libqos/ahci.h |   2 +
 6 files changed, 433 insertions(+), 172 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/9 hw/ide/ahci: cancel in-flight buffered reads on command engine restart

ATAPI CD reads are issued through ide_buffered_readv() (cd_read_sector() and ide_atapi_cmd_read_dma_cb() in hw/ide/atapi.c). The PIO path discards the returned aiocb; the DMA path stores it in s->bus->dma->aiocb.

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while such a read is still in flight. Stopping the engine unmaps the command list (ahci_unmap_clb_address()) and restarting it re-maps the list and clears AHCIDevice.cur_cmd to NULL, but nothing tears down the outstanding read. This path does not run ide_reset(), so the drive's transfer state is preserved and the read still completes. Its callbacks then dereference the stale or NULL cur_cmd in the AHCI transfer helpers:

PIO: cd_read_sector_cb() -> ide_atapi_cmd_reply_end() ->

       ide_transfer_start_norecurse() -> ahci_pio_transfer()

DMA: ide_atapi_cmd_read_dma_cb() -> ahci_dma_rw_buf() ->
ahci_populate_sglist()

Both crash with a NULL cur_cmd; the PIO variant has been seen in the field.

Cancel the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() as the ATAPI DEVICE RESET command does. It runs the completion callback with -ECANCELED (which tears down s->bus->dma->aiocb for the DMA case) and orphans the buffered request, so the eventual asynchronous completion is a no-op. Merely setting the orphaned flag is not enough: it would leave s->bus->dma->aiocb pointing at a freed aiocb that a later reset would cancel.

Fixes 1d8c11d63154 ("ide: add support for IDEBufferedRequest")
---
 hw/ide/ahci.c | 3 +++
 1 file changed, 3 insertions(+)
hw/ide/ahci.c +3 -0
index c2b4432b94..8e3684cd3c 100644
@@ -740,6 +740,9 @@ static bool ahci_map_clb_address(AHCIDevice *ad)
 
 static void ahci_unmap_clb_address(AHCIDevice *ad)
 {
+    /* Cancel in-flight reads that would complete against a cleared cur_cmd. */
+    ide_cancel_dma_sync(&ad->port.ifs[0]);
+
     if (ad->lst == NULL) {
         trace_ahci_unmap_clb_address_null(ad->hba, ad->port_no);
         return;
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH 1/9] hw/ide/ahci: cancel in-flight buffered reads on command engine restart

Hi Denis,

Quoted 42 lines · Denis V. Lunev via qemu development

ATAPI CD reads are issued through ide_buffered_readv() (cd_read_sector() and ide_atapi_cmd_read_dma_cb() in hw/ide/atapi.c). The PIO path discards the returned aiocb; the DMA path stores it in s->bus->dma->aiocb.

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while such a read is still in flight. Stopping the engine unmaps the command list (ahci_unmap_clb_address()) and restarting it re-maps the list and clears AHCIDevice.cur_cmd to NULL, but nothing tears down the outstanding read. This path does not run ide_reset(), so the drive's transfer state is preserved and the read still completes. Its callbacks then dereference the stale or NULL cur_cmd in the AHCI transfer helpers:

   PIO: cd_read_sector_cb() -> ide_atapi_cmd_reply_end() ->
        ide_transfer_start_norecurse() -> ahci_pio_transfer()

   DMA: ide_atapi_cmd_read_dma_cb() -> ahci_dma_rw_buf() ->
        ahci_populate_sglist()

Both crash with a NULL cur_cmd; the PIO variant has been seen in the field.

Cancel the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() as the ATAPI DEVICE RESET command does. It runs the completion callback with -ECANCELED (which tears down s->bus->dma->aiocb for the DMA case) and orphans the buffered request, so the eventual asynchronous completion is a no-op. Merely setting the orphaned flag is not enough: it would leave s->bus->dma->aiocb pointing at a freed aiocb that a later reset would cancel.

Fixes 1d8c11d63154 ("ide: add support for IDEBufferedRequest")
---
  hw/ide/ahci.c | 3 +++
  1 file changed, 3 insertions(+)
hw/ide/ahci.c +0 -0
index c2b4432b94..8e3684cd3c 100644

hw/ide/ahci.c static bool ahci_map_clb_address(AHCIDevice *ad) · line 744

Quoted 5 lines
hw/ide/ahci.c +2 -0
@@ -740,6 +740,9 @@ static bool ahci_map_clb_address(AHCIDevice *ad)
 
  static void ahci_unmap_clb_address(AHCIDevice *ad)
  {
+    /* Cancel in-flight reads that would complete against a cleared cur_cmd. */
+    ide_cancel_dma_sync(&ad->port.ifs[0]);

ide_bus_active_if(&ad->port.ifs) would be safer, otherwise the patch looks reasonable.

Quoted 4 lines
hw/ide/ahci.c +1 -0
+
      if (ad->lst == NULL) {
          trace_ahci_unmap_clb_address_null(ad->hba, ad->port_no);
          return;
#3 lore raw

Re: [PATCH 1/9] hw/ide/ahci: cancel in-flight buffered reads on command engine restart

Quoted 46 lines · Philippe Mathieu-Daudé

Hi Denis,

Quoted 43 lines · Denis V. Lunev via qemu development

ATAPI CD reads are issued through ide_buffered_readv() (cd_read_sector() and ide_atapi_cmd_read_dma_cb() in hw/ide/atapi.c). The PIO path discards the returned aiocb; the DMA path stores it in s->bus->dma->aiocb.

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while such a read is still in flight. Stopping the engine unmaps the command list (ahci_unmap_clb_address()) and restarting it re-maps the list and clears AHCIDevice.cur_cmd to NULL, but nothing tears down the outstanding read. This path does not run ide_reset(), so the drive's transfer state is preserved and the read still completes. Its callbacks then dereference the stale or NULL cur_cmd in the AHCI transfer helpers:

   PIO: cd_read_sector_cb() -> ide_atapi_cmd_reply_end() ->
        ide_transfer_start_norecurse() -> ahci_pio_transfer()

   DMA: ide_atapi_cmd_read_dma_cb() -> ahci_dma_rw_buf() ->
        ahci_populate_sglist()

Both crash with a NULL cur_cmd; the PIO variant has been seen in the field.

Cancel the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() as the ATAPI DEVICE RESET command does. It runs the completion callback with -ECANCELED (which tears down s->bus->dma->aiocb for the DMA case) and orphans the buffered request, so the eventual asynchronous completion is a no-op. Merely setting the orphaned flag is not enough: it would leave s->bus->dma->aiocb pointing at a freed aiocb that a later reset would cancel.

Fixes 1d8c11d63154 ("ide: add support for IDEBufferedRequest")
---
  hw/ide/ahci.c | 3 +++
  1 file changed, 3 insertions(+)
hw/ide/ahci.c +0 -0
index c2b4432b94..8e3684cd3c 100644
@@ -740,6 +740,9 @@ static bool ahci_map_clb_address(AHCIDevice *ad)
 
Quoted 6 lines

hw/ide/ahci.c static bool ahci_map_clb_address(AHCIDevice *ad) · line 744

Quoted 4 lines
hw/ide/ahci.c +2 -0
  static void ahci_unmap_clb_address(AHCIDevice *ad)
  {
+    /* Cancel in-flight reads that would complete against a cleared cur_cmd. */
+    ide_cancel_dma_sync(&ad->port.ifs[0]);

ide_bus_active_if(&ad->port.ifs) would be safer, otherwise the patch looks reasonable.

Anything else for the series?

Should I respin or this change could be cooked at merge?

Thank you in advance,
    Den

#4 lore raw

Re: [PATCH 1/9] hw/ide/ahci: cancel in-flight buffered reads on command engine restart

Hi Denis,

Quoted 53 lines · Denis V. Lunev
Quoted 46 lines · Philippe Mathieu-Daudé

Hi Denis,

Quoted 43 lines · Denis V. Lunev via qemu development

ATAPI CD reads are issued through ide_buffered_readv() (cd_read_sector() and ide_atapi_cmd_read_dma_cb() in hw/ide/atapi.c). The PIO path discards the returned aiocb; the DMA path stores it in s->bus->dma->aiocb.

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while such a read is still in flight. Stopping the engine unmaps the command list (ahci_unmap_clb_address()) and restarting it re-maps the list and clears AHCIDevice.cur_cmd to NULL, but nothing tears down the outstanding read. This path does not run ide_reset(), so the drive's transfer state is preserved and the read still completes. Its callbacks then dereference the stale or NULL cur_cmd in the AHCI transfer helpers:

    PIO: cd_read_sector_cb() -> ide_atapi_cmd_reply_end() ->
         ide_transfer_start_norecurse() -> ahci_pio_transfer()

    DMA: ide_atapi_cmd_read_dma_cb() -> ahci_dma_rw_buf() ->
         ahci_populate_sglist()

Both crash with a NULL cur_cmd; the PIO variant has been seen in the field.

Cancel the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() as the ATAPI DEVICE RESET command does. It runs the completion callback with -ECANCELED (which tears down s->bus->dma->aiocb for the DMA case) and orphans the buffered request, so the eventual asynchronous completion is a no-op. Merely setting the orphaned flag is not enough: it would leave s->bus->dma->aiocb pointing at a freed aiocb that a later reset would cancel.

Fixes 1d8c11d63154 ("ide: add support for IDEBufferedRequest")
---
   hw/ide/ahci.c | 3 +++
   1 file changed, 3 insertions(+)
hw/ide/ahci.c +0 -0
index c2b4432b94..8e3684cd3c 100644
@@ -740,6 +740,9 @@ static bool ahci_map_clb_address(AHCIDevice *ad)
 
Quoted 6 lines

hw/ide/ahci.c static bool ahci_map_clb_address(AHCIDevice *ad) · line 744

Quoted 4 lines
hw/ide/ahci.c +2 -0
   static void ahci_unmap_clb_address(AHCIDevice *ad)
   {
+    /* Cancel in-flight reads that would complete against a cleared cur_cmd. */
+    ide_cancel_dma_sync(&ad->port.ifs[0]);

ide_bus_active_if(&ad->port.ifs) would be safer, otherwise the patch looks reasonable.

Quoted 3 lines

Anything else for the series?

Should I respin or this change could be cooked at merge?

I took this patch via my hw-misc tree (using ide_bus_active_if) because it I could understand it and it looked trivial enough to me; however I don't have enough knowledge (don't feel confident at all) to look at the rest, I'm sorry :|

Regards,

Phil.

#5 lore raw

Re: [PATCH 1/9] hw/ide/ahci: cancel in-flight buffered reads on command engine restart

Quoted 59 lines · Philippe Mathieu-Daudé

Hi Denis,

Quoted 53 lines · Denis V. Lunev
Quoted 46 lines · Philippe Mathieu-Daudé

Hi Denis,

Quoted 43 lines · Denis V. Lunev via qemu development

ATAPI CD reads are issued through ide_buffered_readv() (cd_read_sector() and ide_atapi_cmd_read_dma_cb() in hw/ide/atapi.c). The PIO path discards the returned aiocb; the DMA path stores it in s->bus->dma->aiocb.

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while such a read is still in flight. Stopping the engine unmaps the command list (ahci_unmap_clb_address()) and restarting it re-maps the list and clears AHCIDevice.cur_cmd to NULL, but nothing tears down the outstanding read. This path does not run ide_reset(), so the drive's transfer state is preserved and the read still completes. Its callbacks then dereference the stale or NULL cur_cmd in the AHCI transfer helpers:

    PIO: cd_read_sector_cb() -> ide_atapi_cmd_reply_end() ->
         ide_transfer_start_norecurse() -> ahci_pio_transfer()

    DMA: ide_atapi_cmd_read_dma_cb() -> ahci_dma_rw_buf() ->
         ahci_populate_sglist()

Both crash with a NULL cur_cmd; the PIO variant has been seen in the field.

Cancel the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() as the ATAPI DEVICE RESET command does. It runs the completion callback with -ECANCELED (which tears down s->bus->dma->aiocb for the DMA case) and orphans the buffered request, so the eventual asynchronous completion is a no-op. Merely setting the orphaned flag is not enough: it would leave s->bus->dma->aiocb pointing at a freed aiocb that a later reset would cancel.

Fixes 1d8c11d63154 ("ide: add support for IDEBufferedRequest")
---
   hw/ide/ahci.c | 3 +++
   1 file changed, 3 insertions(+)
hw/ide/ahci.c +0 -0
index c2b4432b94..8e3684cd3c 100644
@@ -740,6 +740,9 @@ static bool ahci_map_clb_address(AHCIDevice *ad)
 
Quoted 6 lines

hw/ide/ahci.c static bool ahci_map_clb_address(AHCIDevice *ad) · line 744

Quoted 4 lines
hw/ide/ahci.c +2 -0
   static void ahci_unmap_clb_address(AHCIDevice *ad)
   {
+    /* Cancel in-flight reads that would complete against a cleared cur_cmd. */
+    ide_cancel_dma_sync(&ad->port.ifs[0]);

ide_bus_active_if(&ad->port.ifs) would be safer, otherwise the patch looks reasonable.

Quoted 3 lines

Anything else for the series?

Should I respin or this change could be cooked at merge?

Quoted 8 lines

I took this patch via my hw-misc tree (using ide_bus_active_if) because it I could understand it and it looked trivial enough to me; however I don't have enough knowledge (don't feel confident at all) to look at the rest, I'm sorry :|

Regards,

Phil.

In that case you should also take next one - this is unit test for the fix.

Den

#6 lore raw

PATCH 2/9 tests/qtest/ahci: test ATAPI read completing after engine restart

In reply to Denis V. Lunev via qemu development

Add a regression test for the crash that occurs when a buffered ATAPI read completes after the command engine has been restarted. Issue an ATAPI READ_10 against a blkdebug-backed CD, suspend the backend read so it stays in flight, stop and restart the port's command engine (which re-maps the command list and clears cur_cmd), then release the read.

The PIO and DMA reply paths fault in different AHCI helpers (ahci_pio_transfer() vs ahci_dma_rw_buf()), so cover both. The DMA variant is the reliable guard: on engine restart check_cmd() can re-arm cur_cmd before the old read completes, so the PIO variant does not fault in every build.

The test only asserts that qemu survives a subsequent register access; if the blkdebug breakpoint ever failed to park the read it would pass without exercising the bug, as with the existing break/resume tests.

---
 tests/qtest/ahci-test.c | 67 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
tests/qtest/ahci-test.c +67 -0
index 5c32ff2002..44799eea15 100644
@@ -1625,6 +1625,69 @@ static void test_cdrom_pio_multi(void)
     ahci_test_cdrom_read10(3, false);
 }
 
+/*
+ * Regression test: a buffered ATAPI read completing after a command
+ * engine restart must not dereference the cleared cur_cmd. Cover both
+ * PIO and DMA; the DMA variant is the reliable guard.
+ */
+static void test_atapi_engine_restart_in_flight(bool dma)
+{
+    AHCIQState *ahci;
+    AHCICommand *cmd;
+    unsigned char *tx;
+    char *iso;
+    int fd;
+    uint8_t port;
+    uint64_t buffer;
+    uint64_t iso_size = (uint64_t)ATAPI_SECTOR_SIZE * 2;
+
+    fd = prepare_iso(iso_size, &tx, &iso);
+
+    ahci = ahci_boot_and_enable("-drive if=none,id=drive0,"
+                                "file=blkdebug::%s,format=raw,readonly=on "
+                                "-M q35 "
+                                "-device ide-cd,drive=drive0 ", iso);
+    port = ahci_port_select(ahci);
+
+    buffer = ahci_alloc(ahci, ATAPI_SECTOR_SIZE);
+    qtest_memset(ahci->parent->qts, buffer, 0x00, ATAPI_SECTOR_SIZE);
+
+    /* Suspend the next backend read so the ATAPI read stays in flight. */
+    g_free(qtest_hmp(ahci->parent->qts,
+                     "qemu-io drive0 \"break read_aio rd\""));
+
+    cmd = ahci_atapi_command_create(CMD_ATAPI_READ_10, ATAPI_SECTOR_SIZE,
+                                    dma);
+    ahci_command_adjust(cmd, 0, buffer, ATAPI_SECTOR_SIZE, 0);
+    ahci_command_commit(ahci, cmd, port);
+    ahci_command_issue_async(ahci, cmd);
+
+    /* Stop and restart the command engine to re-map the command list. */
+    ahci_px_clr(ahci, port, AHCI_PX_CMD, AHCI_PX_CMD_ST);
+    ahci_px_set(ahci, port, AHCI_PX_CMD, AHCI_PX_CMD_ST);
+
+    g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"resume rd\""));
+
+    /* Round-trip through the device to confirm qemu is still alive. */
+    ahci_px_rreg(ahci, port, AHCI_PX_TFD);
+
+    ahci_command_free(cmd);
+    ahci_free(ahci, buffer);
+    g_free(tx);
+    ahci_shutdown(ahci);
+    remove_iso(fd, iso);
+}
+
+static void test_atapi_engine_restart_pio(void)
+{
+    test_atapi_engine_restart_in_flight(false);
+}
+
+static void test_atapi_engine_restart_dma(void)
+{
+    test_atapi_engine_restart_in_flight(true);
+}
+
 /* Regression test: Test that a READ_CD command with a BCL of 0 but a size of 0
  * completes as a NOP instead of erroring out. */
 static void test_atapi_bcl(void)
@@ -2042,6 +2105,10 @@ int main(int argc, char **argv)
 
     qtest_add_func("/ahci/cdrom/pio/bcl", test_atapi_bcl);
     qtest_add_func("/ahci/cdrom/eject", test_atapi_tray);
+    qtest_add_func("/ahci/cdrom/engine_restart/pio",
+                   test_atapi_engine_restart_pio);
+    qtest_add_func("/ahci/cdrom/engine_restart/dma",
+                   test_atapi_engine_restart_dma);
 
     ret = g_test_run();
 
Signature · 1 line
2.53.0
#7 lore raw

PATCH 3/9 tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test

In reply to Denis V. Lunev via qemu development

cdrom_pio_impl() and test_cdrom_dma() duplicate the same image setup and data-integrity check around two different transfer mechanisms. Fold them into a single cdrom_read_impl(nblocks, flags) helper, with a CDROM_PIO/CDROM_DMA flag selecting the transfer, so further read coverage can be added once for both paths.

No functional change: /ide/cdrom/pio, pio_large and dma run exactly as before.

---
 tests/qtest/ide-test.c | 146 +++++++++++++++++++----------------------
 1 file changed, 68 insertions(+), 78 deletions(-)
tests/qtest/ide-test.c +68 -78
index 721e78170b..0fcfd0688c 100644
@@ -982,6 +982,7 @@ static void send_scsi_cdb_read10(QPCIDevice *dev, QPCIBar ide_bar,
     }
 }
 
+
 static void nsleep(QTestState *qts, int64_t nsecs)
 {
     const struct timespec val = { .tv_nsec = nsecs };
@@ -1034,8 +1035,12 @@ static void ide_wait_intr(QTestState *qts, int irq)
     g_assert_not_reached();
 }
 
-static void cdrom_pio_impl(int nblocks)
+#define CDROM_PIO 0
+#define CDROM_DMA (1 << 0)
+
+static void cdrom_read_impl(int nblocks, unsigned flags)
 {
+    bool dma = flags & CDROM_DMA;
     QTestState *qts;
     QPCIDevice *dev;
     QPCIBar bmdma_bar, ide_bar;
@@ -1063,57 +1068,75 @@ static void cdrom_pio_impl(int nblocks)
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
     qtest_irq_intercept_in(qts, "ioapic");
 
-    /* PACKET command on device 0 */
-    qpci_io_writeb(dev, ide_bar, reg_device, 0);
-    qpci_io_writeb(dev, ide_bar, reg_lba_middle, BYTE_COUNT_LIMIT & 0xFF);
-    qpci_io_writeb(dev, ide_bar, reg_lba_high, (BYTE_COUNT_LIMIT >> 8 & 0xFF));
-    qpci_io_writeb(dev, ide_bar, reg_command, CMD_PACKET);
-    /* HP0: Check_Status_A State */
-    nsleep(qts, 400);
-    data = ide_wait_clear(qts, BSY);
-    /* HP1: Send_Packet State */
-    assert_bit_set(data, DRQ | DRDY);
-    assert_bit_clear(data, ERR | DF | BSY);
-
-    /* SCSI CDB (READ10) -- read n*2048 bytes from block 0 */
-    send_scsi_cdb_read10(dev, ide_bar, 0, nblocks);
-
-    /* Read data back: occurs in bursts of 'BYTE_COUNT_LIMIT' bytes.
-     * If BYTE_COUNT_LIMIT is odd, we transfer BYTE_COUNT_LIMIT - 1 bytes.
-     * We allow an odd limit only when the remaining transfer size is
-     * less than BYTE_COUNT_LIMIT. However, SCSI's read10 command can only
-     * request n blocks, so our request size is always even.
-     * For this reason, we assume there is never a hanging byte to fetch. */
-    g_assert(!(rxsize & 1));
-    limit = BYTE_COUNT_LIMIT & ~1;
-    for (i = 0; i < DIV_ROUND_UP(rxsize, limit); i++) {
-        size_t offset = i * (limit / 2);
-        size_t rem = (rxsize / 2) - offset;
-
-        /* HP3: INTRQ_Wait */
-        ide_wait_intr(qts, IDE_PRIMARY_IRQ);
+    if (dma) {
+        uintptr_t guest_buf = guest_alloc(&guest_malloc, rxsize);
+        PrdtEntry prdt[1];
+
+        prdt[0].addr = cpu_to_le32(guest_buf);
+        prdt[0].size = cpu_to_le32(rxsize | PRDT_EOT);
 
-        /* HP2: Check_Status_B (and clear IRQ) */
+        send_dma_request_dev(qts, dev, bmdma_bar, ide_bar, CMD_PACKET, 0,
+                             nblocks, prdt, ARRAY_SIZE(prdt),
+                             send_scsi_cdb_read10);
+
+        qtest_memread(qts, guest_buf, rx, rxsize);
+    } else {
+        /* PACKET command on device 0 */
+        qpci_io_writeb(dev, ide_bar, reg_device, 0);
+        qpci_io_writeb(dev, ide_bar, reg_lba_middle, BYTE_COUNT_LIMIT & 0xFF);
+        qpci_io_writeb(dev, ide_bar, reg_lba_high,
+                       (BYTE_COUNT_LIMIT >> 8 & 0xFF));
+        qpci_io_writeb(dev, ide_bar, reg_command, CMD_PACKET);
+        /* HP0: Check_Status_A State */
+        nsleep(qts, 400);
         data = ide_wait_clear(qts, BSY);
+        /* HP1: Send_Packet State */
         assert_bit_set(data, DRQ | DRDY);
         assert_bit_clear(data, ERR | DF | BSY);
 
-        /* HP4: Transfer_Data */
-        for (j = 0; j < MIN((limit / 2), rem); j++) {
-            rx[offset + j] = cpu_to_le16(qpci_io_readw(dev, ide_bar,
-                                                       reg_data));
+        /* SCSI CDB (READ10) -- read n*2048 bytes from block 0 */
+        send_scsi_cdb_read10(dev, ide_bar, 0, nblocks);
+
+        /*
+         * Read data back: occurs in bursts of 'BYTE_COUNT_LIMIT' bytes.
+         * If BYTE_COUNT_LIMIT is odd, we transfer BYTE_COUNT_LIMIT - 1 bytes.
+         * We allow an odd limit only when the remaining transfer size is
+         * less than BYTE_COUNT_LIMIT. However, SCSI's read10 command can only
+         * request n blocks, so our request size is always even.
+         * For this reason, we assume there is never a hanging byte to fetch.
+         */
+        g_assert(!(rxsize & 1));
+        limit = BYTE_COUNT_LIMIT & ~1;
+        for (i = 0; i < DIV_ROUND_UP(rxsize, limit); i++) {
+            size_t offset = i * (limit / 2);
+            size_t rem = (rxsize / 2) - offset;
+
+            /* HP3: INTRQ_Wait */
+            ide_wait_intr(qts, IDE_PRIMARY_IRQ);
+
+            /* HP2: Check_Status_B (and clear IRQ) */
+            data = ide_wait_clear(qts, BSY);
+            assert_bit_set(data, DRQ | DRDY);
+            assert_bit_clear(data, ERR | DF | BSY);
+
+            /* HP4: Transfer_Data */
+            for (j = 0; j < MIN((limit / 2), rem); j++) {
+                rx[offset + j] = cpu_to_le16(qpci_io_readw(dev, ide_bar,
+                                                           reg_data));
+            }
         }
-    }
 
-    /* Check for final completion IRQ */
-    ide_wait_intr(qts, IDE_PRIMARY_IRQ);
+        /* Check for final completion IRQ */
+        ide_wait_intr(qts, IDE_PRIMARY_IRQ);
 
-    /* Sanity check final state */
-    data = ide_wait_clear(qts, DRQ);
-    assert_bit_set(data, DRDY);
-    assert_bit_clear(data, DRQ | ERR | DF | BSY);
+        /* Sanity check final state */
+        data = ide_wait_clear(qts, DRQ);
+        assert_bit_set(data, DRDY);
+        assert_bit_clear(data, DRQ | ERR | DF | BSY);
+    }
 
     g_assert_cmpint(memcmp(pattern, rx, rxsize), ==, 0);
+
     g_free(pattern);
     g_free(rx);
     test_bmdma_teardown(qts);
@@ -1122,51 +1145,18 @@ static void cdrom_pio_impl(int nblocks)
 
 static void test_cdrom_pio(void)
 {
-    cdrom_pio_impl(1);
+    cdrom_read_impl(1, CDROM_PIO);
 }
 
 static void test_cdrom_pio_large(void)
 {
     /* Test a few loops of the PIO DRQ mechanism. */
-    cdrom_pio_impl(BYTE_COUNT_LIMIT * 4 / ATAPI_BLOCK_SIZE);
+    cdrom_read_impl(BYTE_COUNT_LIMIT * 4 / ATAPI_BLOCK_SIZE, CDROM_PIO);
 }
 
-
 static void test_cdrom_dma(void)
 {
-    QTestState *qts;
-    static const size_t len = ATAPI_BLOCK_SIZE;
-    size_t ret;
-    char *pattern = g_malloc(ATAPI_BLOCK_SIZE * 16);
-    char *rx = g_malloc0(len);
-    uintptr_t guest_buf;
-    PrdtEntry prdt[1];
-    FILE *fh;
-
-    qts = ide_test_start(
-            "-drive if=none,file=%s,media=cdrom,format=raw,id=sr0,index=0 "
-            "-device ide-cd,drive=sr0,bus=ide.0", tmp_path[0]);
-    qtest_irq_intercept_in(qts, "ioapic");
-
-    guest_buf = guest_alloc(&guest_malloc, len);
-    prdt[0].addr = cpu_to_le32(guest_buf);
-    prdt[0].size = cpu_to_le32(len | PRDT_EOT);
-
-    generate_pattern(pattern, ATAPI_BLOCK_SIZE * 16, ATAPI_BLOCK_SIZE);
-    fh = fopen(tmp_path[0], "wb+");
-    ret = fwrite(pattern, ATAPI_BLOCK_SIZE, 16, fh);
-    g_assert_cmpint(ret, ==, 16);
-    fclose(fh);
-
-    send_dma_request(qts, CMD_PACKET, 0, 1, prdt, 1, send_scsi_cdb_read10);
-
-    /* Read back data from guest memory into local qtest memory */
-    qtest_memread(qts, guest_buf, rx, len);
-    g_assert_cmpint(memcmp(pattern, rx, len), ==, 0);
-
-    g_free(pattern);
-    g_free(rx);
-    test_bmdma_teardown(qts);
+    cdrom_read_impl(1, CDROM_DMA);
 }
 
 int main(int argc, char **argv)
Signature · 1 line
2.53.0
#8 lore raw

PATCH 4/9 tests/qtest/ide-test: add a multi-sector ATAPI DMA read test

In reply to Denis V. Lunev via qemu development

test_cdrom_pio_large() already exercises a multi-sector PIO read. Add the DMA counterpart through the same cdrom_read_impl() helper so the multi-block ATAPI DMA read path gets equivalent coverage.

---
 tests/qtest/ide-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/ide-test.c +6 -0
index 0fcfd0688c..2c674edf15 100644
@@ -1159,6 +1159,11 @@ static void test_cdrom_dma(void)
     cdrom_read_impl(1, CDROM_DMA);
 }
 
+static void test_cdrom_dma_large(void)
+{
+    cdrom_read_impl(BYTE_COUNT_LIMIT * 4 / ATAPI_BLOCK_SIZE, CDROM_DMA);
+}
+
 int main(int argc, char **argv)
 {
     const char *base;
@@ -1218,6 +1223,7 @@ int main(int argc, char **argv)
     qtest_add_func("/ide/cdrom/pio", test_cdrom_pio);
     qtest_add_func("/ide/cdrom/pio_large", test_cdrom_pio_large);
     qtest_add_func("/ide/cdrom/dma", test_cdrom_dma);
+    qtest_add_func("/ide/cdrom/dma_large", test_cdrom_dma_large);
 
     ret = g_test_run();
 
Signature · 1 line
2.53.0
#9 lore raw

PATCH 5/9 tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads

In reply to Denis V. Lunev via qemu development

READ CD with the field-selector set to 0xf8 returns full 2352-byte raw sectors (sync + header + 2048 data + EDC/ECC), driving the ATAPI raw read path that READ10 never touches. Add a send_scsi_cdb_read_cd() helper and a CDROM_RAW flag to cdrom_read_impl(), then exercise both PIO and DMA. The PIO case uses a byte-count limit spanning several raw sectors so the device must rebuffer mid-burst, and each sector's 2048-byte payload is verified at its in-sector offset.

---
 tests/qtest/ide-test.c | 75 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 69 insertions(+), 6 deletions(-)
tests/qtest/ide-test.c +69 -6
index 2c674edf15..b36b0f8875 100644
@@ -48,6 +48,10 @@
 
 #define ATAPI_BLOCK_SIZE 2048
 
+/* Raw READ CD sector: 12 sync + 4 header + 2048 data + 288 EDC/ECC. */
+#define ATAPI_RAW_SIZE   2352
+#define ATAPI_RAW_DATA   16
+
 /* How many bytes to receive via ATAPI PIO at one time.
  * Must be less than 0xFFFF. */
 #define BYTE_COUNT_LIMIT 5120
@@ -982,6 +986,40 @@ static void send_scsi_cdb_read10(QPCIDevice *dev, QPCIBar ide_bar,
     }
 }
 
+typedef struct ReadCDCDB {
+    uint8_t opcode;
+    uint8_t sector_type;
+    uint32_t lba;
+    uint8_t length[3];
+    uint8_t main_channel;
+    uint8_t sub_channel;
+    uint8_t control;
+} __attribute__((__packed__)) ReadCDCDB;
+
+static void send_scsi_cdb_read_cd(QPCIDevice *dev, QPCIBar ide_bar,
+                                  uint64_t lba, int nblocks)
+{
+    ReadCDCDB pkt = { };
+    int i;
+
+    g_assert_cmpint(lba, <=, UINT32_MAX);
+    g_assert_cmpint(nblocks, >=, 0);
+    g_assert_cmpint(nblocks, <=, 0xffffff);
+
+    /* Construct SCSI CDB packet */
+    pkt.opcode = 0xbe;
+    pkt.lba = cpu_to_be32(lba);
+    pkt.length[0] = (nblocks >> 16) & 0xff;
+    pkt.length[1] = (nblocks >> 8) & 0xff;
+    pkt.length[2] = nblocks & 0xff;
+    pkt.main_channel = 0xf8; /* sync + headers + user data + EDC/ECC: 2352 */
+
+    /* Send Packet */
+    for (i = 0; i < sizeof(ReadCDCDB) / 2; i++) {
+        qpci_io_writew(dev, ide_bar, reg_data,
+                       le16_to_cpu(((uint16_t *)&pkt)[i]));
+    }
+}
 
 static void nsleep(QTestState *qts, int64_t nsecs)
 {
@@ -1037,10 +1075,12 @@ static void ide_wait_intr(QTestState *qts, int irq)
 
 #define CDROM_PIO 0
 #define CDROM_DMA (1 << 0)
+#define CDROM_RAW (1 << 1)
 
 static void cdrom_read_impl(int nblocks, unsigned flags)
 {
     bool dma = flags & CDROM_DMA;
+    bool raw = flags & CDROM_RAW;
     QTestState *qts;
     QPCIDevice *dev;
     QPCIBar bmdma_bar, ide_bar;
@@ -1048,8 +1088,11 @@ static void cdrom_read_impl(int nblocks, unsigned flags)
     int patt_blocks = MAX(16, nblocks);
     size_t patt_len = ATAPI_BLOCK_SIZE * patt_blocks;
     char *pattern = g_malloc(patt_len);
-    size_t rxsize = ATAPI_BLOCK_SIZE * nblocks;
+    unsigned xfer = raw ? ATAPI_RAW_SIZE : ATAPI_BLOCK_SIZE;
+    size_t rxsize = xfer * nblocks;
     uint16_t *rx = g_malloc0(rxsize);
+    void (*send_cdb)(QPCIDevice *, QPCIBar, uint64_t, int) =
+        raw ? send_scsi_cdb_read_cd : send_scsi_cdb_read10;
     int i, j;
     uint8_t data;
     uint16_t limit;
@@ -1076,8 +1119,7 @@ static void cdrom_read_impl(int nblocks, unsigned flags)
         prdt[0].size = cpu_to_le32(rxsize | PRDT_EOT);
 
         send_dma_request_dev(qts, dev, bmdma_bar, ide_bar, CMD_PACKET, 0,
-                             nblocks, prdt, ARRAY_SIZE(prdt),
-                             send_scsi_cdb_read10);
+                             nblocks, prdt, ARRAY_SIZE(prdt), send_cdb);
 
         qtest_memread(qts, guest_buf, rx, rxsize);
     } else {
@@ -1094,8 +1136,7 @@ static void cdrom_read_impl(int nblocks, unsigned flags)
         assert_bit_set(data, DRQ | DRDY);
         assert_bit_clear(data, ERR | DF | BSY);
 
-        /* SCSI CDB (READ10) -- read n*2048 bytes from block 0 */
-        send_scsi_cdb_read10(dev, ide_bar, 0, nblocks);
+        send_cdb(dev, ide_bar, 0, nblocks);
 
         /*
          * Read data back: occurs in bursts of 'BYTE_COUNT_LIMIT' bytes.
@@ -1135,7 +1176,17 @@ static void cdrom_read_impl(int nblocks, unsigned flags)
         assert_bit_clear(data, DRQ | ERR | DF | BSY);
     }
 
-    g_assert_cmpint(memcmp(pattern, rx, rxsize), ==, 0);
+    if (raw) {
+        /* The 2048-byte payload of each raw sector sits past its header. */
+        for (i = 0; i < nblocks; i++) {
+            uint8_t *sec = (uint8_t *)rx + i * ATAPI_RAW_SIZE + ATAPI_RAW_DATA;
+
+            g_assert_cmpint(memcmp(sec, pattern + i * ATAPI_BLOCK_SIZE,
+                                   ATAPI_BLOCK_SIZE), ==, 0);
+        }
+    } else {
+        g_assert_cmpint(memcmp(pattern, rx, rxsize), ==, 0);
+    }
 
     g_free(pattern);
     g_free(rx);
@@ -1164,6 +1215,16 @@ static void test_cdrom_dma_large(void)
     cdrom_read_impl(BYTE_COUNT_LIMIT * 4 / ATAPI_BLOCK_SIZE, CDROM_DMA);
 }
 
+static void test_cdrom_pio_raw(void)
+{
+    cdrom_read_impl(4, CDROM_RAW);
+}
+
+static void test_cdrom_dma_raw(void)
+{
+    cdrom_read_impl(4, CDROM_DMA | CDROM_RAW);
+}
+
 int main(int argc, char **argv)
 {
     const char *base;
@@ -1224,6 +1285,8 @@ int main(int argc, char **argv)
     qtest_add_func("/ide/cdrom/pio_large", test_cdrom_pio_large);
     qtest_add_func("/ide/cdrom/dma", test_cdrom_dma);
     qtest_add_func("/ide/cdrom/dma_large", test_cdrom_dma_large);
+    qtest_add_func("/ide/cdrom/pio_raw", test_cdrom_pio_raw);
+    qtest_add_func("/ide/cdrom/dma_raw", test_cdrom_dma_raw);
 
     ret = g_test_run();
 
Signature · 1 line
2.53.0
#10 lore raw

PATCH 6/9 tests/qtest/libqos/ahci: support raw (2352-byte) READ CD

In reply to Denis V. Lunev via qemu development

ahci_exec() always builds ATAPI commands with a 2048-byte logical sector size, so it cannot drive a READ CD that returns full 2352-byte raw sectors. Add an atapi_raw option that sets the READ CD field-selector to 0xf8 and the command's sector size to 2352 before the transfer is sized, so the derived block count stays correct while the buffer and byte counts cover the raw sectors.

---
 tests/qtest/libqos/ahci.c | 5 +++++
 tests/qtest/libqos/ahci.h | 2 ++
 2 files changed, 7 insertions(+)
tests/qtest/libqos/ahci.c +5 -0
index 0621a6c477..e3019a1eca 100644
@@ -707,6 +707,11 @@ void ahci_exec(AHCIQState *ahci, uint8_t port,
     if (opts->atapi) {
         uint16_t bcl = opts->set_bcl ? opts->bcl : ATAPI_SECTOR_SIZE;
         cmd = ahci_atapi_command_create(op, bcl, opts->atapi_dma);
+        if (opts->atapi_raw) {
+            /* request full 2352-byte raw sectors; sector_size must match */
+            cmd->atapi_cmd[9] = 0xf8;
+            cmd->sector_size = ATAPI_RAW_SECTOR_SIZE;
+        }
     } else {
         cmd = ahci_command_create(op);
     }
tests/qtest/libqos/ahci.h +2 -0
index a0487a1557..2a48a7523d 100644
@@ -242,6 +242,7 @@
 
 #define AHCI_SECTOR_SIZE                (512)
 #define ATAPI_SECTOR_SIZE              (2048)
+#define ATAPI_RAW_SECTOR_SIZE          (2352)
 
 #define AHCI_SIGNATURE_CDROM     (0xeb140101)
 #define AHCI_SIGNATURE_DISK      (0x00000101)
@@ -485,6 +486,7 @@ typedef struct AHCIOpts {
     uint64_t buffer;    /* Pointer to source or destination guest buffer */
     bool atapi;         /* ATAPI command? */
     bool atapi_dma;     /* Use DMA for ATAPI? */
+    bool atapi_raw;     /* READ CD returning 2352-byte raw sectors */
     bool error;
     int (*pre_cb)(AHCIQState*, AHCICommand*, const struct AHCIOpts *);
     int (*mid_cb)(AHCIQState*, AHCICommand*, const struct AHCIOpts *);
Signature · 1 line
2.53.0
#11 lore raw

PATCH 7/9 tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads

In reply to Denis V. Lunev via qemu development

Add /ahci/cdrom/{pio,dma}/raw: read several sectors with READ CD in raw mode (atapi_raw), so the ATAPI 2352-byte unpack path is exercised through the AHCI delivery, which IDE coverage does not reach. Each sector's 2048-byte payload is verified at its in-sector offset.

The PIO case uses a byte-count limit of one raw sector per DRQ burst:
libqos asserts a one-sector PIO transfer, and the multi-sector unpack loop is already covered by the IDE raw test.

---
 tests/qtest/ahci-test.c | 68 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
tests/qtest/ahci-test.c +68 -0
index 44799eea15..58bc04b3ef 100644
@@ -1565,6 +1565,31 @@ static int ahci_cb_cmp_buff(AHCIQState *ahci, AHCICommand *cmd,
     return 0;
 }
 
+static int ahci_cb_cmp_raw(AHCIQState *ahci, AHCICommand *cmd,
+                           const AHCIOpts *opts)
+{
+    unsigned char *tx = opts->opaque;
+    unsigned char *rx;
+    unsigned i, nsectors;
+
+    if (!opts->size) {
+        return 0;
+    }
+
+    nsectors = opts->size / ATAPI_RAW_SECTOR_SIZE;
+    rx = g_malloc0(opts->size);
+    qtest_bufread(ahci->parent->qts, opts->buffer, rx, opts->size);
+    /* Each raw sector carries its 2048-byte payload past a 16-byte header. */
+    for (i = 0; i < nsectors; i++) {
+        g_assert_cmphex(memcmp(rx + i * ATAPI_RAW_SECTOR_SIZE + 16,
+                               tx + i * ATAPI_SECTOR_SIZE,
+                               ATAPI_SECTOR_SIZE), ==, 0);
+    }
+    g_free(rx);
+
+    return 0;
+}
+
 static void ahci_test_cdrom(int nsectors, bool dma, uint8_t cmd,
                             bool override_bcl, uint16_t bcl)
 {
@@ -1625,6 +1650,47 @@ static void test_cdrom_pio_multi(void)
     ahci_test_cdrom_read10(3, false);
 }
 
+static void ahci_test_cdrom_raw(int nsectors, bool dma)
+{
+    AHCIQState *ahci;
+    unsigned char *tx;
+    char *iso;
+    int fd;
+    AHCIOpts opts = {
+        .size = (uint64_t)ATAPI_RAW_SECTOR_SIZE * nsectors,
+        .atapi = true,
+        .atapi_dma = dma,
+        .atapi_raw = true,
+        .set_bcl = true,
+        .bcl = ATAPI_RAW_SECTOR_SIZE, /* one raw sector per DRQ burst */
+        .post_cb = ahci_cb_cmp_raw,
+    };
+    uint64_t iso_size = (uint64_t)ATAPI_SECTOR_SIZE * (nsectors + 1);
+
+    fd = prepare_iso(iso_size, &tx, &iso);
+    opts.opaque = tx;
+
+    ahci = ahci_boot_and_enable("-drive if=none,id=drive0,file=%s,format=raw "
+                                "-M q35 "
+                                "-device ide-cd,drive=drive0 ", iso);
+
+    ahci_exec(ahci, ahci_port_select(ahci), CMD_ATAPI_READ_CD, &opts);
+
+    g_free(tx);
+    ahci_shutdown(ahci);
+    remove_iso(fd, iso);
+}
+
+static void test_cdrom_dma_raw(void)
+{
+    ahci_test_cdrom_raw(3, true);
+}
+
+static void test_cdrom_pio_raw(void)
+{
+    ahci_test_cdrom_raw(3, false);
+}
+
 /*
  * Regression test: a buffered ATAPI read completing after a command
  * engine restart must not dereference the cleared cur_cmd. Cover both
@@ -2100,8 +2166,10 @@ int main(int argc, char **argv)
 
     qtest_add_func("/ahci/cdrom/dma/single", test_cdrom_dma);
     qtest_add_func("/ahci/cdrom/dma/multi", test_cdrom_dma_multi);
+    qtest_add_func("/ahci/cdrom/dma/raw", test_cdrom_dma_raw);
     qtest_add_func("/ahci/cdrom/pio/single", test_cdrom_pio);
     qtest_add_func("/ahci/cdrom/pio/multi", test_cdrom_pio_multi);
+    qtest_add_func("/ahci/cdrom/pio/raw", test_cdrom_pio_raw);
 
     qtest_add_func("/ahci/cdrom/pio/bcl", test_atapi_bcl);
     qtest_add_func("/ahci/cdrom/eject", test_atapi_tray);
Signature · 1 line
2.53.0
#12 lore raw

PATCH 8/9 hw/ide/atapi: read the whole elementary transfer asynchronously

In reply to Denis V. Lunev via qemu development

An ATAPI PIO read whose byte-count limit spans more than one CD sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did this with a synchronous blk_pread(), which runs blk_wait_while_drained() before issuing the request.

If a drain is in progress when that completion runs -- as happens when a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while an ATAPI read is in flight on the same QEMU -- the nested read is queued until the drained section ends while the outer completion still holds blk->in_flight. bdrv_drain_all_begin() then waits forever for that in_flight count to drop: the main loop is wedged in the drain with the BQL held, and every other QMP/monitor operation blocks behind it.

Read the whole elementary transfer in a single asynchronous request up front instead, so no read is ever issued in the middle of a burst. cd_read_sector() now reads all the sectors a burst spans (the raw 2352-byte case is unpacked in place on completion) and cd_read_sector_sync() is removed. The DMA path already batched its reads and is unchanged.

---
 hw/ide/atapi.c | 180 +++++++++++++++++++++++--------------------------
 1 file changed, 84 insertions(+), 96 deletions(-)
hw/ide/atapi.c +84 -96
index a42b748521..0ea149ad8c 100644
@@ -88,46 +88,14 @@ static void cd_data_to_raw(uint8_t *buf, int lba)
     memset(buf, 0, 288);
 }
 
-static int
-cd_read_sector_sync(IDEState *s)
-{
-    int ret;
-    block_acct_start(blk_get_stats(s->blk), &s->acct,
-                     ATAPI_SECTOR_SIZE, BLOCK_ACCT_READ);
-
-    trace_cd_read_sector_sync(s->lba);
-
-    switch (s->cd_sector_size) {
-    case 2048:
-        ret = blk_pread(s->blk, (int64_t)s->lba << ATAPI_SECTOR_BITS,
-                        ATAPI_SECTOR_SIZE, s->io_buffer, 0);
-        break;
-    case 2352:
-        ret = blk_pread(s->blk, (int64_t)s->lba << ATAPI_SECTOR_BITS,
-                        ATAPI_SECTOR_SIZE, s->io_buffer + 16, 0);
-        if (ret >= 0) {
-            cd_data_to_raw(s->io_buffer, s->lba);
-        }
-        break;
-    default:
-        block_acct_invalid(blk_get_stats(s->blk), BLOCK_ACCT_READ);
-        return -EIO;
-    }
-
-    if (ret < 0) {
-        block_acct_failed(blk_get_stats(s->blk), &s->acct);
-    } else {
-        block_acct_done(blk_get_stats(s->blk), &s->acct);
-        s->lba++;
-        s->io_buffer_index = 0;
-    }
-
-    return ret;
-}
-
 static void cd_read_sector_cb(void *opaque, int ret)
 {
     IDEState *s = opaque;
+    int et = s->elementary_transfer_size;
+    int skip = s->io_buffer_index;
+    int nsec = DIV_ROUND_UP(skip + et, s->cd_sector_size);
+    uint8_t *buf;
+    int i;
 
     trace_cd_read_sector_cb(s->lba, ret);
 
@@ -140,34 +108,64 @@ static void cd_read_sector_cb(void *opaque, int ret)
     block_acct_done(blk_get_stats(s->blk), &s->acct);
 
     if (s->cd_sector_size == 2352) {
-        cd_data_to_raw(s->io_buffer, s->lba);
+        /* unpack back-to-front so a sector never clobbers an unmoved one */
+        for (i = nsec - 1; i >= 0; i--) {
+            memmove(s->io_buffer + i * 2352 + 16, s->io_buffer + i * 2048,
+                    ATAPI_SECTOR_SIZE);
+            cd_data_to_raw(s->io_buffer + i * 2352, s->lba + i);
+        }
     }
 
-    s->lba++;
-    s->io_buffer_index = 0;
     s->status &= ~BUSY_STAT;
 
-    ide_atapi_cmd_reply_end(s);
+    s->nsector = (s->nsector & ~7) | ATAPI_INT_REASON_IO;
+    s->lcyl = et & 0xff;
+    s->hcyl = (et >> 8) & 0xff;
+    ide_bus_set_irq(s->bus);
+
+    /* a boundary sector shared with the next burst is re-read there */
+    buf = s->io_buffer + skip;
+    s->packet_transfer_size -= et;
+    s->lba += (skip + et) / s->cd_sector_size;
+    s->io_buffer_index = (skip + et) % s->cd_sector_size;
+    s->elementary_transfer_size = 0;
+
+    if (ide_transfer_start_norecurse(s, buf, et, ide_atapi_cmd_reply_end)) {
+        ide_atapi_cmd_reply_end(s);
+    }
 }
 
+/*
+ * Read the whole elementary transfer (one DRQ burst) in a single async
+ * request. No read is issued mid-burst, so unlike the old synchronous
+ * rebuffer it cannot deadlock against a concurrent drain.
+ */
 static int cd_read_sector(IDEState *s)
 {
-    void *buf;
+    int et = s->elementary_transfer_size;
+    int skip = s->io_buffer_index;
+    int nsec = DIV_ROUND_UP(skip + et, s->cd_sector_size);
 
     if (s->cd_sector_size != 2048 && s->cd_sector_size != 2352) {
         block_acct_invalid(blk_get_stats(s->blk), BLOCK_ACCT_READ);
         return -EINVAL;
     }
 
-    buf = (s->cd_sector_size == 2352) ? s->io_buffer + 16 : s->io_buffer;
-    qemu_iovec_init_buf(&s->qiov, buf, ATAPI_SECTOR_SIZE);
+    /* a burst is bounded by the byte count limit, so it fits io_buffer */
+    assert(nsec * s->cd_sector_size <= s->io_buffer_total_len);
+
+    /*
+     * Read the payload packed at the front of io_buffer; the 2352 raw case is
+     * unpacked into place on completion.
+     */
+    qemu_iovec_init_buf(&s->qiov, s->io_buffer, nsec * ATAPI_SECTOR_SIZE);
 
     trace_cd_read_sector(s->lba);
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
-                     ATAPI_SECTOR_SIZE, BLOCK_ACCT_READ);
+                     nsec * ATAPI_SECTOR_SIZE, BLOCK_ACCT_READ);
 
-    ide_buffered_readv(s, (int64_t)s->lba << 2, &s->qiov, 4,
+    ide_buffered_readv(s, (int64_t)s->lba << 2, &s->qiov, nsec * 4,
                        cd_read_sector_cb, s);
 
     s->status |= BUSY_STAT;
@@ -222,59 +220,49 @@ static uint16_t atapi_byte_count_limit(IDEState *s)
 void ide_atapi_cmd_reply_end(IDEState *s)
 {
     int byte_count_limit, size, ret;
-    while (s->packet_transfer_size > 0) {
-        trace_ide_atapi_cmd_reply_end(s, s->packet_transfer_size,
-                                      s->elementary_transfer_size,
-                                      s->io_buffer_index);
-
-        /* see if a new sector must be read */
-        if (s->lba != -1 && s->io_buffer_index >= s->cd_sector_size) {
-            if (!s->elementary_transfer_size) {
-                ret = cd_read_sector(s);
-                if (ret < 0) {
-                    ide_atapi_io_error(s, ret);
-                }
-                return;
-            } else {
-                /* rebuffering within an elementary transfer is
-                 * only possible with a sync request because we
-                 * end up with a race condition otherwise */
-                ret = cd_read_sector_sync(s);
-                if (ret < 0) {
-                    ide_atapi_io_error(s, ret);
-                    return;
-                }
+
+    trace_ide_atapi_cmd_reply_end(s, s->packet_transfer_size,
+                                  s->elementary_transfer_size,
+                                  s->io_buffer_index);
+
+    if (s->lba != -1 && s->packet_transfer_size > 0) {
+        byte_count_limit = atapi_byte_count_limit(s);
+        trace_ide_atapi_cmd_reply_end_bcl(s, byte_count_limit);
+        size = s->packet_transfer_size;
+        if (size > byte_count_limit) {
+            /* byte count limit must be even if this case */
+            if (byte_count_limit & 1) {
+                byte_count_limit--;
             }
+            size = byte_count_limit;
         }
-        if (s->elementary_transfer_size > 0) {
-            /* there are some data left to transmit in this elementary
-               transfer */
-            size = s->cd_sector_size - s->io_buffer_index;
-            if (size > s->elementary_transfer_size)
-                size = s->elementary_transfer_size;
-        } else {
-            /* a new transfer is needed */
-            s->nsector = (s->nsector & ~7) | ATAPI_INT_REASON_IO;
-            ide_bus_set_irq(s->bus);
-            byte_count_limit = atapi_byte_count_limit(s);
-            trace_ide_atapi_cmd_reply_end_bcl(s, byte_count_limit);
-            size = s->packet_transfer_size;
-            if (size > byte_count_limit) {
-                /* byte count limit must be even if this case */
-                if (byte_count_limit & 1)
-                    byte_count_limit--;
-                size = byte_count_limit;
-            }
-            s->lcyl = size & 0xff;
-            s->hcyl = size >> 8;
-            s->elementary_transfer_size = size;
-            /* we cannot transmit more than one sector at a time */
-            if (s->lba != -1) {
-                if (size > (s->cd_sector_size - s->io_buffer_index))
-                    size = (s->cd_sector_size - s->io_buffer_index);
+        s->elementary_transfer_size = size;
+        ret = cd_read_sector(s);
+        if (ret < 0) {
+            ide_atapi_io_error(s, ret);
+        }
+        return;
+    }
+
+    while (s->packet_transfer_size > 0) {
+        /* a new transfer is needed */
+        s->nsector = (s->nsector & ~7) | ATAPI_INT_REASON_IO;
+        ide_bus_set_irq(s->bus);
+        byte_count_limit = atapi_byte_count_limit(s);
+        trace_ide_atapi_cmd_reply_end_bcl(s, byte_count_limit);
+        size = s->packet_transfer_size;
+        if (size > byte_count_limit) {
+            /* byte count limit must be even if this case */
+            if (byte_count_limit & 1) {
+                byte_count_limit--;
             }
-            trace_ide_atapi_cmd_reply_end_new(s, s->status);
+            size = byte_count_limit;
         }
+        s->lcyl = size & 0xff;
+        s->hcyl = size >> 8;
+        s->elementary_transfer_size = size;
+        trace_ide_atapi_cmd_reply_end_new(s, s->status);
+
         s->packet_transfer_size -= size;
         s->elementary_transfer_size -= size;
         s->io_buffer_index += size;
@@ -329,7 +317,7 @@ static void ide_atapi_cmd_read_pio(IDEState *s, int lba, int nb_sectors,
     s->lba = lba;
     s->packet_transfer_size = nb_sectors * sector_size;
     s->elementary_transfer_size = 0;
-    s->io_buffer_index = sector_size;
+    s->io_buffer_index = 0;
     s->cd_sector_size = sector_size;
 
     ide_atapi_cmd_reply_end(s);
Signature · 1 line
2.53.0
#13 lore raw

PATCH 9/9 tests/qtest/ahci: regression test for ATAPI read vs. drain

In reply to Denis V. Lunev via qemu development

Add /ahci/cdrom/drain/{pio,dma}: issue a multi-sector ATAPI read whose byte-count limit spans two sectors so the device must rebuffer in the middle of the DRQ burst, hold the backend read in flight with a blkdebug delay, and fire x-blockdev-set-iothread -- which runs bdrv_drain_all_begin() exactly like a guest reset does through virtio_blk_stop_ioeventfd().

On the unfixed PIO path the nested sector fetch is queued behind the drain and the main loop wedges, so the test hangs. The DMA variant never rebuffers and serves as a sanity twin.

---
 tests/qtest/ahci-test.c | 66 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
tests/qtest/ahci-test.c +66 -0
index 58bc04b3ef..71d23fe56a 100644
@@ -1754,6 +1754,70 @@ static void test_atapi_engine_restart_dma(void)
     test_atapi_engine_restart_in_flight(true);
 }
 
+/*
+ * Regression test: a multi-sector ATAPI read fetches its later sectors from
+ * inside the first read's completion; a concurrent drain (as a guest reset
+ * triggers via bdrv_drain_all_begin) must not wedge on that nested read.
+ * blkdebug keeps the read in flight across x-blockdev-set-iothread.
+ */
+static void test_atapi_drain_in_flight(bool dma)
+{
+    AHCIQState *ahci;
+    AHCICommand *cmd;
+    unsigned char *tx;
+    char *iso;
+    int fd;
+    uint8_t port;
+    uint64_t buffer;
+    uint16_t bcl = ATAPI_SECTOR_SIZE * 2;
+    uint64_t iso_size = (uint64_t)ATAPI_SECTOR_SIZE * 3;
+
+    fd = prepare_iso(iso_size, &tx, &iso);
+
+    /* 1s read delay: a wide margin so the drain starts before it completes */
+    ahci = ahci_boot_and_enable(
+        "-blockdev driver=file,node-name=file0,filename=%s,read-only=on "
+        "-blockdev driver=blkdebug,node-name=cd0,image=file0,read-only=on,"
+        "inject-error.0.event=none,inject-error.0.iotype=read,"
+        "inject-error.0.errno=0,inject-error.0.delay-ns=1000000000 "
+        "-M q35 "
+        "-device ide-cd,drive=cd0 ", iso);
+    port = ahci_port_select(ahci);
+
+    buffer = ahci_alloc(ahci, bcl);
+    qtest_memset(ahci->parent->qts, buffer, 0x00, bcl);
+
+    cmd = ahci_atapi_command_create(CMD_ATAPI_READ_10, bcl, dma);
+    ahci_command_adjust(cmd, 0, buffer, bcl, 0);
+    ahci_command_commit(ahci, cmd, port);
+    ahci_command_issue_async(ahci, cmd);
+
+    /* Drain (all nodes) while the delayed read is still in flight. */
+    qtest_qmp_assert_success(ahci->parent->qts,
+        "{ 'execute': 'x-blockdev-set-iothread',"
+        "  'arguments': { 'node-name': 'cd0', 'iothread': null,"
+        "                 'force': true } }");
+
+    /* Round-trip through the device to confirm qemu is still alive. */
+    ahci_px_rreg(ahci, port, AHCI_PX_TFD);
+
+    ahci_command_free(cmd);
+    ahci_free(ahci, buffer);
+    g_free(tx);
+    ahci_shutdown(ahci);
+    remove_iso(fd, iso);
+}
+
+static void test_atapi_drain_pio(void)
+{
+    test_atapi_drain_in_flight(false);
+}
+
+static void test_atapi_drain_dma(void)
+{
+    test_atapi_drain_in_flight(true);
+}
+
 /* Regression test: Test that a READ_CD command with a BCL of 0 but a size of 0
  * completes as a NOP instead of erroring out. */
 static void test_atapi_bcl(void)
@@ -2177,6 +2241,8 @@ int main(int argc, char **argv)
                    test_atapi_engine_restart_pio);
     qtest_add_func("/ahci/cdrom/engine_restart/dma",
                    test_atapi_engine_restart_dma);
+    qtest_add_func("/ahci/cdrom/drain/pio", test_atapi_drain_pio);
+    qtest_add_func("/ahci/cdrom/drain/dma", test_atapi_drain_dma);
 
     ret = g_test_run();
 
Signature · 1 line
2.53.0
#14 lore raw
In reply to Denis V. Lunev via qemu development
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
 hw/ide/ahci.c             |   3 +
 hw/ide/atapi.c            | 180 +++++++++++++++-----------------
 tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
 tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
 tests/qtest/libqos/ahci.c |   8 ++
 tests/qtest/libqos/ahci.h |   2 +
 6 files changed, 433 insertions(+), 172 deletions(-)

ping

#15 lore raw
In reply to Denis V. Lunev via qemu development
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
 hw/ide/ahci.c             |   3 +
 hw/ide/atapi.c            | 180 +++++++++++++++-----------------
 tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
 tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
 tests/qtest/libqos/ahci.c |   8 ++
 tests/qtest/libqos/ahci.h |   2 +
 6 files changed, 433 insertions(+), 172 deletions(-)

ping v2

#16 lore raw
In reply to Denis V. Lunev via qemu development
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
 hw/ide/ahci.c             |   3 +
 hw/ide/atapi.c            | 180 +++++++++++++++-----------------
 tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
 tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
 tests/qtest/libqos/ahci.c |   8 ++
 tests/qtest/libqos/ahci.h |   2 +
 6 files changed, 433 insertions(+), 172 deletions(-)

patches 1-2 landed via Philippe, thanks a lot.

What about the rest?

This is real obscure problem observable in production.

Thank you in advance,
    Den

#17 lore raw
Quoted 80 lines · Denis V. Lunev
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
  hw/ide/ahci.c             |   3 +
  hw/ide/atapi.c            | 180 +++++++++++++++-----------------
  tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
  tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
  tests/qtest/libqos/ahci.c |   8 ++
  tests/qtest/libqos/ahci.h |   2 +
  6 files changed, 433 insertions(+), 172 deletions(-)
Quoted 5 lines

patches 1-2 landed via Philippe, thanks a lot.

What about the rest?

This is real obscure problem observable in production.

Since IDE is listed as "Odd Fixes" and no listed maintainers commented for a month on a fix, maybe you could step up as a new one... My 2 cents ;)

Kevin, as core block/ maintainer, any objection?

#18 lore raw
In reply to Denis V. Lunev via qemu development
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
 hw/ide/ahci.c             |   3 +
 hw/ide/atapi.c            | 180 +++++++++++++++-----------------
 tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
 tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
 tests/qtest/libqos/ahci.c |   8 ++
 tests/qtest/libqos/ahci.h |   2 +
 6 files changed, 433 insertions(+), 172 deletions(-)

Hi!

if nobody is able to take a look, will it be fine if I'll send pull request from my name for this?

Den

#19 lore raw
Quoted 80 lines · Denis V. Lunev
Quoted 71 lines · Denis V. Lunev

An in-flight ATAPI buffered CD read (ide_buffered_readv(), used by both the PIO and DMA reply paths in hw/ide/atapi.c) can collide with a concurrent port operation, and it breaks in two distinct ways. Both are guest-triggerable and reproduce on current master.

Neither was found by inspection: both are real issues reported against the qemu-kvm 10.0 downstream -- the crash from production, and the deadlock from the downstream live-migration validation suite -- on CD-ROMs backed by high-latency network storage, where a backend read outlives the guest's SATA command timeout.

1. Command engine restart -> NULL-deref crash (patches 1-2)

A guest can stop and restart a port's command engine (PxCMD.ST 1 -> 0 -> 1) while a read is in flight. The restart re-maps the command list and clears AHCIDevice.cur_cmd, but nothing tears down the outstanding read; its completion then dereferences the cleared cur_cmd in ahci_pio_transfer() / ahci_dma_rw_buf(). This is not the COMRESET path -- an engine restart does not run ide_reset(), so the drive's transfer state is preserved and the read still runs.

Patch 1 cancels the outstanding I/O when the command list is unmapped, reusing ide_cancel_dma_sync() exactly as ATAPI DEVICE RESET already does from the same MMIO context. Patch 2 is a qtest covering the PIO and DMA reply paths.

2. Concurrent drain -> main-loop deadlock (patch 8, repro in patch 9)

A multi-sector PIO read whose byte-count limit spans more than one sector must fetch the later sectors of a DRQ burst from inside the completion of the first, asynchronous read. cd_read_sector_sync() did that with a synchronous blk_pread(), which queues behind any drain in progress. When a guest reset reaches virtio_blk_stop_ioeventfd() -> bdrv_drain_all_begin() while such a read is in flight, the nested read is queued while the outer completion still holds blk->in_flight, so the drain never finishes:
the main loop wedges in the drain with the BQL held and every QMP/monitor operation blocks behind it.

Patch 8 reads the whole elementary transfer in a single asynchronous request, so no read is ever issued mid-burst; cd_read_sector_sync() is removed and the DMA path is unchanged. Patch 9 reproduces the hang via x-blockdev-set-iothread, which takes the same bdrv_drain_all_begin() path a guest reset does.

Patches 3-7 are the test groundwork the deadlock fix needs: they fold the IDE CD-ROM read tests into one parametrized helper and add the previously missing coverage -- multi-sector DMA, and raw 2352-byte READ CD on both IDE and AHCI -- so the rewritten ATAPI read path (including its in-place 2352 unpack) is exercised on both consumers. The test patches pass with or without patch 8, keeping the series bisectable.

Patches 1-2 stand alone as the engine-restart crash fix; they are included here because the deadlock work shares and extends the same ATAPI read path and its test scaffolding.

checkpatch is clean on all nine patches; the new and existing IDE and AHCI qtests pass.

Denis V. Lunev (9):
hw/ide/ahci: cancel in-flight buffered reads on command engine restart tests/qtest/ahci: test ATAPI read completing after engine restart tests/qtest/ide-test: parametrize the ATAPI CD-ROM read test tests/qtest/ide-test: add a multi-sector ATAPI DMA read test tests/qtest/ide-test: cover raw (2352-byte) ATAPI CD reads tests/qtest/libqos/ahci: support raw (2352-byte) READ CD tests/qtest/ahci: cover raw (2352-byte) ATAPI CD reads hw/ide/atapi: read the whole elementary transfer asynchronously tests/qtest/ahci: regression test for ATAPI read vs. drain

Quoted 8 lines
  hw/ide/ahci.c             |   3 +
  hw/ide/atapi.c            | 180 +++++++++++++++-----------------
  tests/qtest/ahci-test.c   | 201 ++++++++++++++++++++++++++++++++++++
  tests/qtest/ide-test.c    | 211 ++++++++++++++++++++++++--------------
  tests/qtest/libqos/ahci.c |   8 ++
  tests/qtest/libqos/ahci.h |   2 +
  6 files changed, 433 insertions(+), 172 deletions(-)
Quoted 4 lines

Hi!

if nobody is able to take a look, will it be fine if I'll send pull request from my name for this?

qemu-block@ is Cc'ed so no objection on my side but first send a MAINTAINERS patch stepping up for IDE, so you can include as first patch of the pullreq?