Skip to contentSkip to conversation

tests/qtest/ahci: fix a racy deadlock in the engine stop test

PATCH In review Denis V. Lunev 3 participants 4 messages Aug 30 – Sep 10

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/qtest/ahci: fix a racy deadlock in the engine stop test

From: Denis V. Lunev <[email protected]>

test_write_engine_stop_in_flight() resumes the suspended backend write and goes straight on to restart the command engine and read the second sector back. Whether the write has retired by then is a race. While it is still outstanding the drive is busy, and handle_cmd() drops a command that arrives in that state instead of queueing it. Nothing retries the drop, so the PxCI slot stays set and no D2H FIS is posted. ahci_command_wait() polls both with no timeout, so the test deadlocks until the harness kills it.

Drain the port before issuing the read, so the abandoned write has retired and the engine is idle. Host load stretches the gap between the resume and the write completing, which is why the deadlock only shows up on a loaded machine.

Fixes d05ae87e7a6a ("tests/qtest/ahci: regression test for a PIO write vs. engine stop")
---
 tests/qtest/ahci-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/ahci-test.c +6 -0
index b143862ce7..2552fa39f1 100644
@@ -1914,6 +1914,12 @@ static void test_write_engine_stop_in_flight(void)
 
     g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"resume wr\""));
 
+    /*
+     * Retire the abandoned write. handle_cmd() drops a command that arrives
+     * while the drive is still busy and nothing retries it.
+     */
+    g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"aio_flush\""));
+
     /* Round-trip through the device to confirm qemu is still alive. */
     ahci_px_rreg(ahci, port, AHCI_PX_TFD);
 
Signature · 1 line
2.53.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 28 lines · Denis V. Lunev

From: Denis V. Lunev <[email protected]>

test_write_engine_stop_in_flight() resumes the suspended backend write and goes straight on to restart the command engine and read the second sector back. Whether the write has retired by then is a race. While it is still outstanding the drive is busy, and handle_cmd() drops a command that arrives in that state instead of queueing it. Nothing retries the drop, so the PxCI slot stays set and no D2H FIS is posted. ahci_command_wait() polls both with no timeout, so the test deadlocks until the harness kills it.

Drain the port before issuing the read, so the abandoned write has retired and the engine is idle. Host load stretches the gap between the resume and the write completing, which is why the deadlock only shows up on a loaded machine.

Fixes d05ae87e7a6a ("tests/qtest/ahci: regression test for a PIO write vs. engine stop")
---
 tests/qtest/ahci-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/ahci-test.c +0 -0
index b143862ce7..2552fa39f1 100644
@@ -1914,6 +1914,12 @@ static void test_write_engine_stop_in_flight(void)
 

tests/qtest/ahci-test.c static void test_write_engine_stop_in_flight(void) · line 1924

Quoted 10 lines
tests/qtest/ahci-test.c +6 -0
     g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"resume wr\""));
 
+    /*
+     * Retire the abandoned write. handle_cmd() drops a command that arrives
+     * while the drive is still busy and nothing retries it.
+     */
+    g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"aio_flush\""));
+
     /* Round-trip through the device to confirm qemu is still alive. */
     ahci_px_rreg(ahci, port, AHCI_PX_TFD);
#2 lore raw
Quoted 32 lines · Fabiano Rosas
Quoted 30 lines · Denis V. Lunev

From: Denis V. Lunev <[email protected]>

test_write_engine_stop_in_flight() resumes the suspended backend write and goes straight on to restart the command engine and read the second sector back. Whether the write has retired by then is a race. While it is still outstanding the drive is busy, and handle_cmd() drops a command that arrives in that state instead of queueing it. Nothing retries the drop, so the PxCI slot stays set and no D2H FIS is posted. ahci_command_wait() polls both with no timeout, so the test deadlocks until the harness kills it.

Drain the port before issuing the read, so the abandoned write has retired and the engine is idle. Host load stretches the gap between the resume and the write completing, which is why the deadlock only shows up on a loaded machine.

Fixes d05ae87e7a6a ("tests/qtest/ahci: regression test for a PIO write vs. engine stop")
---
 tests/qtest/ahci-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/ahci-test.c +0 -0
index b143862ce7..2552fa39f1 100644
@@ -1914,6 +1914,12 @@ static void test_write_engine_stop_in_flight(void)
 
     g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"resume wr\""));
 
Quoted 9 lines

tests/qtest/ahci-test.c static void test_write_engine_stop_in_flight(void) · line 1924

Quoted 8 lines
tests/qtest/ahci-test.c +6 -0
+    /*
+     * Retire the abandoned write. handle_cmd() drops a command that arrives
+     * while the drive is still busy and nothing retries it.
+     */
+    g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"aio_flush\""));
+
     /* Round-trip through the device to confirm qemu is still alive. */
     ahci_px_rreg(ahci, port, AHCI_PX_TFD);

Hi, Fabiano!

Would you mind to get this patch through test tree? I do not have enough IDE material except this patch for now.

Thank you in advance,
    Den

#3 lore raw
Quoted 42 lines · Denis V. Lunev
Quoted 32 lines · Fabiano Rosas
Quoted 30 lines · Denis V. Lunev

From: Denis V. Lunev <[email protected]>

test_write_engine_stop_in_flight() resumes the suspended backend write and goes straight on to restart the command engine and read the second sector back. Whether the write has retired by then is a race. While it is still outstanding the drive is busy, and handle_cmd() drops a command that arrives in that state instead of queueing it. Nothing retries the drop, so the PxCI slot stays set and no D2H FIS is posted. ahci_command_wait() polls both with no timeout, so the test deadlocks until the harness kills it.

Drain the port before issuing the read, so the abandoned write has retired and the engine is idle. Host load stretches the gap between the resume and the write completing, which is why the deadlock only shows up on a loaded machine.

Fixes d05ae87e7a6a ("tests/qtest/ahci: regression test for a PIO write vs. engine stop")
---
 tests/qtest/ahci-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/ahci-test.c +0 -0
index b143862ce7..2552fa39f1 100644
@@ -1914,6 +1914,12 @@ static void test_write_engine_stop_in_flight(void)
 
     g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"resume wr\""));
 
Quoted 9 lines

tests/qtest/ahci-test.c static void test_write_engine_stop_in_flight(void) · line 1924

Quoted 8 lines
tests/qtest/ahci-test.c +6 -0
+    /*
+     * Retire the abandoned write. handle_cmd() drops a command that arrives
+     * while the drive is still busy and nothing retries it.
+     */
+    g_free(qtest_hmp(ahci->parent->qts, "qemu-io drive0 \"aio_flush\""));
+
     /* Round-trip through the device to confirm qemu is still alive. */
     ahci_px_rreg(ahci, port, AHCI_PX_TFD);
Quoted 8 lines

Hi, Fabiano!

Would you mind to get this patch through test tree? I do not have enough IDE material except this patch for now.

Thank you in advance,
    Den

Yep, no worries. I'm just waiting for my leak fixes series to be ready and I'll send a PR.