record/replay: fix race condition on test_aarch64_reverse_debug
PATCH 1/1 record/replay: fix race condition on test_aarch64_reverse_debug
Ensures EVENT_INSTRUCTION written to replay.bin before EVENT_SHUTDOWN_HOST_QMP
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2921
---
During the record pass, test_reverse_debug writes a sequence of instructions to replay.bin. Presumably due to a race condition or host's async implementation details, the resulting file looks like:
...
12: EVENT_CP_CLOCK_WARP_ACCOUNT(31) no additional data 13: EVENT_INSTRUCTION(0) + 59 -> 44298
14: EVENT_CP_CLOCK_WARP_ACCOUNT(31) no additional data 15: EVENT_SHUTDOWN_HOST_QMP_QUIT(12)
16: EVENT_INSTRUCTION(0) + 5587988 -> 5632286 17: EVENT_SHUTDOWN_HOST_SIGNAL(14)
18: EVENT_END(39)
Reached 162 of 162 bytes
Here, SHUTDOWN_HOST_QMP_QUIT is written before the last instruction event. During the replay pass, QUIT is executed before the last instruction, which causes the VM to shut down. As a result, the QMP and GDB connections are broken, and the test cannot execute its final steps.
Adding replay_save_instructions ensures EVENT_INSTRUCTION is written before EVENT_SHUTDOWN_HOST_QMP_QUIT.
Tested on my arm64. This does not fix the bug on x86_64. The x86_64 case seems similar, but slightly different.
replay/replay.c | 2 ++ tests/functional/test_aarch64_reverse_debug.py | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)
replay/replay.c +2 -0
@@ -263,6 +263,8 @@ bool replay_has_interrupt(void)void replay_shutdown_request (ShutdownCause cause) { +replay_save_instructions (); +if (replay_mode ==REPLAY_MODE_RECORD ) {g_assert (replay_mutex_locked ());replay_put_event (EVENT_SHUTDOWN + cause);
tests/functional/test_aarch64_reverse_debug.py +0 -1
@@ -26,7 +26,6 @@ class ReverseDebugging_AArch64(ReverseDebugging): 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'), '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7') - @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2921") def test_aarch64_virt(self): self.set_machine('virt') self.cpu = 'cortex-a53'
Signature · 1 line
2.34.1
Quoted 19 lines · Vladimir Lukianov
Ensures
EVENT_INSTRUCTIONwritten to replay.bin beforeEVENT_SHUTDOWN_HOST_QMPResolves: https://gitlab.com/qemu-project/qemu/-/issues/2921
Signed-off-by Vladimir Lukianov <[email protected]>---
During the record pass,test_reverse_debugwrites a sequence of instructions to replay.bin. Presumably due to a race condition or host's async implementation details, the resulting file looks like:...
12:EVENT_CP_CLOCK_WARP_ACCOUNT(31)no additional data 13:EVENT_INSTRUCTION(0)+ 59 -> 44298
14:EVENT_CP_CLOCK_WARP_ACCOUNT(31)no additional data 15:EVENT_SHUTDOWN_HOST_QMP_QUIT(12)
16:EVENT_INSTRUCTION(0)+ 5587988 -> 5632286 17:EVENT_SHUTDOWN_HOST_SIGNAL(14)
18:EVENT_END(39)
Reached 162 of 162 bytes
Quoted 5 lines
Here,
SHUTDOWN_HOST_QMP_QUITis written before the last instruction event. During the replay pass, QUIT is executed before the last instruction, which causes the VM to shut down. As a result, the QMP and GDB connections are broken, and the test cannot execute its final steps.
Seems reasonable to me.
Quoted 6 lines
Adding
replay_save_instructionsensuresEVENT_INSTRUCTIONis written beforeEVENT_SHUTDOWN_HOST_QMP_QUIT.Tested on my arm64. This does not fix the bug on
x86_64. Thex86_64case seems similar, but slightly different.
Hmm I can't run the functional tests due to missing avocado bits. How did you run the tests?
Quoted 30 lines
replay/replay.c | 2 ++ tests/functional/test_aarch64_reverse_debug.py | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)replay/replay.c +2
-0@@ -263,6 +263,8 @@ bool replay_has_interrupt(void)void replay_shutdown_request (ShutdownCause cause) { +replay_save_instructions (); +if (replay_mode ==REPLAY_MODE_RECORD ) {g_assert (replay_mutex_locked ());replay_put_event (EVENT_SHUTDOWN + cause);tests/functional/test_aarch64_reverse_debug.py +0
-1@@ -26,7 +26,6 @@ class ReverseDebugging_AArch64(ReverseDebugging): 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'), '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7') - @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2921") def test_aarch64_virt(self): self.set_machine('virt') self.cpu = 'cortex-a53'
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Hi, thanks for responding!
Usual flow was
source build/pyvenv/bin/activate
export PYTHONBREAKPOINT="ipdb.set_trace"
export
QEMU_TEST_QEMU_BINARY=/home/lukvladimir/dev/qemu/build/qemu-system-aarch64
If missing python packages / avocado
pip install -e python/
pip install avocado-framework
Then
QEMU_TEST_FLAKY_TESTS=1 avocado -V runtests/functional/test_aarch64_reverse_debug.py
Or
QEMU_TEST_FLAKY_TESTS=1 tests/functional/test_aarch64_reverse_debug.py
Quoted 78 lines · Alex Bennée
Quoted 1 line · Vladimir Lukianov
Ensures
EVENT_INSTRUCTIONwritten to replay.bin before
EVENT_SHUTDOWN_HOST_QMPQuoted 18 lines
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2921
Signed-off-by Vladimir Lukianov <[email protected]>---
During the record pass,test_reverse_debugwrites a sequence of instructions to replay.bin. Presumably due to a race condition or host's async implementation details, the resulting file looks like:...
12:EVENT_CP_CLOCK_WARP_ACCOUNT(31)no additional data 13:EVENT_INSTRUCTION(0)+ 59 -> 44298
14:EVENT_CP_CLOCK_WARP_ACCOUNT(31)no additional data 15:EVENT_SHUTDOWN_HOST_QMP_QUIT(12)
16:EVENT_INSTRUCTION(0)+ 5587988 -> 5632286 17:EVENT_SHUTDOWN_HOST_SIGNAL(14)
18:EVENT_END(39)
Reached 162 of 162 bytesQuoted 5 lines
Here,
SHUTDOWN_HOST_QMP_QUITis written before the last instruction event. During the replay pass, QUIT is executed before the last instruction, which causes the VM to shut down. As a result, the QMP and GDB connections are broken, and the test cannot execute its final steps.Seems reasonable to me.
Quoted 6 lines
Adding
replay_save_instructionsensuresEVENT_INSTRUCTIONis written beforeEVENT_SHUTDOWN_HOST_QMP_QUIT.Tested on my arm64. This does not fix the bug on
x86_64. Thex86_64case seems similar, but slightly different.Hmm I can't run the functional tests due to missing avocado bits. How did you run the tests?
Review
Quoted 19 lines
replay/replay.c | 2 ++ tests/functional/test_aarch64_reverse_debug.py | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)replay/replay.c +2
-0@@ -263,6 +263,8 @@ bool replay_has_interrupt(void)void replay_shutdown_request (ShutdownCause cause) { +replay_save_instructions (); +if (replay_mode ==REPLAY_MODE_RECORD ) {g_assert (replay_mutex_locked ());replay_put_event (EVENT_SHUTDOWN + cause);(quoted diff) +0
-0
b/tests/functional/test_aarch64_reverse_debug.pytests/functional/test_aarch64_reverse_debug.py class ReverseDebugging_AArch64(ReverseDebugging): · line 27
Quoted 6 lines
index
58d45328..0ac1ccb0100755tests/functional/test_aarch64_reverse_debug.py +0
-0@@ -26,7 +26,6 @@ class ReverseDebugging_AArch64(ReverseDebugging): 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),'
7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')tests/functional/test_aarch64_reverse_debug.py class ReverseDebugging_AArch64(ReverseDebugging): · line 28
Quoted 2 lines
tests/functional/test_aarch64_reverse_debug.py +0
-1- @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2921")
tests/functional/test_aarch64_reverse_debug.py class ReverseDebugging_AArch64(ReverseDebugging): · line 31
Quoted 3 lines
tests/functional/test_aarch64_reverse_debug.py +0
-0def test_aarch64_virt(self): self.set_machine('virt') self.cpu = 'cortex-a53'--
Alex Bennée
Virtualisation Tech Lead @ Linaro
Alex Bennée
AppliedQuoted 4 lines · Vladimir Lukianov
Ensures
EVENT_INSTRUCTIONwritten to replay.bin beforeEVENT_SHUTDOWN_HOST_QMPResolves: https://gitlab.com/qemu-project/qemu/-/issues/2921
Signed-off-by Vladimir Lukianov <[email protected]>
Queued to pr/031025-10.2-maintainer-1, thanks.
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro