iotests: Fix intermittent 030 hang
cover iotests: Fix intermittent 030 hang
Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.
Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increase pause_wait() timeout
python/qemu/machine.py | 1 + tests/qemu-iotests/iotests.py | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
PATCH 1/3 iotests.py: Enable faulthandler
With this, you can send SIGABRT to a hanging test case and you'll get a Python stack trace so you know where it was hanging.
--- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+)
tests/qemu-iotests/iotests.py +3 -0
@@ -30,12 +30,15 @@ import logging import atexit import io from collections import OrderedDict +import faulthandler sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest assert sys.version_info >= (3,6) +faulthandler.enable() + # This will not work if arguments contain spaces but is necessary if we # want to support the override options that ./check supports. qemu_img_args = [os.environ.get('QEMU_IMG_PROG', 'qemu-img')]
Signature · 1 line
2.20.1
Re: [PATCH 1/3] iotests.py: Enable faulthandler
Quoted 4 lines · Kevin Wolf
With this, you can send SIGABRT to a hanging test case and you'll get a Python stack trace so you know where it was hanging.
Signed-off-by Kevin Wolf <[email protected]>
PATCH 2/3 python/qemu: Kill QEMU process if 'quit' doesn't work
In reply to Kevin WolfWith a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.
Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.
--- python/qemu/machine.py | 1 + 1 file changed, 1 insertion(+)
python/qemu/machine.py +1 -0
@@ -358,6 +358,7 @@ class QEMUMachine(object): if not has_quit: self._qmp.cmd('quit') self._qmp.close() + self._popen.wait(timeout=3) except: self._popen.kill() self._popen.wait()
Signature · 1 line
2.20.1
Re: [PATCH 2/3] python/qemu: Kill QEMU process if 'quit' doesn't work
Quoted 7 lines · Kevin Wolf
With a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.
Quoted 5 lines
Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.
Signed-off-by Kevin Wolf <[email protected]>
Quoted 17 lines
--- python/qemu/machine.py | 1 + 1 file changed, 1 insertion(+)python/qemu/machine.py +1
-0@@ -358,6 +358,7 @@ class QEMUMachine(object): if not has_quit: self._qmp.cmd('quit') self._qmp.close() + self._popen.wait(timeout=3) except: self._popen.kill() self._popen.wait()
Signature · 1 line
—js
Re: [PATCH 2/3] python/qemu: Kill QEMU process if 'quit' doesn't work
In reply to Kevin WolfQuoted 22 lines · Kevin Wolf
With a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.
Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.
Signed-off-by Kevin Wolf <[email protected]>--- python/qemu/machine.py | 1 + 1 file changed, 1 insertion(+)python/qemu/machine.py +0
-0
python/qemu/machine.py class QEMUMachine(object): · line 364
Quoted 7 lines
python/qemu/machine.py +1
-0@@ -358,6 +358,7 @@ class QEMUMachine(object): if not has_quit: self._qmp.cmd('quit') self._qmp.close() + self._popen.wait(timeout=3) except: self._popen.kill() self._popen.wait()
PATCH 3/3 iotests: Increase pause_wait() timeout
In reply to Kevin WolfWaiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/type', 'mirror') def pause_wait(self, job_id='job0'): - with Timeout(1, "Timeout waiting for job to pause"): + with Timeout(3, "Timeout waiting for job to pause"): while True: result = self.vm.qmp('query-block-jobs') found = False
Signature · 1 line
2.20.1
Re: [PATCH 3/3] iotests: Increase pause_wait() timeout
Quoted 5 lines · Kevin Wolf
Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.
Signed-off-by Kevin Wolf <[email protected]>
Quoted 18 lines
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/iotests.py +1
-1@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/type', 'mirror') def pause_wait(self, job_id='job0'): - with Timeout(1, "Timeout waiting for job to pause"): + with Timeout(3, "Timeout waiting for job to pause"): while True: result = self.vm.qmp('query-block-jobs') found = False
Re: [PATCH 3/3] iotests: Increase pause_wait() timeout
In reply to Kevin WolfQuoted 15 lines · Kevin Wolf
Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/iotests.py +0
-0
tests/qemu-iotests/iotests.py class QMPTestCase(unittest.TestCase): · line 931
Quoted 8 lines
tests/qemu-iotests/iotests.py +1
-1@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/type', 'mirror') def pause_wait(self, job_id='job0'): - with Timeout(1, "Timeout waiting for job to pause"): + with Timeout(3, "Timeout waiting for job to pause"): while True: result = self.vm.qmp('query-block-jobs') found = False
I wonder if this might be more accurate:
load_timeout = math.ceil(os.getloadavg()[0])
with Timeout(1 + load_timeout, "Timeout waiting for job to pause"):
...
Anyhow:
Re: [PATCH 3/3] iotests: Increase pause_wait() timeout
Quoted 27 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Kevin Wolf
Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.
Signed-off-by Kevin Wolf <[email protected]>---
tests/qemu-iotests/iotests.py| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/iotests.py +0
-0@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/type', 'mirror')Quoted 7 lines
def
pause_wait(self, job_id='job0'):
- withTimeout(1, "Timeout waiting for job to pause"):
+ withTimeout(3, "Timeout waiting for job to pause"):
while True:
result =self.vm.qmp('query-block-jobs')found = FalseI wonder if this might be more accurate:
Quoted 7 lines
load_timeout= math.ceil(os.getloadavg()[0])
withTimeout(1 + load_timeout, "Timeout waiting for job to pause"):
...Anyhow:
Reviewed-by Philippe Mathieu-Daudé <[email protected]>
They're just tests ... and this is just a worst-case timeout. I don't think we need to get too cute with it. This only affects cases where the test (or the binary) is broken and we have to wait without getting a response for 3 seconds before being able to declare that the test is indeed broken.
Optimizing this waiting time is probably not helpful; as when the test is passing we'll never see this delay.
--js
Quoted 6 lines · Kevin Wolf
Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.
Quoted 9 lines
Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increasepause_wait()timeoutpython/qemu/machine.py | 1 + tests/qemu-iotests/iotests.py | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-)
Peter Maydell
AppliedQuoted 7 lines · Kevin Wolf
Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.
Quoted 8 lines
Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increasepause_wait()timeoutpython/qemu/machine.py | 1 + tests/qemu-iotests/iotests.py | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-)
Applied to master since I've been seeing this hang off-and-on.
thanks
-- PMM