Skip to contentSkip to conversation

iotests: Fix intermittent 030 hang

PATCH Applied Kevin Wolf 4 participants 12 messages Mar 13, 2020 – Mar 23, 2020

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

cover iotests: Fix intermittent 030 hang

Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.

Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increase pause_wait() timeout

 python/qemu/machine.py        | 1 +
 tests/qemu-iotests/iotests.py | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/3 iotests.py: Enable faulthandler

With this, you can send SIGABRT to a hanging test case and you'll get a Python stack trace so you know where it was hanging.

---
 tests/qemu-iotests/iotests.py | 3 +++
 1 file changed, 3 insertions(+)
tests/qemu-iotests/iotests.py +3 -0
index 23043baa26..b859c303a2 100644
@@ -30,12 +30,15 @@ import logging
 import atexit
 import io
 from collections import OrderedDict
+import faulthandler
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import qtest
 
 assert sys.version_info >= (3,6)
 
+faulthandler.enable()
+
 # This will not work if arguments contain spaces but is necessary if we
 # want to support the override options that ./check supports.
 qemu_img_args = [os.environ.get('QEMU_IMG_PROG', 'qemu-img')]
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH 2/3 python/qemu: Kill QEMU process if 'quit' doesn't work

In reply to Kevin Wolf

With a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.

Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.

---
 python/qemu/machine.py | 1 +
 1 file changed, 1 insertion(+)
python/qemu/machine.py +1 -0
index 183d8f3d38..c837ee8723 100644
@@ -358,6 +358,7 @@ class QEMUMachine(object):
                     if not has_quit:
                         self._qmp.cmd('quit')
                     self._qmp.close()
+                    self._popen.wait(timeout=3)
                 except:
                     self._popen.kill()
             self._popen.wait()
Signature · 1 line
2.20.1
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] python/qemu: Kill QEMU process if 'quit' doesn't work

Quoted 7 lines · Kevin Wolf

With a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.

Quoted 5 lines

Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.

Quoted 17 lines
---
 python/qemu/machine.py | 1 +
 1 file changed, 1 insertion(+)
python/qemu/machine.py +1 -0
index 183d8f3d38..c837ee8723 100644
@@ -358,6 +358,7 @@ class QEMUMachine(object):
                     if not has_quit:
                         self._qmp.cmd('quit')
                     self._qmp.close()
+                    self._popen.wait(timeout=3)
                 except:
                     self._popen.kill()
             self._popen.wait()
Signature · 1 line
—js
Reviewed-by on v1 2/3
#5 lore raw

Re: [PATCH 2/3] python/qemu: Kill QEMU process if 'quit' doesn't work

In reply to Kevin Wolf
Quoted 22 lines · Kevin Wolf

With a QEMU bug, it can happen that the QEMU process doesn't react to a 'quit' QMP command. If we got an exception during previous QMP communication (e.g. iotests Timeout expiring), we could also be in an inconsistent state where after sending 'quit' we immediately read an old response and close the socket even though the 'quit' command wasn't processed yet. Both cases would lead to a hanging test.

Fix this by waiting for the QEMU process to exit after sending 'quit' with a timeout, and if it doesn't happen within three seconds, send SIGKILL.

---
  python/qemu/machine.py | 1 +
  1 file changed, 1 insertion(+)
python/qemu/machine.py +0 -0
index 183d8f3d38..c837ee8723 100644

python/qemu/machine.py class QEMUMachine(object): · line 364

Quoted 7 lines
python/qemu/machine.py +1 -0
@@ -358,6 +358,7 @@ class QEMUMachine(object):
                      if not has_quit:
                          self._qmp.cmd('quit')
                      self._qmp.close()
+                    self._popen.wait(timeout=3)
                  except:
                      self._popen.kill()
              self._popen.wait()
author
#6 lore raw

PATCH 3/3 iotests: Increase pause_wait() timeout

In reply to Kevin Wolf

Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.

---
 tests/qemu-iotests/iotests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
index b859c303a2..7bc4934cd2 100644
@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase):
         self.assert_qmp(event, 'data/type', 'mirror')
 
     def pause_wait(self, job_id='job0'):
-        with Timeout(1, "Timeout waiting for job to pause"):
+        with Timeout(3, "Timeout waiting for job to pause"):
             while True:
                 result = self.vm.qmp('query-block-jobs')
                 found = False
Signature · 1 line
2.20.1
Reviewed-by on v1 3/3
#7 lore raw

Re: [PATCH 3/3] iotests: Increase pause_wait() timeout

Quoted 5 lines · Kevin Wolf

Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.

Quoted 18 lines
---
 tests/qemu-iotests/iotests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
index b859c303a2..7bc4934cd2 100644
@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase):
         self.assert_qmp(event, 'data/type', 'mirror')
 
     def pause_wait(self, job_id='job0'):
-        with Timeout(1, "Timeout waiting for job to pause"):
+        with Timeout(3, "Timeout waiting for job to pause"):
             while True:
                 result = self.vm.qmp('query-block-jobs')
                 found = False
Reviewed-by on v1 3/3
#8 lore raw

Re: [PATCH 3/3] iotests: Increase pause_wait() timeout

In reply to Kevin Wolf
Quoted 15 lines · Kevin Wolf

Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.

---
  tests/qemu-iotests/iotests.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +0 -0
index b859c303a2..7bc4934cd2 100644

tests/qemu-iotests/iotests.py class QMPTestCase(unittest.TestCase): · line 931

Quoted 8 lines
tests/qemu-iotests/iotests.py +1 -1
@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase):
          self.assert_qmp(event, 'data/type', 'mirror')
 
      def pause_wait(self, job_id='job0'):
-        with Timeout(1, "Timeout waiting for job to pause"):
+        with Timeout(3, "Timeout waiting for job to pause"):
              while True:
                  result = self.vm.qmp('query-block-jobs')
                  found = False

I wonder if this might be more accurate:

load_timeout = math.ceil(os.getloadavg()[0])
with Timeout(1 + load_timeout, "Timeout waiting for job to pause"):

    ...

Anyhow:

#9 lore raw

Re: [PATCH 3/3] iotests: Increase pause_wait() timeout

Quoted 27 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Kevin Wolf

Waiting for only 1 second proved to be too short on a loaded system, resulting in false positives when testing pull requests. Increase the timeout a bit to make this less likely.

---
  tests/qemu-iotests/iotests.py | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +0 -0
index b859c303a2..7bc4934cd2 100644
@@ -925,7 +925,7 @@ class QMPTestCase(unittest.TestCase):
           self.assert_qmp(event, 'data/type', 'mirror')
Quoted 7 lines

        def pause_wait(self, job_id='job0'):
-        with Timeout(1, "Timeout waiting for job to pause"):
+        with Timeout(3, "Timeout waiting for job to pause"):
              while True:
                  result = self.vm.qmp('query-block-jobs') found = False

I wonder if this might be more accurate:

Quoted 7 lines

  load_timeout = math.ceil(os.getloadavg()[0])
  with Timeout(1 + load_timeout, "Timeout waiting for job to pause"):
    ...

Anyhow:

They're just tests ... and this is just a worst-case timeout. I don't think we need to get too cute with it. This only affects cases where the test (or the binary) is broken and we have to wait without getting a response for 3 seconds before being able to declare that the test is indeed broken.

Optimizing this waiting time is probably not helpful; as when the test is passing we'll never see this delay.

--js

Tested-by on the v1 series
#10 lore raw
In reply to Kevin Wolf
Quoted 6 lines · Kevin Wolf

Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.

Quoted 9 lines

Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increase pause_wait() timeout

  python/qemu/machine.py        | 1 +
  tests/qemu-iotests/iotests.py | 5 ++++-
  2 files changed, 5 insertions(+), 1 deletion(-)
#11 lore raw
In reply to Kevin Wolf
Quoted 7 lines · Kevin Wolf

Peter ran into a 030 hang while testing a pull request. This turned out to be two bugs in the test suite at once: First was the test failing because a timeout was apparently too short, second was that the timeout would actually cause the test to hang instead of failing. This series should fix both.

Quoted 8 lines

Kevin Wolf (3):
iotests.py: Enable faulthandler
python/qemu: Kill QEMU process if 'quit' doesn't work
iotests: Increase pause_wait() timeout

 python/qemu/machine.py        | 1 +
 tests/qemu-iotests/iotests.py | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

Applied to master since I've been seeing this hang off-and-on.

thanks
-- PMM