Skip to contentSkip to conversation

iotests: add enhanced debugging info to qemu-img failures

PATCH Superseded John Snow 3 participants 30 messages Mar 17, 2022 – Mar 21, 2022

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

v4 cover iotests: add enhanced debugging info to qemu-img failures

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Here's a summary of the changes to help you update any tests you may have in flight:

qemu_img()                   => qemu_img().returncode
qemu_img_pipe()              => qemu_img().stdout
qemu_img_pipe_and_status()   => qemu_img()

json.loads(qemu_img_pipe())) => qemu_img_json()
qemu_img_log() => qemu_img_log()

v4:

- Combine both series into one big series again.

002/18:[0007] [FC] 'python/utils: add VerboseProcessError'
Added a comment and a temporary var to explain the "elif stderr is not None" thing.
(Kept R-Bs because I live dangerously.)

006/18:[0014] [FC] 'iotests: add qemu_img_json()'
Made this wayyyy less stupid. Thanks Hanna.

010/18:[0010] [FC] 'iotests: add qemu_img_map() function'
Fixed test output for 211.

018/18:[down] 'iotests: make qemu_img_log and img_info_log raise on error'
New, enforce a good return code by default for *all* qemu-img calls, logged or not.

John Snow (18):
python/utils: add add_visual_margin() text decoration utility python/utils: add VerboseProcessError
iotests: Remove explicit checks for qemu_img() == 0
iotests: make qemu_img raise on non-zero rc by default
iotests: fortify compare_images() against crashes
iotests: add qemu_img_json()
iotests: use qemu_img_json() when applicable
iotests: add qemu_img_info() iotests/remove-bitmap-from-backing: use qemu_img_info()
iotests: add qemu_img_map() function
iotests: change supports_quorum to use qemu_img
iotests: replace unchecked calls to qemu_img_pipe() iotests/149: Remove qemu_img_pipe() call
iotests: remove remaining calls to qemu_img_pipe()
iotests: use qemu_img() in has_working_luks()
iotests: replace qemu_img_log('create', ...) calls
iotests: remove qemu_img_pipe_and_status()
iotests: make qemu_img_log and img_info_log raise on error

 python/qemu/utils/__init__.py                 | 117 ++++++++++++
 tests/qemu-iotests/041                        |   5 +-
 tests/qemu-iotests/065                        |   7 +-
 tests/qemu-iotests/149                        |   7 +-
 tests/qemu-iotests/149.out                    |  21 ---
 tests/qemu-iotests/163                        |   9 +-
 tests/qemu-iotests/194                        |   4 +-
 tests/qemu-iotests/202                        |   4 +-
 tests/qemu-iotests/203                        |   4 +-
 tests/qemu-iotests/211                        |   6 +-
 tests/qemu-iotests/211.out                    |  10 +-
 tests/qemu-iotests/216                        |   6 +-
 tests/qemu-iotests/218                        |   2 +-
 tests/qemu-iotests/224                        |  11 +-
 tests/qemu-iotests/228                        |  12 +-
 tests/qemu-iotests/234                        |   4 +-
 tests/qemu-iotests/237                        |   3 +-
 tests/qemu-iotests/237.out                    |   3 -
 tests/qemu-iotests/242                        |   7 +-
 tests/qemu-iotests/255                        |   8 +-
 tests/qemu-iotests/255.out                    |   4 -
 tests/qemu-iotests/257                        |  11 +-
 tests/qemu-iotests/258                        |   4 +-
 tests/qemu-iotests/262                        |   2 +-
 tests/qemu-iotests/266                        |   2 +-
 tests/qemu-iotests/274                        |  17 +-
 tests/qemu-iotests/274.out                    |  29 ---
 tests/qemu-iotests/280                        |   2 +-
 tests/qemu-iotests/280.out                    |   1 -
 tests/qemu-iotests/296                        |  12 +-
 tests/qemu-iotests/303                        |   2 +-
 tests/qemu-iotests/310                        |  13 +-
 tests/qemu-iotests/iotests.py                 | 169 +++++++++++++-----
 tests/qemu-iotests/tests/block-status-cache   |  14 +-
 .../qemu-iotests/tests/graph-changes-while-io |   7 +-
 tests/qemu-iotests/tests/image-fleecing       |  10 +-
 .../tests/mirror-ready-cancel-error           |   6 +-
 tests/qemu-iotests/tests/mirror-top-perms     |   3 +-
 .../qemu-iotests/tests/parallels-read-bitmap  |   6 +-
 .../tests/remove-bitmap-from-backing          |  14 +-
 .../qemu-iotests/tests/stream-error-on-reset  |   4 +-
 41 files changed, 353 insertions(+), 229 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v4 1/18 python/utils: add add_visual_margin() text decoration utility

Quoted 1 line
Quoted 1 line
Quoted 1 line

print(add_visual_margin(msg, width=72, name="Commit Message"))

┏━ Commit Message ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┃ add_visual_margin() takes a chunk of text and wraps it in a visual
┃ container that force-wraps to a specified width. An optional title
┃ label may be given, and any of the individual glyphs used to draw the
┃ box may be replaced or specified as well.
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
---
 python/qemu/utils/__init__.py | 78 +++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
python/qemu/utils/__init__.py +78 -0
index 7f1a5138c4..5babf40df2 100644
@@ -15,7 +15,10 @@
 # the COPYING file in the top-level directory.
 #
 
+import os
 import re
+import shutil
+import textwrap
 from typing import Optional
 
 # pylint: disable=import-error
@@ -23,6 +26,7 @@
 
 
 __all__ = (
+    'add_visual_margin',
     'get_info_usernet_hostfwd_port',
     'kvm_available',
     'list_accel',
@@ -43,3 +47,77 @@ def get_info_usernet_hostfwd_port(info_usernet_output: str) -> Optional[int]:
         if match is not None:
             return int(match[1])
     return None
+
+
+# pylint: disable=too-many-arguments
+def add_visual_margin(
+        content: str = '',
+        width: Optional[int] = None,
+        name: Optional[str] = None,
+        padding: int = 1,
+        upper_left: str = '┏',
+        lower_left: str = '┗',
+        horizontal: str = '━',
+        vertical: str = '┃',
+) -> str:
+    """
+    Decorate and wrap some text with a visual decoration around it.
+
+    This function assumes that the text decoration characters are single
+    characters that display using a single monospace column.
+
+    ┏━ Example ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+    ┃ This is what this function looks like with text content that's
+    ┃ wrapped to 72 characters. The right-hand margin is left open to
+    ┃ acommodate the occasional unicode character that might make
+    ┃ predicting the total "visual" width of a line difficult. This
+    ┃ provides a visual distinction that's good-enough, though.
+    ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+    :param content: The text to wrap and decorate.
+    :param width:
+        The number of columns to use, including for the decoration
+        itself. The default (None) uses the the available width of the
+        current terminal, or a fallback of 72 lines. A negative number
+        subtracts a fixed-width from the default size. The default obeys
+        the COLUMNS environment variable, if set.
+    :param name: A label to apply to the upper-left of the box.
+    :param padding: How many columns of padding to apply inside.
+    :param upper_left: Upper-left single-width text decoration character.
+    :param lower_left: Lower-left single-width text decoration character.
+    :param horizontal: Horizontal single-width text decoration character.
+    :param vertical: Vertical single-width text decoration character.
+    """
+    if width is None or width < 0:
+        avail = shutil.get_terminal_size(fallback=(72, 24))[0]
+        if width is None:
+            _width = avail
+        else:
+            _width = avail + width
+    else:
+        _width = width
+
+    prefix = vertical + (' ' * padding)
+
+    def _bar(name: Optional[str], top: bool = True) -> str:
+        ret = upper_left if top else lower_left
+        if name is not None:
+            ret += f"{horizontal} {name} "
+
+        filler_len = _width - len(ret)
+        ret += f"{horizontal * filler_len}"
+        return ret
+
+    def _wrap(line: str) -> str:
+        return os.linesep.join(
+            textwrap.wrap(
+                line, width=_width - padding, initial_indent=prefix,
+                subsequent_indent=prefix, replace_whitespace=False,
+                drop_whitespace=True, break_on_hyphens=False)
+        )
+
+    return os.linesep.join((
+        _bar(name, top=True),
+        os.linesep.join(_wrap(line) for line in content.splitlines()),
+        _bar(None, top=False),
+    ))
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v4 2/18 python/utils: add VerboseProcessError

In reply to John Snow

This adds an Exception that extends the Python stdlib subprocess.CalledProcessError.

The difference is that the str() method of this exception also adds the stdout/stderr logs. In effect, if this exception goes unhandled, Python will print the output in a visually distinct wrapper to the terminal so that it's easy to spot in a sea of traceback information.

---
 python/qemu/utils/__init__.py | 39 +++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
python/qemu/utils/__init__.py +39 -0
index 5babf40df2..904eb0de33 100644
@@ -18,6 +18,7 @@
 import os
 import re
 import shutil
+from subprocess import CalledProcessError
 import textwrap
 from typing import Optional
 
@@ -26,6 +27,7 @@
 
 
 __all__ = (
+    'VerboseProcessError',
     'add_visual_margin',
     'get_info_usernet_hostfwd_port',
     'kvm_available',
@@ -121,3 +123,40 @@ def _wrap(line: str) -> str:
         os.linesep.join(_wrap(line) for line in content.splitlines()),
         _bar(None, top=False),
     ))
+
+
+class VerboseProcessError(CalledProcessError):
+    """
+    The same as CalledProcessError, but more verbose.
+
+    This is useful for debugging failed calls during test executions.
+    The return code, signal (if any), and terminal output will be displayed
+    on unhandled exceptions.
+    """
+    def summary(self) -> str:
+        """Return the normal CalledProcessError str() output."""
+        return super().__str__()
+
+    def __str__(self) -> str:
+        lmargin = '  '
+        width = -len(lmargin)
+        sections = []
+
+        # Does self.stdout contain both stdout and stderr?
+        has_combined_output = self.stderr is None
+
+        name = 'output' if has_combined_output else 'stdout'
+        if self.stdout:
+            sections.append(add_visual_margin(self.stdout, width, name))
+        else:
+            sections.append(f"{name}: N/A")
+
+        if self.stderr:
+            sections.append(add_visual_margin(self.stderr, width, 'stderr'))
+        elif not has_combined_output:
+            sections.append("stderr: N/A")
+
+        return os.linesep.join((
+            self.summary(),
+            textwrap.indent(os.linesep.join(sections), prefix=lmargin),
+        ))
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v4 3/18 iotests: Remove explicit checks for qemu_img() == 0

In reply to John Snow

qemu_img() returning zero ought to be the rule, not the exception. Remove all explicit checks against the condition in preparation for making non-zero returns an Exception.

---
 tests/qemu-iotests/163                              |  9 +++------
 tests/qemu-iotests/216                              |  6 +++---
 tests/qemu-iotests/218                              |  2 +-
 tests/qemu-iotests/224                              | 11 +++++------
 tests/qemu-iotests/228                              | 12 ++++++------
 tests/qemu-iotests/257                              |  3 +--
 tests/qemu-iotests/258                              |  4 ++--
 tests/qemu-iotests/310                              | 13 ++++++-------
 tests/qemu-iotests/tests/block-status-cache         |  3 +--
 tests/qemu-iotests/tests/graph-changes-while-io     |  7 +++----
 tests/qemu-iotests/tests/image-fleecing             | 10 +++++-----
 tests/qemu-iotests/tests/mirror-ready-cancel-error  |  6 ++----
 tests/qemu-iotests/tests/mirror-top-perms           |  3 +--
 tests/qemu-iotests/tests/remove-bitmap-from-backing |  8 ++++----
 tests/qemu-iotests/tests/stream-error-on-reset      |  4 ++--
 15 files changed, 45 insertions(+), 56 deletions(-)
tests/qemu-iotests/163 +3 -6
index b8bfc95358..e4cd4b230f 100755
@@ -107,8 +107,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
 
         if iotests.imgfmt == 'raw':
             return
-        self.assertEqual(qemu_img('check', test_img), 0,
-                         "Verifying image corruption")
+        qemu_img('check', test_img)
 
     def test_empty_image(self):
         qemu_img('resize',  '-f', iotests.imgfmt, '--shrink', test_img,
@@ -130,8 +129,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         qemu_img('resize',  '-f', iotests.imgfmt, '--shrink', test_img,
                  self.shrink_size)
 
-        self.assertEqual(qemu_img("compare", test_img, check_img), 0,
-                         "Verifying image content")
+        qemu_img("compare", test_img, check_img)
 
         self.image_verify()
 
@@ -146,8 +144,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         qemu_img('resize',  '-f', iotests.imgfmt, '--shrink', test_img,
                  self.shrink_size)
 
-        self.assertEqual(qemu_img("compare", test_img, check_img), 0,
-                         "Verifying image content")
+        qemu_img("compare", test_img, check_img)
 
         self.image_verify()
 
tests/qemu-iotests/216 +3 -3
index c02f8d2880..88b385afa3 100755
@@ -51,10 +51,10 @@ with iotests.FilePath('base.img') as base_img_path, \
     log('--- Setting up images ---')
     log('')
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
     assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                    '-F', iotests.imgfmt, top_img_path) == 0
+    qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+             '-F', iotests.imgfmt, top_img_path)
     assert qemu_io_silent(top_img_path,  '-c', 'write -P 2 1M 1M') == 0
 
     log('Done')
tests/qemu-iotests/218 +1 -1
index 4922b4d3b6..853ed52b34 100755
@@ -145,7 +145,7 @@ log('')
 with iotests.VM() as vm, \
      iotests.FilePath('src.img') as src_img_path:
 
-    assert qemu_img('create', '-f', iotests.imgfmt, src_img_path, '64M') == 0
+    qemu_img('create', '-f', iotests.imgfmt, src_img_path, '64M')
     assert qemu_io_silent('-f', iotests.imgfmt, src_img_path,
                           '-c', 'write -P 42 0M 64M') == 0
 
tests/qemu-iotests/224 +5 -6
index 38dd153625..c31c55b49d 100755
@@ -47,12 +47,11 @@ for filter_node_name in False, True:
          iotests.FilePath('top.img') as top_img_path, \
          iotests.VM() as vm:
 
-        assert qemu_img('create', '-f', iotests.imgfmt,
-                        base_img_path, '64M') == 0
-        assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                        '-F', iotests.imgfmt, mid_img_path) == 0
-        assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
-                        '-F', iotests.imgfmt, top_img_path) == 0
+        qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
+        qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+                 '-F', iotests.imgfmt, mid_img_path)
+        qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
+                 '-F', iotests.imgfmt, top_img_path)
 
         # Something to commit
         assert qemu_io_silent(mid_img_path, '-c', 'write -P 1 0 1M') == 0
tests/qemu-iotests/228 +6 -6
index a5eda2e149..f79bae0267 100755
@@ -54,11 +54,11 @@ with iotests.FilePath('base.img') as base_img_path, \
      iotests.FilePath('top.img') as top_img_path, \
      iotests.VM() as vm:
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
     # Choose a funny way to describe the backing filename
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b',
-                    'file:' + base_img_path, '-F', iotests.imgfmt,
-                    top_img_path) == 0
+    qemu_img('create', '-f', iotests.imgfmt, '-b',
+             'file:' + base_img_path, '-F', iotests.imgfmt,
+             top_img_path)
 
     vm.launch()
 
@@ -172,8 +172,8 @@ with iotests.FilePath('base.img') as base_img_path, \
     # (because qemu cannot "canonicalize"/"resolve" the backing
     # filename unless the backing file is opened implicitly with the
     # overlay)
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                    '-F', iotests.imgfmt, top_img_path) == 0
+    qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+             '-F', iotests.imgfmt, top_img_path)
 
     # You can only reliably override backing options by using a node
     # reference (or by specifying file.filename, but, well...)
tests/qemu-iotests/257 +1 -2
index c72c82a171..fb5359c581 100755
@@ -240,8 +240,7 @@ def compare_images(image, reference, baseimg=None, expected_match=True):
     """
     expected_ret = 0 if expected_match else 1
     if baseimg:
-        assert qemu_img("rebase", "-u", "-b", baseimg, '-F', iotests.imgfmt,
-                        image) == 0
+        qemu_img("rebase", "-u", "-b", baseimg, '-F', iotests.imgfmt, image)
     ret = qemu_img("compare", image, reference)
     log('qemu_img compare "{:s}" "{:s}" ==> {:s}, {:s}'.format(
         image, reference,
tests/qemu-iotests/258 +2 -2
index a6618208a8..7798a04d7d 100755
@@ -75,13 +75,13 @@ def test_concurrent_finish(write_to_stream_node):
 
         # It is important to use raw for the base layer (so that
         # permissions are just handed through to the protocol layer)
-        assert qemu_img('create', '-f', 'raw', node0_path, '64M') == 0
+        qemu_img('create', '-f', 'raw', node0_path, '64M')
 
         stream_throttle=None
         commit_throttle=None
 
         for path in [node1_path, node2_path, node3_path, node4_path]:
-            assert qemu_img('create', '-f', iotests.imgfmt, path, '64M') == 0
+            qemu_img('create', '-f', iotests.imgfmt, path, '64M')
 
         if write_to_stream_node:
             # This is what (most of the time) makes commit finish
tests/qemu-iotests/310 +6 -7
index 33c3411869..e3bfedc7fd 100755
@@ -43,15 +43,15 @@ with iotests.FilePath('base.img') as base_img_path, \
     log('--- Setting up images ---')
     log('')
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
     assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
     assert qemu_io_silent(base_img_path, '-c', 'write -P 1 3M 1M') == 0
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                    '-F', iotests.imgfmt, mid_img_path) == 0
+    qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+             '-F', iotests.imgfmt, mid_img_path)
     assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 2M 1M') == 0
     assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 4M 1M') == 0
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
-                    '-F', iotests.imgfmt, top_img_path) == 0
+    qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
+             '-F', iotests.imgfmt, top_img_path)
     assert qemu_io_silent(top_img_path, '-c', 'write -P 2 1M 1M') == 0
 
 #      0 1 2 3 4
@@ -105,8 +105,7 @@ with iotests.FilePath('base.img') as base_img_path, \
     log('')
 
     # Detach backing to check that we can read the data from the top level now
-    assert qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt,
-                    top_img_path) == 0
+    qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt, top_img_path)
 
     assert qemu_io_silent(top_img_path, '-c', 'read -P 0 0 1M') == 0
     assert qemu_io_silent(top_img_path, '-c', 'read -P 2 1M 1M') == 0
tests/qemu-iotests/tests/block-status-cache +1 -2
index 6fa10bb8f8..40e648e251 100755
@@ -35,8 +35,7 @@ nbd_sock = os.path.join(iotests.sock_dir, 'nbd.sock')
 class TestBscWithNbd(iotests.QMPTestCase):
     def setUp(self) -> None:
         """Just create an empty image with a read-only NBD server on it"""
-        assert qemu_img_create('-f', iotests.imgfmt, test_img,
-                               str(image_size)) == 0
+        qemu_img_create('-f', iotests.imgfmt, test_img, str(image_size))
 
         # Pass --allocation-depth to enable the qemu:allocation-depth context,
         # which we are going to query to provoke a block-status inquiry with
tests/qemu-iotests/tests/graph-changes-while-io +3 -4
index 567e8cf21e..7664f33689 100755
@@ -34,16 +34,15 @@ def do_qemu_img_bench() -> None:
     """
     Do some I/O requests on `nbd_sock`.
     """
-    assert qemu_img('bench', '-f', 'raw', '-c', '2000000',
-                    f'nbd+unix:///node0?socket={nbd_sock}') == 0
+    qemu_img('bench', '-f', 'raw', '-c', '2000000',
+             f'nbd+unix:///node0?socket={nbd_sock}')
 
 
 class TestGraphChangesWhileIO(QMPTestCase):
     def setUp(self) -> None:
         # Create an overlay that can be added at runtime on top of the
         # null-co block node that will receive I/O
-        assert qemu_img_create('-f', imgfmt, '-F', 'raw', '-b', 'null-co://',
-                               top) == 0
+        qemu_img_create('-f', imgfmt, '-F', 'raw', '-b', 'null-co://', top)
 
         # QSD instance with a null-co block node in an I/O thread,
         # exported over NBD (on `nbd_sock`, export name "node0")
tests/qemu-iotests/tests/image-fleecing +5 -5
index c56278639c..b7e5076104 100755
@@ -63,18 +63,18 @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
     log('--- Setting up images ---')
     log('')
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
     if bitmap:
-        assert qemu_img('bitmap', '--add', base_img_path, 'bitmap0') == 0
+        qemu_img('bitmap', '--add', base_img_path, 'bitmap0')
 
     if use_snapshot_access_filter:
         assert use_cbw
-        assert qemu_img('create', '-f', 'raw', fleece_img_path, '64M') == 0
+        qemu_img('create', '-f', 'raw', fleece_img_path, '64M')
     else:
-        assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
+        qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M')
 
     if push_backup:
-        assert qemu_img('create', '-f', 'qcow2', target_img_path, '64M') == 0
+        qemu_img('create', '-f', 'qcow2', target_img_path, '64M')
 
     for p in patterns:
         qemu_io('-f', iotests.imgfmt,
tests/qemu-iotests/tests/mirror-ready-cancel-error +2 -4
index 770ffca379..1d0e333b5e 100755
@@ -31,10 +31,8 @@ target = os.path.join(iotests.test_dir, 'target.img')
 
 class TestMirrorReadyCancelError(iotests.QMPTestCase):
     def setUp(self) -> None:
-        assert iotests.qemu_img_create('-f', iotests.imgfmt, source,
-                                       str(image_size)) == 0
-        assert iotests.qemu_img_create('-f', iotests.imgfmt, target,
-                                       str(image_size)) == 0
+        iotests.qemu_img_create('-f', iotests.imgfmt, source, str(image_size))
+        iotests.qemu_img_create('-f', iotests.imgfmt, target, str(image_size))
 
         # Ensure that mirror will copy something before READY so the
         # target format layer will forward the pre-READY flush to its
tests/qemu-iotests/tests/mirror-top-perms +1 -2
index b5849978c4..6ac8d5efcc 100755
@@ -34,8 +34,7 @@ source = os.path.join(iotests.test_dir, 'source.img')
 
 class TestMirrorTopPerms(iotests.QMPTestCase):
     def setUp(self):
-        assert qemu_img('create', '-f', iotests.imgfmt, source,
-                        str(image_size)) == 0
+        qemu_img('create', '-f', iotests.imgfmt, source, str(image_size))
         self.vm = iotests.VM()
         self.vm.add_drive(source)
         self.vm.add_blockdev(f'null-co,node-name=null,size={image_size}')
tests/qemu-iotests/tests/remove-bitmap-from-backing +4 -4
index 3c397b08ea..fee3141340 100755
@@ -27,11 +27,11 @@ iotests.script_initialize(supported_fmts=['qcow2'],
 top, base = iotests.file_path('top', 'base')
 size = '1M'
 
-assert qemu_img_create('-f', iotests.imgfmt, base, size) == 0
-assert qemu_img_create('-f', iotests.imgfmt, '-b', base,
-                       '-F', iotests.imgfmt, top, size) == 0
+qemu_img_create('-f', iotests.imgfmt, base, size)
+qemu_img_create('-f', iotests.imgfmt, '-b', base,
+                '-F', iotests.imgfmt, top, size)
 
-assert qemu_img('bitmap', '--add', base, 'bitmap0') == 0
+qemu_img('bitmap', '--add', base, 'bitmap0')
 # Just assert that our method of checking bitmaps in the image works.
 assert 'bitmaps' in qemu_img_pipe('info', base)
 
tests/qemu-iotests/tests/stream-error-on-reset +2 -2
index 7eaedb24d7..389ae822b8 100755
@@ -54,9 +54,9 @@ class TestStreamErrorOnReset(QMPTestCase):
           to it will result in an error
         - top image is attached to a virtio-scsi device
         """
-        assert qemu_img_create('-f', imgfmt, base, str(image_size)) == 0
+        qemu_img_create('-f', imgfmt, base, str(image_size))
         assert qemu_io_silent('-c', f'write 0 {data_size}', base) == 0
-        assert qemu_img_create('-f', imgfmt, top, str(image_size)) == 0
+        qemu_img_create('-f', imgfmt, top, str(image_size))
 
         self.vm = iotests.VM()
         self.vm.add_args('-accel', 'tcg') # Make throttling work properly
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH v4 4/18 iotests: make qemu_img raise on non-zero rc by default

In reply to John Snow

re-write qemu_img() as a function that will by default raise a VerboseProcessException (extended from CalledProcessException) on non-zero return codes. This will produce a stack trace that will show the command line arguments and return code from the failed process run.

Users that want something more flexible (there appears to be only one) can use check=False and manage the return themselves. However, when the return code is negative, the Exception will be raised no matter what. This is done under the belief that there's no legitimate reason, even in negative tests, to see a crash from qemu-img.

---
 tests/qemu-iotests/257        |  8 +++--
 tests/qemu-iotests/iotests.py | 56 ++++++++++++++++++++++++++++++-----
 2 files changed, 54 insertions(+), 10 deletions(-)
tests/qemu-iotests/257 +5 -3
index fb5359c581..e7e7a2317e 100755
@@ -241,11 +241,13 @@ def compare_images(image, reference, baseimg=None, expected_match=True):
     expected_ret = 0 if expected_match else 1
     if baseimg:
         qemu_img("rebase", "-u", "-b", baseimg, '-F', iotests.imgfmt, image)
-    ret = qemu_img("compare", image, reference)
+
+    sub = qemu_img("compare", image, reference, check=False)
+
     log('qemu_img compare "{:s}" "{:s}" ==> {:s}, {:s}'.format(
         image, reference,
-        "Identical" if ret == 0 else "Mismatch",
-        "OK!" if ret == expected_ret else "ERROR!"),
+        "Identical" if sub.returncode == 0 else "Mismatch",
+        "OK!" if sub.returncode == expected_ret else "ERROR!"),
         filters=[iotests.filter_testfiles])
 
 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
tests/qemu-iotests/iotests.py +49 -7

82 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#5 lore raw

PATCH v4 5/18 iotests: fortify compare_images() against crashes

In reply to John Snow

Fortify compare_images() to be more discerning about the status codes it receives. If qemu_img() returns an exit code that implies it didn't actually perform the comparison, treat that as an exceptional circumstance and force the caller to be aware of the peril.

If a negative test is desired (perhaps to test how qemu_img compare behaves on malformed images, for instance), it is still possible to catch the exception in the test and deal with that circumstance manually.

---
 tests/qemu-iotests/iotests.py | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
tests/qemu-iotests/iotests.py +16 -5
index ec4568b24a..7057db0686 100644
@@ -506,11 +506,22 @@ def qemu_nbd_popen(*args):
             p.kill()
             p.wait()
 
-def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
-    '''Return True if two image files are identical'''
-    res = qemu_img('compare', '-f', fmt1,
-                   '-F', fmt2, img1, img2, check=False)
-    return res.returncode == 0
+def compare_images(img1: str, img2: str,
+                   fmt1: str = imgfmt, fmt2: str = imgfmt) -> bool:
+    """
+    Compare two images with QEMU_IMG; return True if they are identical.
+
+    :raise CalledProcessError:
+        when qemu-img crashes or returns a status code of anything other
+        than 0 (identical) or 1 (different).
+    """
+    try:
+        qemu_img('compare', '-f', fmt1, '-F', fmt2, img1, img2)
+        return True
+    except subprocess.CalledProcessError as exc:
+        if exc.returncode == 1:
+            return False
+        raise
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
Signature · 1 line
2.34.1
author
#6 lore raw

PATCH v4 6/18 iotests: add qemu_img_json()

In reply to John Snow

qemu_img_json() is a new helper built on top of qemu_img() that tries to pull a valid JSON document out of the stdout stream.

In the event that the return code is negative (the program crashed), or the code is greater than zero and did not produce valid JSON output, the VerboseProcessError raised by qemu_img() is re-raised.

In the event that the return code is zero but we can't parse valid JSON, allow the JSON deserialization error to be raised.

---
 tests/qemu-iotests/iotests.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
tests/qemu-iotests/iotests.py +32 -0
index 7057db0686..9d23066701 100644
@@ -276,6 +276,38 @@ def ordered_qmp(qmsg, conv_keys=True):
 def qemu_img_create(*args: str) -> subprocess.CompletedProcess[str]:
     return qemu_img('create', *args)
 
+def qemu_img_json(*args: str) -> Any:
+    """
+    Run qemu-img and return its output as deserialized JSON.
+
+    :raise CalledProcessError:
+        When qemu-img crashes, or returns a non-zero exit code without
+        producing a valid JSON document to stdout.
+    :raise JSONDecoderError:
+        When qemu-img returns 0, but failed to produce a valid JSON document.
+
+    :return: A deserialized JSON object; probably a dict[str, Any].
+    """
+    try:
+        res = qemu_img(*args, combine_stdio=False)
+    except subprocess.CalledProcessError as exc:
+        # Terminated due to signal. Don't bother.
+        if exc.returncode < 0:
+            raise
+
+        # Commands like 'check' can return failure (exit codes 2 and 3)
+        # to indicate command completion, but with errors found. For
+        # multi-command flexibility, ignore the exact error codes and
+        # *try* to load JSON.
+        try:
+            return json.loads(exc.stdout)
+        except json.JSONDecodeError:
+            # Nope. This thing is toast. Raise the /process/ error.
+            pass
+        raise
+
+    return json.loads(res.stdout)
+
 def qemu_img_measure(*args):
     return json.loads(qemu_img_pipe("measure", "--output", "json", *args))
 
Signature · 1 line
2.34.1
Reviewed-by on v4 6/18
#7 lore raw

Re: [PATCH v4 06/18] iotests: add qemu_img_json()

Quoted 24 lines · John Snow

qemu_img_json() is a new helper built on top of qemu_img() that tries to pull a valid JSON document out of the stdout stream.

In the event that the return code is negative (the program crashed), or the code is greater than zero and did not produce valid JSON output, the VerboseProcessError raised by qemu_img() is re-raised.

In the event that the return code is zero but we can't parse valid JSON, allow the JSON deserialization error to be raised.

---
 tests/qemu-iotests/iotests.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
tests/qemu-iotests/iotests.py +1 -0
index 7057db0686..9d23066701 100644
@@ -276,6 +276,38 @@ def ordered_qmp(qmsg, conv_keys=True):
 def qemu_img_create(*args: str) -> subprocess.CompletedProcess[str]:
     return qemu_img('create', *args)
 
+def qemu_img_json(*args: str) -> Any:

tests/qemu-iotests/iotests.py def ordered_qmp(qmsg, conv_keys=True): · line 289

Quoted 10 lines
tests/qemu-iotests/iotests.py +10 -0
+    """
+    Run qemu-img and return its output as deserialized JSON.
+
+    :raise CalledProcessError:
+        When qemu-img crashes, or returns a non-zero exit code without
+        producing a valid JSON document to stdout.
+    :raise JSONDecoderError:
+        When qemu-img returns 0, but failed to produce a valid JSON document.
+
+    :return: A deserialized JSON object; probably a dict[str, Any].

Interesting choice to type the function as '-> Any', but document that we expect a more specific '-> dict[str, Any]' for our known usage of qemu-img. But it makes sense to me (in case a future qemu-img --output=json produces something that is JSON but not a dict).

Quoted 10 lines
tests/qemu-iotests/iotests.py +10 -0
+    """
+    try:
+        res = qemu_img(*args, combine_stdio=False)
+    except subprocess.CalledProcessError as exc:
+        # Terminated due to signal. Don't bother.
+        if exc.returncode < 0:
+            raise
+
+        # Commands like 'check' can return failure (exit codes 2 and 3)
+        # to indicate command completion, but with errors found. For

tests/qemu-iotests/iotests.py def ordered_qmp(qmsg, conv_keys=True): · line 309

Quoted 10 lines
tests/qemu-iotests/iotests.py +10 -0
+        # multi-command flexibility, ignore the exact error codes and
+        # *try* to load JSON.
+        try:
+            return json.loads(exc.stdout)
+        except json.JSONDecodeError:
+            # Nope. This thing is toast. Raise the /process/ error.
+            pass
+        raise
+
+    return json.loads(res.stdout)

The comments were very helpful.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#8 lore raw

PATCH v4 7/18 iotests: use qemu_img_json() when applicable

In reply to John Snow

qemu_img_json() gives better diagnostic information on failure.

---
 tests/qemu-iotests/iotests.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +4 -4
index 9d23066701..628dfd8886 100644
@@ -308,11 +308,11 @@ def qemu_img_json(*args: str) -> Any:
 
     return json.loads(res.stdout)
 
-def qemu_img_measure(*args):
-    return json.loads(qemu_img_pipe("measure", "--output", "json", *args))
+def qemu_img_measure(*args: str) -> Any:
+    return qemu_img_json("measure", "--output", "json", *args)
 
-def qemu_img_check(*args):
-    return json.loads(qemu_img_pipe("check", "--output", "json", *args))
+def qemu_img_check(*args: str) -> Any:
+    return qemu_img_json("check", "--output", "json", *args)
 
 def qemu_img_pipe(*args: str) -> str:
     '''Run qemu-img and return its output'''
Signature · 1 line
2.34.1
author
#9 lore raw

PATCH v4 8/18 iotests: add qemu_img_info()

In reply to John Snow

Add qemu_img_info() by analogy with qemu_img_measure() and qemu_img_check(). Modify image_size() to use this function instead to take advantage of the better diagnostic information on failure provided (ultimately) by qemu_img().

---
 tests/qemu-iotests/065        |  5 ++---
 tests/qemu-iotests/242        |  5 ++---
 tests/qemu-iotests/iotests.py | 15 +++++++++++----
 3 files changed, 15 insertions(+), 10 deletions(-)
tests/qemu-iotests/065 +2 -3
index f7c1b68dad..9466ce7df4 100755
@@ -24,7 +24,7 @@ import os
 import re
 import json
 import iotests
-from iotests import qemu_img, qemu_img_pipe
+from iotests import qemu_img, qemu_img_info, qemu_img_pipe
 import unittest
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -49,8 +49,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
     human_compare = None
 
     def test_json(self):
-        data = json.loads(qemu_img_pipe('info', '--output=json', test_img))
-        data = data['format-specific']
+        data = qemu_img_info(test_img)['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.json_compare)
 
tests/qemu-iotests/242 +2 -3
index 96a30152b0..547bf382e3 100755
@@ -22,7 +22,7 @@
 import iotests
 import json
 import struct
-from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \
+from iotests import qemu_img_create, qemu_io, qemu_img_info, \
     file_path, img_info_log, log, filter_qemu_io
 
 iotests.script_initialize(supported_fmts=['qcow2'],
@@ -39,8 +39,7 @@ flag_offset = 0x5000f
 def print_bitmap(extra_args):
     log('qemu-img info dump:\n')
     img_info_log(disk, extra_args=extra_args)
-    result = json.loads(qemu_img_pipe('info', '--force-share',
-                                      '--output=json', disk))
+    result = qemu_img_info('--force-share', disk)
     if 'bitmaps' in result['format-specific']['data']:
         bitmaps = result['format-specific']['data']['bitmaps']
         log('The same bitmaps in JSON format:')
tests/qemu-iotests/iotests.py +11 -4
index 628dfd8886..646b4b0bd4 100644
@@ -314,6 +314,9 @@ def qemu_img_measure(*args: str) -> Any:
 def qemu_img_check(*args: str) -> Any:
     return qemu_img_json("check", "--output", "json", *args)
 
+def qemu_img_info(*args: str) -> Any:
+    return qemu_img_json('info', "--output", "json", *args)
+
 def qemu_img_pipe(*args: str) -> str:
     '''Run qemu-img and return its output'''
     return qemu_img_pipe_and_status(*args)[0]
@@ -564,10 +567,14 @@ def create_image(name, size):
             file.write(sector)
             i = i + 512
 
-def image_size(img):
-    '''Return image's virtual size'''
-    r = qemu_img_pipe('info', '--output=json', '-f', imgfmt, img)
-    return json.loads(r)['virtual-size']
+def image_size(img: str) -> int:
+    """Return image's virtual size"""
+    value = qemu_img_info('-f', imgfmt, img)['virtual-size']
+    if not isinstance(value, int):
+        type_name = type(value).__name__
+        raise TypeError("Expected 'int' for 'virtual-size', "
+                        f"got '{value}' of type '{type_name}'")
+    return value
 
 def is_str(val):
     return isinstance(val, str)
Signature · 1 line
2.34.1
author
#10 lore raw

PATCH v4 9/18 iotests/remove-bitmap-from-backing: use qemu_img_info()

In reply to John Snow

This removes two more usages of qemu_img_pipe() and replaces them with calls to qemu_img(), which provides better diagnostic information on failure.

---
 tests/qemu-iotests/tests/remove-bitmap-from-backing | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/tests/remove-bitmap-from-backing +3 -3
index fee3141340..15be32dcb9 100755
@@ -19,7 +19,7 @@
 #
 
 import iotests
-from iotests import log, qemu_img_create, qemu_img, qemu_img_pipe
+from iotests import log, qemu_img_create, qemu_img, qemu_img_info
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           unsupported_imgopts=['compat'])
@@ -33,7 +33,7 @@ qemu_img_create('-f', iotests.imgfmt, '-b', base,
 
 qemu_img('bitmap', '--add', base, 'bitmap0')
 # Just assert that our method of checking bitmaps in the image works.
-assert 'bitmaps' in qemu_img_pipe('info', base)
+assert 'bitmaps' in qemu_img_info(base)['format-specific']['data']
 
 vm = iotests.VM().add_drive(top, 'backing.node-name=base')
 vm.launch()
@@ -68,5 +68,5 @@ if result != {'return': {}}:
 
 vm.shutdown()
 
-if 'bitmaps' in qemu_img_pipe('info', base):
+if 'bitmaps' in qemu_img_info(base)['format-specific']['data']:
     log('ERROR: Bitmap is still in the base image')
Signature · 1 line
2.34.1
author
#11 lore raw

PATCH v4 10/18 iotests: add qemu_img_map() function

In reply to John Snow

Add a qemu_img_map() function by analogy with qemu_img_measure(), qemu_img_check(), and qemu_img_info() that all return JSON information.

Replace calls to qemu_img_pipe('map', '--output=json', ...) with this new function, which provides better diagnostic information on failure.

Note: The output for iotest 211 changes, because logging JSON after it was deserialized by Python behaves a little differently than logging the raw JSON document string itself.
(iotests.log() sorts the keys for Python 3.6 support.)

---
 tests/qemu-iotests/041                         |  5 ++---
 tests/qemu-iotests/211                         |  6 +++---
 tests/qemu-iotests/211.out                     | 10 +++-------
 tests/qemu-iotests/iotests.py                  |  3 +++
 tests/qemu-iotests/tests/block-status-cache    | 11 ++++-------
 tests/qemu-iotests/tests/parallels-read-bitmap |  6 ++----
 6 files changed, 17 insertions(+), 24 deletions(-)
tests/qemu-iotests/041 +2 -3
index db9f5dc540..3e16acee56 100755
@@ -24,7 +24,7 @@ import os
 import re
 import json
 import iotests
-from iotests import qemu_img, qemu_img_pipe, qemu_io
+from iotests import qemu_img, qemu_img_map, qemu_io
 
 backing_img = os.path.join(iotests.test_dir, 'backing.img')
 target_backing_img = os.path.join(iotests.test_dir, 'target-backing.img')
@@ -1360,8 +1360,7 @@ class TestFilters(iotests.QMPTestCase):
 
         self.vm.qmp('blockdev-del', node_name='target')
 
-        target_map = qemu_img_pipe('map', '--output=json', target_img)
-        target_map = json.loads(target_map)
+        target_map = qemu_img_map(target_img)
 
         assert target_map[0]['start'] == 0
         assert target_map[0]['length'] == 512 * 1024
tests/qemu-iotests/211 +3 -3
index f52cadade1..1a3b4596c8 100755
@@ -59,7 +59,7 @@ with iotests.FilePath('t.vdi') as disk_path, \
     vm.shutdown()
 
     iotests.img_info_log(disk_path)
-    iotests.log(iotests.qemu_img_pipe('map', '--output=json', disk_path))
+    iotests.log(iotests.qemu_img_map(disk_path))
 
     #
     # Successful image creation (explicit defaults)
@@ -83,7 +83,7 @@ with iotests.FilePath('t.vdi') as disk_path, \
     vm.shutdown()
 
     iotests.img_info_log(disk_path)
-    iotests.log(iotests.qemu_img_pipe('map', '--output=json', disk_path))
+    iotests.log(iotests.qemu_img_map(disk_path))
 
     #
     # Successful image creation (with non-default options)
@@ -107,7 +107,7 @@ with iotests.FilePath('t.vdi') as disk_path, \
     vm.shutdown()
 
     iotests.img_info_log(disk_path)
-    iotests.log(iotests.qemu_img_pipe('map', '--output=json', disk_path))
+    iotests.log(iotests.qemu_img_map(disk_path))
 
     #
     # Invalid BlockdevRef
tests/qemu-iotests/211.out +3 -7
index c4425b5982..f02c75409c 100644
@@ -17,8 +17,7 @@ file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 1048576
 
-[{ "start": 0, "length": 134217728, "depth": 0, "present": true, "zero": true, "data": false}]
-
+[{"data": false, "depth": 0, "length": 134217728, "present": true, "start": 0, "zero": true}]
 === Successful image creation (explicit defaults) ===
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
@@ -36,8 +35,7 @@ file format: IMGFMT
 virtual size: 64 MiB (67108864 bytes)
 cluster_size: 1048576
 
-[{ "start": 0, "length": 67108864, "depth": 0, "present": true, "zero": true, "data": false}]
-
+[{"data": false, "depth": 0, "length": 67108864, "present": true, "start": 0, "zero": true}]
 === Successful image creation (with non-default options) ===
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
@@ -55,9 +53,7 @@ file format: IMGFMT
 virtual size: 32 MiB (33554432 bytes)
 cluster_size: 1048576
 
-[{ "start": 0, "length": 3072, "depth": 0, "present": true, "zero": false, "data": true, "offset": 1024},
-{ "start": 3072, "length": 33551360, "depth": 0, "present": true, "zero": true, "data": true, "offset": 4096}]
-
+[{"data": true, "depth": 0, "length": 3072, "offset": 1024, "present": true, "start": 0, "zero": false}, {"data": true, "depth": 0, "length": 33551360, "offset": 4096, "present": true, "start": 3072, "zero": true}]
 === Invalid BlockdevRef ===
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "this doesn't exist", "size": 33554432}}}
tests/qemu-iotests/iotests.py +3 -0
index 646b4b0bd4..58fb1734b5 100644
@@ -317,6 +317,9 @@ def qemu_img_check(*args: str) -> Any:
 def qemu_img_info(*args: str) -> Any:
     return qemu_img_json('info', "--output", "json", *args)
 
+def qemu_img_map(*args: str) -> Any:
+    return qemu_img_json('map', "--output", "json", *args)
+
 def qemu_img_pipe(*args: str) -> str:
     '''Run qemu-img and return its output'''
     return qemu_img_pipe_and_status(*args)[0]
tests/qemu-iotests/tests/block-status-cache +4 -7
index 40e648e251..5a7bc2c149 100755
@@ -22,7 +22,7 @@
 import os
 import signal
 import iotests
-from iotests import qemu_img_create, qemu_img_pipe, qemu_nbd
+from iotests import qemu_img_create, qemu_img_map, qemu_nbd
 
 
 image_size = 1 * 1024 * 1024
@@ -76,8 +76,7 @@ class TestBscWithNbd(iotests.QMPTestCase):
         # to allocate the first sector to facilitate alignment probing), and
         # then the rest to be zero.  The BSC will thus contain (if anything)
         # one range covering the first sector.
-        map_pre = qemu_img_pipe('map', '--output=json', '--image-opts',
-                                nbd_img_opts)
+        map_pre = qemu_img_map('--image-opts', nbd_img_opts)
 
         # qemu:allocation-depth maps for want_zero=false.
         # want_zero=false should (with the file driver, which the server is
@@ -111,14 +110,12 @@ class TestBscWithNbd(iotests.QMPTestCase):
         # never loop too many times here.
         for _ in range(2):
             # (Ignore the result, this is just to contaminate the cache)
-            qemu_img_pipe('map', '--output=json', '--image-opts',
-                          nbd_img_opts_alloc_depth)
+            qemu_img_map('--image-opts', nbd_img_opts_alloc_depth)
 
         # Now let's see whether the cache reports everything as data, or
         # whether we get correct information (i.e. the same as we got on our
         # first attempt).
-        map_post = qemu_img_pipe('map', '--output=json', '--image-opts',
-                                 nbd_img_opts)
+        map_post = qemu_img_map('--image-opts', nbd_img_opts)
 
         if map_pre != map_post:
             print('ERROR: Map information differs before and after querying ' +
tests/qemu-iotests/tests/parallels-read-bitmap +2 -4
index af6b9c5db3..38ab5fa5b2 100755
@@ -18,9 +18,8 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-import json
 import iotests
-from iotests import qemu_nbd_popen, qemu_img_pipe, log, file_path
+from iotests import qemu_nbd_popen, qemu_img_map, log, file_path
 
 iotests.script_initialize(supported_fmts=['parallels'])
 
@@ -36,8 +35,7 @@ iotests.unarchive_sample_image('parallels-with-bitmap', disk)
 
 with qemu_nbd_popen('--read-only', f'--socket={nbd_sock}',
                     f'--bitmap={bitmap}', '-f', iotests.imgfmt, disk):
-    out = qemu_img_pipe('map', '--output=json', '--image-opts', nbd_opts)
-    chunks = json.loads(out)
+    chunks = qemu_img_map('--image-opts', nbd_opts)
     cluster = 64 * 1024
 
     log('dirty clusters (cluster size is 64K):')
Signature · 1 line
2.34.1
Reviewed-by on v4 10/18
#12 lore raw

Re: [PATCH v4 10/18] iotests: add qemu_img_map() function

Quoted 17 lines · John Snow

Add a qemu_img_map() function by analogy with qemu_img_measure(), qemu_img_check(), and qemu_img_info() that all return JSON information.

Replace calls to qemu_img_pipe('map', '--output=json', ...) with this new function, which provides better diagnostic information on failure.

Note: The output for iotest 211 changes, because logging JSON after it was deserialized by Python behaves a little differently than logging the raw JSON document string itself.
(iotests.log() sorts the keys for Python 3.6 support.)

---

+++ b/tests/qemu-iotests/211.out

tests/qemu-iotests/211.out file format: IMGFMT · line 56

Quoted 7 lines
tests/qemu-iotests/211.out +1 -3
@@ -55,9 +53,7 @@ file format: IMGFMT
 virtual size: 32 MiB (33554432 bytes)
 cluster_size: 1048576
 
-[{ "start": 0, "length": 3072, "depth": 0, "present": true, "zero": false, "data": true, "offset": 1024},
-{ "start": 3072, "length": 33551360, "depth": 0, "present": true, "zero": true, "data": true, "offset": 4096}]
-
+[{"data": true, "depth": 0, "length": 3072, "offset": 1024, "present": true, "start": 0, "zero": false}, {"data": true, "depth": 0, "length": 33551360, "offset": 4096, "present": true, "start": 3072, "zero": true}]

The change in format can produce really long lines for a more complex map, which can introduce its own problems in legibility. But I can live with it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#13 lore raw

Re: [PATCH v4 10/18] iotests: add qemu_img_map() function

Quoted 34 lines · Eric Blake
Quoted 17 lines · John Snow

Add a qemu_img_map() function by analogy with qemu_img_measure(), qemu_img_check(), and qemu_img_info() that all return JSON information.

Replace calls to qemu_img_pipe('map', '--output=json', ...) with this new function, which provides better diagnostic information on failure.

Note: The output for iotest 211 changes, because logging JSON after it was deserialized by Python behaves a little differently than logging the raw JSON document string itself.
(iotests.log() sorts the keys for Python 3.6 support.)

---

+++ b/tests/qemu-iotests/211.out

tests/qemu-iotests/211.out file format: IMGFMT · line 58

Quoted 4 lines
tests/qemu-iotests/211.out +0 -1
@@ -55,9 +53,7 @@ file format: IMGFMT
 virtual size: 32 MiB (33554432 bytes)
 cluster_size: 1048576
 
-[{ "start": 0, "length": 3072, "depth": 0, "present": true, "zero":

false, "data": true, "offset": 1024},

Review file format: IMGFMT · line 59

Quoted 1 line
(quoted diff) +0 -1
-{ "start": 3072, "length": 33551360, "depth": 0, "present": true,

"zero": true, "data": true, "offset": 4096}]

Review file format: IMGFMT · line 56

Quoted 2 lines
(quoted diff) +1 -1
-
+[{"data": true, "depth": 0, "length": 3072, "offset": 1024, "present":

true, "start": 0, "zero": false}, {"data": true, "depth": 0, "length":
33551360, "offset": 4096, "present": true, "start": 3072, "zero": true}]

The change in format can produce really long lines for a more complex map, which can introduce its own problems in legibility. But I can live with it.

Quoted 6 lines

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org

Yeah, we don't have to print out the entire thing, either. We could also pretty-print it if we want to.

(Once we drop 3.6 (which I know is contested as to when we can do it) we can remove a lot of our special QMP sorting code and just start printing the raw JSON objects, which makes dealing with qmp a lot easier in diff-based tests.)

The point was more just to remove any copy-pastables using the JSON and provide only the "one good way". This patch in and of itself is otherwise pretty lateral.

Quoted 1 line
author
#14 lore raw

PATCH v4 11/18 iotests: change supports_quorum to use qemu_img

In reply to John Snow

Similar to other recent changes: use the qemu_img() invocation that supports throwing loud, nasty exceptions when it fails for surprising reasons.

(Why would "--help" ever fail? I don't know, but eliminating *all* calls to qemu-img that do not go through qemu_img() is my goal, so qemu_img_pipe() has to be removed.)

---
 tests/qemu-iotests/iotests.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/qemu-iotests/iotests.py +2 -2
index 58fb1734b5..17065fc176 100644
@@ -1427,8 +1427,8 @@ def _verify_imgopts(unsupported: Sequence[str] = ()) -> None:
         notrun(f'not suitable for this imgopts: {imgopts}')
 
 
-def supports_quorum():
-    return 'quorum' in qemu_img_pipe('--help')
+def supports_quorum() -> bool:
+    return 'quorum' in qemu_img('--help').stdout
 
 def verify_quorum():
     '''Skip test suite if quorum support is not available'''
Signature · 1 line
2.34.1
author
#15 lore raw

PATCH v4 12/18 iotests: replace unchecked calls to qemu_img_pipe()

In reply to John Snow

qemu_img_pipe() discards the return code from qemu-img in favor of returning just its output. Some tests using this function don't save, log, or check the output either, though, which is unsafe.

Replace all of these calls with a checked version.

Tests affected are 194, 202, 203, 234, 262, and 303.

---
 tests/qemu-iotests/194 | 4 ++--
 tests/qemu-iotests/202 | 4 ++--
 tests/qemu-iotests/203 | 4 ++--
 tests/qemu-iotests/234 | 4 ++--
 tests/qemu-iotests/262 | 2 +-
 tests/qemu-iotests/303 | 2 +-
 6 files changed, 10 insertions(+), 10 deletions(-)
tests/qemu-iotests/194 +2 -2
index e44b8df728..68894371f5 100755
@@ -33,8 +33,8 @@ with iotests.FilePath('source.img') as source_img_path, \
      iotests.VM('dest') as dest_vm:
 
     img_size = '1G'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, source_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, dest_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, source_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, dest_img_path, img_size)
 
     iotests.log('Launching VMs...')
     (source_vm.add_drive(source_img_path)
tests/qemu-iotests/202 +2 -2
index 8eb5f32d15..b784dcd791 100755
@@ -35,8 +35,8 @@ with iotests.FilePath('disk0.img') as disk0_img_path, \
      iotests.VM() as vm:
 
     img_size = '10M'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk0_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk1_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk0_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk1_img_path, img_size)
 
     iotests.log('Launching VM...')
     vm.launch()
tests/qemu-iotests/203 +2 -2
index ea30e50497..ab80fd0e44 100755
@@ -33,8 +33,8 @@ with iotests.FilePath('disk0.img') as disk0_img_path, \
      iotests.VM() as vm:
 
     img_size = '10M'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk0_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk1_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk0_img_path, img_size)
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk1_img_path, img_size)
 
     iotests.log('Launching VM...')
     (vm.add_object('iothread,id=iothread0')
tests/qemu-iotests/234 +2 -2
index cb5f1753e0..a9f764bb2c 100755
@@ -34,8 +34,8 @@ with iotests.FilePath('img') as img_path, \
      iotests.VM(path_suffix='a') as vm_a, \
      iotests.VM(path_suffix='b') as vm_b:
 
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, backing_path, '64M')
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, img_path, '64M')
+    iotests.qemu_img_create('-f', iotests.imgfmt, backing_path, '64M')
+    iotests.qemu_img_create('-f', iotests.imgfmt, img_path, '64M')
 
     os.mkfifo(fifo_a)
     os.mkfifo(fifo_b)
tests/qemu-iotests/262 +1 -1
index 32d69988ef..2294fd5ecb 100755
@@ -51,7 +51,7 @@ with iotests.FilePath('img') as img_path, \
 
         vm.add_device('virtio-blk,drive=%s,iothread=iothread0' % root)
 
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, img_path, '64M')
+    iotests.qemu_img_create('-f', iotests.imgfmt, img_path, '64M')
 
     os.mkfifo(fifo)
 
tests/qemu-iotests/303 +1 -1
index 16c2e10827..93aa5ce9b7 100755
@@ -38,7 +38,7 @@ def create_bitmap(bitmap_number, disabled):
     if disabled:
         args.append('--disable')
 
-    iotests.qemu_img_pipe(*args)
+    iotests.qemu_img(*args)
 
 
 def write_to_disk(offset, size):
Signature · 1 line
2.34.1
author
#16 lore raw

PATCH v4 13/18 iotests/149: Remove qemu_img_pipe() call

In reply to John Snow

qemu_img_pipe calls blank their output when the command being run is a 'create' call and the command succeeds. Thus, the normative output for this command in iotest 149 is to print a blank line. We can remove the logging from this invocation and use a checked invocation, but we still need to inspect the actual output to see if we want to retroactively skip the test due to missing cipher support.

---
 tests/qemu-iotests/149     |  7 +++++--
 tests/qemu-iotests/149.out | 21 ---------------------
 2 files changed, 5 insertions(+), 23 deletions(-)
tests/qemu-iotests/149 +5 -2
index d49646ca60..9bb96d6a1d 100755
@@ -265,8 +265,11 @@ def qemu_img_create(config, size_mb):
             "%dM" % size_mb]
 
     iotests.log("qemu-img " + " ".join(args), filters=[iotests.filter_test_dir])
-    iotests.log(check_cipher_support(config, iotests.qemu_img_pipe(*args)),
-                filters=[iotests.filter_test_dir])
+    try:
+        iotests.qemu_img(*args)
+    except subprocess.CalledProcessError as exc:
+        check_cipher_support(config, exc.output)
+        raise
 
 def qemu_io_image_args(config, dev=False):
     """Get the args for access an image or device with qemu-io"""
tests/qemu-iotests/149.out +0 -21
index ab879596ce..2cc5b82f7c 100644
@@ -61,7 +61,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-sha1.img
 # ================= qemu-img aes-256-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-aes-256-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-sha1.img qiotest-145-aes-256-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -180,7 +179,6 @@ unlink TEST_DIR/luks-twofish-256-xts-plain64-sha1.img
 # ================= qemu-img twofish-256-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=twofish-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-twofish-256-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-twofish-256-xts-plain64-sha1.img qiotest-145-twofish-256-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -299,7 +297,6 @@ unlink TEST_DIR/luks-serpent-256-xts-plain64-sha1.img
 # ================= qemu-img serpent-256-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=serpent-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-serpent-256-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-serpent-256-xts-plain64-sha1.img qiotest-145-serpent-256-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -418,7 +415,6 @@ unlink TEST_DIR/luks-cast5-128-cbc-plain64-sha1.img
 # ================= qemu-img cast5-128-cbc-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=cast5-128,cipher-mode=cbc,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-cast5-128-cbc-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-cast5-128-cbc-plain64-sha1.img qiotest-145-cast5-128-cbc-plain64-sha1
 # Write test pattern 0xa7
@@ -538,7 +534,6 @@ unlink TEST_DIR/luks-aes-256-cbc-plain-sha1.img
 # ================= qemu-img aes-256-cbc-plain-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=cbc,ivgen-alg=plain,hash-alg=sha1 TEST_DIR/luks-aes-256-cbc-plain-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-cbc-plain-sha1.img qiotest-145-aes-256-cbc-plain-sha1
 # Write test pattern 0xa7
@@ -657,7 +652,6 @@ unlink TEST_DIR/luks-aes-256-cbc-plain64-sha1.img
 # ================= qemu-img aes-256-cbc-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=cbc,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-aes-256-cbc-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-cbc-plain64-sha1.img qiotest-145-aes-256-cbc-plain64-sha1
 # Write test pattern 0xa7
@@ -776,7 +770,6 @@ unlink TEST_DIR/luks-aes-256-cbc-essiv-sha256-sha1.img
 # ================= qemu-img aes-256-cbc-essiv-sha256-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=cbc,ivgen-alg=essiv,hash-alg=sha1,ivgen-hash-alg=sha256 TEST_DIR/luks-aes-256-cbc-essiv-sha256-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-cbc-essiv-sha256-sha1.img qiotest-145-aes-256-cbc-essiv-sha256-sha1
 # Write test pattern 0xa7
@@ -895,7 +888,6 @@ unlink TEST_DIR/luks-aes-256-xts-essiv-sha256-sha1.img
 # ================= qemu-img aes-256-xts-essiv-sha256-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=essiv,hash-alg=sha1,ivgen-hash-alg=sha256 TEST_DIR/luks-aes-256-xts-essiv-sha256-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-essiv-sha256-sha1.img qiotest-145-aes-256-xts-essiv-sha256-sha1
 # Write test pattern 0xa7
@@ -1014,7 +1006,6 @@ unlink TEST_DIR/luks-aes-128-xts-plain64-sha256-sha1.img
 # ================= qemu-img aes-128-xts-plain64-sha256-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-128,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-aes-128-xts-plain64-sha256-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-128-xts-plain64-sha256-sha1.img qiotest-145-aes-128-xts-plain64-sha256-sha1
 # Write test pattern 0xa7
@@ -1133,7 +1124,6 @@ unlink TEST_DIR/luks-aes-192-xts-plain64-sha256-sha1.img
 # ================= qemu-img aes-192-xts-plain64-sha256-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-192,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-aes-192-xts-plain64-sha256-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-192-xts-plain64-sha256-sha1.img qiotest-145-aes-192-xts-plain64-sha256-sha1
 # Write test pattern 0xa7
@@ -1252,7 +1242,6 @@ unlink TEST_DIR/luks-twofish-128-xts-plain64-sha1.img
 # ================= qemu-img twofish-128-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=twofish-128,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-twofish-128-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-twofish-128-xts-plain64-sha1.img qiotest-145-twofish-128-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -1372,7 +1361,6 @@ unlink TEST_DIR/luks-serpent-128-xts-plain64-sha1.img
 # ================= qemu-img serpent-128-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=serpent-128,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-serpent-128-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-serpent-128-xts-plain64-sha1.img qiotest-145-serpent-128-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -1491,7 +1479,6 @@ unlink TEST_DIR/luks-serpent-192-xts-plain64-sha1.img
 # ================= qemu-img serpent-192-xts-plain64-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=serpent-192,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha1 TEST_DIR/luks-serpent-192-xts-plain64-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-serpent-192-xts-plain64-sha1.img qiotest-145-serpent-192-xts-plain64-sha1
 # Write test pattern 0xa7
@@ -1612,7 +1599,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-sha224.img
 # ================= qemu-img aes-256-xts-plain64-sha224 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha224 TEST_DIR/luks-aes-256-xts-plain64-sha224.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-sha224.img qiotest-145-aes-256-xts-plain64-sha224
 # Write test pattern 0xa7
@@ -1731,7 +1717,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-sha256.img
 # ================= qemu-img aes-256-xts-plain64-sha256 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha256 TEST_DIR/luks-aes-256-xts-plain64-sha256.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-sha256.img qiotest-145-aes-256-xts-plain64-sha256
 # Write test pattern 0xa7
@@ -1850,7 +1835,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-sha384.img
 # ================= qemu-img aes-256-xts-plain64-sha384 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha384 TEST_DIR/luks-aes-256-xts-plain64-sha384.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-sha384.img qiotest-145-aes-256-xts-plain64-sha384
 # Write test pattern 0xa7
@@ -1969,7 +1953,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-sha512.img
 # ================= qemu-img aes-256-xts-plain64-sha512 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=sha512 TEST_DIR/luks-aes-256-xts-plain64-sha512.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-sha512.img qiotest-145-aes-256-xts-plain64-sha512
 # Write test pattern 0xa7
@@ -2088,7 +2071,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain64-ripemd160.img
 # ================= qemu-img aes-256-xts-plain64-ripemd160 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain64,hash-alg=ripemd160 TEST_DIR/luks-aes-256-xts-plain64-ripemd160.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain64-ripemd160.img qiotest-145-aes-256-xts-plain64-ripemd160
 # Write test pattern 0xa7
@@ -2281,7 +2263,6 @@ unlink TEST_DIR/luks-aes-256-xts-plain-sha1-pwallslots.img
 # ================= qemu-img aes-256-xts-plain-sha1-pwallslots =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=c2xvdDE=,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=xts,ivgen-alg=plain,hash-alg=sha1 TEST_DIR/luks-aes-256-xts-plain-sha1-pwallslots.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-xts-plain-sha1-pwallslots.img qiotest-145-aes-256-xts-plain-sha1-pwallslots
 # Write test pattern 0xa7
@@ -2400,7 +2381,6 @@ unlink TEST_DIR/luks-aes-256-cbc-essiv-auto-sha1.img
 # ================= qemu-img aes-256-cbc-essiv-auto-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=cbc,ivgen-alg=essiv,hash-alg=sha1 TEST_DIR/luks-aes-256-cbc-essiv-auto-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-cbc-essiv-auto-sha1.img qiotest-145-aes-256-cbc-essiv-auto-sha1
 # Write test pattern 0xa7
@@ -2519,7 +2499,6 @@ unlink TEST_DIR/luks-aes-256-cbc-plain64-sha256-sha1.img
 # ================= qemu-img aes-256-cbc-plain64-sha256-sha1 =================
 # Create image
 qemu-img create -f luks --object secret,id=sec0,data=MTIzNDU2,format=base64 -o key-secret=sec0,iter-time=10,cipher-alg=aes-256,cipher-mode=cbc,ivgen-alg=plain64,hash-alg=sha1,ivgen-hash-alg=sha256 TEST_DIR/luks-aes-256-cbc-plain64-sha256-sha1.img 4194304M
-
 # Open dev
 sudo cryptsetup -q -v luksOpen TEST_DIR/luks-aes-256-cbc-plain64-sha256-sha1.img qiotest-145-aes-256-cbc-plain64-sha256-sha1
 # Write test pattern 0xa7
Signature · 1 line
2.34.1
author
#17 lore raw

PATCH v4 14/18 iotests: remove remaining calls to qemu_img_pipe()

In reply to John Snow

As part of moving all python iotest invocations of qemu-img onto a single qemu_img() implementation, remove a few lingering uses of qemu_img_pipe() from outside of iotests.py itself.

Several cases here rely on the knowledge that qemu_img_pipe() suppresses *all* output on a successful case when the command being issued is 'create'.

065: This call's output is inspected, but it appears as if it's expected
to succeed. Replace this call with the checked qemu_img() variant instead to get better diagnostics if/when qemu-img itself fails.

237: "create" call output isn't actually logged. Use qemu_img_create()
instead, which checks the return code. Remove the empty lines from the test output.

296: Two calls;
-create: Expected to succeed. Like other create calls, the output
isn't actually logged. Switch to a checked variant (qemu_img_create) instead. The output for this test is a mixture of both test styles, so actually replace the blank line for readability.
-amend: This is expected to fail. Log the output.

After this patch, the only uses of qemu_img_pipe are internal to iotests.py and will be removed in subsequent patches.

---
 tests/qemu-iotests/065     |  4 ++--
 tests/qemu-iotests/237     |  3 +--
 tests/qemu-iotests/237.out |  3 ---
 tests/qemu-iotests/296     | 12 ++++++------
 4 files changed, 9 insertions(+), 13 deletions(-)
tests/qemu-iotests/065 +2 -2
index 9466ce7df4..ba94e19349 100755
@@ -24,7 +24,7 @@ import os
 import re
 import json
 import iotests
-from iotests import qemu_img, qemu_img_info, qemu_img_pipe
+from iotests import qemu_img, qemu_img_info
 import unittest
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -54,7 +54,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
         self.assertEqual(data['data'], self.json_compare)
 
     def test_human(self):
-        data = qemu_img_pipe('info', '--output=human', test_img).split('\n')
+        data = qemu_img('info', '--output=human', test_img).stdout.split('\n')
         data = data[(data.index('Format specific information:') + 1)
                     :data.index('')]
         for field in data:
tests/qemu-iotests/237 +1 -2
index 43dfd3bd40..5ea13eb01f 100755
@@ -165,8 +165,7 @@ with iotests.FilePath('t.vmdk') as disk_path, \
     iotests.log("")
 
     for path in [ extent1_path, extent2_path, extent3_path ]:
-        msg = iotests.qemu_img_pipe('create', '-f', imgfmt, path, '0')
-        iotests.log(msg, [iotests.filter_testfiles])
+        iotests.qemu_img_create('-f', imgfmt, path, '0')
 
     vm.add_blockdev('driver=file,filename=%s,node-name=ext1' % (extent1_path))
     vm.add_blockdev('driver=file,filename=%s,node-name=ext2' % (extent2_path))
tests/qemu-iotests/237.out +0 -3
index aeb9724492..62b8865677 100644
@@ -129,9 +129,6 @@ Job failed: Cannot find device='this doesn't exist' nor node-name='this doesn't
 
 === Other subformats ===
 
-
-
-
 == Missing extent ==
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
tests/qemu-iotests/296 +6 -6
index f80ef3434a..0d21b740a7 100755
@@ -76,7 +76,7 @@ class EncryptionSetupTestCase(iotests.QMPTestCase):
     # create the encrypted block device using qemu-img
     def createImg(self, file, secret):
 
-        output = iotests.qemu_img_pipe(
+        iotests.qemu_img(
             'create',
             '--object', *secret.to_cmdline_object(),
             '-f', iotests.imgfmt,
@@ -84,8 +84,7 @@ class EncryptionSetupTestCase(iotests.QMPTestCase):
             '-o', 'iter-time=10',
             file,
             '1M')
-
-        iotests.log(output, filters=[iotests.filter_test_dir])
+        iotests.log('')
 
     # attempts to add a key using qemu-img
     def addKey(self, file, secret, new_secret):
@@ -99,7 +98,7 @@ class EncryptionSetupTestCase(iotests.QMPTestCase):
                 }
             }
 
-        output = iotests.qemu_img_pipe(
+        output = iotests.qemu_img(
             'amend',
             '--object', *secret.to_cmdline_object(),
             '--object', *new_secret.to_cmdline_object(),
@@ -108,8 +107,9 @@ class EncryptionSetupTestCase(iotests.QMPTestCase):
             '-o', 'new-secret=' + new_secret.id(),
             '-o', 'iter-time=10',
 
-            "json:" + json.dumps(image_options)
-            )
+            "json:" + json.dumps(image_options),
+            check=False  # Expected to fail. Log output.
+        ).stdout
 
         iotests.log(output, filters=[iotests.filter_test_dir])
 
Signature · 1 line
2.34.1
author
#18 lore raw

PATCH v4 15/18 iotests: use qemu_img() in has_working_luks()

In reply to John Snow

Admittedly a mostly lateral move, but qemu_img() is essentially the replacement for qemu_img_pipe_and_status(). It will give slightly better diagnostics on crash.

---
 tests/qemu-iotests/iotests.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
tests/qemu-iotests/iotests.py +9 -9
index 17065fc176..8491ff0477 100644
@@ -1444,20 +1444,20 @@ def has_working_luks() -> Tuple[bool, str]:
     """
 
     img_file = f'{test_dir}/luks-test.luks'
-    (output, status) = \
-        qemu_img_pipe_and_status('create', '-f', 'luks',
-                                 '--object', luks_default_secret_object,
-                                 '-o', luks_default_key_secret_opt,
-                                 '-o', 'iter-time=10',
-                                 img_file, '1G')
+    res = qemu_img('create', '-f', 'luks',
+                   '--object', luks_default_secret_object,
+                   '-o', luks_default_key_secret_opt,
+                   '-o', 'iter-time=10',
+                   img_file, '1G',
+                   check=False)
     try:
         os.remove(img_file)
     except OSError:
         pass
 
-    if status != 0:
-        reason = output
-        for line in output.splitlines():
+    if res.returncode:
+        reason = res.stdout
+        for line in res.stdout.splitlines():
             if img_file + ':' in line:
                 reason = line.split(img_file + ':', 1)[1].strip()
                 break
Signature · 1 line
2.34.1
author
#19 lore raw

PATCH v4 16/18 iotests: replace qemu_img_log('create', ...) calls

In reply to John Snow

qemu_img_log() calls into qemu_img_pipe(), which always removes output for 'create' commands on success anyway. Replace all of these calls to the simpler qemu_img_create(...) which doesn't log, but raises a detailed exception object on failure instead.

Blank lines are removed from output files where appropriate.

---
 tests/qemu-iotests/255     |  8 ++++----
 tests/qemu-iotests/255.out |  4 ----
 tests/qemu-iotests/274     | 17 ++++++++---------
 tests/qemu-iotests/274.out | 29 -----------------------------
 tests/qemu-iotests/280     |  2 +-
 tests/qemu-iotests/280.out |  1 -
 6 files changed, 13 insertions(+), 48 deletions(-)
tests/qemu-iotests/255 +4 -4
index 3d6d0e80cb..f86fa851b6 100755
@@ -42,8 +42,8 @@ with iotests.FilePath('t.qcow2') as disk_path, \
     size_str = str(size)
 
     iotests.create_image(base_path, size)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, mid_path, size_str)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, disk_path, size_str)
+    iotests.qemu_img_create('-f', iotests.imgfmt, mid_path, size_str)
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, size_str)
 
     # Create a backing chain like this:
     # base <- [throttled: bps-read=4096] <- mid <- overlay
@@ -92,8 +92,8 @@ with iotests.FilePath('src.qcow2') as src_path, \
     size = 128 * 1024 * 1024
     size_str = str(size)
 
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, src_path, size_str)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, dst_path, size_str)
+    iotests.qemu_img_create('-f', iotests.imgfmt, src_path, size_str)
+    iotests.qemu_img_create('-f', iotests.imgfmt, dst_path, size_str)
 
     iotests.log(iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 1M',
                                 src_path),
tests/qemu-iotests/255.out +0 -4
index 11a05a5213..2e837cbb5f 100644
@@ -3,8 +3,6 @@ Finishing a commit job with background reads
 
 === Create backing chain and start VM ===
 
-
-
 === Start background read requests ===
 
 === Run a commit job ===
@@ -21,8 +19,6 @@ Closing the VM while a job is being cancelled
 
 === Create images and start VM ===
 
-
-
 wrote 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
tests/qemu-iotests/274 +8 -9
index 080a90f10f..2495e051a2 100755
@@ -31,12 +31,11 @@ size_long = 2 * 1024 * 1024
 size_diff = size_long - size_short
 
 def create_chain() -> None:
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, base,
-                         str(size_long))
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base,
-                         '-F', iotests.imgfmt, mid, str(size_short))
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid,
-                         '-F', iotests.imgfmt, top, str(size_long))
+    iotests.qemu_img_create('-f', iotests.imgfmt, base, str(size_long))
+    iotests.qemu_img_create('-f', iotests.imgfmt, '-b', base,
+                            '-F', iotests.imgfmt, mid, str(size_short))
+    iotests.qemu_img_create('-f', iotests.imgfmt, '-b', mid,
+                            '-F', iotests.imgfmt, top, str(size_long))
 
     iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base)
 
@@ -160,9 +159,9 @@ with iotests.FilePath('base') as base, \
             ('off',      '512k', '256k', '500k', '436k')]:
 
         iotests.log('=== preallocation=%s ===' % prealloc)
-        iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, base_size)
-        iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base,
-                             '-F', iotests.imgfmt, top, top_size_old)
+        iotests.qemu_img_create('-f', iotests.imgfmt, base, base_size)
+        iotests.qemu_img_create('-f', iotests.imgfmt, '-b', base,
+                                '-F', iotests.imgfmt, top, top_size_old)
         iotests.qemu_io_log('-c', 'write -P 1 %s 64k' % off, base)
 
         # After this, top_size_old to base_size should be allocated/zeroed.
tests/qemu-iotests/274.out +0 -29
index 1ce40d839a..acd8b166a6 100644
@@ -1,7 +1,4 @@
 == Commit tests ==
-
-
-
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -63,9 +60,6 @@ read 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing HMP commit (top -> mid) ===
-
-
-
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -92,9 +86,6 @@ read 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing QMP active commit (top -> mid) ===
-
-
-
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -127,9 +118,6 @@ read 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing qemu-img commit (top -> base) ===
-
-
-
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -154,9 +142,6 @@ read 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing QMP active commit (top -> base) ===
-
-
-
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -190,8 +175,6 @@ read 1048576/1048576 bytes at offset 1048576
 
 == Resize tests ==
 === preallocation=off ===
-
-
 wrote 65536/65536 bytes at offset 5368709120
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -207,8 +190,6 @@ read 65536/65536 bytes at offset 5368709120
 { "start": 1073741824, "length": 7516192768, "depth": 0, "present": true, "zero": true, "data": false}]
 
 === preallocation=metadata ===
-
-
 wrote 65536/65536 bytes at offset 33285996544
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -229,8 +210,6 @@ read 65536/65536 bytes at offset 33285996544
 { "start": 34896609280, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 2685075456}]
 
 === preallocation=falloc ===
-
-
 wrote 65536/65536 bytes at offset 9437184
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -246,8 +225,6 @@ read 65536/65536 bytes at offset 9437184
 { "start": 5242880, "length": 10485760, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=full ===
-
-
 wrote 65536/65536 bytes at offset 11534336
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -263,8 +240,6 @@ read 65536/65536 bytes at offset 11534336
 { "start": 8388608, "length": 4194304, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=off ===
-
-
 wrote 65536/65536 bytes at offset 259072
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -281,8 +256,6 @@ read 65536/65536 bytes at offset 259072
 { "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
 
 === preallocation=off ===
-
-
 wrote 65536/65536 bytes at offset 344064
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -298,8 +271,6 @@ read 65536/65536 bytes at offset 344064
 { "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
 
 === preallocation=off ===
-
-
 wrote 65536/65536 bytes at offset 446464
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
tests/qemu-iotests/280 +1 -1
index 628f3c33ca..5f50500fdb 100755
@@ -33,7 +33,7 @@ with iotests.FilePath('base') as base_path , \
      iotests.FilePath('top') as top_path, \
      iotests.VM() as vm:
 
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, base_path, '64M')
+    iotests.qemu_img_create('-f', iotests.imgfmt, base_path, '64M')
 
     iotests.log('=== Launch VM ===')
     vm.add_object('iothread,id=iothread0')
tests/qemu-iotests/280.out +0 -1
index e39164c579..c75f437c00 100644
@@ -1,4 +1,3 @@
-
 === Launch VM ===
 Enabling migration QMP events on VM...
 {"return": {}}
Signature · 1 line
2.34.1
author
#20 lore raw

PATCH v4 17/18 iotests: remove qemu_img_pipe_and_status()

In reply to John Snow

With the exceptional 'create' calls removed in the prior commit, change qemu_img_log() and img_info_log() to call qemu_img() directly instead.

For now, allow these calls to qemu-img to return non-zero on the basis that any unusual output will be logged anyway. The very next commit begins to enforce a successful exit code by default even for the logged functions.

---
 tests/qemu-iotests/iotests.py | 26 +++++++-------------------
 1 file changed, 7 insertions(+), 19 deletions(-)
tests/qemu-iotests/iotests.py +7 -19
index 8491ff0477..c75c7470e2 100644
@@ -207,15 +207,6 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
 
     return result
 
-def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:
-    """
-    Run qemu-img and return both its output and its exit code
-    """
-    is_create = bool(args and args[0] == 'create')
-    full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
-    return qemu_tool_pipe_and_status('qemu-img', full_args,
-                                     drop_successful_output=is_create)
-
 def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
              ) -> subprocess.CompletedProcess[str]:
     """
@@ -320,17 +311,14 @@ def qemu_img_info(*args: str) -> Any:
 def qemu_img_map(*args: str) -> Any:
     return qemu_img_json('map', "--output", "json", *args)
 
-def qemu_img_pipe(*args: str) -> str:
-    '''Run qemu-img and return its output'''
-    return qemu_img_pipe_and_status(*args)[0]
-
-def qemu_img_log(*args):
-    result = qemu_img_pipe(*args)
-    log(result, filters=[filter_testfiles])
+def qemu_img_log(*args: str) -> subprocess.CompletedProcess[str]:
+    result = qemu_img(*args, check=False)
+    log(result.stdout, filters=[filter_testfiles])
     return result
 
-def img_info_log(filename, filter_path=None, use_image_opts=False,
-                 extra_args=()):
+def img_info_log(filename: str, filter_path: Optional[str] = None,
+                 use_image_opts: bool = False, extra_args: Sequence[str] = (),
+                 ) -> None:
     args = ['info']
     if use_image_opts:
         args.append('--image-opts')
@@ -339,7 +327,7 @@ def img_info_log(filename, filter_path=None, use_image_opts=False,
     args += extra_args
     args.append(filename)
 
-    output = qemu_img_pipe(*args)
+    output = qemu_img(*args, check=False).stdout
     if not filter_path:
         filter_path = filename
     log(filter_img_info(output, filter_path))
Signature · 1 line
2.34.1
author
#21 lore raw

PATCH v4 18/18 iotests: make qemu_img_log and img_info_log raise on error

In reply to John Snow

Add a check: bool = True parameter to both functions and make their qemu_img() invocations raise on error by default.

users of img_info_log:
206, 207, 210, 211, 212, 213, 237, 242, 266, 274, 302

users of qemu_img_log:
044, 209, 274, 302, 304

iotests 242 and 266 need to use check=False for their negative tests. iotests 206, 210, 211, 212, 213, 237, 274 and 302 continue working normally.

As of this commit, all calls to QEMU_IMG made from iotests enforce a return code of zero by default unless explicitly disabled or suppressed by passing check=False or with an exception handler.

---
 tests/qemu-iotests/242        | 2 +-
 tests/qemu-iotests/266        | 2 +-
 tests/qemu-iotests/iotests.py | 8 +++++---
 3 files changed, 7 insertions(+), 5 deletions(-)
tests/qemu-iotests/242 +1 -1
index 547bf382e3..b3afd36d72 100755
@@ -100,7 +100,7 @@ add_bitmap(1, True, False)
 log('Write an unknown bitmap flag \'{}\' into a new QCOW2 image at offset {}'
     .format(hex(bitmap_flag_unknown), flag_offset))
 toggle_flag(flag_offset)
-img_info_log(disk)
+img_info_log(disk, check=False)
 toggle_flag(flag_offset)
 log('Unset the unknown bitmap flag \'{}\' in the bitmap directory entry:\n'
     .format(hex(bitmap_flag_unknown)))
tests/qemu-iotests/266 +1 -1
index 71ce81d0df..8fc3807ac5 100755
@@ -137,7 +137,7 @@ def main():
             iotests.log('')
 
             vm.shutdown()
-            iotests.img_info_log(file_path)
+            iotests.img_info_log(file_path, check=False)
 
 
 iotests.script_main(main,
tests/qemu-iotests/iotests.py +5 -3
index c75c7470e2..6cd8374c81 100644
@@ -311,13 +311,15 @@ def qemu_img_info(*args: str) -> Any:
 def qemu_img_map(*args: str) -> Any:
     return qemu_img_json('map', "--output", "json", *args)
 
-def qemu_img_log(*args: str) -> subprocess.CompletedProcess[str]:
-    result = qemu_img(*args, check=False)
+def qemu_img_log(*args: str, check: bool = True
+                 ) -> subprocess.CompletedProcess[str]:
+    result = qemu_img(*args, check=check)
     log(result.stdout, filters=[filter_testfiles])
     return result
 
 def img_info_log(filename: str, filter_path: Optional[str] = None,
                  use_image_opts: bool = False, extra_args: Sequence[str] = (),
+                 check: bool = True,
                  ) -> None:
     args = ['info']
     if use_image_opts:
@@ -327,7 +329,7 @@ def img_info_log(filename: str, filter_path: Optional[str] = None,
     args += extra_args
     args.append(filename)
 
-    output = qemu_img(*args, check=False).stdout
+    output = qemu_img(*args, check=check).stdout
     if not filter_path:
         filter_path = filename
     log(filter_img_info(output, filter_path))
Signature · 1 line
2.34.1
#22 lore raw
In reply to John Snow
Quoted 11 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

Quoted 7 lines

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks!  Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

author
#23 lore raw
Quoted 17 lines · Hanna Reitz
Quoted 16 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

Quoted 9 lines
Quoted 2 lines

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks! Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Thanks so much!

I have more works-in-progress, but I want to be kind to your time. (And tolerance level for Python.)

Important:

- 4 patches that switch to async qmp permanently. Almost no code, it's just a policy thing, but it could affect iotests. Not for this freeze now, but it'd help me a lot if you could take the time to ack it next week so I can stage them and push forward with splitting the qmp library out of the tree. I need to rebase and resend, which I'll do in just a bit.

Not urgent:

- Another 15ish patches for unifying qemu-io calls like i did for qemu-img here. Stalled somewhat because I couldn't convincingly unify the qemu_io calls that keep the pipe open, so I will probably just leave those calls alone for now, unless I get a New Idea. Should I send them to the list and you'll get to them whenever you get to them, or would you prefer I wait a while?

- Another 15ish patches that split the "skip files" list for pylint/mypy into separate skip-lists per tool and then drastically reduces their size such that only a handful of files remain in each skiplist. Same question
here: Should I send these to the list and someone'll get to it whenever they do, or would you prefer I wait?

Thanks again.
--js

Quoted 1 line
#24 lore raw
Quoted 43 lines · John Snow

On Fri, Mar 18, 2022, 9:36 AM Hanna Reitz <[email protected]> wrote:

On 18.03.22 00:49, John Snow wrote:
> Hiya!
>
> This series effectively replaces qemu_img_pipe_and_status() with a > rewritten function named qemu_img() that raises an exception on non-zero
> return code by default. By the end of the series, every last invocation
> of the qemu-img binary ultimately goes through qemu_img(). >
> The exception that this function raises includes stdout/stderr output
> when the traceback is printed in a a little decorated text box so that
> it stands out from the jargony Python traceback readout. >
> (You can test what this looks like for yourself, or at least you could,
> by disabling ztsd support and then running qcow2 iotest 065.) >
> Negative tests are still possible in two ways:
>
> - Passing check=False to qemu_img, qemu_img_log, or img_info_log > - Catching and handling the CalledProcessError exception at the callsite.

Thanks!  Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Thanks so much!

I have more works-in-progress, but I want to be kind to your time. (And tolerance level for Python.)

Important:

Quoted 7 lines

- 4 patches that switch to async qmp permanently. Almost no code, it's
just a policy thing, but it could affect iotests. Not for this freeze now, but it'd help me a lot if you could take the time to ack it next week so I can stage them and push forward with splitting the qmp library out of the tree. I need to rebase and resend, which I'll do in just a bit.

Not urgent:

- Another 15ish patches for unifying qemu-io calls like i did for
qemu-img here. Stalled somewhat because I couldn't convincingly unify the qemu_io calls that keep the pipe open, so I will probably just leave those calls alone for now, unless I get a New Idea. Should I send them to the list and you'll get to them whenever you get to them, or would you prefer I wait a while?

I don’t mind you sending them.

Quoted 2 lines

- Another 15ish patches that split the "skip files" list for
pylint/mypy into separate skip-lists per tool and then drastically reduces their size such that only a handful of files remain in each skiplist. Same question here: Should I send these to the list and someone'll get to it whenever they do, or would you prefer I wait?

Same reply. :)

Hanna

author
#25 lore raw
In reply to Hanna Reitz
Quoted 18 lines · Hanna Reitz
Quoted 16 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

Quoted 9 lines
Quoted 2 lines

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks! Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Actually, hold it -- this looks like it is causing problems with the Gitlab CI. I need to investigate these.
https://gitlab.com/jsnow/qemu/-/pipelines/495155073/failures

... and, ugh, naturally the nice error diagnostics are suppressed here so I can't see them. Well, there's one more thing to try and fix somehow.

--js

#26 lore raw
Quoted 26 lines · John Snow
Quoted 17 lines · Hanna Reitz
Quoted 16 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

Quoted 8 lines
Quoted 2 lines

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks! Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Quoted 7 lines

Actually, hold it -- this looks like it is causing problems with the Gitlab CI. I need to investigate these.
https://gitlab.com/jsnow/qemu/-/pipelines/495155073/failures

... and, ugh, naturally the nice error diagnostics are suppressed here so I can't see them. Well, there's one more thing to try and fix somehow.

I hope this patch by Thomas fixes the logging at least:

https://lists.nongnu.org/archive/html/qemu-devel/2022-03/msg02946.html

Let’s see.

Hanna

#27 lore raw
Quoted 32 lines · Hanna Reitz
Quoted 31 lines · John Snow
Quoted 17 lines · Hanna Reitz
Quoted 16 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero return code by default. By the end of the series, every last invocation of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could, by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

Quoted 9 lines
Quoted 3 lines

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks!  Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Actually, hold it -- this looks like it is causing problems with the Gitlab CI. I need to investigate these.
https://gitlab.com/jsnow/qemu/-/pipelines/495155073/failures

Quoted 7 lines
Quoted 3 lines

... and, ugh, naturally the nice error diagnostics are suppressed here so I can't see them. Well, there's one more thing to try and fix somehow.

I hope this patch by Thomas fixes the logging at least:

https://lists.nongnu.org/archive/html/qemu-devel/2022-03/msg02946.html

So I found three issues:

1. check-patch wrongfully complains about the comment added in in “python/utils: add add_visual_margin() text decoration utility” that shows an example for how the output looks.  It complains the lines consisting mostly of “━━━━━━━━” were too long.  I believe that’s because it counts bytes, not characters.

Not fatal, i.e. doesn’t break the pipeline.  We should ignore that.

2. riscv64-debian-cross-container breaks, but that looks pre-existing.  apt complains about some dependencies.

Also marked as allowed-to-fail, so I believe we should also just ignore that.  (Seems to fail on master, too.)

3. The rest are runs complaining about subprocess.CompletedProcess[str].  Looks like the same issue I was facing for ec88eed8d14088b36a3495710368b8d1a3c33420, where I had to specify the type as a string.

Indeed this is fixed by something like https://gitlab.com/hreitz/qemu/-/commit/87615eb536bdca7babe8eb4a35fd4ea810d1da24 .  Maybe squash that in?  (If it’s the correct way to go about this?)

Hanna

author
#28 lore raw
Quoted 46 lines · Hanna Reitz
Quoted 35 lines · Hanna Reitz
Quoted 34 lines · John Snow
Quoted 20 lines · Hanna Reitz
Quoted 19 lines · John Snow

Hiya!

This series effectively replaces qemu_img_pipe_and_status() with a rewritten function named qemu_img() that raises an exception on non-zero
return code by default. By the end of the series, every last invocation
of the qemu-img binary ultimately goes through qemu_img().

The exception that this function raises includes stdout/stderr output when the traceback is printed in a a little decorated text box so that it stands out from the jargony Python traceback readout.

(You can test what this looks like for yourself, or at least you could,
by disabling ztsd support and then running qcow2 iotest 065.)

Negative tests are still possible in two ways:

Quoted 9 lines
Quoted 3 lines

- Passing check=False to qemu_img, qemu_img_log, or img_info_log
- Catching and handling the CalledProcessError exception at the callsite.

Thanks! Applied to my block branch:

https://gitlab.com/hreitz/qemu/-/commits/block

Hanna

Actually, hold it -- this looks like it is causing problems with the Gitlab CI. I need to investigate these.
https://gitlab.com/jsnow/qemu/-/pipelines/495155073/failures

Quoted 7 lines
Quoted 3 lines

... and, ugh, naturally the nice error diagnostics are suppressed here so I can't see them. Well, there's one more thing to try and fix somehow.

I hope this patch by Thomas fixes the logging at least:

https://lists.nongnu.org/archive/html/qemu-devel/2022-03/msg02946.html

So I found three issues:

Quoted 8 lines

1. check-patch wrongfully complains about the comment added in in “python/utils: add add_visual_margin() text decoration utility” that shows an example for how the output looks. It complains the lines consisting mostly of “━━━━━━━━” were too long. I believe that’s because it counts bytes, not characters.

Not fatal, i.e. doesn’t break the pipeline. We should ignore that.

Agree. (Though I did shorten the lines in my re-spin to see if I could make it shut up, but it didn't work. Ignoring it is.)

Quoted 6 lines

2. riscv64-debian-cross-container breaks, but that looks pre-existing. apt complains about some dependencies.

Also marked as allowed-to-fail, so I believe we should also just ignore that. (Seems to fail on master, too.)

Yeah, I don't think this is me.

Quoted 5 lines

3. The rest are runs complaining about subprocess.CompletedProcess[str]. Looks like the same issue I was facing for ec88eed8d14088b36a3495710368b8d1a3c33420, where I had to specify the type as a string.

Quoted 7 lines

Indeed this is fixed by something like

https://gitlab.com/hreitz/qemu/-/commit/87615eb536bdca7babe8eb4a35fd4ea810d1da24
. Maybe squash that in? (If it’s the correct way to go about this?)

Hanna

Yep, sorry for not replying. I respun the series and tested it, but it became "way too Saturday" for me to hit send on the respin. Will do so today.

(Annoying: I test under python 3.6, but I didn't *run the iotests with 3.6*, which is where this problem shows up. Meh.)

#29 lore raw
Quoted 90 lines · John Snow

On Mon, Mar 21, 2022, 11:50 AM Hanna Reitz <[email protected]> wrote:

On 21.03.22 14:14, Hanna Reitz wrote:
> On 18.03.22 22:14, John Snow wrote:
>> On Fri, Mar 18, 2022 at 9:36 AM Hanna Reitz <[email protected]> wrote:
>>> On 18.03.22 00:49, John Snow wrote:
>>>> Hiya!
>>>>
>>>> This series effectively replaces qemu_img_pipe_and_status() with a
>>>> rewritten function named qemu_img() that raises an exception on >>>> non-zero
>>>> return code by default. By the end of the series, every last >>>> invocation
>>>> of the qemu-img binary ultimately goes through qemu_img(). >>>>
>>>> The exception that this function raises includes stdout/stderr output
>>>> when the traceback is printed in a a little decorated text box so that
>>>> it stands out from the jargony Python traceback readout. >>>>
>>>> (You can test what this looks like for yourself, or at least you >>>> could,
>>>> by disabling ztsd support and then running qcow2 iotest 065.) >>>>
>>>> Negative tests are still possible in two ways:
>>>>
>>>> - Passing check=False to qemu_img, qemu_img_log, or img_info_log >>>> - Catching and handling the CalledProcessError exception at the >>>> callsite.
>>> Thanks!  Applied to my block branch:
>>>
>>> https://gitlab.com/hreitz/qemu/-/commits/block
>>>
>>> Hanna
>>>
>> Actually, hold it -- this looks like it is causing problems with the
>> Gitlab CI. I need to investigate these.
>> https://gitlab.com/jsnow/qemu/-/pipelines/495155073/failures
>>
>> ... and, ugh, naturally the nice error diagnostics are suppressed here
>> so I can't see them. Well, there's one more thing to try and fix >> somehow.
>
> I hope this patch by Thomas fixes the logging at least:
>
>
https://lists.nongnu.org/archive/html/qemu-devel/2022-03/msg02946.html

So I found three issues:

1. check-patch wrongfully complains about the comment added in in “python/utils: add add_visual_margin() text decoration utility” that shows an example for how the output looks.  It complains the lines consisting mostly of “━━━━━━━━” were too long.  I believe that’s because
it counts bytes, not characters.

Not fatal, i.e. doesn’t break the pipeline.  We should ignore that.

Agree. (Though I did shorten the lines in my re-spin to see if I could make it shut up, but it didn't work. Ignoring it is.)

2. riscv64-debian-cross-container breaks, but that looks pre-existing.
apt complains about some dependencies.

Also marked as allowed-to-fail, so I believe we should also just ignore
that.  (Seems to fail on master, too.)

Yeah, I don't think this is me.

3. The rest are runs complaining about subprocess.CompletedProcess[str].  Looks like the same issue I was facing for ec88eed8d14088b36a3495710368b8d1a3c33420, where I had to specify the type as a string.

Indeed this is fixed by something like
https://gitlab.com/hreitz/qemu/-/commit/87615eb536bdca7babe8eb4a35fd4ea810d1da24

Quoted 6 lines

.  Maybe squash that in?  (If it’s the correct way to go about this?)

Hanna

Yep, sorry for not replying. I respun the series and tested it, but it became "way too Saturday" for me to hit send on the respin. Will do so today.

Great, thanks!

Hanna