Skip to contentSkip to conversation

Convert avocado tests to normal Python unittests

PATCH Superseded Thomas Huth 5 participants 36 messages Jul 16, 2024 – Jul 29, 2024 Arm/Amlogic Meson SoC support

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Convert avocado tests to normal Python unittests

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export PYTHONPATH=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Thomas Huth (11):
tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Add python-based tests to the meson build system tests/functional: Implement fetch_asset() method for downloading
assets
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Set up logging
tests/functional: Convert the s390x avocado tests into standalone
tests
gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  22 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  18 +-
 tests/functional/meson.build                  | 112 +++++
 tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  28 ++
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_mem_addr_space.py}                   |  53 +--
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_version.py}                   |   8 +-
 tests/meson.build                             |   1 +
 28 files changed, 795 insertions(+), 320 deletions(-)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/11 tests/functional: Add base classes for the upcoming pytest-based tests

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

---
 tests/functional/qemu_test/__init__.py | 326 +++++++++++++++++++++++++
 1 file changed, 326 insertions(+)
 create mode 100644 tests/functional/qemu_test/__init__.py
tests/functional/qemu_test/__init__.py new file +326 -0
new file mode 100644
index 0000000000..e73705d40a
@@ -0,0 +1,326 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import uuid
+import unittest
+
+from pathlib import Path
+from qemu.machine import QEMUMachine
+from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
+                        tcg_available)
+
+def _source_dir():
+    # Determine top-level directory of the QEMU sources
+    return Path(__file__).parent.parent.parent.parent
+
+def _build_dir():
+    root = os.getenv('QEMU_BUILD_ROOT')
+    if root is not None:
+        return Path(root)
+    # Makefile.mtest only exists in build dir, so if it is available, use CWD
+    if os.path.exists('Makefile.mtest'):
+        return Path(os.getcwd())
+
+    root = os.path.join(_source_dir(), 'build')
+    if os.path.exists(root):
+        return Path(root)
+
+    raise Exception("Cannot identify build dir, set QEMU_BUILD_ROOT")
+
+def has_cmd(name, args=None):
+    """
+    This function is for use in a @skipUnless decorator, e.g.:
+
+        @skipUnless(*has_cmd('sudo -n', ('sudo', '-n', 'true')))
+        def test_something_that_needs_sudo(self):
+            ...
+    """
+
+    if args is None:
+        args = ('which', name)
+
+    try:
+        _, stderr, exitcode = run_cmd(args)
+    except Exception as e:
+        exitcode = -1
+        stderr = str(e)
+
+    if exitcode != 0:
+        cmd_line = ' '.join(args)
+        err = f'{name} required, but "{cmd_line}" failed: {stderr.strip()}'
+        return (False, err)
+    else:
+        return (True, '')
+
+def has_cmds(*cmds):
+    """
+    This function is for use in a @skipUnless decorator and
+    allows checking for the availability of multiple commands, e.g.:
+
+        @skipUnless(*has_cmds(('cmd1', ('cmd1', '--some-parameter')),
+                              'cmd2', 'cmd3'))
+        def test_something_that_needs_cmd1_and_cmd2(self):
+            ...
+    """
+
+    for cmd in cmds:
+        if isinstance(cmd, str):
+            cmd = (cmd,)
+
+        ok, errstr = has_cmd(*cmd)
+        if not ok:
+            return (False, errstr)
+
+    return (True, '')
+
+def run_cmd(args):
+    subp = subprocess.Popen(args,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            universal_newlines=True)
+    stdout, stderr = subp.communicate()
+    ret = subp.returncode
+
+    return (stdout, stderr, ret)
+
+def is_readable_executable_file(path):
+    return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
+
+def _console_interaction(test, success_message, failure_message,
+                         send_string, keep_sending=False, vm=None):
+    assert not keep_sending or send_string
+    if vm is None:
+        vm = test.vm
+    console = vm.console_file
+    console_logger = logging.getLogger('console')
+    while True:
+        if send_string:
+            vm.console_socket.sendall(send_string.encode())
+            if not keep_sending:
+                send_string = None # send only once
+        try:
+            msg = console.readline().decode().strip()
+        except UnicodeDecodeError:
+            msg = None
+        if not msg:
+            continue
+        console_logger.debug(msg)
+        if success_message is None or success_message in msg:
+            break
+        if failure_message and failure_message in msg:
+            console.close()
+            fail = 'Failure message found in console: "%s". Expected: "%s"' % \
+                    (failure_message, success_message)
+            test.fail(fail)
+
+def interrupt_interactive_console_until_pattern(test, success_message,
+                                                failure_message=None,
+                                                interrupt_string='\r'):
+    """
+    Keep sending a string to interrupt a console prompt, while logging the
+    console output. Typical use case is to break a boot loader prompt, such:
+
+        Press a key within 5 seconds to interrupt boot process.
+        5
+        4
+        3
+        2
+        1
+        Booting default image...
+
+    :param test: a  test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    :param interrupt_string: a string to send to the console before trying
+                             to read a new line
+    """
+    _console_interaction(test, success_message, failure_message,
+                         interrupt_string, True)
+
+def wait_for_console_pattern(test, success_message, failure_message=None,
+                             vm=None):
+    """
+    Waits for messages to appear on the console, while logging the content
+
+    :param test: a test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    """
+    _console_interaction(test, success_message, failure_message, None, vm=vm)
+
+def exec_command(test, command):
+    """
+    Send a command to a console (appending CRLF characters), while logging
+    the content.
+
+    :param test: a test containing a VM.
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param command: the command to send
+    :type command: str
+    """
+    _console_interaction(test, None, None, command + '\r')
+
+def exec_command_and_wait_for_pattern(test, command,
+                                      success_message, failure_message=None):
+    """
+    Send a command to a console (appending CRLF characters), then wait
+    for success_message to appear on the console, while logging the.
+    content. Mark the test as failed if failure_message is found instead.
+
+    :param test: a test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param command: the command to send
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    """
+    _console_interaction(test, success_message, failure_message, command + '\r')
+
+class QemuBaseTest(unittest.TestCase):
+
+    qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+
+    BUILD_DIR = _build_dir()
+
+    workdir = None
+    log = logging.getLogger('qemu-test')
+
+    def setUp(self, bin_prefix):
+        self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+
+        self.workdir = os.path.join(self.BUILD_DIR, 'tests/functional')
+        self.workdir = os.path.join(self.workdir, self.id())
+        if not os.path.exists(self.workdir):
+            os.makedirs(self.workdir)
+
+    def main():
+        tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
+                                   test_output_log = pycotap.LogMode.LogToError)
+        unittest.main(testRunner = tr)
+
+
+class QemuSystemTest(QemuBaseTest):
+    """Facilitates system emulation tests."""
+
+    cpu = None
+    machine = None
+    _machinehelp = None
+
+    def setUp(self):
+        self._vms = {}
+
+        super().setUp('qemu-system-')
+
+    def set_machine(self, machinename):
+        # TODO: We should use QMP to get the list of available machines
+        if not self._machinehelp:
+            self._machinehelp = run_cmd([self.qemu_bin, '-M', 'help'])[0];
+        if self._machinehelp.find(machinename) < 0:
+            self.skipTest('no support for machine ' + machinename)
+        self.machine = machinename
+
+    def require_accelerator(self, accelerator):
+        """
+        Requires an accelerator to be available for the test to continue
+
+        It takes into account the currently set qemu binary.
+
+        If the check fails, the test is canceled.  If the check itself
+        for the given accelerator is not available, the test is also
+        canceled.
+
+        :param accelerator: name of the accelerator, such as "kvm" or "tcg"
+        :type accelerator: str
+        """
+        checker = {'tcg': tcg_available,
+                   'kvm': kvm_available}.get(accelerator)
+        if checker is None:
+            self.skipTest("Don't know how to check for the presence "
+                          "of accelerator %s" % accelerator)
+        if not checker(qemu_bin=self.qemu_bin):
+            self.skipTest("%s accelerator does not seem to be "
+                          "available" % accelerator)
+
+    def require_netdev(self, netdevname):
+        netdevhelp = run_cmd([self.qemu_bin,
+                             '-M', 'none', '-netdev', 'help'])[0];
+        if netdevhelp.find('\n' + netdevname + '\n') < 0:
+            self.skipTest('no support for " + netdevname + " networking')
+
+    def require_device(self, devicename):
+        devhelp = run_cmd([self.qemu_bin,
+                           '-M', 'none', '-device', 'help'])[0];
+        if devhelp.find(devicename) < 0:
+            self.skipTest('no support for device ' + devicename)
+
+    def _new_vm(self, name, *args):
+        vm = QEMUMachine(self.qemu_bin, base_temp_dir=self.workdir)
+        self.log.debug('QEMUMachine "%s" created', name)
+        self.log.debug('QEMUMachine "%s" temp_dir: %s', name, vm.temp_dir)
+        self.log.debug('QEMUMachine "%s" log_dir: %s', name, vm.log_dir)
+        if args:
+            vm.add_args(*args)
+        return vm
+
+    @property
+    def vm(self):
+        return self.get_vm(name='default')
+
+    def get_vm(self, *args, name=None):
+        if not name:
+            name = str(uuid.uuid4())
+        if self._vms.get(name) is None:
+            self._vms[name] = self._new_vm(name, *args)
+            if self.cpu is not None:
+                self._vms[name].add_args('-cpu', self.cpu)
+            if self.machine is not None:
+                self._vms[name].set_machine(self.machine)
+        return self._vms[name]
+
+    def set_vm_arg(self, arg, value):
+        """
+        Set an argument to list of extra arguments to be given to the QEMU
+        binary. If the argument already exists then its value is replaced.
+
+        :param arg: the QEMU argument, such as "-cpu" in "-cpu host"
+        :type arg: str
+        :param value: the argument value, such as "host" in "-cpu host"
+        :type value: str
+        """
+        if not arg or not value:
+            return
+        if arg not in self.vm.args:
+            self.vm.args.extend([arg, value])
+        else:
+            idx = self.vm.args.index(arg) + 1
+            if idx < len(self.vm.args):
+                self.vm.args[idx] = value
+            else:
+                self.vm.args.append(value)
+
+    def tearDown(self):
+        for vm in self._vms.values():
+            vm.shutdown()
+        super().tearDown()
Signature · 1 line
2.45.2
#2 lore raw

PATCH 2/11 tests/functional: Convert simple avocado tests into standalone python tests

In reply to Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
 .../test_cpu_queries.py}                      |  7 ++-
 .../test_empty_cpu_model.py}                  |  7 ++-
 .../test_mem_addr_space.py}                   | 53 +++----------------
 .../test_pc_cpu_hotplug_props.py}             | 11 ++--
 .../test_virtio_version.py}                   |  8 +--
 5 files changed, 29 insertions(+), 57 deletions(-)
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
tests/functional/test_cpu_queries.py renamed from tests/avocado/cpu_queries.py +6 -1
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/cpu_queries.py
rename to tests/functional/test_cpu_queries.py
index d3faa14720..97dd600c01
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Sanity check of query-cpu-* results
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -8,7 +10,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class QueryCPUModelExpansion(QemuSystemTest):
     """
@@ -33,3 +35,6 @@ def test(self):
             e = self.vm.cmd('query-cpu-model-expansion', model=model,
                             type='full')
             self.assertEqual(e['model']['name'], c['name'])
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_empty_cpu_model.py renamed from tests/avocado/empty_cpu_model.py +6 -1
old mode 100644
new mode 100755
similarity index 84%
rename from tests/avocado/empty_cpu_model.py
rename to tests/functional/test_empty_cpu_model.py
index d906ef3d3c..0081b06d85
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Check for crash when using empty -cpu option
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -7,7 +9,7 @@
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class EmptyCPUModel(QemuSystemTest):
     def test(self):
@@ -17,3 +19,6 @@ def test(self):
         self.vm.wait()
         self.assertEqual(self.vm.exitcode(), 1, "QEMU exit code should be 1")
         self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +6 -47

155 lines not on this page · raw message

tests/functional/test_pc_cpu_hotplug_props.py renamed from tests/avocado/pc_cpu_hotplug_props.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/pc_cpu_hotplug_props.py
rename to tests/functional/test_pc_cpu_hotplug_props.py
index 4bd3e02665..9d5a37cb17
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # Ensure CPU die-id can be omitted on -device
 #
@@ -20,16 +21,16 @@
 # License along with this library; if not, see <http://www.gnu.org/licenses/>.
 #
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class OmittedCPUProps(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=cpu:qemu64
-    """
+
     def test_no_die_id(self):
         self.vm.add_args('-nodefaults', '-S')
         self.vm.add_args('-smp', '1,sockets=2,cores=2,threads=2,maxcpus=8')
         self.vm.add_args('-device', 'qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0')
         self.vm.launch()
         self.assertEqual(len(self.vm.cmd('query-cpus-fast')), 2)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_virtio_version.py renamed from tests/avocado/virtio_version.py +5 -3
old mode 100644
new mode 100755
similarity index 98%
rename from tests/avocado/virtio_version.py
rename to tests/functional/test_virtio_version.py
index afe5e828b5..eb23060564
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 """
 Check compatibility of virtio device types
 """
@@ -12,7 +13,7 @@
 import os
 
 from qemu.machine import QEMUMachine
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 # Virtio Device IDs:
 VIRTIO_NET = 1
@@ -60,8 +61,6 @@ class VirtioVersionCheck(QemuSystemTest):
     Check if virtio-version-specific device types result in the
     same device tree created by `disable-modern` and
     `disable-legacy`.
-
-    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
@@ -173,3 +172,6 @@ def test_modern_only_devs(self):
         self.check_modern_only('virtio-mouse-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-tablet-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-keyboard-pci', VIRTIO_INPUT)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v1 2/11
#3 lore raw

Re: [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests

Quoted 42 lines · Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
 .../test_cpu_queries.py}                      |  7 ++-
 .../test_empty_cpu_model.py}                  |  7 ++-
 .../test_mem_addr_space.py}                   | 53 +++----------------
 .../test_pc_cpu_hotplug_props.py}             | 11 ++--
 .../test_virtio_version.py}                   |  8 +--
 5 files changed, 29 insertions(+), 57 deletions(-)
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +2 -0
old mode 100644
new mode 100755
similarity index 92%
rename from tests/avocado/mem-addr-space-check.py
rename to tests/functional/test_mem_addr_space.py
index 85541ea051..bb0cf062ca
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Check for crash when using memory beyond the available guest processor
 # address space.
 #

tests/functional/test_mem_addr_space.py line 12

Quoted 5 lines
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +0 -2
@@ -8,8 +10,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 
-from avocado_qemu import QemuSystemTest
-import signal

Nit-pick - cleanup of an unrelated existing bug - 'signal' wasn't used. Suggest doing it in a separate patch

Aside from that:

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw

PATCH 3/11 tests/functional: Convert avocado tests that just need a small adjustment

In reply to Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
 .../test_info_usernet.py}                     | 11 ++-
 .../test_ppc_74xx.py}                         | 74 ++++++++-----------
 .../version.py => functional/test_version.py} | 13 ++--
 3 files changed, 47 insertions(+), 51 deletions(-)
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
tests/functional/test_info_usernet.py renamed from tests/avocado/info_usernet.py +7 -4
old mode 100644
new mode 100755
similarity index 87%
rename from tests/avocado/info_usernet.py
rename to tests/functional/test_info_usernet.py
index e1aa7a6e0a..ded973253a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test for the hmp command "info usernet"
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -8,18 +10,16 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 from qemu.utils import get_info_usernet_hostfwd_port
 
 
 class InfoUsernet(QemuSystemTest):
-    """
-    :avocado: tags=machine:none
-    """
 
     def test_hostfwd(self):
         self.require_netdev('user')
+        self.machine = 'none'
         self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
@@ -31,3 +31,6 @@ def test_hostfwd(self):
         self.assertGreater(port, 0,
                            ('Found a redirected port that is not greater than'
                             ' zero'))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_74xx.py renamed from tests/avocado/ppc_74xx.py +32 -42

167 lines not on this page · raw message

tests/functional/test_version.py renamed from tests/avocado/version.py +8 -5
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..5e566d76b1
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Version check example test
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
 class Version(QemuSystemTest):
-    """
-    :avocado: tags=quick
-    :avocado: tags=machine:none
-    """
+
     def test_qmp_human_info_version(self):
+        self.machine = 'none'
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
                           command_line='info version')
         self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v1 3/11
#5 lore raw

Re: [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment

Quoted 6 lines · Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

Quoted 11 lines
---
 .../test_info_usernet.py}                     | 11 ++-
 .../test_ppc_74xx.py}                         | 74 ++++++++-----------
 .../version.py => functional/test_version.py} | 13 ++--
 3 files changed, 47 insertions(+), 51 deletions(-)
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

PATCH 4/11 tests/functional: Add python-based tests to the meson build system

In reply to Thomas Huth

Integrate the new python-based test framework with the meson build system. Since these tests now require the pycotap module, make sure that it gets installed in the venv.

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use pycotap for running the tests instead).

---
 pythondeps.toml              |  3 +-
 tests/Makefile.include       | 18 ++++++++-
 tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 4 files changed, 95 insertions(+), 2 deletions(-)
 create mode 100644 tests/functional/meson.build
pythondeps.toml +2 -1
index f6e590fdd8..c018b4d74a 100644
@@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
 sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
-[avocado]
+[tests]
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
 avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
+pycotap = { accepted = ">=1.1.0" }
tests/Makefile.include +17 -1
index d39d5dd6a4..2bdf607977 100644
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest            Run qtest tests"
+	@echo " $(MAKE) check-functional       Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARG  Run functional tests for a given target"
 	@echo " $(MAKE) check-unit             Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
 	@echo " $(MAKE) check-block            Run block tests"
@@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
 
 $(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(MKVENV_ENSUREGROUP) $< avocado
+	$(MKVENV_ENSUREGROUP) $< tests
 	$(call quiet-command, touch $@)
 
 $(TESTS_RESULTS_DIR):
@@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+# Make sure that pycotap is installed before running any functional tests:
+ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
+do-meson-check: check-venv
+endif
+
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@make SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@make SPEED=thorough check-func check-func-quick
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
tests/functional/meson.build new file +75 -0

75 lines not on this page · raw message

tests/meson.build +1 -0
index acb6807094..3345ad2098 100644
@@ -85,3 +85,4 @@ subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')
Signature · 1 line
2.45.2
Reviewed-by on v1 4/11
#7 lore raw

Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system

Quoted 9 lines · Thomas Huth

Integrate the new python-based test framework with the meson build system. Since these tests now require the pycotap module, make sure that it gets installed in the venv.

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use pycotap for running the tests instead).

Just one touch-up below.

Quoted 54 lines
---
 pythondeps.toml              |  3 +-
 tests/Makefile.include       | 18 ++++++++-
 tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 4 files changed, 95 insertions(+), 2 deletions(-)
 create mode 100644 tests/functional/meson.build
pythondeps.toml +2 -1
index f6e590fdd8..c018b4d74a 100644
@@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
 sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
-[avocado]
+[tests]
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
 avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
+pycotap = { accepted = ">=1.1.0" }
tests/Makefile.include +2 -0
index d39d5dd6a4..2bdf607977 100644
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest            Run qtest tests"
+	@echo " $(MAKE) check-functional       Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARG  Run functional tests for

a given target"

tests/Makefile.include +4 -1
 	@echo " $(MAKE) check-unit             Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
 	@echo " $(MAKE) check-block            Run block tests"
@@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
 
 $(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(MKVENV_ENSUREGROUP) $< avocado
+	$(MKVENV_ENSUREGROUP) $< tests
 	$(call quiet-command, touch $@)
 
 $(TESTS_RESULTS_DIR):
@@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+# Make sure that pycotap is installed before running any functional tests:
+ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
+do-meson-check: check-venv

tests/Makefile.include check-acceptance-deprecated-warning: · line 169

Quoted 10 lines
tests/Makefile.include +10 -0
+endif
+
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@make SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@make SPEED=thorough check-func check-func-quick

I think these^ two should use $(MAKE) instead:

make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.

Quoted 73 lines
tests/Makefile.include +1 -0
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
tests/functional/meson.build new file +63 -0
new file mode 100644
index 0000000000..11352b5bb5
@@ -0,0 +1,75 @@
+# QEMU functional tests:
+# Tests that are put in the 'quick' category are run by default during
+# 'make check'. Everything that should not be run during 'make check'
+# (e.g. tests that fetch assets from the internet) should be put into
+# the 'thorough' category instead.
+
+# Most tests run too slow with TCI enabled, so skip the functional tests there
+if get_option('tcg_interpreter')
+  subdir_done()
+endif
+
+# Timeouts for individual tests that can be slow e.g. with debugging enabled
+test_timeouts = {
+  'ppc_74xx' : 90,
+}
+
+tests_generic = [
+  'empty_cpu_model',
+  'info_usernet',
+  'version',
+]
+
+tests_ppc_quick = [
+  'ppc_74xx',
+]
+
+tests_x86_64_quick = [
+  'cpu_queries',
+  'mem_addr_space',
+  'pc_cpu_hotplug_props',
+  'virtio_version',
+]
+
+foreach speed : ['quick', 'thorough']
+  foreach dir : target_dirs
+    if not dir.endswith('-softmmu')
+      continue
+    endif
+
+    target_base = dir.split('-')[0]
+    test_emulator = emulators['qemu-system-' + target_base]
+
+    if speed == 'quick'
+      suites = ['func-quick', 'func-' + target_base]
+      target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
+    else
+      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
+      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
+    endif
+
+    test_deps = roms
+    test_env = environment()
+    if have_tools
+      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
+      test_deps += [qemu_img]
+    endif
+    test_env.set('QEMU_TEST_QEMU_BINARY',
+                 meson.global_build_root() / 'qemu-system-' + target_base)
+    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
+    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+                               meson.current_source_dir())
+
+    foreach test : target_tests

tests/functional/meson.build line 73

Quoted 10 lines
tests/functional/meson.build new file +10 -0
+      test('func-@0@/@1@'.format(target_base, test),
+           python,
+           depends: [test_deps, test_emulator, emulator_modules],
+           env: test_env,
+           args: [meson.current_source_dir() / 'test_' + test + '.py'],
+           protocol: 'tap',
+           timeout: test_timeouts.get(test, 60),
+           priority: test_timeouts.get(test, 60),
+           suite: suites)
+    endforeach

2/19 qemu:func-thorough+func-s390x-thorough+thorough / func-s390x/s390x_ccw_virtio OK 48.82s 2 subtests passe

func, thorough, func, thorough, thorough, func
s390x, s390x, s390x

=)

I know, not much we can do...

Quoted 11 lines
tests/functional/meson.build new file +2 -0
+  endforeach
+endforeach
tests/meson.build +1 -0
index acb6807094..3345ad2098 100644
@@ -85,3 +85,4 @@ subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')
#8 lore raw

Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system

Quoted 83 lines · Fabiano Rosas
Quoted 9 lines · Thomas Huth

Integrate the new python-based test framework with the meson build system. Since these tests now require the pycotap module, make sure that it gets installed in the venv.

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use pycotap for running the tests instead).

Just one touch-up below.

Quoted 54 lines
---
  pythondeps.toml              |  3 +-
  tests/Makefile.include       | 18 ++++++++-
  tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
  tests/meson.build            |  1 +
  4 files changed, 95 insertions(+), 2 deletions(-)

create mode 100644 tests/functional/meson.build

pythondeps.toml +2 -1
index f6e590fdd8..c018b4d74a 100644
@@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
  sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
  sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
-[avocado]
+[tests]
  # Note that qemu.git/python/ is always implicitly installed.
  # Prefer an LTS version when updating the accepted versions of
  # avocado-framework, for example right now the limit is 92.x.
  avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
  pycdlib = { accepted = ">=1.11.0" }
+pycotap = { accepted = ">=1.1.0" }
tests/Makefile.include +2 -0
index d39d5dd6a4..2bdf607977 100644
@@ -9,6 +9,8 @@ check-help:
  	@echo "Individual test suites:"
  	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
  	@echo " $(MAKE) check-qtest            Run qtest tests"
+	@echo " $(MAKE) check-functional       Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARG  Run functional tests for

a given target"

  	@echo " $(MAKE) check-unit             Run qobject tests"
  	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
  	@echo " $(MAKE) check-block            Run block tests"
tests/Makefile.include +4 -1
@@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
 
  $(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
  	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(MKVENV_ENSUREGROUP) $< avocado
+	$(MKVENV_ENSUREGROUP) $< tests
  	$(call quiet-command, touch $@)
 
  $(TESTS_RESULTS_DIR):
@@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
 
  check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+# Make sure that pycotap is installed before running any functional tests:
+ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
+do-meson-check: check-venv

tests/Makefile.include check-acceptance-deprecated-warning: · line 169

Quoted 10 lines
tests/Makefile.include +10 -0
+endif
+
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@make SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@make SPEED=thorough check-func check-func-quick

I think these^ two should use $(MAKE) instead:

Quoted 2 lines

make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.

Right, thanks! I'll change it in the next version.

Thomas

Reviewed-by on v1 4/11
#9 lore raw

Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system

In reply to Thomas Huth
Quoted 8 lines · Thomas Huth

Integrate the new python-based test framework with the meson build system. Since these tests now require the pycotap module, make sure that it gets installed in the venv.

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use pycotap for running the tests instead).

Quoted 8 lines
---
 pythondeps.toml              |  3 +-
 tests/Makefile.include       | 18 ++++++++-
 tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 4 files changed, 95 insertions(+), 2 deletions(-)
 create mode 100644 tests/functional/meson.build

Strictly speaking this patch probably ought to be #2, otherwise we have a bisection window where we've converted some tests but not run them.

Quoted 22 lines
pythondeps.toml +2 -1
index f6e590fdd8..c018b4d74a 100644
@@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
 sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
-[avocado]
+[tests]
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
 avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
+pycotap = { accepted = ">=1.1.0" }
tests/Makefile.include +0 -0
index d39d5dd6a4..2bdf607977 100644

tests/Makefile.include check-help: · line 13

Quoted 5 lines
tests/Makefile.include +2 -0
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest            Run qtest tests"
+	@echo " $(MAKE) check-functional       Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARG  Run functional tests for a given target"

We could increase whitespace by 2 to fit TARGET, or shorten all cases to TGT ?

tests/Makefile.include quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ · line 113

Quoted 5 lines
tests/Makefile.include +0 -0
 	@echo " $(MAKE) check-unit             Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
 	@echo " $(MAKE) check-block            Run block tests"
@@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
 

The above is a minor non-functional point though so

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#10 lore raw

PATCH 5/11 tests/functional: Implement fetch_asset() method for downloading assets

In reply to Thomas Huth

In the new python test framework, we cannot use the fetch_asset() function from Avocado anymore, so we have to provide our own implementation now instead. Thus add such a function based on the urllib python module for this purpose.

---
 tests/functional/qemu_test/__init__.py | 36 ++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
tests/functional/qemu_test/__init__.py +36 -0
index e73705d40a..77c3b14d34 100644
@@ -11,6 +11,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import hashlib
+import urllib.request
 import logging
 import os
 import pycotap
@@ -23,6 +25,7 @@
 import unittest
 
 from pathlib import Path
+from shutil import copyfileobj
 from qemu.machine import QEMUMachine
 from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
                         tcg_available)
@@ -215,6 +218,39 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+    def check_hash(self, file_name, expected_hash):
+        if not expected_hash:
+            return True
+        if len(expected_hash) == 40:
+            sum_prog = 'sha1sum'
+        elif len(expected_hash) == 64:
+            sum_prog = 'sha256sum'
+        elif len(expected_hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")
+        checksum = subprocess.check_output([sum_prog, file_name]).split()[0]
+        return expected_hash == checksum.decode("utf-8")
+
+    def fetch_asset(self, url, asset_hash):
+        cache_dir = os.path.expanduser("~/.cache/qemu/download")
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+        fname = os.path.join(cache_dir,
+                             hashlib.sha256(url.encode("utf-8")).hexdigest())
+        if os.path.exists(fname) and self.check_hash(fname, asset_hash):
+            return fname
+        self.log.info("Downloading %s to %s...", url, fname)
+        dl_fname = fname + ".download"
+        with urllib.request.urlopen(url) as src:
+            with open(dl_fname, "wb+") as dst:
+                copyfileobj(src, dst)
+        if not self.check_hash(dl_fname, asset_hash):
+            os.remove(dl_fname)
+            raise Exception("Hash of " + url + " does not match")
+        os.rename(dl_fname, fname)
+        return fname
+
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
Signature · 1 line
2.45.2
#11 lore raw

Re: [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets

Quoted 52 lines · Thomas Huth

In the new python test framework, we cannot use the fetch_asset() function from Avocado anymore, so we have to provide our own implementation now instead. Thus add such a function based on the urllib python module for this purpose.

---
 tests/functional/qemu_test/__init__.py | 36 ++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
tests/functional/qemu_test/__init__.py +20 -0
index e73705d40a..77c3b14d34 100644
@@ -11,6 +11,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import hashlib
+import urllib.request
 import logging
 import os
 import pycotap
@@ -23,6 +25,7 @@
 import unittest
 
 from pathlib import Path
+from shutil import copyfileobj
 from qemu.machine import QEMUMachine
 from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
                         tcg_available)
@@ -215,6 +218,39 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+    def check_hash(self, file_name, expected_hash):
+        if not expected_hash:
+            return True
+        if len(expected_hash) == 40:
+            sum_prog = 'sha1sum'
+        elif len(expected_hash) == 64:
+            sum_prog = 'sha256sum'
+        elif len(expected_hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")
+        checksum = subprocess.check_output([sum_prog, file_name]).split()[0]
+        return expected_hash == checksum.decode("utf-8")
+
+    def fetch_asset(self, url, asset_hash):
+        cache_dir = os.path.expanduser("~/.cache/qemu/download")
+        if not os.path.exists(cache_dir):

tests/functional/qemu_test/__init__.py def setUp(self, bin_prefix): · line 247

Quoted 10 lines
tests/functional/qemu_test/__init__.py +10 -0
+            os.makedirs(cache_dir)
+        fname = os.path.join(cache_dir,
+                             hashlib.sha256(url.encode("utf-8")).hexdigest())
+        if os.path.exists(fname) and self.check_hash(fname, asset_hash):
+            return fname
+        self.log.info("Downloading %s to %s...", url, fname)
+        dl_fname = fname + ".download"
+        with urllib.request.urlopen(url) as src:
+            with open(dl_fname, "wb+") as dst:
+                copyfileobj(src, dst)

For cleanliness we should probably wrap this and delete the file on error, eg

try:

     with open(dl_fname,....) as dst
        ....

except:
os.remove(dl_fname)
raise

Quoted 12 lines
tests/functional/qemu_test/__init__.py +6 -0
+        if not self.check_hash(dl_fname, asset_hash):
+            os.remove(dl_fname)
+            raise Exception("Hash of " + url + " does not match")
+        os.rename(dl_fname, fname)
+        return fname
+
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

PATCH 6/11 tests/functional: Convert some tests that download files via fetch_asset()

In reply to Thomas Huth

Now that we've got a working fetch_asset() function, we can convert some Avocado tests that use this function for downloading their required files.

---
 MAINTAINERS                                   | 12 +++----
 tests/functional/meson.build                  | 25 +++++++++++++++
 .../test_arm_n8x0.py}                         | 25 +++++++--------
 .../test_avr_mega2560.py}                     | 11 ++++---
 .../test_loongarch64_virt.py}                 | 16 ++++++----
 .../test_mips64el_loongson3v.py}              | 26 +++++++--------
 .../test_netdev_ethtool.py}                   | 32 ++++++-------------
 .../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
 8 files changed, 89 insertions(+), 77 deletions(-)
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
MAINTAINERS +6 -6
index 7d9811458c..533e22b548 100644
@@ -221,7 +221,7 @@ S: Maintained
 F: docs/system/target-avr.rst
 F: gdb-xml/avr-cpu.xml
 F: target/avr/
-F: tests/avocado/machine_avr6.py
+F: tests/functional/test_avr_mega2560.py
 
 CRIS TCG CPUs
 M: Edgar E. Iglesias <[email protected]>
@@ -264,7 +264,7 @@ M: Song Gao <[email protected]>
 S: Maintained
 F: target/loongarch/
 F: tests/tcg/loongarch64/
-F: tests/avocado/machine_loongarch.py
+F: tests/functional/test_loongarch64_virt.py
 
 M68K TCG CPUs
 M: Laurent Vivier <[email protected]>
@@ -883,7 +883,7 @@ F: include/hw/display/blizzard.h
 F: include/hw/input/lm832x.h
 F: include/hw/input/tsc2xxx.h
 F: include/hw/misc/cbus.h
-F: tests/avocado/machine_arm_n8x0.py
+F: tests/functional/test_arm_n8x0.py
 F: docs/system/arm/nseries.rst
 
 Palm
@@ -1392,7 +1392,7 @@ F: hw/mips/loongson3_bootp.h
 F: hw/mips/loongson3_virt.c
 F: include/hw/intc/loongson_ipi.h
 F: include/hw/intc/loongson_liointc.h
-F: tests/avocado/machine_mips_loongson3v.py
+F: tests/functional/test_mips64el_loongson3v.py
 
 Boston
 M: Paul Burton <[email protected]>
@@ -1418,7 +1418,7 @@ PowerPC Machines
 L: [email protected]
 S: Orphan
 F: hw/ppc/ppc405*
-F: tests/avocado/ppc_405.py
+F: tests/functional/test_ppc_405.py
 
 Bamboo
 L: [email protected]
@@ -2483,7 +2483,7 @@ R: Sriram Yagnaraman <[email protected]>
 S: Maintained
 F: docs/system/devices/igb.rst
 F: hw/net/igb*
-F: tests/avocado/netdev-ethtool.py
+F: tests/functional/test_netdev_ethtool.py
 F: tests/qtest/igb-test.c
 F: tests/qtest/libqos/igb.c
 
tests/functional/meson.build +25 -0
index 11352b5bb5..947f9ac74f 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'netdev_ethtool' : 180,
   'ppc_74xx' : 90,
 }
 
@@ -20,10 +21,30 @@ tests_generic = [
   'version',
 ]
 
+tests_arm_thorough = [
+  'arm_n8x0',
+]
+
+tests_avr_thorough = [
+  'avr_mega2560',
+]
+
+tests_loongarch64_thorough = [
+  'loongarch64_virt',
+]
+
+tests_mips64el_thorough = [
+  'mips64el_loongson3v',
+]
+
 tests_ppc_quick = [
   'ppc_74xx',
 ]
 
+tests_ppc_thorough = [
+  'ppc_405',
+]
+
 tests_x86_64_quick = [
   'cpu_queries',
   'mem_addr_space',
@@ -31,6 +52,10 @@ tests_x86_64_quick = [
   'virtio_version',
 ]
 
+tests_x86_64_thorough = [
+  'netdev_ethtool',
+]
+
 foreach speed : ['quick', 'thorough']
   foreach dir : target_dirs
     if not dir.endswith('-softmmu')
tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +12 -13
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..d451c80a73
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,9 +12,9 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class N8x0Machine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
@@ -32,18 +34,15 @@ def __do_test_n8x0(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'TSC2005 driver initializing')
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n800(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n800
-        """
+        self.set_machine('n800')
         self.__do_test_n8x0()
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n810(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n810
-        """
+        self.set_machine('n810')
         self.__do_test_n8x0()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_avr_mega2560.py renamed from tests/avocado/machine_avr6.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/machine_avr6.py
rename to tests/functional/test_avr_mega2560.py
index 5485db79c6..b9b89367c6
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # QEMU AVR integration tests
 #
@@ -19,16 +20,12 @@
 
 import time
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class AVR6Machine(QemuSystemTest):
     timeout = 5
 
     def test_freertos(self):
-        """
-        :avocado: tags=arch:avr
-        :avocado: tags=machine:arduino-mega-2560-v3
-        """
         """
         https://github.com/seharris/qemu-avr-tests/raw/master/free-rtos/Demo/AVR_ATMega2560_GCC/demo.elf
         constantly prints out 'ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX'
@@ -39,6 +36,7 @@ def test_freertos(self):
         rom_hash = '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4'
         rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
 
+        self.set_machine('arduino-mega-2560-v3')
         self.vm.add_args('-bios', rom_path)
         self.vm.add_args('-nographic')
         self.vm.launch()
@@ -48,3 +46,6 @@ def test_freertos(self):
 
         self.assertIn('ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX',
                 self.vm.get_log())
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_loongarch64_virt.py renamed from tests/avocado/machine_loongarch.py +9 -7
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/machine_loongarch.py
rename to tests/functional/test_loongarch64_virt.py
index 8de308f2d6..3dc8bb3516
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
 # LoongArch virt test.
@@ -5,9 +7,9 @@
 # Copyright (c) 2023 Loongson Technology Corporation Limited
 #
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
 
 class LoongArchMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -21,10 +23,7 @@ def wait_for_console_pattern(self, success_message, vm=None):
 
     def test_loongarch64_devices(self):
 
-        """
-        :avocado: tags=arch:loongarch64
-        :avocado: tags=machine:virt
-        """
+        self.set_machine('virt')
 
         kernel_url = ('https://github.com/yangxiaojuan-loongson/qemu-binary/'
                       'releases/download/2024-05-30/vmlinuz.efi')
@@ -56,3 +55,6 @@ def test_loongarch64_devices(self):
         self.wait_for_console_pattern('Run /sbin/init as init process')
         exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
                                           'processor		: 3')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mips64el_loongson3v.py renamed from tests/avocado/machine_mips_loongson3v.py +12 -14
old mode 100644
new mode 100755
similarity index 55%
rename from tests/avocado/machine_mips_loongson3v.py
rename to tests/functional/test_mips64el_loongson3v.py
index 5194cf18c9..f8d40b48a9
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional tests for the Generic Loongson-3 Platform.
 #
 # Copyright (c) 2021 Jiaxun Yang <[email protected]>
@@ -10,30 +12,26 @@
 import os
 import time
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class MipsLoongson3v(QemuSystemTest):
     timeout = 60
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_pmon_serial_console(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=endian:little
-        :avocado: tags=machine:loongson3-virt
-        :avocado: tags=cpu:Loongson-3A1000
-        :avocado: tags=device:liointc
-        :avocado: tags=device:goldfish_rtc
-        """
+        self.set_machine('loongson3-virt')
 
-        pmon_hash = '7c8b45dd81ccfc55ff28f5aa267a41c3'
+        pmon_hash = 'fcdf6bb2cb7885a4a62f31fcb0d5e368bac7b6cea28f40c6dfa678af22fea20a'
         pmon_path = self.fetch_asset('https://github.com/loongson-community/pmon/'
                                     'releases/download/20210112/pmon-3avirt.bin',
-                                     asset_hash=pmon_hash, algorithm='md5')
+                                     asset_hash=pmon_hash)
 
         self.vm.set_console()
         self.vm.add_args('-bios', pmon_path)
         self.vm.launch()
         wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_netdev_ethtool.py renamed from tests/avocado/netdev-ethtool.py +10 -22
old mode 100644
new mode 100755
similarity index 81%
rename from tests/avocado/netdev-ethtool.py
rename to tests/functional/test_netdev_ethtool.py
index 5f33288f81..341c087e36
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # ethtool tests for emulated network devices
 #
 # This test leverages ethtool's --test sequence to validate network
@@ -5,15 +7,11 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-late
 
-from avocado import skip
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skip
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class NetDevEthtool(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=machine:q35
-    """
 
     # Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
     timeout = 45
@@ -25,11 +23,10 @@ def get_asset(self, name, sha1):
                     'kE4nCFLdQcoBF9t/download?'
                     'path=%2Fnetdev-ethtool&files=' )
         url = base_url + name
-        # use explicit name rather than failing to neatly parse the
-        # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(url, asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
+        self.set_machine('q35')
 
         # This custom kernel has drivers for all the supported network
         # devices we can emulate in QEMU
@@ -68,15 +65,9 @@ def common_test_code(self, netdev, extra_args=None):
         self.vm.kill()
 
     def test_igb(self):
-        """
-        :avocado: tags=device:igb
-        """
         self.common_test_code("igb")
 
     def test_igb_nomsi(self):
-        """
-        :avocado: tags=device:igb
-        """
         self.common_test_code("igb", "pci=nomsi")
 
     # It seems the other popular cards we model in QEMU currently fail
@@ -88,14 +79,11 @@ def test_igb_nomsi(self):
 
     @skip("Incomplete reg 0x00178 support")
     def test_e1000(self):
-        """
-        :avocado: tags=device:e1000
-        """
         self.common_test_code("e1000")
 
     @skip("Incomplete reg 0x00178 support")
     def test_i82550(self):
-        """
-        :avocado: tags=device:i82550
-        """
         self.common_test_code("i82550")
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_405.py renamed from tests/avocado/ppc_405.py +9 -10
old mode 100644
new mode 100755
similarity index 73%
rename from tests/avocado/ppc_405.py
rename to tests/functional/test_ppc_405.py
index 4e7e01aa76..f2368ada80
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that the U-Boot firmware boots on ppc 405 machines and check the console
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -5,10 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class Ppc405Machine(QemuSystemTest):
 
@@ -26,11 +27,9 @@ def do_test_ppc405(self):
         exec_command_and_wait_for_pattern(self, 'reset', 'AMCC PowerPC 405EP')
 
     def test_ppc_ref405ep(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:ref405ep
-        :avocado: tags=cpu:405ep
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('ref405ep')
         self.do_test_ppc405()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v1 6/11
#13 lore raw

Re: [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset()

Quoted 19 lines · Thomas Huth

Now that we've got a working fetch_asset() function, we can convert some Avocado tests that use this function for downloading their required files.

---
 MAINTAINERS                                   | 12 +++----
 tests/functional/meson.build                  | 25 +++++++++++++++
 .../test_arm_n8x0.py}                         | 25 +++++++--------
 .../test_avr_mega2560.py}                     | 11 ++++---
 .../test_loongarch64_virt.py}                 | 16 ++++++----
 .../test_mips64el_loongson3v.py}              | 26 +++++++--------
 .../test_netdev_ethtool.py}                   | 32 ++++++-------------
 .../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
 8 files changed, 89 insertions(+), 77 deletions(-)
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
Quoted 8 lines
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

PATCH 7/11 tests/functional: Add a function for extracting files from an archive

In reply to Thomas Huth

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

---
 tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/functional/qemu_test/utils.py
tests/functional/qemu_test/utils.py new file +21 -0
new file mode 100644
index 0000000000..4eb5e5d5e5
@@ -0,0 +1,21 @@
+# Utilities for python-based QEMU tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# Authors:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import tarfile
+
+def archive_extract(archive, dest_dir, member=None):
+    with tarfile.open(archive) as tf:
+        if hasattr(tarfile, 'data_filter'):
+            tf.extraction_filter = getattr(tarfile, 'data_filter',
+                                           (lambda member, path: member))
+        if member:
+            tf.extract(member=member, path=dest_dir)
+        else:
+            tf.extractall(path=dest_dir)
Signature · 1 line
2.45.2
Reviewed-by on v1 7/11
#15 lore raw

Re: [PATCH 07/11] tests/functional: Add a function for extracting files from an archive

Quoted 7 lines · Thomas Huth

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

Quoted 5 lines
---
 tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/functional/qemu_test/utils.py

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#16 lore raw

PATCH 8/11 tests/functional: Convert some avocado tests that needed avocado.utils.archive

In reply to Thomas Huth

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

---
 MAINTAINERS                                   |  6 ++---
 tests/functional/meson.build                  |  6 +++++
 .../test_arm_canona1100.py}                   | 21 +++++++++-------
 .../test_ppc_bamboo.py}                       | 23 ++++++++---------
 .../test_sparc64_sun4u.py}                    | 25 +++++++++++--------
 5 files changed, 46 insertions(+), 35 deletions(-)
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
MAINTAINERS +3 -3
index 533e22b548..025227954c 100644
@@ -730,7 +730,7 @@ S: Odd Fixes
 F: include/hw/arm/digic.h
 F: hw/*/digic*
 F: include/hw/*/digic*
-F: tests/avocado/machine_arm_canona1100.py
+F: tests/functional/test_arm_canona1100.py
 F: docs/system/arm/digic.rst
 
 Goldfish RTC
@@ -1425,7 +1425,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/ppc440_bamboo.c
 F: hw/pci-host/ppc4xx_pci.c
-F: tests/avocado/ppc_bamboo.py
+F: tests/functional/test_ppc_bamboo.py
 
 e500
 L: [email protected]
@@ -1711,7 +1711,7 @@ F: include/hw/pci-host/sabre.h
 F: hw/pci-bridge/simba.c
 F: include/hw/pci-bridge/simba.h
 F: pc-bios/openbios-sparc64
-F: tests/avocado/machine_sparc64_sun4u.py
+F: tests/functional/test_sparc64_sun4u.py
 
 Sun4v
 M: Artyom Tarasenko <[email protected]>
tests/functional/meson.build +6 -0
index 947f9ac74f..c8fc9f6c07 100644
@@ -22,6 +22,7 @@ tests_generic = [
 ]
 
 tests_arm_thorough = [
+  'arm_canona1100',
   'arm_n8x0',
 ]
 
@@ -43,6 +44,11 @@ tests_ppc_quick = [
 
 tests_ppc_thorough = [
   'ppc_405',
+  'ppc_bamboo',
+]
+
+tests_sparc64_thorough = [
+  'sparc64_sun4u',
 ]
 
 tests_x86_64_quick = [
tests/functional/test_arm_canona1100.py renamed from tests/avocado/machine_arm_canona1100.py +12 -9
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_canona1100.py
rename to tests/functional/test_arm_canona1100.py
index a42d8b0f2b..de6169f2af
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots the canon-a1100 machine with firmware
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -8,9 +10,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
 class CanonA1100Machine(QemuSystemTest):
     """Boots the barebox firmware and checks that the console is operational"""
@@ -18,18 +20,19 @@ class CanonA1100Machine(QemuSystemTest):
     timeout = 90
 
     def test_arm_canona1100(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:canon-a1100
-        :avocado: tags=device:pflash_cfi02
-        """
+        self.set_machine('canon-a1100')
+
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day18.tar.xz')
         tar_hash = '068b5fc4242b29381acee94713509f8a876e9db6'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, dest_dir=self.workdir,
+                        member="day18/barebox.canon-a1100.bin")
         self.vm.set_console()
         self.vm.add_args('-bios',
                          self.workdir + '/day18/barebox.canon-a1100.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'running /env/bin/init')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_bamboo.py renamed from tests/avocado/ppc_bamboo.py +11 -12
old mode 100644
new mode 100755
similarity index 75%
rename from tests/avocado/ppc_bamboo.py
rename to tests/functional/test_ppc_bamboo.py
index a81be3d608..652b31e95a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on the ppc bamboo board and check the console
 #
 # Copyright (c) 2021 Red Hat
@@ -5,30 +7,24 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class BambooMachine(QemuSystemTest):
 
     timeout = 90
 
     def test_ppc_bamboo(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:bamboo
-        :avocado: tags=cpu:440epb
-        :avocado: tags=device:rtl8139
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('bamboo')
         self.require_accelerator("tcg")
         self.require_netdev('user')
         tar_url = ('http://landley.net/aboriginal/downloads/binaries/'
                    'system-image-powerpc-440fp.tar.gz')
         tar_hash = '53e5f16414b195b82d2c70272f81c2eedb39bad9'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir +
                                    '/system-image-powerpc-440fp/linux',
@@ -40,3 +36,6 @@ def test_ppc_bamboo(self):
         exec_command_and_wait_for_pattern(self, 'ping 10.0.2.2',
                                           '10.0.2.2 is alive!')
         exec_command_and_wait_for_pattern(self, 'halt', 'System Halted')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_sparc64_sun4u.py renamed from tests/avocado/machine_sparc64_sun4u.py +14 -11
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/machine_sparc64_sun4u.py
rename to tests/functional/test_sparc64_sun4u.py
index d333c0ae91..cd89d4a1d6
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,27 +12,28 @@
 
 import os
 
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
-from boot_linux_console import LinuxKernelTest
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
-class Sun4uMachine(LinuxKernelTest):
+class Sun4uMachine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
 
     timeout = 90
 
     def test_sparc64_sun4u(self):
-        """
-        :avocado: tags=arch:sparc64
-        :avocado: tags=machine:sun4u
-        """
+        self.set_machine('sun4u')
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day23.tar.xz')
         tar_hash = '142db83cd974ffadc4f75c8a5cad5bcc5722c240'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        kernel_name = 'day23/vmlinux'
+        archive_extract(file_path, self.workdir, kernel_name)
         self.vm.set_console()
-        self.vm.add_args('-kernel', self.workdir + '/day23/vmlinux',
-                         '-append', self.KERNEL_COMMON_COMMAND_LINE)
+        self.vm.add_args('-kernel', os.path.join(self.workdir, kernel_name),
+                         '-append', 'printk.time=0')
         self.vm.launch()
         wait_for_console_pattern(self, 'Starting logging: OK')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v1 8/11
#17 lore raw

Re: [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive

Quoted 5 lines · Thomas Huth

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

Quoted 13 lines
---
 MAINTAINERS                                   |  6 ++---
 tests/functional/meson.build                  |  6 +++++
 .../test_arm_canona1100.py}                   | 21 +++++++++-------
 .../test_ppc_bamboo.py}                       | 23 ++++++++---------
 .../test_sparc64_sun4u.py}                    | 25 +++++++++++--------
 5 files changed, 46 insertions(+), 35 deletions(-)
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw

PATCH 9/11 tests/functional: Set up logging

In reply to Thomas Huth

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/__init__.py | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
tests/functional/qemu_test/__init__.py +23 -1
index 77c3b14d34..cc49fd4c94 100644
@@ -208,7 +208,7 @@ class QemuBaseTest(unittest.TestCase):
     BUILD_DIR = _build_dir()
 
     workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
 
     def setUp(self, bin_prefix):
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -218,6 +218,18 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.workdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)
+
+    def tearDown(self):
+        self.log.removeHandler(self._log_fh)
+
     def check_hash(self, file_name, expected_hash):
         if not expected_hash:
             return True
@@ -269,6 +281,15 @@ def setUp(self):
 
         super().setUp('qemu-system-')
 
+        console_log = logging.getLogger('console')
+        console_log.setLevel(logging.DEBUG)
+        self._console_log_fh = logging.FileHandler(os.path.join(self.workdir,
+                                                   'console.log'), mode='w')
+        self._console_log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
+        self._console_log_fh.setFormatter(fileFormatter)
+        console_log.addHandler(self._console_log_fh)
+
     def set_machine(self, machinename):
         # TODO: We should use QMP to get the list of available machines
         if not self._machinehelp:
@@ -359,4 +380,5 @@ def set_vm_arg(self, arg, value):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
+        logging.getLogger('console').removeHandler(self._console_log_fh)
         super().tearDown()
Signature · 1 line
2.45.2
Reviewed-by on v1 9/11
#19 lore raw

Re: [PATCH 09/11] tests/functional: Set up logging

Quoted 7 lines · Thomas Huth

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/__init__.py | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#20 lore raw

PATCH 10/11 tests/functional: Convert the s390x avocado tests into standalone tests

In reply to Thomas Huth

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
 MAINTAINERS                                   |  4 +-
 tests/functional/meson.build                  |  6 ++
 tests/functional/qemu_test/utils.py           |  7 ++
 .../test_s390x_ccw_virtio.py}                 | 32 ++++-----
 .../test_s390x_topology.py}                   | 70 +++++++------------
 5 files changed, 52 insertions(+), 67 deletions(-)
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index 025227954c..cbefb6fb81 100644
@@ -1739,7 +1739,7 @@ S: Supported
 F: hw/s390x/
 F: include/hw/s390x/
 F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: [email protected]
 
@@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
 F: target/s390x/kvm/stsi-topology.c
 F: docs/devel/s390-cpu-topology.rst
 F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
 
 X86 Machines
 ------------
tests/functional/meson.build +6 -0
index c8fc9f6c07..f6de9af8a2 100644
@@ -13,6 +13,7 @@ endif
 test_timeouts = {
   'netdev_ethtool' : 180,
   'ppc_74xx' : 90,
+  's390x_ccw_virtio' : 180,
 }
 
 tests_generic = [
@@ -47,6 +48,11 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_s390x_thorough = [
+  's390x_ccw_virtio',
+  's390x_topology',
+]
+
 tests_sparc64_thorough = [
   'sparc64_sun4u',
 ]
tests/functional/qemu_test/utils.py +7 -0
index 4eb5e5d5e5..8115d9d1da 100644
@@ -8,6 +8,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import lzma
+import shutil
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):
@@ -19,3 +21,8 @@ def archive_extract(archive, dest_dir, member=None):
             tf.extract(member=member, path=dest_dir)
         else:
             tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+    with lzma.open(xz_path, 'rb') as lzma_in:
+        with open(output_path, 'wb') as raw_out:
+            shutil.copyfileobj(lzma_in, raw_out)
tests/functional/test_s390x_ccw_virtio.py renamed from tests/avocado/machine_s390_ccw_virtio.py +12 -20
old mode 100644
new mode 100755
similarity index 95%
rename from tests/avocado/machine_s390_ccw_virtio.py
rename to tests/functional/test_s390x_ccw_virtio.py
index 26e938c9e9..b2cbaacd21
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots an s390x Linux guest with ccw and PCI devices
 # attached and checks whether the devices are recognized by Linux
 #
@@ -12,11 +14,10 @@
 import os
 import tempfile
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import lzma_uncompress
 
 class S390CCWVirtioMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -41,11 +42,7 @@ def clear_guest_dmesg(self):
         self.dmesg_clear_count += 1
 
     def test_s390x_devices(self):
-
-        """
-        :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390-ccw-virtio
-        """
+        self.set_machine('s390-ccw-virtio')
 
         kernel_url = ('https://snapshot.debian.org/archive/debian/'
                       '20201126T092837Z/dists/buster/main/installer-s390x/'
@@ -160,15 +157,7 @@ def test_s390x_devices(self):
 
 
     def test_s390x_fedora(self):
-
-        """
-        :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390-ccw-virtio
-        :avocado: tags=device:virtio-gpu
-        :avocado: tags=device:virtio-crypto
-        :avocado: tags=device:virtio-net
-        :avocado: tags=flaky
-        """
+        self.set_machine('s390-ccw-virtio')
 
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
                       '/fedora-secondary/releases/31/Server/s390x/os'
@@ -182,7 +171,7 @@ def test_s390x_fedora(self):
         initrd_hash = '3de45d411df5624b8d8ef21cd0b44419ab59b12f'
         initrd_path_xz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
         initrd_path = os.path.join(self.workdir, 'initrd-raw.img')
-        archive.lzma_uncompress(initrd_path_xz, initrd_path)
+        lzma_uncompress(initrd_path_xz, initrd_path)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE + ' audit=0 '
@@ -275,3 +264,6 @@ def test_s390x_fedora(self):
         exec_command_and_wait_for_pattern(self,
                         'while ! (dmesg -c | grep Start.virtcrypto_remove) ; do'
                         ' sleep 1 ; done', 'Start virtcrypto_remove.')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_s390x_topology.py renamed from tests/avocado/s390_topology.py +25 -45

165 lines not on this page · raw message

Signature · 1 line
2.45.2
#21 lore raw

Re: [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 81 lines · Thomas Huth

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
 MAINTAINERS                                   |  4 +-
 tests/functional/meson.build                  |  6 ++
 tests/functional/qemu_test/utils.py           |  7 ++
 .../test_s390x_ccw_virtio.py}                 | 32 ++++-----
 .../test_s390x_topology.py}                   | 70 +++++++------------
 5 files changed, 52 insertions(+), 67 deletions(-)
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index 025227954c..cbefb6fb81 100644
@@ -1739,7 +1739,7 @@ S: Supported
 F: hw/s390x/
 F: include/hw/s390x/
 F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: [email protected]
 
@@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
 F: target/s390x/kvm/stsi-topology.c
 F: docs/devel/s390-cpu-topology.rst
 F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
 
 X86 Machines
 ------------
tests/functional/meson.build +6 -0
index c8fc9f6c07..f6de9af8a2 100644
@@ -13,6 +13,7 @@ endif
 test_timeouts = {
   'netdev_ethtool' : 180,
   'ppc_74xx' : 90,
+  's390x_ccw_virtio' : 180,
 }
 
 tests_generic = [
@@ -47,6 +48,11 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_s390x_thorough = [
+  's390x_ccw_virtio',
+  's390x_topology',
+]
+
 tests_sparc64_thorough = [
   'sparc64_sun4u',
 ]
tests/functional/qemu_test/utils.py +2 -0
index 4eb5e5d5e5..8115d9d1da 100644
@@ -8,6 +8,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import lzma
+import shutil
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):

tests/functional/qemu_test/utils.py def archive_extract(archive, dest_dir, member=None): · line 28

Quoted 8 lines
tests/functional/qemu_test/utils.py +5 -0
@@ -19,3 +21,8 @@ def archive_extract(archive, dest_dir, member=None):
             tf.extract(member=member, path=dest_dir)
         else:
             tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+    with lzma.open(xz_path, 'rb') as lzma_in:
+        with open(output_path, 'wb') as raw_out:
+            shutil.copyfileobj(lzma_in, raw_out)

Avocado short-circuited if output_path already existed for speed. Worth doing the same.

The inner 'with' should be surrounded by a try/except that does os.remove(output_path) on error. Avocado didn't have this safety net either, but we should add it.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#22 lore raw

PATCH 11/11 gitlab-ci: Add "check-functional" to the build tests

In reply to Thomas Huth

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

---
 .gitlab-ci.d/buildtest-template.yml |  3 +-
 .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
 2 files changed, 32 insertions(+), 31 deletions(-)
.gitlab-ci.d/buildtest-template.yml +2 -1
index 8f7ebfaed8..54cae74a8e 100644
@@ -71,7 +71,7 @@
     reports:
       junit: build/meson-logs/testlog.junit.xml
 
-.avocado_test_job_template:
+.functional_test_job_template:
   extends: .common_test_job_template
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -98,6 +98,7 @@
         du -chs ${CI_PROJECT_DIR}/avocado-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+    - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
.gitlab-ci.d/buildtest.yml +30 -30

171 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by on v1 11/11
#23 lore raw

Re: [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests

Quoted 3 lines · Thomas Huth

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

I was thinking of suggesting we make 'check-functional' run both avocado and new tests, but that's pointless extra work if we fully convert all the avocado tests to the new approach in a short time frame.

Quoted 6 lines
---
 .gitlab-ci.d/buildtest-template.yml |  3 +-
 .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
 2 files changed, 32 insertions(+), 31 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Tested-by on the v1 series
#24 lore raw
In reply to Thomas Huth
Quoted 47 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

Quoted 4 lines

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export PYTHONPATH=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

For the whole series as is

as it does what you claim it does here when I tried it.

Quoted 3 lines

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

As an example, one dir name appears to be:

__main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too

IOW, could we make this dir name be:

test_mem_addr_space.phybits_ok_pentium_pae

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#25 lore raw
Quoted 1 line · Daniel P. Berrangé

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 7 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

...

Quoted 20 lines
Quoted 15 lines

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

Quoted 4 lines

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export PYTHONPATH=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

For the whole series as is

as it does what you claim it does here when I tried it.

Thanks!

Quoted 14 lines
Quoted 3 lines

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

As an example, one dir name appears to be:

__main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too

Quoted 3 lines

IOW, could we make this dir name be:

test_mem_addr_space.phybits_ok_pentium_pae

I can try to change that, indeed ... but the boilerplate code will increase a little bit, I guess, since I cannot simply rely on the unittest.id() function in that case anymore...

Thomas

#26 lore raw
Quoted 53 lines · Thomas Huth
Quoted 1 line · Daniel P. Berrangé

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 7 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

...

Quoted 20 lines
Quoted 15 lines

Now if you want to try out these patches: Apply the patches, then recompile and then run:

  make check-functional

You can also run single targets e.g. with:

  make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

Quoted 4 lines

  export PYTHONPATH=$HOME/qemu/python
  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
  export PYTHONPATH=$HOME/qemu/build
  ~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

For the whole series as is

as it does what you claim it does here when I tried it.

Thanks!

Quoted 14 lines
Quoted 3 lines

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

As an example, one dir name appears to be:

   __main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too

Quoted 5 lines
Quoted 3 lines

IOW, could we make this dir name be:

   test_mem_addr_space.phybits_ok_pentium_pae

I can try to change that, indeed ... but the boilerplate code will increase a little bit, I guess, since I cannot simply rely on the unittest.id() function in that case anymore...

After looking at this for a while, I think it's maybe best to ditch the idea of making the .py files directly runnable and run the tests via a simple pycotap runner instead. Then you get proper module names:

$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..2

Thomas

#27 lore raw
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Thomas Huth
Quoted 1 line · Daniel P. Berrangé

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 7 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

...

Quoted 20 lines
Quoted 15 lines

Now if you want to try out these patches: Apply the patches, then recompile and then run:

  make check-functional

You can also run single targets e.g. with:

  make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

Quoted 4 lines

  export PYTHONPATH=$HOME/qemu/python
  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
  export PYTHONPATH=$HOME/qemu/build
  ~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

For the whole series as is

as it does what you claim it does here when I tried it.

Thanks!

Quoted 17 lines
Quoted 3 lines

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

As an example, one dir name appears to be:

   __main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too

IOW, could we make this dir name be:

   test_mem_addr_space.phybits_ok_pentium_pae

Quoted 7 lines
Quoted 3 lines

I can try to change that, indeed ... but the boilerplate code will increase a little bit, I guess, since I cannot simply rely on the unittest.id() function in that case anymore...

After looking at this for a while, I think it's maybe best to ditch the idea of making the .py files directly runnable and run the tests via a simple pycotap runner instead. Then you get proper module names:

I'd really not want to loose that. To me, eliminating the test harness entirely when debugging is the single biggest improvement of this new approach, especially when I want to 'strace' the test without extraneous processes.

Quoted 5 lines
$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..

With the following change, you get the same output with direct execution, by making argv look the same as you'd get when running your pycotap example.

tests/functional/qemu_test/__init__.py +4 -1
index cc49fd4c94..3a3e65252d 100644
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        unittest.main(testRunner = tr)
+        import sys
+        import os.path
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
 
 
 class QemuSystemTest(QemuBaseTest):

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#28 lore raw
Quoted 87 lines · Daniel P. Berrangé
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Thomas Huth
Quoted 1 line · Daniel P. Berrangé

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 7 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

...

Quoted 20 lines
Quoted 15 lines

Now if you want to try out these patches: Apply the patches, then recompile and then run:

  make check-functional

You can also run single targets e.g. with:

  make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

Quoted 4 lines

  export PYTHONPATH=$HOME/qemu/python export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 export PYTHONPATH=$HOME/qemu/build ~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

For the whole series as is

as it does what you claim it does here when I tried it.

Thanks!

Quoted 17 lines
Quoted 3 lines

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

As an example, one dir name appears to be:

   __main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too

IOW, could we make this dir name be:

   test_mem_addr_space.phybits_ok_pentium_pae

Quoted 7 lines
Quoted 3 lines

I can try to change that, indeed ... but the boilerplate code will increase a little bit, I guess, since I cannot simply rely on the unittest.id() function in that case anymore...

After looking at this for a while, I think it's maybe best to ditch the idea of making the .py files directly runnable and run the tests via a simple pycotap runner instead. Then you get proper module names:

I'd really not want to loose that. To me, eliminating the test harness entirely when debugging is the single biggest improvement of this new approach, especially when I want to 'strace' the test without extraneous processes.

Quoted 5 lines
$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..

With the following change, you get the same output with direct execution, by making argv look the same as you'd get when running your pycotap example.

tests/functional/qemu_test/__init__.py +0 -0
index cc49fd4c94..3a3e65252d 100644

tests/functional/qemu_test/__init__.py def fetch_asset(self, url, asset_hash): · line 272

Quoted 8 lines
tests/functional/qemu_test/__init__.py +4 -1
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
      def main():
          tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                     test_output_log = pycotap.LogMode.LogToError)
-        unittest.main(testRunner = tr)
+        import sys
+        import os.path
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])

Sweet, thank you very much, looks like this will do the job!

Thomas

#29 lore raw
In reply to Thomas Huth

Il mar 16 lug 2024, 13:26 Thomas Huth <[email protected]> ha scritto:

Quoted 9 lines

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

I think we should take another look at that. Avocado 92 should work, though I am not sure if it's also depending on "imp", and if I recall correctly the problem with more recent version was more that it conflicted with distros that didn't have it packaged. That should also not be a problem anymore with the pythondeps.toml mechanism.

Additionally, the whole "make check" test suite in QEMU is using the

Quoted 5 lines

meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

This is true.

Paolo

Quoted 166 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export PYTHONPATH=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Thomas Huth (11):
tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Add python-based tests to the meson build system tests/functional: Implement fetch_asset() method for downloading
assets
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Set up logging
tests/functional: Convert the s390x avocado tests into standalone
tests
gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  22 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  18 +-
 tests/functional/meson.build                  | 112 +++++
 tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  28 ++
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_mem_addr_space.py}                   |  53 +--
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_version.py}                   |   8 +-
 tests/meson.build                             |   1 +
 28 files changed, 795 insertions(+), 320 deletions(-)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_arm_canona1100.py =>
functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py}
(71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py}
(90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py}
(89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py =>
functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py}
(87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py =>
functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py =>
functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py =>
functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py =>
functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py =>
functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py}
(75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py =>
functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py =>
functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py =>
functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py =>
functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755

--
2.45.2

#30 lore raw
Quoted 9 lines · Paolo Bonzini

Il mar 16 lug 2024, 13:26 Thomas Huth <[email protected] <mailto:[email protected]>> ha scritto:

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

Quoted 6 lines

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

I think we should take another look at that. Avocado 92 should work, though I am not sure if it's also depending on "imp"

FWIW, the "imp" problem had been fixed here:

https://github.com/avocado-framework/avocado/commit/dfc340f1a0b0aeddbf6699e49e94b75ed080137c

That had just been released with Avocado version 99, so updating to 92 does not really help.

Thomas

#31 lore raw
In reply to Thomas Huth
Quoted 6 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

Hi Thomas,

Let me start by saying that this is my attempt to explain the context of the lack of updates on QEMU when it comes to the Avocado update, and point to the actual updates.

Quoted 4 lines

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky. Then 103.0 LTS was released, and there was extensive work to test the QEMU tests before that release was cut. Additionally, there was further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2]. I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2]. A side node is that, because 103.0 is an LTS release, it will receive the needed bug fixes and updates that we deem necessary, including things we find in QEMU tests. In fact, 103.1[3] is in the works.

Quoted 6 lines

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson.

Quoted 8 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Quoted 8 lines

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself. This means way more libraries and features that go into a common set of utilities and features (more on that later) than it currently exists in avocado.utils.

This brings the autils[4] initiative into the picture. We're about 80% done with the project structure, and after that, it will be a common utility project (such as the cloudinit and ssh) which can be released automatically when the maintainer votes (through GitHub) that a new release is needed. With that, we can have all the existing QEMU tests (and more) running in either meson or Avocado (for users that want Avocado runner only features, such as dependency resolution, container support, etc).

Now, if you think any of this makes sense, I'd love to cooperate with you on an update that does both: update Avocado, and allow for the execution of all tests through meson.

Thanks and I'm very very sorry for the delay in this update.
- Cleber.

[1] https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/avocado_test_updates_lts?ref_type=heads
[2] https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478
[3] https://github.com/avocado-framework/avocado/pull/5953
[4] https://github.com/avocado-framework/autils

#32 lore raw
Quoted 1 line · Cleber Rosa

On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <[email protected]> wrote:

...

Quoted 4 lines
Quoted 3 lines

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Quoted 13 lines

So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky. Then 103.0 LTS was released, and there was extensive work to test the QEMU tests before that release was cut. Additionally, there was further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2]. I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2]. A side node is that, because 103.0 is an LTS release, it will receive the needed bug fixes and updates that we deem necessary, including things we find in QEMU tests. In fact, 103.1[3] is in the works.

Hi Cleber,

thanks for the explanation, but we really need to replace v88 rather *now* since v88 does not work with the latest versions of Python anymore (there is a work-around on Fedora fortunately, but it's completely broken on Ubuntu 24.04 as far as I know). So even a single-threaded execution with v103 would have been better than waiting forever for your update to land. The problem with v88 being broken has been raised a couple of times already, but it's incredibly hard to get a response from you Avocado folks, so with hardly any help from the Avocado side, and nobody on the QEMU side being really familiar with the Avocado stuff, and with the meson test runner being used by all other subsystems in QEMU already, I think it's best if we continue with this series here.

Quoted 7 lines
Quoted 6 lines

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

Quoted 7 lines

Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson.

Ok, good to know, we could maybe use that eventually for the tests that really require the Avocado framework (i.e. the cloud-init based tests).

For the others, as Daniel said in an earlier mail, it's much more convenient if you can also run the tests directly instead, without such a layer in between - that makes debugging way easier.

Quoted 8 lines
Quoted 7 lines

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Quoted 4 lines

So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself.

I didn't really see such a movement in recent times yet ... Could you point to an example?

Quoted 9 lines

This means way more libraries and features that go into a
common set of utilities and features (more on that later) than it currently exists in avocado.utils.

This brings the autils[4] initiative into the picture. We're about 80% done with the project structure, and after that, it will be a common utility project (such as the cloudinit and ssh) which can be released automatically when the maintainer votes (through GitHub) that a new release is needed.

autils sounds promising, but I just had a look at the repository, and there does not seem to be that much code available there yet, so I guess it will take still quite a long time 'til that's ready?

Thomas

#33 lore raw
Quoted 24 lines · Thomas Huth
Quoted 1 line · Cleber Rosa

On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <[email protected]> wrote:

...

Quoted 4 lines
Quoted 3 lines

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Quoted 13 lines

So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky. Then 103.0 LTS was released, and there was extensive work to test the QEMU tests before that release was cut. Additionally, there was further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2]. I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2]. A side node is that, because 103.0 is an LTS release, it will receive the needed bug fixes and updates that we deem necessary, including things we find in QEMU tests. In fact, 103.1[3] is in the works.

Hi Cleber,

Quoted 12 lines

thanks for the explanation, but we really need to replace v88 rather *now* since v88 does not work with the latest versions of Python anymore (there is a work-around on Fedora fortunately, but it's completely broken on Ubuntu 24.04 as far as I know). So even a single-threaded execution with v103 would have been better than waiting forever for your update to land. The problem with v88 being broken has been raised a couple of times already, but it's incredibly hard to get a response from you Avocado folks, so with hardly any help from the Avocado side, and nobody on the QEMU side being really familiar with the Avocado stuff, and with the meson test runner being used by all other subsystems in QEMU already, I think it's best if we continue with this series here.

Hi Thomas,

I agree with the urgency. I've posted the updates here:

https://lists.gnu.org/archive/html/qemu-devel/2024-07/msg06236.html

Sorry for the delay, and I hope this gives more time to make the evolution towards a solution that better suits QEMU.

Regards,
- Cleber.

Quoted 30 lines
Quoted 7 lines
Quoted 6 lines

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

Quoted 7 lines

Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson.

Ok, good to know, we could maybe use that eventually for the tests that really require the Avocado framework (i.e. the cloud-init based tests).

For the others, as Daniel said in an earlier mail, it's much more convenient if you can also run the tests directly instead, without such a layer in between - that makes debugging way easier.

Quoted 8 lines
Quoted 7 lines

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Quoted 8 lines
Quoted 4 lines

So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself.

I didn't really see such a movement in recent times yet ... Could you point to an example?

Indeed, there is no public evidence of that, yet.

Quoted 10 lines
Quoted 9 lines

This means way more libraries and features that go into a
common set of utilities and features (more on that later) than it currently exists in avocado.utils.

This brings the autils[4] initiative into the picture. We're about 80% done with the project structure, and after that, it will be a common utility project (such as the cloudinit and ssh) which can be released automatically when the maintainer votes (through GitHub) that a new release is needed.

Quoted 4 lines

autils sounds promising, but I just had a look at the repository, and there does not seem to be that much code available there yet, so I guess it will take still quite a long time 'til that's ready?

Like I said previously, we were working on the project infrastructure. We haven't ported much code over there yet. There are very few things missing, and all of them have been assigned to be worked on. It's been tracked here:

https://github.com/avocado-framework/autils/issues/1

My estimation is that 2-4 weeks for the project infrastructure to be complete.

Regards,
- Cleber.

Quoted 2 lines

Thomas

#34 lore raw
In reply to Cleber Rosa
Quoted 11 lines · Cleber Rosa
Quoted 4 lines · Thomas Huth

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky.

IMHO the problems with updating avocado in QEMU are a sign that we're suffering from excess complexity, as it shouldn't be this difficult to update the test harness. That we need to stick on an LTS release to protect us from further instability further suggests that we would benefit from a much simpler solution here.

Quoted 7 lines
Quoted 6 lines

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

Quoted 7 lines

Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson.

Looking back to when we introduced avocado, we still had our original home grown build system, and multiple different ways to run integrate testing betwen our unit tests / qtests and block iotests. Adding avocado to the mix introduced a third way to run tests. All round it was rather a mess of inconsistent approaches to testing, but that was a fact of life with our old home grown build/test process.

Today we have adopted a much more standardized build system via meson. While we do still have the block iotests harness, we now expose every test to meson individually to reduce what the intermediate harness needs to be involved in doing. Thomas' series here is bringing the same simplified integration to our functional tests, such that each functional test is known to meson individually. Overall we'll now have a consistent approach to integrating tests into meson, without intermediate harnesses adding their own logic.

I think this will make it easier for QEMU maintainers to understand how the tests are working, and make it easier to maintain them and troubleshoot failures.

So overall I'm not seeing a compelling reason for us to offer users a choice between meson & avocado how they run tests. Such choice is a maint burden over the long term, as we need to validate that both options keep working. Focusing on a single option reduces the complexity and maint work and gives us more flexiblity to evolve the code over time.

Quoted 17 lines
Quoted 8 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Quoted 8 lines

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Quoted 6 lines

So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself. This means way more libraries and features that go into a common set of utilities and features (more on that later) than it currently exists in avocado.utils.

The series here is showing how simple it is to adapt our existing tests to the new execution approach, so shouldn't have a significant impact on ability of people to contribute futher tests in future.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#35 lore raw
Quoted 14 lines · Daniel P. Berrangé
Quoted 11 lines · Cleber Rosa
Quoted 4 lines · Thomas Huth

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky.

Quoted 6 lines

IMHO the problems with updating avocado in QEMU are a sign that we're suffering from excess complexity, as it shouldn't be this difficult to update the test harness. That we need to stick on an LTS release to protect us from further instability further suggests that we would benefit from a much simpler solution here.

Hi Daniel,

Some other projects using Avocado have picked different, non-LTS versions, to use as their pinned versions. The consequences of using rolling releases versus LTS-like releases is not something specific to one software package IMO. The suggestion to use LTS was to minimize the impact on users writing and running tests, while allowing for only needed fixes or very important features (say supporting a new Python version) to be addressed.

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

Quoted 7 lines

Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson.

Quoted 7 lines

Looking back to when we introduced avocado, we still had our original home grown build system, and multiple different ways to run integrate testing betwen our unit tests / qtests and block iotests. Adding avocado to the mix introduced a third way to run tests. All round it was rather a mess of inconsistent approaches to testing, but that was a fact of life with our old home grown build/test process.

Agreed.

Quoted 9 lines

Today we have adopted a much more standardized build system via meson. While we do still have the block iotests harness, we now expose every test to meson individually to reduce what the intermediate harness needs to be involved in doing. Thomas' series here is bringing the same simplified integration to our functional tests, such that each functional test is known to meson individually. Overall we'll now have a consistent approach to integrating tests into meson, without intermediate harnesses adding their own logic.

Quoted 4 lines

I think this will make it easier for QEMU maintainers to understand how the tests are working, and make it easier to maintain them and troubleshoot failures.

I think this is a very good way forward. I don't think anything (simple tests here) should depend on what they don't need to depend on.

Quoted 7 lines

So overall I'm not seeing a compelling reason for us to offer users a choice between meson & avocado how they run tests. Such choice is a maint burden over the long term, as we need to validate that both options keep working. Focusing on a single option reduces the complexity and maint work and gives us more flexiblity to evolve the code over time.

If meson is capable of doing all that can be done right now, then I believe you're right. No reason to use duplicate software. Now, If you mean that having "single option" may imply duplicating code that already exists and is maintained elsewhere, into the QEMU repo, then you're wrong. This can be easily judged by how easy it is to convert all current tests and their features (timeout, download management, etc).

Another level of judgement that I believe would be helpful to the decision making process here is how this impacts growing the test coverage. Tests that manipulate both QEMU and the guest are still very very simple compared to what can be found on what I'll call "real life integration testing" of QEMU (read Avocado-VT like tests). But they present a glimpse of the problem. Having such tests requires a good amount of code to either be used from a project/repo that addresses that specifically, or be duplicated inside QEMU.

Quoted 24 lines
Quoted 17 lines
Quoted 8 lines

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Quoted 8 lines

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Quoted 6 lines

So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself. This means way more libraries and features that go into a common set of utilities and features (more on that later) than it currently exists in avocado.utils.

Quoted 4 lines

The series here is showing how simple it is to adapt our existing tests to the new execution approach, so shouldn't have a significant impact on ability of people to contribute futher tests in future.

I am on the side of the unified execution approach. But, will it really be simple to convert all tests? "Real life integration" tests too? I would love to see that, but I can't see how that can be done without the woes mentioned before (duplicating other code or limiting the further toolset and capabilities of test writers).

Best regards,
- Cleber.