Skip to contentSkip to conversation

Convert avocado tests to normal Python unittests

PATCH Superseded Thomas Huth 6 participants 49 messages Jul 24, 2024 – Jul 29, 2024

Subsystems (3)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover Convert avocado tests to normal Python unittests

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

v2:
- Addressed review feedback from v1
- Add pycotap as a wheel instead of trying to install it on demand when running "make check-functional" (works much better now!)
- Converted much more tests
- Lots of other small improvements here and there

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Thomas Huth (23):
python: Install pycotap in our venv if necessary tests/functional: Add base classes for the upcoming pytest-based tests tests/Makefile.include: Increase the level of indentation in the help
text
tests/functional: Prepare the meson build system for the functional
tests
tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Implement fetch_asset() method for downloading
assets
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Set up logging
tests/functional: Convert the s390x avocado tests into standalone
tests
tests/functional: Convert the x86_cpu_model_versions test tests/functional: Convert the microblaze avocado tests into standalone
tests
tests/functional: Convert the riscv_opensbi avocado test into a
standalone test
tests/functional: Convert the virtio_gpu avocado test into a
standalone test
tests/functional: Convert most ppc avocado tests into standalone tests tests/functional: Convert the ppc_amiga avocado test into a standalone
test
tests/functional: Convert the ppc_hv avocado test into a standalone
test
tests/functional: Convert the m68k nextcube test with tesseract tests/functional: Convert the acpi-bits test into a standalone test tests/functional: Convert the rx_gdbsim avocado test into a standalone
test
gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  32 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 python/wheels/pycotap-1.3.1-py3-none-any.whl  | Bin 0 -> 5119 bytes
 pythondeps.toml                               |   1 +
 tests/Makefile.include                        |  41 +-
 tests/avocado/machine_microblaze.py           |  61 ---
 tests/avocado/riscv_opensbi.py                |  63 ---
 tests/avocado/tesseract_utils.py              |  46 --
 .../acpi-bits/bits-config/bits-cfg.txt        |   0
 .../acpi-bits/bits-tests/smbios.py2           |   0
 .../acpi-bits/bits-tests/smilatency.py2       |   0
 .../acpi-bits/bits-tests/testacpi.py2         |   0
 .../acpi-bits/bits-tests/testcpuid.py2        |   0
 tests/functional/meson.build                  | 150 +++++++
 tests/functional/qemu_test/__init__.py        | 393 ++++++++++++++++++
 tests/functional/qemu_test/tesseract.py       |  35 ++
 tests/functional/qemu_test/utils.py           |  47 +++
 .../test_acpi_bits.py}                        |  29 +-
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_m68k_nextcube.py}                    |  20 +-
 .../test_mem_addr_space.py}                   |  52 +--
 .../functional/test_microblaze_s3adsp1800.py  |  38 ++
 .../test_microblazeel_s3adsp1800.py           |  41 ++
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../test_ppc64_hv.py}                         |  36 +-
 .../test_ppc64_powernv.py}                    |  45 +-
 .../test_ppc64_pseries.py}                    |  45 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_40p.py}                          |  37 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_amiga.py}                        |  33 +-
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_ppc_mpc8544ds.py}                    |  19 +-
 .../test_ppc_virtex_ml507.py}                 |  19 +-
 tests/functional/test_riscv_opensbi.py        |  36 ++
 .../test_rx_gdbsim.py}                        |  34 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_gpu.py}                       |  34 +-
 .../test_virtio_version.py}                   |   8 +-
 .../test_x86_cpu_model_versions.py}           |  63 +--
 tests/meson.build                             |   1 +
 53 files changed, 1187 insertions(+), 758 deletions(-)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
 delete mode 100644 tests/avocado/machine_microblaze.py
 delete mode 100644 tests/avocado/riscv_opensbi.py
 delete mode 100644 tests/avocado/tesseract_utils.py
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (86%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (80%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (83%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_amiga.py => functional/test_ppc_amiga.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (78%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_riscv_opensbi.py
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (88%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
Signature · 1 line
2.45.2
#1 lore raw

PATCH v2 1/23 python: Install pycotap in our venv if necessary

The upcoming functional tests will require pycotap for providing TAP output from the python-based tests. Since we want to be able to run some of the tests offline by default, too, let's install it along with meson in our venv if necessary (it's size is only 5 kB, so adding the wheel here should not really be a problem).

The wheel file has been obtained with:

pip download --only-binary :all: --dest . --no-cache pycotap

---
 python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes
 pythondeps.toml                              |   1 +
 2 files changed, 1 insertion(+)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
python/wheels/pycotap-1.3.1-py3-none-any.whl new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9c2c7d259365c8a0487228c6878e2c0b10ba6bdd
GIT binary patch
pythondeps.toml +1 -0
index f6e590fdd8..0bc4144d4b 100644
@@ -20,6 +20,7 @@
 [meson]
 # The install key should match the version in python/wheels/
 meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
+pycotap = { accepted = ">=1.1.0", installed = "1.3.1" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
Signature · 1 line
2.45.2
Tested-by on v2 1/23
#2 lore raw

Re: [PATCH v2 01/23] python: Install pycotap in our venv if necessary

Quoted 8 lines · Thomas Huth

The upcoming functional tests will require pycotap for providing TAP output from the python-based tests. Since we want to be able to run some of the tests offline by default, too, let's install it along with meson in our venv if necessary (it's size is only 5 kB, so adding the wheel here should not really be a problem).

The wheel file has been obtained with:

Quoted 8 lines
  pip download --only-binary :all: --dest . --no-cache pycotap
---
  python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes
  pythondeps.toml                              |   1 +
  2 files changed, 1 insertion(+)

create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl

#3 lore raw

PATCH v2 2/23 tests/functional: Add base classes for the upcoming pytest-based tests

In reply to Thomas Huth

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

---
 tests/functional/qemu_test/__init__.py | 328 +++++++++++++++++++++++++
 1 file changed, 328 insertions(+)
 create mode 100644 tests/functional/qemu_test/__init__.py
tests/functional/qemu_test/__init__.py new file +328 -0
new file mode 100644
index 0000000000..fc98222c52
@@ -0,0 +1,328 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import uuid
+import unittest
+
+from pathlib import Path
+from qemu.machine import QEMUMachine
+from qemu.utils import kvm_available, tcg_available
+
+def _source_dir():
+    # Determine top-level directory of the QEMU sources
+    return Path(__file__).parent.parent.parent.parent
+
+def _build_dir():
+    root = os.getenv('QEMU_BUILD_ROOT')
+    if root is not None:
+        return Path(root)
+    # Makefile.mtest only exists in build dir, so if it is available, use CWD
+    if os.path.exists('Makefile.mtest'):
+        return Path(os.getcwd())
+
+    root = os.path.join(_source_dir(), 'build')
+    if os.path.exists(root):
+        return Path(root)
+
+    raise Exception("Cannot identify build dir, set QEMU_BUILD_ROOT")
+
+BUILD_DIR = _build_dir()
+
+def has_cmd(name, args=None):
+    """
+    This function is for use in a @skipUnless decorator, e.g.:
+
+        @skipUnless(*has_cmd('sudo -n', ('sudo', '-n', 'true')))
+        def test_something_that_needs_sudo(self):
+            ...
+    """
+
+    if args is None:
+        args = ('which', name)
+
+    try:
+        _, stderr, exitcode = run_cmd(args)
+    except Exception as e:
+        exitcode = -1
+        stderr = str(e)
+
+    if exitcode != 0:
+        cmd_line = ' '.join(args)
+        err = f'{name} required, but "{cmd_line}" failed: {stderr.strip()}'
+        return (False, err)
+    else:
+        return (True, '')
+
+def has_cmds(*cmds):
+    """
+    This function is for use in a @skipUnless decorator and
+    allows checking for the availability of multiple commands, e.g.:
+
+        @skipUnless(*has_cmds(('cmd1', ('cmd1', '--some-parameter')),
+                              'cmd2', 'cmd3'))
+        def test_something_that_needs_cmd1_and_cmd2(self):
+            ...
+    """
+
+    for cmd in cmds:
+        if isinstance(cmd, str):
+            cmd = (cmd,)
+
+        ok, errstr = has_cmd(*cmd)
+        if not ok:
+            return (False, errstr)
+
+    return (True, '')
+
+def run_cmd(args):
+    subp = subprocess.Popen(args,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            universal_newlines=True)
+    stdout, stderr = subp.communicate()
+    ret = subp.returncode
+
+    return (stdout, stderr, ret)
+
+def is_readable_executable_file(path):
+    return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
+
+def _console_interaction(test, success_message, failure_message,
+                         send_string, keep_sending=False, vm=None):
+    assert not keep_sending or send_string
+    if vm is None:
+        vm = test.vm
+    console = vm.console_file
+    console_logger = logging.getLogger('console')
+    while True:
+        if send_string:
+            vm.console_socket.sendall(send_string.encode())
+            if not keep_sending:
+                send_string = None # send only once
+        try:
+            msg = console.readline().decode().strip()
+        except UnicodeDecodeError:
+            msg = None
+        if not msg:
+            continue
+        console_logger.debug(msg)
+        if success_message is None or success_message in msg:
+            break
+        if failure_message and failure_message in msg:
+            console.close()
+            fail = 'Failure message found in console: "%s". Expected: "%s"' % \
+                    (failure_message, success_message)
+            test.fail(fail)
+
+def interrupt_interactive_console_until_pattern(test, success_message,
+                                                failure_message=None,
+                                                interrupt_string='\r'):
+    """
+    Keep sending a string to interrupt a console prompt, while logging the
+    console output. Typical use case is to break a boot loader prompt, such:
+
+        Press a key within 5 seconds to interrupt boot process.
+        5
+        4
+        3
+        2
+        1
+        Booting default image...
+
+    :param test: a  test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    :param interrupt_string: a string to send to the console before trying
+                             to read a new line
+    """
+    _console_interaction(test, success_message, failure_message,
+                         interrupt_string, True)
+
+def wait_for_console_pattern(test, success_message, failure_message=None,
+                             vm=None):
+    """
+    Waits for messages to appear on the console, while logging the content
+
+    :param test: a test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    """
+    _console_interaction(test, success_message, failure_message, None, vm=vm)
+
+def exec_command(test, command):
+    """
+    Send a command to a console (appending CRLF characters), while logging
+    the content.
+
+    :param test: a test containing a VM.
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param command: the command to send
+    :type command: str
+    """
+    _console_interaction(test, None, None, command + '\r')
+
+def exec_command_and_wait_for_pattern(test, command,
+                                      success_message, failure_message=None):
+    """
+    Send a command to a console (appending CRLF characters), then wait
+    for success_message to appear on the console, while logging the.
+    content. Mark the test as failed if failure_message is found instead.
+
+    :param test: a test containing a VM that will have its console
+                 read and probed for a success or failure message
+    :type test: :class:`qemu_test.QemuSystemTest`
+    :param command: the command to send
+    :param success_message: if this message appears, test succeeds
+    :param failure_message: if this message appears, test fails
+    """
+    _console_interaction(test, success_message, failure_message, command + '\r')
+
+class QemuBaseTest(unittest.TestCase):
+
+    qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+    arch = None
+
+    workdir = None
+    log = logging.getLogger('qemu-test')
+
+    def setUp(self, bin_prefix):
+        self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+        self.arch = self.qemu_bin.split('-')[-1]
+
+        self.workdir = os.path.join(BUILD_DIR, 'tests/functional', self.arch,
+                                    self.id())
+        if not os.path.exists(self.workdir):
+            os.makedirs(self.workdir)
+
+    def main():
+        tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
+                                   test_output_log = pycotap.LogMode.LogToError)
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
+
+
+class QemuSystemTest(QemuBaseTest):
+    """Facilitates system emulation tests."""
+
+    cpu = None
+    machine = None
+    _machinehelp = None
+
+    def setUp(self):
+        self._vms = {}
+
+        super().setUp('qemu-system-')
+
+    def set_machine(self, machinename):
+        # TODO: We should use QMP to get the list of available machines
+        if not self._machinehelp:
+            self._machinehelp = run_cmd([self.qemu_bin, '-M', 'help'])[0];
+        if self._machinehelp.find(machinename) < 0:
+            self.skipTest('no support for machine ' + machinename)
+        self.machine = machinename
+
+    def require_accelerator(self, accelerator):
+        """
+        Requires an accelerator to be available for the test to continue
+
+        It takes into account the currently set qemu binary.
+
+        If the check fails, the test is canceled.  If the check itself
+        for the given accelerator is not available, the test is also
+        canceled.
+
+        :param accelerator: name of the accelerator, such as "kvm" or "tcg"
+        :type accelerator: str
+        """
+        checker = {'tcg': tcg_available,
+                   'kvm': kvm_available}.get(accelerator)
+        if checker is None:
+            self.skipTest("Don't know how to check for the presence "
+                          "of accelerator %s" % accelerator)
+        if not checker(qemu_bin=self.qemu_bin):
+            self.skipTest("%s accelerator does not seem to be "
+                          "available" % accelerator)
+
+    def require_netdev(self, netdevname):
+        netdevhelp = run_cmd([self.qemu_bin,
+                             '-M', 'none', '-netdev', 'help'])[0];
+        if netdevhelp.find('\n' + netdevname + '\n') < 0:
+            self.skipTest('no support for " + netdevname + " networking')
+
+    def require_device(self, devicename):
+        devhelp = run_cmd([self.qemu_bin,
+                           '-M', 'none', '-device', 'help'])[0];
+        if devhelp.find(devicename) < 0:
+            self.skipTest('no support for device ' + devicename)
+
+    def _new_vm(self, name, *args):
+        vm = QEMUMachine(self.qemu_bin, base_temp_dir=self.workdir)
+        self.log.debug('QEMUMachine "%s" created', name)
+        self.log.debug('QEMUMachine "%s" temp_dir: %s', name, vm.temp_dir)
+        self.log.debug('QEMUMachine "%s" log_dir: %s', name, vm.log_dir)
+        if args:
+            vm.add_args(*args)
+        return vm
+
+    @property
+    def vm(self):
+        return self.get_vm(name='default')
+
+    def get_vm(self, *args, name=None):
+        if not name:
+            name = str(uuid.uuid4())
+        if self._vms.get(name) is None:
+            self._vms[name] = self._new_vm(name, *args)
+            if self.cpu is not None:
+                self._vms[name].add_args('-cpu', self.cpu)
+            if self.machine is not None:
+                self._vms[name].set_machine(self.machine)
+        return self._vms[name]
+
+    def set_vm_arg(self, arg, value):
+        """
+        Set an argument to list of extra arguments to be given to the QEMU
+        binary. If the argument already exists then its value is replaced.
+
+        :param arg: the QEMU argument, such as "-cpu" in "-cpu host"
+        :type arg: str
+        :param value: the argument value, such as "host" in "-cpu host"
+        :type value: str
+        """
+        if not arg or not value:
+            return
+        if arg not in self.vm.args:
+            self.vm.args.extend([arg, value])
+        else:
+            idx = self.vm.args.index(arg) + 1
+            if idx < len(self.vm.args):
+                self.vm.args[idx] = value
+            else:
+                self.vm.args.append(value)
+
+    def tearDown(self):
+        for vm in self._vms.values():
+            vm.shutdown()
+        super().tearDown()
Signature · 1 line
2.45.2
Reviewed-by Tested-by on v2 2/23
#4 lore raw

Re: [PATCH v2 02/23] tests/functional: Add base classes for the upcoming pytest-based tests

Quoted 4 lines · Thomas Huth

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

Quoted 8 lines

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

---
  tests/functional/qemu_test/__init__.py | 328 +++++++++++++++++++++++++
  1 file changed, 328 insertions(+)

create mode 100644 tests/functional/qemu_test/__init__.py

#5 lore raw

PATCH v2 3/23 tests/Makefile.include: Increase the level of indentation in the help text

In reply to Thomas Huth

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

---
 tests/Makefile.include | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
tests/Makefile.include +15 -15
index 6618bfed70..167d8f29a4 100644
@@ -3,28 +3,28 @@
 .PHONY: check-help
 check-help:
 	@echo "Regression testing targets:"
-	@echo " $(MAKE) check                  Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
-	@echo " $(MAKE) bench                  Run speed tests"
+	@echo " $(MAKE) check                    Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
+	@echo " $(MAKE) bench                    Run speed tests"
 	@echo
 	@echo "Individual test suites:"
-	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
-	@echo " $(MAKE) check-qtest            Run qtest tests"
-	@echo " $(MAKE) check-unit             Run qobject tests"
-	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
-	@echo " $(MAKE) check-block            Run block tests"
+	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
+	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-unit               Run qobject tests"
+	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
+	@echo " $(MAKE) check-block              Run block tests"
 ifneq ($(filter $(all-check-targets), check-softfloat),)
-	@echo " $(MAKE) check-tcg              Run TCG tests"
-	@echo " $(MAKE) check-softfloat        Run FPU emulation tests"
+	@echo " $(MAKE) check-tcg                Run TCG tests"
+	@echo " $(MAKE) check-softfloat          Run FPU emulation tests"
 endif
-	@echo " $(MAKE) check-avocado          Run avocado (integration) tests for currently configured targets"
+	@echo " $(MAKE) check-avocado            Run avocado (integration) tests for currently configured targets"
 	@echo
-	@echo " $(MAKE) check-report.junit.xml Generates an aggregated XML test report"
-	@echo " $(MAKE) check-venv             Creates a Python venv for tests"
-	@echo " $(MAKE) check-clean            Clean the tests and related data"
+	@echo " $(MAKE) check-report.junit.xml   Generates an aggregated XML test report"
+	@echo " $(MAKE) check-venv               Creates a Python venv for tests"
+	@echo " $(MAKE) check-clean              Clean the tests and related data"
 	@echo
 	@echo "The following are useful for CI builds"
-	@echo " $(MAKE) check-build            Build most test binaries"
-	@echo " $(MAKE) get-vm-images          Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
+	@echo " $(MAKE) check-build              Build most test binaries"
+	@echo " $(MAKE) get-vm-images            Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
 	@echo
 	@echo
 	@echo "The variable SPEED can be set to control the gtester speed setting."
Signature · 1 line
2.45.2
Reviewed-by on v2 3/23
#6 lore raw

Re: [PATCH v2 03/23] tests/Makefile.include: Increase the level of indentation in the help text

Quoted 8 lines · Thomas Huth

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

---
  tests/Makefile.include | 30 +++++++++++++++---------------
  1 file changed, 15 insertions(+), 15 deletions(-)
#7 lore raw

PATCH v2 4/23 tests/functional: Prepare the meson build system for the functional tests

In reply to Thomas Huth

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

---
 tests/Makefile.include       | 11 ++++++
 tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 3 files changed, 78 insertions(+)
 create mode 100644 tests/functional/meson.build
tests/Makefile.include +11 -0
index 167d8f29a4..d0c2ae30b7 100644
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-functional         Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARGET  Run functional tests for a given target"
 	@echo " $(MAKE) check-unit               Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
 	@echo " $(MAKE) check-block              Run block tests"
@@ -152,6 +154,15 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@$(MAKE) SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@$(MAKE) SPEED=thorough check-func check-func-quick
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
tests/functional/meson.build new file +66 -0

66 lines not on this page · raw message

tests/meson.build +1 -0
index acb6807094..3345ad2098 100644
@@ -85,3 +85,4 @@ subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')
Signature · 1 line
2.45.2
Tested-by on v2 4/23
#8 lore raw

Re: [PATCH v2 04/23] tests/functional: Prepare the meson build system for the functional tests

Quoted 11 lines · Thomas Huth

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

Quoted 7 lines
---
  tests/Makefile.include       | 11 ++++++
  tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
  tests/meson.build            |  1 +
  3 files changed, 78 insertions(+)

create mode 100644 tests/functional/meson.build

#9 lore raw

PATCH v2 5/23 tests/functional: Convert simple avocado tests into standalone python tests

In reply to Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
 tests/functional/meson.build                  |  5 ++
 .../test_cpu_queries.py}                      |  7 ++-
 .../test_empty_cpu_model.py}                  |  7 ++-
 .../test_mem_addr_space.py}                   | 52 +++----------------
 .../test_pc_cpu_hotplug_props.py}             | 11 ++--
 .../test_virtio_version.py}                   |  8 +--
 6 files changed, 34 insertions(+), 56 deletions(-)
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
tests/functional/meson.build +5 -0
index 7fc9c4e513..a8cc0e6330 100644
@@ -14,9 +14,14 @@ test_timeouts = {
 }
 
 tests_generic = [
+  'empty_cpu_model',
 ]
 
 tests_x86_64_quick = [
+  'cpu_queries',
+  'mem_addr_space',
+  'pc_cpu_hotplug_props',
+  'virtio_version',
 ]
 
 tests_x86_64_thorough = [
tests/functional/test_cpu_queries.py renamed from tests/avocado/cpu_queries.py +6 -1
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/cpu_queries.py
rename to tests/functional/test_cpu_queries.py
index d3faa14720..97dd600c01
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Sanity check of query-cpu-* results
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -8,7 +10,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class QueryCPUModelExpansion(QemuSystemTest):
     """
@@ -33,3 +35,6 @@ def test(self):
             e = self.vm.cmd('query-cpu-model-expansion', model=model,
                             type='full')
             self.assertEqual(e['model']['name'], c['name'])
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_empty_cpu_model.py renamed from tests/avocado/empty_cpu_model.py +6 -1
old mode 100644
new mode 100755
similarity index 84%
rename from tests/avocado/empty_cpu_model.py
rename to tests/functional/test_empty_cpu_model.py
index d906ef3d3c..0081b06d85
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Check for crash when using empty -cpu option
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -7,7 +9,7 @@
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class EmptyCPUModel(QemuSystemTest):
     def test(self):
@@ -17,3 +19,6 @@ def test(self):
         self.vm.wait()
         self.assertEqual(self.vm.exitcode(), 1, "QEMU exit code should be 1")
         self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +6 -46

154 lines not on this page · raw message

tests/functional/test_pc_cpu_hotplug_props.py renamed from tests/avocado/pc_cpu_hotplug_props.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/pc_cpu_hotplug_props.py
rename to tests/functional/test_pc_cpu_hotplug_props.py
index 4bd3e02665..9d5a37cb17
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # Ensure CPU die-id can be omitted on -device
 #
@@ -20,16 +21,16 @@
 # License along with this library; if not, see <http://www.gnu.org/licenses/>.
 #
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class OmittedCPUProps(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=cpu:qemu64
-    """
+
     def test_no_die_id(self):
         self.vm.add_args('-nodefaults', '-S')
         self.vm.add_args('-smp', '1,sockets=2,cores=2,threads=2,maxcpus=8')
         self.vm.add_args('-device', 'qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0')
         self.vm.launch()
         self.assertEqual(len(self.vm.cmd('query-cpus-fast')), 2)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_virtio_version.py renamed from tests/avocado/virtio_version.py +5 -3
old mode 100644
new mode 100755
similarity index 98%
rename from tests/avocado/virtio_version.py
rename to tests/functional/test_virtio_version.py
index afe5e828b5..eb23060564
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 """
 Check compatibility of virtio device types
 """
@@ -12,7 +13,7 @@
 import os
 
 from qemu.machine import QEMUMachine
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 # Virtio Device IDs:
 VIRTIO_NET = 1
@@ -60,8 +61,6 @@ class VirtioVersionCheck(QemuSystemTest):
     Check if virtio-version-specific device types result in the
     same device tree created by `disable-modern` and
     `disable-legacy`.
-
-    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
@@ -173,3 +172,6 @@ def test_modern_only_devs(self):
         self.check_modern_only('virtio-mouse-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-tablet-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-keyboard-pci', VIRTIO_INPUT)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#10 lore raw

Re: [PATCH v2 05/23] tests/functional: Convert simple avocado tests into standalone python tests

Hi Thomas,

Quoted 29 lines · Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
  tests/functional/meson.build                  |  5 ++
  .../test_cpu_queries.py}                      |  7 ++-
  .../test_empty_cpu_model.py}                  |  7 ++-
  .../test_mem_addr_space.py}                   | 52 +++----------------
  .../test_pc_cpu_hotplug_props.py}             | 11 ++--
  .../test_virtio_version.py}                   |  8 +--
  6 files changed, 34 insertions(+), 56 deletions(-)

rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755
rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755
rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755
rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755
rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755

tests/functional/test_mem_addr_space.py class MemAddrCheck(QemuSystemTest): · line 26

Quoted 5 lines
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +0 -2
@@ -22,9 +24,6 @@ class MemAddrCheck(QemuSystemTest):
      # for all 32-bit cases, pci64_hole_size is 0.
      def test_phybits_low_pse36(self):
          """
-        :avocado: tags=machine:q35
-        :avocado: tags=arch:x86_64

Could we keep the tags, but renaming as:

          qemu-test-tag=machine:q35
          qemu-test-tag=arch:x86_64

?

I'm OK to implement the equivalent feature in QEMU functional framework.

Quoted 3 lines
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +0 -0
          With pse36 feature ON, a processor has 36 bits of addressing. So it can
          access up to a maximum of 64GiB of memory. Memory hotplug region begins
          at 4 GiB boundary when "above_4g_mem_size" is 0 (this would be true when
#11 lore raw

Re: [PATCH v2 05/23] tests/functional: Convert simple avocado tests into standalone python tests

Quoted 40 lines · Philippe Mathieu-Daudé

Hi Thomas,

Quoted 30 lines · Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
  tests/functional/meson.build                  |  5 ++ .../test_cpu_queries.py}                      |  7 ++- .../test_empty_cpu_model.py}                  |  7 ++- .../test_mem_addr_space.py}                   | 52 +++---------------- .../test_pc_cpu_hotplug_props.py}             | 11 ++-- .../test_virtio_version.py}                   |  8 +--

  6 files changed, 34 insertions(+), 56 deletions(-)

  rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755
  rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755
  rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755
  rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755
  rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755

tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +0 -0
@@ -22,9 +24,6 @@ class MemAddrCheck(QemuSystemTest):
       # for all 32-bit cases, pci64_hole_size is 0.
Quoted 4 lines

      def test_phybits_low_pse36(self):
          """
-        :avocado: tags=machine:q35
-        :avocado: tags=arch:x86_64

Could we keep the tags, but renaming as:

Quoted 4 lines

          qemu-test-tag=machine:q35
          qemu-test-tag=arch:x86_64

?

Bah, let's just remove that for now, since this feature implementation might use a different format.

Quoted 7 lines

I'm OK to implement the equivalent feature in QEMU functional framework.

Quoted 3 lines

          With pse36 feature ON, a processor has 36 bits of addressing. So it can access up to a maximum of 64GiB of memory. Memory hotplug region begins at 4 GiB boundary when "above_4g_mem_size" is 0 (this would be true when

Reviewed-by on v2 5/23
#12 lore raw

Re: [PATCH v2 05/23] tests/functional: Convert simple avocado tests into standalone python tests

In reply to Thomas Huth
Quoted 53 lines · Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
  tests/functional/meson.build                  |  5 ++
  .../test_cpu_queries.py}                      |  7 ++-
  .../test_empty_cpu_model.py}                  |  7 ++-
  .../test_mem_addr_space.py}                   | 52 +++----------------
  .../test_pc_cpu_hotplug_props.py}             | 11 ++--
  .../test_virtio_version.py}                   |  8 +--
  6 files changed, 34 insertions(+), 56 deletions(-)

rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755
rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755
rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755
rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755
rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755

tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +3 -1
old mode 100644
new mode 100755
similarity index 93%
rename from tests/avocado/mem-addr-space-check.py
rename to tests/functional/test_mem_addr_space.py
index d3974599f4..bb0cf062ca
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
  # Check for crash when using memory beyond the available guest processor
  # address space.
  #
@@ -8,7 +10,7 @@
  #
  # SPDX-License-Identifier: GPL-2.0-or-later
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
  import time
 
  class MemAddrCheck(QemuSystemTest):

tests/functional/test_mem_addr_space.py class MemAddrCheck(QemuSystemTest): · line 25

Quoted 4 lines
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +0 -1
@@ -22,9 +24,6 @@ class MemAddrCheck(QemuSystemTest):
      # for all 32-bit cases, pci64_hole_size is 0.
      def test_phybits_low_pse36(self):
          """
-        :avocado: tags=machine:q35

Pre-existing, we should use set_machine('q35') in this file tests. (Can be cleaned later.)

#13 lore raw

Re: [PATCH v2 05/23] tests/functional: Convert simple avocado tests into standalone python tests

Quoted 29 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
  tests/functional/meson.build                  |  5 ++ .../test_cpu_queries.py}                      |  7 ++- .../test_empty_cpu_model.py}                  |  7 ++- .../test_mem_addr_space.py}                   | 52 +++---------------- .../test_pc_cpu_hotplug_props.py}             | 11 ++-- .../test_virtio_version.py}                   |  8 +--

  6 files changed, 34 insertions(+), 56 deletions(-)
Quoted 10 lines

  rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755
  rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755
  rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755
  rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755
  rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755

Quoted 2 lines
#14 lore raw

PATCH v2 6/23 tests/functional: Convert avocado tests that just need a small adjustment

In reply to Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
 tests/functional/meson.build                  |  6 ++
 .../test_info_usernet.py}                     | 11 ++-
 .../test_ppc_74xx.py}                         | 74 ++++++++-----------
 .../version.py => functional/test_version.py} | 13 ++--
 4 files changed, 53 insertions(+), 51 deletions(-)
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
tests/functional/meson.build +6 -0
index a8cc0e6330..8a8fa0ab99 100644
@@ -15,6 +15,12 @@ test_timeouts = {
 
 tests_generic = [
   'empty_cpu_model',
+  'info_usernet',
+  'version',
+]
+
+tests_ppc_quick = [
+  'ppc_74xx',
 ]
 
 tests_x86_64_quick = [
tests/functional/test_info_usernet.py renamed from tests/avocado/info_usernet.py +7 -4
old mode 100644
new mode 100755
similarity index 87%
rename from tests/avocado/info_usernet.py
rename to tests/functional/test_info_usernet.py
index e1aa7a6e0a..ded973253a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test for the hmp command "info usernet"
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -8,18 +10,16 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 from qemu.utils import get_info_usernet_hostfwd_port
 
 
 class InfoUsernet(QemuSystemTest):
-    """
-    :avocado: tags=machine:none
-    """
 
     def test_hostfwd(self):
         self.require_netdev('user')
+        self.machine = 'none'
         self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
@@ -31,3 +31,6 @@ def test_hostfwd(self):
         self.assertGreater(port, 0,
                            ('Found a redirected port that is not greater than'
                             ' zero'))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_74xx.py renamed from tests/avocado/ppc_74xx.py +32 -42

167 lines not on this page · raw message

tests/functional/test_version.py renamed from tests/avocado/version.py +8 -5
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..5e566d76b1
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Version check example test
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
 class Version(QemuSystemTest):
-    """
-    :avocado: tags=quick
-    :avocado: tags=machine:none
-    """
+
     def test_qmp_human_info_version(self):
+        self.machine = 'none'
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
                           command_line='info version')
         self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v2 6/23
#15 lore raw

Re: [PATCH v2 06/23] tests/functional: Convert avocado tests that just need a small adjustment

Quoted 42 lines · Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
  tests/functional/meson.build                  |  6 ++
  .../test_info_usernet.py}                     | 11 ++-
  .../test_ppc_74xx.py}                         | 74 ++++++++-----------
  .../version.py => functional/test_version.py} | 13 ++--
  4 files changed, 53 insertions(+), 51 deletions(-)

rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755
rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755
rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755

tests/functional/test_version.py renamed from tests/avocado/version.py +3 -1
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..5e566d76b1
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
  # Version check example test
  #
  # Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
  # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest

tests/functional/test_version.py line 20

Quoted 10 lines
tests/functional/test_version.py renamed from tests/avocado/version.py +2 -4
 
 
  class Version(QemuSystemTest):
-    """
-    :avocado: tags=quick
-    :avocado: tags=machine:none
-    """
+
      def test_qmp_human_info_version(self):
+        self.machine = 'none'

API is self.set_machine('none'), otherwise:

#16 lore raw

Re: [PATCH v2 06/23] tests/functional: Convert avocado tests that just need a small adjustment

Quoted 22 lines · Philippe Mathieu-Daudé
Quoted 12 lines · Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
  tests/functional/meson.build                  |  6 ++ .../test_info_usernet.py}                     | 11 ++- .../test_ppc_74xx.py}                         | 74 ++++++++----------- .../version.py => functional/test_version.py} | 13 ++--

Quoted 7 lines
  4 files changed, 53 insertions(+), 51 deletions(-)

  rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755
  rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755
  rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755

Quoted 2 lines
#17 lore raw

PATCH v2 7/23 tests/functional: Implement fetch_asset() method for downloading assets

In reply to Thomas Huth

In the new python test framework, we cannot use the fetch_asset() function from Avocado anymore, so we have to provide our own implementation now instead. Thus add such a function based on the urllib python module for this purpose.

---
 tests/functional/qemu_test/__init__.py | 41 ++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
tests/functional/qemu_test/__init__.py +41 -0
index fc98222c52..40a81c3927 100644
@@ -11,6 +11,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import hashlib
+import urllib.request
 import logging
 import os
 import pycotap
@@ -23,6 +25,7 @@
 import unittest
 
 from pathlib import Path
+from shutil import copyfileobj
 from qemu.machine import QEMUMachine
 from qemu.utils import kvm_available, tcg_available
 
@@ -216,6 +219,44 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+    def check_hash(self, file_name, expected_hash):
+        if not expected_hash:
+            return True
+        if len(expected_hash) == 40:
+            sum_prog = 'sha1sum'
+        elif len(expected_hash) == 64:
+            sum_prog = 'sha256sum'
+        elif len(expected_hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")
+        checksum = subprocess.check_output([sum_prog, file_name]).split()[0]
+        return expected_hash == checksum.decode("utf-8")
+
+    def fetch_asset(self, url, asset_hash):
+        cache_dir = os.path.expanduser("~/.cache/qemu/download")
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+        fname = os.path.join(cache_dir,
+                             hashlib.sha256(url.encode("utf-8")).hexdigest())
+        if os.path.exists(fname) and self.check_hash(fname, asset_hash):
+            self.log.debug("Using cached assed %s for %s", fname, url)
+            return fname
+        self.log.info("Downloading %s to %s...", url, fname)
+        dl_fname = fname + ".download"
+        with urllib.request.urlopen(url) as src:
+            try:
+                with open(dl_fname, "wb+") as dst:
+                    copyfileobj(src, dst)
+            except:
+                os.remove(dl_fname)
+                raise
+        if not self.check_hash(dl_fname, asset_hash):
+            os.remove(dl_fname)
+            raise Exception("Hash of " + url + " does not match")
+        os.rename(dl_fname, fname)
+        return fname
+
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
Signature · 1 line
2.45.2
#18 lore raw

PATCH v2 8/23 tests/functional: Convert some tests that download files via fetch_asset()

In reply to Thomas Huth

Now that we've got a working fetch_asset() function, we can convert some Avocado tests that use this function for downloading their required files.

---
 MAINTAINERS                                   | 12 +++----
 tests/functional/meson.build                  | 22 +++++++++++++
 .../test_arm_n8x0.py}                         | 25 +++++++--------
 .../test_avr_mega2560.py}                     | 11 ++++---
 .../test_loongarch64_virt.py}                 | 16 ++++++----
 .../test_mips64el_loongson3v.py}              | 26 +++++++--------
 .../test_netdev_ethtool.py}                   | 32 ++++++-------------
 .../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
 8 files changed, 86 insertions(+), 77 deletions(-)
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
MAINTAINERS +6 -6
index dd01288992..d69de11cfc 100644
@@ -222,7 +222,7 @@ S: Maintained
 F: docs/system/target-avr.rst
 F: gdb-xml/avr-cpu.xml
 F: target/avr/
-F: tests/avocado/machine_avr6.py
+F: tests/functional/test_avr_mega2560.py
 
 CRIS TCG CPUs
 M: Edgar E. Iglesias <[email protected]>
@@ -265,7 +265,7 @@ M: Song Gao <[email protected]>
 S: Maintained
 F: target/loongarch/
 F: tests/tcg/loongarch64/
-F: tests/avocado/machine_loongarch.py
+F: tests/functional/test_loongarch64_virt.py
 
 M68K TCG CPUs
 M: Laurent Vivier <[email protected]>
@@ -884,7 +884,7 @@ F: include/hw/display/blizzard.h
 F: include/hw/input/lm832x.h
 F: include/hw/input/tsc2xxx.h
 F: include/hw/misc/cbus.h
-F: tests/avocado/machine_arm_n8x0.py
+F: tests/functional/test_arm_n8x0.py
 F: docs/system/arm/nseries.rst
 
 Palm
@@ -1393,7 +1393,7 @@ F: hw/mips/loongson3_bootp.h
 F: hw/mips/loongson3_virt.c
 F: include/hw/intc/loongson_ipi.h
 F: include/hw/intc/loongson_liointc.h
-F: tests/avocado/machine_mips_loongson3v.py
+F: tests/functional/test_mips64el_loongson3v.py
 
 Boston
 M: Paul Burton <[email protected]>
@@ -1419,7 +1419,7 @@ PowerPC Machines
 L: [email protected]
 S: Orphan
 F: hw/ppc/ppc405*
-F: tests/avocado/ppc_405.py
+F: tests/functional/test_ppc_405.py
 
 Bamboo
 L: [email protected]
@@ -2486,7 +2486,7 @@ R: Sriram Yagnaraman <[email protected]>
 S: Maintained
 F: docs/system/devices/igb.rst
 F: hw/net/igb*
-F: tests/avocado/netdev-ethtool.py
+F: tests/functional/test_netdev_ethtool.py
 F: tests/qtest/igb-test.c
 F: tests/qtest/libqos/igb.c
 
tests/functional/meson.build +22 -0
index 8a8fa0ab99..7eb5dbfe31 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'netdev_ethtool' : 180,
 }
 
 tests_generic = [
@@ -19,10 +20,30 @@ tests_generic = [
   'version',
 ]
 
+tests_arm_thorough = [
+  'arm_n8x0',
+]
+
+tests_avr_thorough = [
+  'avr_mega2560',
+]
+
+tests_loongarch64_thorough = [
+  'loongarch64_virt',
+]
+
+tests_mips64el_thorough = [
+  'mips64el_loongson3v',
+]
+
 tests_ppc_quick = [
   'ppc_74xx',
 ]
 
+tests_ppc_thorough = [
+  'ppc_405',
+]
+
 tests_x86_64_quick = [
   'cpu_queries',
   'mem_addr_space',
@@ -31,6 +52,7 @@ tests_x86_64_quick = [
 ]
 
 tests_x86_64_thorough = [
+  'netdev_ethtool',
 ]
 
 foreach speed : ['quick', 'thorough']
tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +12 -13
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..d451c80a73
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,9 +12,9 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class N8x0Machine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
@@ -32,18 +34,15 @@ def __do_test_n8x0(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'TSC2005 driver initializing')
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n800(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n800
-        """
+        self.set_machine('n800')
         self.__do_test_n8x0()
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n810(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n810
-        """
+        self.set_machine('n810')
         self.__do_test_n8x0()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_avr_mega2560.py renamed from tests/avocado/machine_avr6.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/machine_avr6.py
rename to tests/functional/test_avr_mega2560.py
index 5485db79c6..b9b89367c6
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # QEMU AVR integration tests
 #
@@ -19,16 +20,12 @@
 
 import time
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class AVR6Machine(QemuSystemTest):
     timeout = 5
 
     def test_freertos(self):
-        """
-        :avocado: tags=arch:avr
-        :avocado: tags=machine:arduino-mega-2560-v3
-        """
         """
         https://github.com/seharris/qemu-avr-tests/raw/master/free-rtos/Demo/AVR_ATMega2560_GCC/demo.elf
         constantly prints out 'ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX'
@@ -39,6 +36,7 @@ def test_freertos(self):
         rom_hash = '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4'
         rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
 
+        self.set_machine('arduino-mega-2560-v3')
         self.vm.add_args('-bios', rom_path)
         self.vm.add_args('-nographic')
         self.vm.launch()
@@ -48,3 +46,6 @@ def test_freertos(self):
 
         self.assertIn('ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX',
                 self.vm.get_log())
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_loongarch64_virt.py renamed from tests/avocado/machine_loongarch.py +9 -7
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/machine_loongarch.py
rename to tests/functional/test_loongarch64_virt.py
index 8de308f2d6..3dc8bb3516
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
 # LoongArch virt test.
@@ -5,9 +7,9 @@
 # Copyright (c) 2023 Loongson Technology Corporation Limited
 #
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
 
 class LoongArchMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -21,10 +23,7 @@ def wait_for_console_pattern(self, success_message, vm=None):
 
     def test_loongarch64_devices(self):
 
-        """
-        :avocado: tags=arch:loongarch64
-        :avocado: tags=machine:virt
-        """
+        self.set_machine('virt')
 
         kernel_url = ('https://github.com/yangxiaojuan-loongson/qemu-binary/'
                       'releases/download/2024-05-30/vmlinuz.efi')
@@ -56,3 +55,6 @@ def test_loongarch64_devices(self):
         self.wait_for_console_pattern('Run /sbin/init as init process')
         exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
                                           'processor		: 3')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mips64el_loongson3v.py renamed from tests/avocado/machine_mips_loongson3v.py +12 -14
old mode 100644
new mode 100755
similarity index 55%
rename from tests/avocado/machine_mips_loongson3v.py
rename to tests/functional/test_mips64el_loongson3v.py
index 5194cf18c9..f8d40b48a9
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional tests for the Generic Loongson-3 Platform.
 #
 # Copyright (c) 2021 Jiaxun Yang <[email protected]>
@@ -10,30 +12,26 @@
 import os
 import time
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class MipsLoongson3v(QemuSystemTest):
     timeout = 60
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_pmon_serial_console(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=endian:little
-        :avocado: tags=machine:loongson3-virt
-        :avocado: tags=cpu:Loongson-3A1000
-        :avocado: tags=device:liointc
-        :avocado: tags=device:goldfish_rtc
-        """
+        self.set_machine('loongson3-virt')
 
-        pmon_hash = '7c8b45dd81ccfc55ff28f5aa267a41c3'
+        pmon_hash = 'fcdf6bb2cb7885a4a62f31fcb0d5e368bac7b6cea28f40c6dfa678af22fea20a'
         pmon_path = self.fetch_asset('https://github.com/loongson-community/pmon/'
                                     'releases/download/20210112/pmon-3avirt.bin',
-                                     asset_hash=pmon_hash, algorithm='md5')
+                                     asset_hash=pmon_hash)
 
         self.vm.set_console()
         self.vm.add_args('-bios', pmon_path)
         self.vm.launch()
         wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_netdev_ethtool.py renamed from tests/avocado/netdev-ethtool.py +10 -22
old mode 100644
new mode 100755
similarity index 81%
rename from tests/avocado/netdev-ethtool.py
rename to tests/functional/test_netdev_ethtool.py
index 5f33288f81..341c087e36
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # ethtool tests for emulated network devices
 #
 # This test leverages ethtool's --test sequence to validate network
@@ -5,15 +7,11 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-late
 
-from avocado import skip
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skip
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class NetDevEthtool(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=machine:q35
-    """
 
     # Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
     timeout = 45
@@ -25,11 +23,10 @@ def get_asset(self, name, sha1):
                     'kE4nCFLdQcoBF9t/download?'
                     'path=%2Fnetdev-ethtool&files=' )
         url = base_url + name
-        # use explicit name rather than failing to neatly parse the
-        # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(url, asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
+        self.set_machine('q35')
 
         # This custom kernel has drivers for all the supported network
         # devices we can emulate in QEMU
@@ -68,15 +65,9 @@ def common_test_code(self, netdev, extra_args=None):
         self.vm.kill()
 
     def test_igb(self):
-        """
-        :avocado: tags=device:igb
-        """
         self.common_test_code("igb")
 
     def test_igb_nomsi(self):
-        """
-        :avocado: tags=device:igb
-        """
         self.common_test_code("igb", "pci=nomsi")
 
     # It seems the other popular cards we model in QEMU currently fail
@@ -88,14 +79,11 @@ def test_igb_nomsi(self):
 
     @skip("Incomplete reg 0x00178 support")
     def test_e1000(self):
-        """
-        :avocado: tags=device:e1000
-        """
         self.common_test_code("e1000")
 
     @skip("Incomplete reg 0x00178 support")
     def test_i82550(self):
-        """
-        :avocado: tags=device:i82550
-        """
         self.common_test_code("i82550")
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_405.py renamed from tests/avocado/ppc_405.py +9 -10
old mode 100644
new mode 100755
similarity index 73%
rename from tests/avocado/ppc_405.py
rename to tests/functional/test_ppc_405.py
index 4e7e01aa76..f2368ada80
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that the U-Boot firmware boots on ppc 405 machines and check the console
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -5,10 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class Ppc405Machine(QemuSystemTest):
 
@@ -26,11 +27,9 @@ def do_test_ppc405(self):
         exec_command_and_wait_for_pattern(self, 'reset', 'AMCC PowerPC 405EP')
 
     def test_ppc_ref405ep(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:ref405ep
-        :avocado: tags=cpu:405ep
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('ref405ep')
         self.do_test_ppc405()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v2 8/23
#19 lore raw

Re: [PATCH v2 08/23] tests/functional: Convert some tests that download files via fetch_asset()

Quoted 59 lines · Thomas Huth

Now that we've got a working fetch_asset() function, we can convert some Avocado tests that use this function for downloading their required files.

---
  MAINTAINERS                                   | 12 +++----
  tests/functional/meson.build                  | 22 +++++++++++++
  .../test_arm_n8x0.py}                         | 25 +++++++--------
  .../test_avr_mega2560.py}                     | 11 ++++---
  .../test_loongarch64_virt.py}                 | 16 ++++++----
  .../test_mips64el_loongson3v.py}              | 26 +++++++--------
  .../test_netdev_ethtool.py}                   | 32 ++++++-------------
  .../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
  8 files changed, 86 insertions(+), 77 deletions(-)

rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%) mode change 100644 => 100755
rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%) mode change 100644 => 100755
rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%) mode change 100644 => 100755
rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%) mode change 100644 => 100755
rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%) mode change 100644 => 100755
rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
mode change 100644 => 100755

tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +5 -3
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..d451c80a73
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
  # Functional test that boots a Linux kernel and checks the console
  #
  # Copyright (c) 2020 Red Hat, Inc.
@@ -10,9 +12,9 @@
 
  import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
  class N8x0Machine(QemuSystemTest):
      """Boots the Linux kernel and checks that the console is operational"""

tests/functional/test_arm_n8x0.py def __do_test_n8x0(self): · line 37

Quoted 5 lines
tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +1 -1
@@ -32,18 +34,15 @@ def __do_test_n8x0(self):
          self.vm.launch()
          wait_for_console_pattern(self, 'TSC2005 driver initializing')
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')

We need to update docs/devel/testing.rst which lists the "most used [test] variables". But the functional testing chapter need to be completely revamped, so meanwhile:

#20 lore raw

PATCH v2 9/23 tests/functional: Add a function for extracting files from an archive

In reply to Thomas Huth

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

---
 tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/functional/qemu_test/utils.py
tests/functional/qemu_test/utils.py new file +21 -0
new file mode 100644
index 0000000000..4eb5e5d5e5
@@ -0,0 +1,21 @@
+# Utilities for python-based QEMU tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# Authors:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import tarfile
+
+def archive_extract(archive, dest_dir, member=None):
+    with tarfile.open(archive) as tf:
+        if hasattr(tarfile, 'data_filter'):
+            tf.extraction_filter = getattr(tarfile, 'data_filter',
+                                           (lambda member, path: member))
+        if member:
+            tf.extract(member=member, path=dest_dir)
+        else:
+            tf.extractall(path=dest_dir)
Signature · 1 line
2.45.2
Reviewed-by on v2 9/23
#21 lore raw

Re: [PATCH v2 09/23] tests/functional: Add a function for extracting files from an archive

Quoted 7 lines · Thomas Huth

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

Quoted 6 lines
---
  tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
  1 file changed, 21 insertions(+)

create mode 100644 tests/functional/qemu_test/utils.py

#22 lore raw

PATCH v2 10/23 tests/functional: Convert some avocado tests that needed avocado.utils.archive

In reply to Thomas Huth

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

---
 MAINTAINERS                                   |  6 ++---
 tests/functional/meson.build                  |  6 +++++
 .../test_arm_canona1100.py}                   | 21 +++++++++-------
 .../test_ppc_bamboo.py}                       | 23 ++++++++---------
 .../test_sparc64_sun4u.py}                    | 25 +++++++++++--------
 5 files changed, 46 insertions(+), 35 deletions(-)
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
MAINTAINERS +3 -3
index d69de11cfc..f107b8eb20 100644
@@ -731,7 +731,7 @@ S: Odd Fixes
 F: include/hw/arm/digic.h
 F: hw/*/digic*
 F: include/hw/*/digic*
-F: tests/avocado/machine_arm_canona1100.py
+F: tests/functional/test_arm_canona1100.py
 F: docs/system/arm/digic.rst
 
 Goldfish RTC
@@ -1426,7 +1426,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/ppc440_bamboo.c
 F: hw/pci-host/ppc4xx_pci.c
-F: tests/avocado/ppc_bamboo.py
+F: tests/functional/test_ppc_bamboo.py
 
 e500
 L: [email protected]
@@ -1712,7 +1712,7 @@ F: include/hw/pci-host/sabre.h
 F: hw/pci-bridge/simba.c
 F: include/hw/pci-bridge/simba.h
 F: pc-bios/openbios-sparc64
-F: tests/avocado/machine_sparc64_sun4u.py
+F: tests/functional/test_sparc64_sun4u.py
 
 Sun4v
 M: Artyom Tarasenko <[email protected]>
tests/functional/meson.build +6 -0
index 7eb5dbfe31..5615a5b2fd 100644
@@ -21,6 +21,7 @@ tests_generic = [
 ]
 
 tests_arm_thorough = [
+  'arm_canona1100',
   'arm_n8x0',
 ]
 
@@ -42,6 +43,11 @@ tests_ppc_quick = [
 
 tests_ppc_thorough = [
   'ppc_405',
+  'ppc_bamboo',
+]
+
+tests_sparc64_thorough = [
+  'sparc64_sun4u',
 ]
 
 tests_x86_64_quick = [
tests/functional/test_arm_canona1100.py renamed from tests/avocado/machine_arm_canona1100.py +12 -9
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_canona1100.py
rename to tests/functional/test_arm_canona1100.py
index a42d8b0f2b..de6169f2af
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots the canon-a1100 machine with firmware
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -8,9 +10,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
 class CanonA1100Machine(QemuSystemTest):
     """Boots the barebox firmware and checks that the console is operational"""
@@ -18,18 +20,19 @@ class CanonA1100Machine(QemuSystemTest):
     timeout = 90
 
     def test_arm_canona1100(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:canon-a1100
-        :avocado: tags=device:pflash_cfi02
-        """
+        self.set_machine('canon-a1100')
+
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day18.tar.xz')
         tar_hash = '068b5fc4242b29381acee94713509f8a876e9db6'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, dest_dir=self.workdir,
+                        member="day18/barebox.canon-a1100.bin")
         self.vm.set_console()
         self.vm.add_args('-bios',
                          self.workdir + '/day18/barebox.canon-a1100.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'running /env/bin/init')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_bamboo.py renamed from tests/avocado/ppc_bamboo.py +11 -12
old mode 100644
new mode 100755
similarity index 75%
rename from tests/avocado/ppc_bamboo.py
rename to tests/functional/test_ppc_bamboo.py
index a81be3d608..652b31e95a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on the ppc bamboo board and check the console
 #
 # Copyright (c) 2021 Red Hat
@@ -5,30 +7,24 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class BambooMachine(QemuSystemTest):
 
     timeout = 90
 
     def test_ppc_bamboo(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:bamboo
-        :avocado: tags=cpu:440epb
-        :avocado: tags=device:rtl8139
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('bamboo')
         self.require_accelerator("tcg")
         self.require_netdev('user')
         tar_url = ('http://landley.net/aboriginal/downloads/binaries/'
                    'system-image-powerpc-440fp.tar.gz')
         tar_hash = '53e5f16414b195b82d2c70272f81c2eedb39bad9'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir +
                                    '/system-image-powerpc-440fp/linux',
@@ -40,3 +36,6 @@ def test_ppc_bamboo(self):
         exec_command_and_wait_for_pattern(self, 'ping 10.0.2.2',
                                           '10.0.2.2 is alive!')
         exec_command_and_wait_for_pattern(self, 'halt', 'System Halted')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_sparc64_sun4u.py renamed from tests/avocado/machine_sparc64_sun4u.py +14 -11
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/machine_sparc64_sun4u.py
rename to tests/functional/test_sparc64_sun4u.py
index d333c0ae91..cd89d4a1d6
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,27 +12,28 @@
 
 import os
 
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
-from boot_linux_console import LinuxKernelTest
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
-class Sun4uMachine(LinuxKernelTest):
+class Sun4uMachine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
 
     timeout = 90
 
     def test_sparc64_sun4u(self):
-        """
-        :avocado: tags=arch:sparc64
-        :avocado: tags=machine:sun4u
-        """
+        self.set_machine('sun4u')
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day23.tar.xz')
         tar_hash = '142db83cd974ffadc4f75c8a5cad5bcc5722c240'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        kernel_name = 'day23/vmlinux'
+        archive_extract(file_path, self.workdir, kernel_name)
         self.vm.set_console()
-        self.vm.add_args('-kernel', self.workdir + '/day23/vmlinux',
-                         '-append', self.KERNEL_COMMON_COMMAND_LINE)
+        self.vm.add_args('-kernel', os.path.join(self.workdir, kernel_name),
+                         '-append', 'printk.time=0')
         self.vm.launch()
         wait_for_console_pattern(self, 'Starting logging: OK')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#23 lore raw

PATCH v2 11/23 tests/functional: Set up logging

In reply to Thomas Huth

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/__init__.py | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)
tests/functional/qemu_test/__init__.py +25 -1
index 40a81c3927..49cce620ab 100644
@@ -208,7 +208,8 @@ class QemuBaseTest(unittest.TestCase):
     arch = None
 
     workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
+    logdir = None
 
     def setUp(self, bin_prefix):
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -219,6 +220,19 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+        self.logdir = self.workdir
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.logdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)
+
+    def tearDown(self):
+        self.log.removeHandler(self._log_fh)
+
     def check_hash(self, file_name, expected_hash):
         if not expected_hash:
             return True
@@ -276,6 +290,15 @@ def setUp(self):
 
         super().setUp('qemu-system-')
 
+        console_log = logging.getLogger('console')
+        console_log.setLevel(logging.DEBUG)
+        self._console_log_fh = logging.FileHandler(os.path.join(self.workdir,
+                                                   'console.log'), mode='w')
+        self._console_log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
+        self._console_log_fh.setFormatter(fileFormatter)
+        console_log.addHandler(self._console_log_fh)
+
     def set_machine(self, machinename):
         # TODO: We should use QMP to get the list of available machines
         if not self._machinehelp:
@@ -366,4 +389,5 @@ def set_vm_arg(self, arg, value):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
+        logging.getLogger('console').removeHandler(self._console_log_fh)
         super().tearDown()
Signature · 1 line
2.45.2
#24 lore raw

PATCH v2 12/23 tests/functional: Convert the s390x avocado tests into standalone tests

In reply to Thomas Huth

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
 MAINTAINERS                                   |  4 +-
 tests/functional/meson.build                  |  6 ++
 tests/functional/qemu_test/utils.py           | 14 ++++
 .../test_s390x_ccw_virtio.py}                 | 32 ++++-----
 .../test_s390x_topology.py}                   | 70 +++++++------------
 5 files changed, 59 insertions(+), 67 deletions(-)
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index f107b8eb20..209fe0f728 100644
@@ -1739,7 +1739,7 @@ S: Supported
 F: hw/s390x/
 F: include/hw/s390x/
 F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: [email protected]
 
@@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
 F: target/s390x/kvm/stsi-topology.c
 F: docs/devel/s390-cpu-topology.rst
 F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
 
 X86 Machines
 ------------
tests/functional/meson.build +6 -0
index 5615a5b2fd..ddbb59604b 100644
@@ -12,6 +12,7 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  's390x_ccw_virtio' : 180,
 }
 
 tests_generic = [
@@ -46,6 +47,11 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_s390x_thorough = [
+  's390x_ccw_virtio',
+  's390x_topology',
+]
+
 tests_sparc64_thorough = [
   'sparc64_sun4u',
 ]
tests/functional/qemu_test/utils.py +14 -0
index 4eb5e5d5e5..a12dac51b6 100644
@@ -8,6 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import lzma
+import os
+import shutil
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):
@@ -19,3 +22,14 @@ def archive_extract(archive, dest_dir, member=None):
             tf.extract(member=member, path=dest_dir)
         else:
             tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with lzma.open(xz_path, 'rb') as lzma_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(lzma_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
tests/functional/test_s390x_ccw_virtio.py renamed from tests/avocado/machine_s390_ccw_virtio.py +12 -20
old mode 100644
new mode 100755
similarity index 95%
rename from tests/avocado/machine_s390_ccw_virtio.py
rename to tests/functional/test_s390x_ccw_virtio.py
index 26e938c9e9..b2cbaacd21
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots an s390x Linux guest with ccw and PCI devices
 # attached and checks whether the devices are recognized by Linux
 #
@@ -12,11 +14,10 @@
 import os
 import tempfile
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import lzma_uncompress
 
 class S390CCWVirtioMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -41,11 +42,7 @@ def clear_guest_dmesg(self):
         self.dmesg_clear_count += 1
 
     def test_s390x_devices(self):
-
-        """
-        :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390-ccw-virtio
-        """
+        self.set_machine('s390-ccw-virtio')
 
         kernel_url = ('https://snapshot.debian.org/archive/debian/'
                       '20201126T092837Z/dists/buster/main/installer-s390x/'
@@ -160,15 +157,7 @@ def test_s390x_devices(self):
 
 
     def test_s390x_fedora(self):
-
-        """
-        :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390-ccw-virtio
-        :avocado: tags=device:virtio-gpu
-        :avocado: tags=device:virtio-crypto
-        :avocado: tags=device:virtio-net
-        :avocado: tags=flaky
-        """
+        self.set_machine('s390-ccw-virtio')
 
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
                       '/fedora-secondary/releases/31/Server/s390x/os'
@@ -182,7 +171,7 @@ def test_s390x_fedora(self):
         initrd_hash = '3de45d411df5624b8d8ef21cd0b44419ab59b12f'
         initrd_path_xz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
         initrd_path = os.path.join(self.workdir, 'initrd-raw.img')
-        archive.lzma_uncompress(initrd_path_xz, initrd_path)
+        lzma_uncompress(initrd_path_xz, initrd_path)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE + ' audit=0 '
@@ -275,3 +264,6 @@ def test_s390x_fedora(self):
         exec_command_and_wait_for_pattern(self,
                         'while ! (dmesg -c | grep Start.virtcrypto_remove) ; do'
                         ' sleep 1 ; done', 'Start virtcrypto_remove.')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_s390x_topology.py renamed from tests/avocado/s390_topology.py +25 -45

165 lines not on this page · raw message

Signature · 1 line
2.45.2
#25 lore raw

PATCH v2 13/23 tests/functional: Convert the x86_cpu_model_versions test

In reply to Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                  |  1 +
 .../test_x86_cpu_model_versions.py}           | 63 ++++++-------------
 2 files changed, 20 insertions(+), 44 deletions(-)
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index ddbb59604b..e0e1be68bb 100644
@@ -61,6 +61,7 @@ tests_x86_64_quick = [
   'mem_addr_space',
   'pc_cpu_hotplug_props',
   'virtio_version',
+  'x86_cpu_model_versions',
 ]
 
 tests_x86_64_thorough = [
tests/functional/test_x86_cpu_model_versions.py renamed from tests/avocado/x86_cpu_model_versions.py +19 -44

154 lines not on this page · raw message

Signature · 1 line
2.45.2
Tested-by Reviewed-by on v2 13/23
#26 lore raw

Re: [PATCH v2 13/23] tests/functional: Convert the x86_cpu_model_versions test

Quoted 9 lines · Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                  |  1 +
  .../test_x86_cpu_model_versions.py}           | 63 ++++++-------------
  2 files changed, 20 insertions(+), 44 deletions(-)

rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%) mode change 100644 => 100755

#27 lore raw

PATCH v2 14/23 tests/functional: Convert the microblaze avocado tests into standalone tests

In reply to Thomas Huth

The machine_microblaze.py file contained two tests, one for each endianess. Since we only support one QEMU target binary per file in the new functional test environment, we have to split this file up into two files now.

---
 MAINTAINERS                                   |  2 +-
 tests/avocado/machine_microblaze.py           | 61 -------------------
 tests/functional/meson.build                  |  8 +++
 .../functional/test_microblaze_s3adsp1800.py  | 38 ++++++++++++
 .../test_microblazeel_s3adsp1800.py           | 41 +++++++++++++
 5 files changed, 88 insertions(+), 62 deletions(-)
 delete mode 100644 tests/avocado/machine_microblaze.py
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
MAINTAINERS +1 -1
index 209fe0f728..02cf39e94d 100644
@@ -1327,7 +1327,7 @@ M: Edgar E. Iglesias <[email protected]>
 S: Maintained
 F: hw/microblaze/petalogix_s3adsp1800_mmu.c
 F: include/hw/char/xilinx_uartlite.h
-F: tests/avocado/machine_microblaze.py
+F: tests/functional/test_microblaze*.py
 
 petalogix_ml605
 M: Edgar E. Iglesias <[email protected]>
tests/avocado/machine_microblaze.py deleted +0 -61

61 lines not on this page · raw message

tests/functional/meson.build +8 -0
index e0e1be68bb..a2c0398b03 100644
@@ -34,6 +34,14 @@ tests_loongarch64_thorough = [
   'loongarch64_virt',
 ]
 
+tests_microblaze_thorough = [
+  'microblaze_s3adsp1800'
+]
+
+tests_microblazeel_thorough = [
+  'microblazeel_s3adsp1800'
+]
+
 tests_mips64el_thorough = [
   'mips64el_loongson3v',
 ]
tests/functional/test_microblaze_s3adsp1800.py new file +38 -0
new file mode 100755
index 0000000000..3e46fa7453
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+#
+# Functional test that boots a microblaze Linux kernel and checks the console
+#
+# Copyright (c) 2018, 2021 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import time
+from qemu_test import exec_command, exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+
+class MicroblazeMachine(QemuSystemTest):
+
+    timeout = 90
+
+    def test_microblaze_s3adsp1800(self):
+        self.set_machine('petalogix-s3adsp1800')
+        tar_url = ('https://qemu-advcal.gitlab.io'
+                   '/qac-best-of-multiarch/download/day17.tar.xz')
+        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        archive_extract(file_path, self.workdir)
+        self.vm.set_console()
+        self.vm.add_args('-kernel', self.workdir + '/day17/ballerina.bin')
+        self.vm.launch()
+        wait_for_console_pattern(self, 'This architecture does not have '
+                                       'kernel memory protection')
+        # Note:
+        # The kernel sometimes gets stuck after the "This architecture ..."
+        # message, that's why we don't test for a later string here. This
+        # needs some investigation by a microblaze wizard one day...
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_microblazeel_s3adsp1800.py new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.45.2
#28 lore raw

PATCH v2 15/23 tests/functional: Convert the riscv_opensbi avocado test into a standalone test

In reply to Thomas Huth

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

---
 tests/avocado/riscv_opensbi.py         | 63 --------------------------
 tests/functional/meson.build           |  8 ++++
 tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
 3 files changed, 44 insertions(+), 63 deletions(-)
 delete mode 100644 tests/avocado/riscv_opensbi.py
 create mode 100755 tests/functional/test_riscv_opensbi.py
tests/avocado/riscv_opensbi.py deleted +0 -63
deleted file mode 100644
index bfff9cc3c3..0000000000
@@ -1,63 +0,0 @@
-# OpenSBI boot test for RISC-V machines
-#
-# Copyright (c) 2022, Ventana Micro
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class RiscvOpenSBI(QemuSystemTest):
-    """
-    :avocado: tags=accel:tcg
-    """
-    timeout = 5
-
-    def boot_opensbi(self):
-        self.vm.set_console()
-        self.vm.launch()
-        wait_for_console_pattern(self, 'Platform Name')
-        wait_for_console_pattern(self, 'Boot HART MEDELEG')
-
-    def test_riscv32_spike(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_spike(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_virt(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_virt(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
tests/functional/meson.build +8 -0
index a2c0398b03..ebc6e2d1c6 100644
@@ -55,6 +55,14 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_riscv32_quick = [
+  'riscv_opensbi',
+]
+
+tests_riscv64_quick = [
+  'riscv_opensbi',
+]
+
 tests_s390x_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/test_riscv_opensbi.py new file +36 -0
new file mode 100755
index 0000000000..d077e40f42
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+#
+# OpenSBI boot test for RISC-V machines
+#
+# Copyright (c) 2022, Ventana Micro
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+
+class RiscvOpenSBI(QemuSystemTest):
+
+    timeout = 5
+
+    def boot_opensbi(self):
+        self.vm.set_console()
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Platform Name')
+        wait_for_console_pattern(self, 'Boot HART MEDELEG')
+
+    def test_riscv_spike(self):
+        self.set_machine('spike')
+        self.boot_opensbi()
+
+    def test_riscv_sifive_u(self):
+        self.set_machine('sifive_u')
+        self.boot_opensbi()
+
+    def test_riscv_virt(self):
+        self.set_machine('virt')
+        self.boot_opensbi()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v2 15/23
#29 lore raw

Re: [PATCH v2 15/23] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 7 lines · Thomas Huth

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Alistair

Quoted 142 lines
---
 tests/avocado/riscv_opensbi.py         | 63 --------------------------
 tests/functional/meson.build           |  8 ++++
 tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
 3 files changed, 44 insertions(+), 63 deletions(-)
 delete mode 100644 tests/avocado/riscv_opensbi.py
 create mode 100755 tests/functional/test_riscv_opensbi.py
tests/avocado/riscv_opensbi.py deleted +0 -63
deleted file mode 100644
index bfff9cc3c3..0000000000
@@ -1,63 +0,0 @@
-# OpenSBI boot test for RISC-V machines
-#
-# Copyright (c) 2022, Ventana Micro
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class RiscvOpenSBI(QemuSystemTest):
-    """
-    :avocado: tags=accel:tcg
-    """
-    timeout = 5
-
-    def boot_opensbi(self):
-        self.vm.set_console()
-        self.vm.launch()
-        wait_for_console_pattern(self, 'Platform Name')
-        wait_for_console_pattern(self, 'Boot HART MEDELEG')
-
-    def test_riscv32_spike(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_spike(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_virt(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_virt(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
tests/functional/meson.build +8 -0
index a2c0398b03..ebc6e2d1c6 100644
@@ -55,6 +55,14 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_riscv32_quick = [
+  'riscv_opensbi',
+]
+
+tests_riscv64_quick = [
+  'riscv_opensbi',
+]
+
 tests_s390x_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/test_riscv_opensbi.py new file +36 -1
new file mode 100755
index 0000000000..d077e40f42
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+#
+# OpenSBI boot test for RISC-V machines
+#
+# Copyright (c) 2022, Ventana Micro
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+
+class RiscvOpenSBI(QemuSystemTest):
+
+    timeout = 5
+
+    def boot_opensbi(self):
+        self.vm.set_console()
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Platform Name')
+        wait_for_console_pattern(self, 'Boot HART MEDELEG')
+
+    def test_riscv_spike(self):
+        self.set_machine('spike')
+        self.boot_opensbi()
+
+    def test_riscv_sifive_u(self):
+        self.set_machine('sifive_u')
+        self.boot_opensbi()
+
+    def test_riscv_virt(self):
+        self.set_machine('virt')
+        self.boot_opensbi()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
--

2.45.2

Reviewed-by Tested-by on v2 15/23
#30 lore raw

Re: [PATCH v2 15/23] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

In reply to Thomas Huth
Quoted 5 lines · Thomas Huth

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Quoted 8 lines
---
  tests/avocado/riscv_opensbi.py         | 63 --------------------------
  tests/functional/meson.build           |  8 ++++
  tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
  3 files changed, 44 insertions(+), 63 deletions(-)

delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

#31 lore raw

PATCH v2 16/23 tests/functional: Convert the virtio_gpu avocado test into a standalone test

In reply to Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                  |  1 +
 .../test_virtio_gpu.py}                       | 34 ++++++++-----------
 2 files changed, 16 insertions(+), 19 deletions(-)
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (88%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index ebc6e2d1c6..91fe835e8f 100644
@@ -82,6 +82,7 @@ tests_x86_64_quick = [
 
 tests_x86_64_thorough = [
   'netdev_ethtool',
+  'virtio_gpu',
 ]
 
 foreach speed : ['quick', 'thorough']
tests/functional/test_virtio_gpu.py renamed from tests/avocado/virtio-gpu.py +15 -19
old mode 100644
new mode 100755
similarity index 88%
rename from tests/avocado/virtio-gpu.py
rename to tests/functional/test_virtio_gpu.py
index 6091f614a4..911b4119c6
@@ -1,14 +1,16 @@
+#!/usr/bin/env python3
+#
 # virtio-gpu tests
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import BUILD_DIR
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import is_readable_executable_file
+from qemu_test import BUILD_DIR
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import is_readable_executable_file
 
 from qemu.utils import kvm_available
 
@@ -28,11 +30,6 @@ def pick_default_vug_bin():
 
 
 class VirtioGPUx86(QemuSystemTest):
-    """
-    :avocado: tags=virtio-gpu
-    :avocado: tags=arch:x86_64
-    :avocado: tags=cpu:host
-    """
 
     KERNEL_COMMAND_LINE = "printk.time=0 console=ttyS0 rdinit=/bin/bash"
     KERNEL_URL = (
@@ -57,9 +54,6 @@ def wait_for_console_pattern(self, success_message, vm=None):
         )
 
     def test_virtio_vga_virgl(self):
-        """
-        :avocado: tags=device:virtio-vga-gl
-        """
         # FIXME: should check presence of virtio, virgl etc
         self.require_accelerator('kvm')
 
@@ -67,6 +61,7 @@ def test_virtio_vga_virgl(self):
         initrd_path = self.fetch_asset(self.INITRD_URL, self.INITRD_HASH)
 
         self.vm.set_console()
+        self.vm.add_args("-cpu", "host")
         self.vm.add_args("-m", "2G")
         self.vm.add_args("-machine", "pc,accel=kvm")
         self.vm.add_args("-device", "virtio-vga-gl")
@@ -83,7 +78,7 @@ def test_virtio_vga_virgl(self):
             self.vm.launch()
         except:
             # TODO: probably fails because we are missing the VirGL features
-            self.cancel("VirGL not enabled?")
+            self.skipTest("VirGL not enabled?")
 
         self.wait_for_console_pattern("as init process")
         exec_command_and_wait_for_pattern(
@@ -92,15 +87,12 @@ def test_virtio_vga_virgl(self):
         self.wait_for_console_pattern("features: +virgl +edid")
 
     def test_vhost_user_vga_virgl(self):
-        """
-        :avocado: tags=device:vhost-user-vga
-        """
         # FIXME: should check presence of vhost-user-gpu, virgl, memfd etc
         self.require_accelerator('kvm')
 
         vug = pick_default_vug_bin()
         if not vug:
-            self.cancel("Could not find vhost-user-gpu")
+            self.skipTest("Could not find vhost-user-gpu")
 
         kernel_path = self.fetch_asset(self.KERNEL_URL, self.KERNEL_HASH)
         initrd_path = self.fetch_asset(self.INITRD_URL, self.INITRD_HASH)
@@ -129,6 +121,7 @@ def test_vhost_user_vga_virgl(self):
         )
 
         self.vm.set_console()
+        self.vm.add_args("-cpu", "host")
         self.vm.add_args("-m", "2G")
         self.vm.add_args("-object", "memory-backend-memfd,id=mem,size=2G")
         self.vm.add_args("-machine", "pc,memory-backend=mem,accel=kvm")
@@ -147,7 +140,7 @@ def test_vhost_user_vga_virgl(self):
             self.vm.launch()
         except:
             # TODO: probably fails because we are missing the VirGL features
-            self.cancel("VirGL not enabled?")
+            self.skipTest("VirGL not enabled?")
         self.wait_for_console_pattern("as init process")
         exec_command_and_wait_for_pattern(self, "/usr/sbin/modprobe virtio_gpu",
                                           "features: +virgl +edid")
@@ -155,3 +148,6 @@ def test_vhost_user_vga_virgl(self):
         qemu_sock.close()
         vugp.terminate()
         vugp.wait()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#32 lore raw

PATCH v2 17/23 tests/functional: Convert most ppc avocado tests into standalone tests

In reply to Thomas Huth

Nothing thrilling in here, just straight forward conversions.

---
 MAINTAINERS                                   |  8 ++--
 tests/functional/meson.build                  | 10 +++++
 .../test_ppc64_powernv.py}                    | 45 +++++--------------
 .../test_ppc64_pseries.py}                    | 45 +++++--------------
 .../test_ppc_40p.py}                          | 37 ++++++---------
 .../test_ppc_mpc8544ds.py}                    | 19 ++++----
 .../test_ppc_virtex_ml507.py}                 | 19 ++++----
 7 files changed, 71 insertions(+), 112 deletions(-)
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (80%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (83%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (78%)
 mode change 100644 => 100755
MAINTAINERS +4 -4
index 02cf39e94d..cec9a209a9 100644
@@ -1449,7 +1449,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/mpc8544ds.c
 F: hw/ppc/mpc8544_guts.c
-F: tests/avocado/ppc_mpc8544ds.py
+F: tests/functional/test_ppc_mpc8544ds.py
 
 New World (mac99)
 M: Mark Cave-Ayland <[email protected]>
@@ -1502,7 +1502,7 @@ F: hw/dma/i82374.c
 F: hw/rtc/m48t59-isa.c
 F: include/hw/isa/pc87312.h
 F: include/hw/rtc/m48t59.h
-F: tests/avocado/ppc_prep_40p.py
+F: tests/functional/test_ppc_40p.py
 
 sPAPR (pseries)
 M: Nicholas Piggin <[email protected]>
@@ -1526,7 +1526,7 @@ F: tests/qtest/spapr*
 F: tests/qtest/libqos/*spapr*
 F: tests/qtest/rtas*
 F: tests/qtest/libqos/rtas*
-F: tests/avocado/ppc_pseries.py
+F: tests/functional/test_ppc64_pseries.py
 F: tests/avocado/ppc_hv_tests.py
 
 PowerNV (Non-Virtualized)
@@ -1558,7 +1558,7 @@ M: Edgar E. Iglesias <[email protected]>
 L: [email protected]
 S: Odd Fixes
 F: hw/ppc/virtex_ml507.c
-F: tests/avocado/ppc_virtex_ml507.py
+F: tests/functional/test_ppc_virtex_ml507.py
 
 sam460ex
 M: BALATON Zoltan <[email protected]>
tests/functional/meson.build +10 -0
index 91fe835e8f..510dddfc49 100644
@@ -12,6 +12,8 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  'ppc_40p' : 240,
+  'ppc64_powernv' : 120,
   's390x_ccw_virtio' : 180,
 }
 
@@ -52,7 +54,15 @@ tests_ppc_quick = [
 
 tests_ppc_thorough = [
   'ppc_405',
+  'ppc_40p',
   'ppc_bamboo',
+  'ppc_mpc8544ds',
+  'ppc_virtex_ml507',
+]
+
+tests_ppc64_thorough = [
+  'ppc64_powernv',
+  'ppc64_pseries',
 ]
 
 tests_riscv32_quick = [
tests/functional/test_ppc64_powernv.py renamed from tests/avocado/ppc_powernv.py +12 -33

86 lines not on this page · raw message

tests/functional/test_ppc64_pseries.py renamed from tests/avocado/ppc_pseries.py +12 -33

92 lines not on this page · raw message

tests/functional/test_ppc_40p.py renamed from tests/avocado/ppc_prep_40p.py +13 -24
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/ppc_prep_40p.py
rename to tests/functional/test_ppc_40p.py
index d4f1eb7e1d..c3f713dc51
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a PReP/40p machine and checks its serial console.
 #
 # Copyright (c) Philippe Mathieu-Daudé <[email protected]>
@@ -7,9 +9,9 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 
 class IbmPrep40pMachine(QemuSystemTest):
@@ -23,13 +25,7 @@ class IbmPrep40pMachine(QemuSystemTest):
     # restricted by GSA ADP Schedule Contract with IBM Corp.
     @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_factory_firmware_and_netbsd(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
-        :avocado: tags=os:netbsd
-        :avocado: tags=slowness:high
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('40p')
         self.require_accelerator("tcg")
         bios_url = ('http://ftpmirror.your.org/pub/misc/'
                     'ftp.software.ibm.com/rs6000/firmware/'
@@ -50,11 +46,7 @@ def test_factory_firmware_and_netbsd(self):
         wait_for_console_pattern(self, 'Model: IBM PPS Model 6015')
 
     def test_openbios_192m(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('40p')
         self.require_accelerator("tcg")
         self.vm.set_console()
         self.vm.add_args('-m', '192') # test fw_cfg
@@ -65,21 +57,18 @@ def test_openbios_192m(self):
         wait_for_console_pattern(self, '>> CPU type PowerPC,604')
 
     def test_openbios_and_netbsd(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
-        :avocado: tags=os:netbsd
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('40p')
         self.require_accelerator("tcg")
         drive_url = ('https://archive.netbsd.org/pub/NetBSD-archive/'
                      'NetBSD-7.1.2/iso/NetBSD-7.1.2-prep.iso')
-        drive_hash = 'ac6fa2707d888b36d6fa64de6e7fe48e'
-        drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash,
-                                      algorithm='md5')
+        drive_hash = 'cc7cb290b06aaa839362deb7bd9f417ac5015557db24088508330f76c3f825ec'
+        drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
         self.vm.set_console()
         self.vm.add_args('-cdrom', drive_path,
                          '-boot', 'd')
 
         self.vm.launch()
         wait_for_console_pattern(self, 'NetBSD/prep BOOT, Revision 1.9')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_mpc8544ds.py renamed from tests/avocado/ppc_mpc8544ds.py +10 -9
old mode 100644
new mode 100755
similarity index 75%
rename from tests/avocado/ppc_mpc8544ds.py
rename to tests/functional/test_ppc_mpc8544ds.py
index b599fb1cc9..6817ed8091
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class Mpc8544dsMachine(QemuSystemTest):
 
@@ -16,19 +18,18 @@ class Mpc8544dsMachine(QemuSystemTest):
     panic_message = 'Kernel panic - not syncing'
 
     def test_ppc_mpc8544ds(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:mpc8544ds
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('mpc8544ds')
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day04.tar.xz')
         tar_hash = 'f46724d281a9f30fa892d458be7beb7d34dc25f9'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, self.workdir, member='creek/creek.bin')
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/creek/creek.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_virtex_ml507.py renamed from tests/avocado/ppc_virtex_ml507.py +10 -9
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/ppc_virtex_ml507.py
rename to tests/functional/test_ppc_virtex_ml507.py
index a73f8ae396..6d1f1455be
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
 
 class VirtexMl507Machine(QemuSystemTest):
 
@@ -16,17 +18,13 @@ class VirtexMl507Machine(QemuSystemTest):
     panic_message = 'Kernel panic - not syncing'
 
     def test_ppc_virtex_ml507(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:virtex-ml507
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('virtex-ml507')
         tar_url = ('https://qemu-advcal.gitlab.io'
                    '/qac-best-of-multiarch/download/day08.tar.xz')
         tar_hash = '74c68f5af7a7b8f21c03097b298f3bb77ff52c1f'
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/hippo/hippo.linux',
                          '-dtb', self.workdir + '/hippo/virtex440-ml507.dtb',
@@ -34,3 +32,6 @@ def test_ppc_virtex_ml507(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by on v2 17/23
#33 lore raw

Re: [PATCH v2 17/23] tests/functional: Convert most ppc avocado tests into standalone tests

Quoted 12 lines · Thomas Huth

Nothing thrilling in here, just straight forward conversions.

---
  MAINTAINERS                                   |  8 ++--
  tests/functional/meson.build                  | 10 +++++
  .../test_ppc64_powernv.py}                    | 45 +++++--------------
  .../test_ppc64_pseries.py}                    | 45 +++++--------------
  .../test_ppc_40p.py}                          | 37 ++++++---------
  .../test_ppc_mpc8544ds.py}                    | 19 ++++----
  .../test_ppc_virtex_ml507.py}                 | 19 ++++----
  7 files changed, 71 insertions(+), 112 deletions(-)
Quoted 10 lines

rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (80%) mode change 100644 => 100755
rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (83%) mode change 100644 => 100755
rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (78%) mode change 100644 => 100755
rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (75%) mode change 100644 => 100755
rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (78%) mode change 100644 => 100755

#34 lore raw

PATCH v2 18/23 tests/functional: Convert the ppc_amiga avocado test into a standalone test

In reply to Thomas Huth

Use the Python standard zipfile module instead of avocado.utils for extracting the ZIP file that we download here, and use the standard subprocess module for running the "tail" command.

---
 tests/functional/meson.build                  |  1 +
 .../test_ppc_amiga.py}                        | 33 ++++++++++---------
 2 files changed, 19 insertions(+), 15 deletions(-)
 rename tests/{avocado/ppc_amiga.py => functional/test_ppc_amiga.py} (54%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index 510dddfc49..74347cf545 100644
@@ -55,6 +55,7 @@ tests_ppc_quick = [
 tests_ppc_thorough = [
   'ppc_405',
   'ppc_40p',
+  'ppc_amiga',
   'ppc_bamboo',
   'ppc_mpc8544ds',
   'ppc_virtex_ml507',
tests/functional/test_ppc_amiga.py renamed from tests/avocado/ppc_amiga.py +18 -15
old mode 100644
new mode 100755
similarity index 54%
rename from tests/avocado/ppc_amiga.py
rename to tests/functional/test_ppc_amiga.py
index b6f866f91d..6fdb334346
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test AmigaNG boards
 #
 # Copyright (c) 2023 BALATON Zoltan
@@ -5,34 +7,35 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado.utils import process
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+import subprocess
+
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern, run_cmd
+from zipfile import ZipFile
 
 class AmigaOneMachine(QemuSystemTest):
 
     timeout = 90
 
     def test_ppc_amigaone(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:amigaone
-        :avocado: tags=device:articia
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('amigaone')
         tar_name = 'A1Firmware_Floppy_05-Mar-2005.zip'
         tar_url = ('https://www.hyperion-entertainment.com/index.php/'
                    'downloads?view=download&format=raw&file=25')
         tar_hash = 'c52e59bc73e31d8bcc3cc2106778f7ac84f6c755'
-        zip_file = self.fetch_asset(tar_name, locations=tar_url,
-                                    asset_hash=tar_hash)
-        archive.extract(zip_file, self.workdir)
-        cmd = f"tail -c 524288 {self.workdir}/floppy_edition/updater.image >{self.workdir}/u-boot-amigaone.bin"
-        process.run(cmd, shell=True)
+        zip_file = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        with ZipFile(zip_file, 'r') as zf:
+            zf.extractall(path=self.workdir)
+        bios_fh = open(self.workdir + "/u-boot-amigaone.bin", "wb")
+        subprocess.run(['tail', '-c', '524288',
+                        self.workdir + "/floppy_edition/updater.image"],
+                        stdout=bios_fh)
 
         self.vm.set_console()
         self.vm.add_args('-bios', self.workdir + '/u-boot-amigaone.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'FLASH:')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#35 lore raw

PATCH v2 19/23 tests/functional: Convert the ppc_hv avocado test into a standalone test

In reply to Thomas Huth

Note: The original Avocado test seems currently to be broken, it hangs when the guest is trying to install additional packages. So mark it as broken for now until it gets fixed.

---
 .../test_ppc64_hv.py}                         | 36 ++++++++-----------
 1 file changed, 14 insertions(+), 22 deletions(-)
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (92%)
 mode change 100644 => 100755
tests/functional/test_ppc64_hv.py renamed from tests/avocado/ppc_hv_tests.py +14 -22
old mode 100644
new mode 100755
similarity index 92%
rename from tests/avocado/ppc_hv_tests.py
rename to tests/functional/test_ppc64_hv.py
index bf8822bb97..acba3e01b9
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Tests that specifically try to exercise hypervisor features of the
 # target machines. powernv supports the Power hypervisor ISA, and
 # pseries supports the nested-HV hypervisor spec.
@@ -7,10 +9,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado import skipIf, skipUnless
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern, exec_command
+from unittest import skipIf, skipUnless
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern, exec_command
 import os
 import time
 import subprocess
@@ -47,7 +49,7 @@ def missing_deps():
 @skipIf(missing_deps(), 'dependencies (%s) not installed' % ','.join(deps))
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck due to console handling problem')
 @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
-@skipUnless(os.getenv('SPEED') == 'slow', 'runtime limited')
+@skip('broken test - guest fails to install packages')
 class HypervisorTest(QemuSystemTest):
 
     timeout = 1000
@@ -88,8 +90,7 @@ def setUp(self):
 
         # Alpine use sha256 so I recalculated this myself
         iso_sha256 = 'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha256,
-                                    algorithm = "sha256")
+        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha256)
 
         self.iso_path = iso_path
         self.vmlinuz = self.extract_from_iso(iso_path, '/boot/vmlinuz-lts')
@@ -159,12 +160,8 @@ def do_test_kvm(self, hpt=False):
         wait_for_console_pattern(self, 'alpine:~#')
 
     def test_hv_pseries(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on")
@@ -174,12 +171,8 @@ def test_hv_pseries(self):
         self.do_stop_alpine()
 
     def test_hv_pseries_kvm(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:kvm
-        """
         self.require_accelerator("kvm")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "kvm")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on,cap-ccf-assist=off")
@@ -189,12 +182,8 @@ def test_hv_pseries_kvm(self):
         self.do_stop_alpine()
 
     def test_hv_powernv(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:powernv
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('powernv')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,bus=pcie.2,addr=0x0,serial=1234,drive=drive0',
                          '-device', 'e1000e,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pcie.0,addr=0x0',
@@ -204,3 +193,6 @@ def test_hv_powernv(self):
         self.do_test_kvm()
         self.do_test_kvm(True)
         self.do_stop_alpine()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#36 lore raw

PATCH v2 20/23 tests/functional: Convert the m68k nextcube test with tesseract

In reply to Thomas Huth

The code that handles running of tesseract needs to be tweaked a little bit to be able to run without the functions from avocado.utils, and while we're at it, drop some legacy stuff that was still there due to Tesseract 3 support that we already dropped a while ago.

---
 tests/avocado/tesseract_utils.py              | 46 -------------------
 tests/functional/meson.build                  |  4 ++
 tests/functional/qemu_test/tesseract.py       | 35 ++++++++++++++
 .../test_m68k_nextcube.py}                    | 20 ++++----
 4 files changed, 50 insertions(+), 55 deletions(-)
 delete mode 100644 tests/avocado/tesseract_utils.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (86%)
 mode change 100644 => 100755
tests/avocado/tesseract_utils.py deleted +0 -46
deleted file mode 100644
index 476f528147..0000000000
@@ -1,46 +0,0 @@
-# ...
-#
-# Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later. See the COPYING file in the top-level directory.
-
-import re
-import logging
-
-from avocado.utils import process
-from avocado.utils.path import find_command, CmdNotFoundError
-
-def tesseract_available(expected_version):
-    try:
-        find_command('tesseract')
-    except CmdNotFoundError:
-        return False
-    res = process.run('tesseract --version')
-    try:
-        version = res.stdout_text.split()[1]
-    except IndexError:
-        version = res.stderr_text.split()[1]
-    return int(version.split('.')[0]) >= expected_version
-
-    match = re.match(r'tesseract\s(\d)', res)
-    if match is None:
-        return False
-    # now this is guaranteed to be a digit
-    return int(match.groups()[0]) >= expected_version
-
-
-def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
-    console_logger = logging.getLogger('tesseract')
-    console_logger.debug(image_path)
-    if tesseract_version == 4:
-        tesseract_args += ' --oem 1'
-    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
-                                                       image_path))
-    lines = []
-    for line in proc.stdout_text.split('\n'):
-        sline = line.strip()
-        if len(sline):
-            console_logger.debug(sline)
-            lines += [sline]
-    return lines
tests/functional/meson.build +4 -0
index 74347cf545..109e0408e4 100644
@@ -36,6 +36,10 @@ tests_loongarch64_thorough = [
   'loongarch64_virt',
 ]
 
+tests_m68k_thorough = [
+  'm68k_nextcube'
+]
+
 tests_microblaze_thorough = [
   'microblaze_s3adsp1800'
 ]
tests/functional/qemu_test/tesseract.py new file +35 -0
new file mode 100644
index 0000000000..c4087b7c11
@@ -0,0 +1,35 @@
+# ...
+#
+# Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import re
+import logging
+
+from . import has_cmd, run_cmd
+
+def tesseract_available(expected_version):
+    if not has_cmd('tesseract'):
+        return False
+    (stdout, stderr, ret) = run_cmd([ 'tesseract', '--version'])
+    if ret:
+        return False
+    version = stdout.split()[1]
+    return int(version.split('.')[0]) >= expected_version
+
+def tesseract_ocr(image_path, tesseract_args=''):
+    console_logger = logging.getLogger('console')
+    console_logger.debug(image_path)
+    (stdout, stderr, ret) = run_cmd(['tesseract', image_path,
+                                     'stdout'])
+    if ret:
+        return None
+    lines = []
+    for line in stdout.split('\n'):
+        sline = line.strip()
+        if len(sline):
+            console_logger.debug(sline)
+            lines += [sline]
+    return lines
tests/functional/test_m68k_nextcube.py renamed from tests/avocado/machine_m68k_nextcube.py +11 -9
old mode 100644
new mode 100755
similarity index 86%
rename from tests/avocado/machine_m68k_nextcube.py
rename to tests/functional/test_m68k_nextcube.py
index 1f3c883910..4595032af5
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a VM and run OCR on the framebuffer
 #
 # Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
@@ -8,10 +10,10 @@
 import os
 import time
 
-from avocado_qemu import QemuSystemTest
-from avocado import skipUnless
+from qemu_test import QemuSystemTest
+from unittest import skipUnless
 
-from tesseract_utils import tesseract_available, tesseract_ocr
+from qemu_test.tesseract import tesseract_available, tesseract_ocr
 
 PIL_AVAILABLE = True
 try:
@@ -21,11 +23,6 @@
 
 
 class NextCubeMachine(QemuSystemTest):
-    """
-    :avocado: tags=arch:m68k
-    :avocado: tags=machine:next-cube
-    :avocado: tags=device:framebuffer
-    """
 
     timeout = 15
 
@@ -48,6 +45,7 @@ def check_bootrom_framebuffer(self, screenshot_path):
 
     @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
     def test_bootrom_framebuffer_size(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
@@ -60,11 +58,15 @@ def test_bootrom_framebuffer_size(self):
     # that it is still alpha-level software.
     @skipUnless(tesseract_available(4), 'tesseract OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
-        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
+        lines = tesseract_ocr(screenshot_path)
         text = '\n'.join(lines)
         self.assertIn('Testing the FPU', text)
         self.assertIn('System test failed. Error code', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#37 lore raw

PATCH v2 21/23 tests/functional: Convert the acpi-bits test into a standalone test

In reply to Thomas Huth

Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore.

---
 .../acpi-bits/bits-config/bits-cfg.txt        |  0
 .../acpi-bits/bits-tests/smbios.py2           |  0
 .../acpi-bits/bits-tests/smilatency.py2       |  0
 .../acpi-bits/bits-tests/testacpi.py2         |  0
 .../acpi-bits/bits-tests/testcpuid.py2        |  0
 tests/functional/meson.build                  |  2 ++
 .../test_acpi_bits.py}                        | 29 ++++++++-----------
 7 files changed, 14 insertions(+), 17 deletions(-)
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (95%)
 mode change 100644 => 100755
tests/functional/acpi-bits/bits-config/bits-cfg.txt renamed from tests/avocado/acpi-bits/bits-config/bits-cfg.txt +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-config/bits-cfg.txt
rename to tests/functional/acpi-bits/bits-config/bits-cfg.txt
tests/functional/acpi-bits/bits-tests/smbios.py2 renamed from tests/avocado/acpi-bits/bits-tests/smbios.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smbios.py2
rename to tests/functional/acpi-bits/bits-tests/smbios.py2
tests/functional/acpi-bits/bits-tests/smilatency.py2 renamed from tests/avocado/acpi-bits/bits-tests/smilatency.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smilatency.py2
rename to tests/functional/acpi-bits/bits-tests/smilatency.py2
tests/functional/acpi-bits/bits-tests/testacpi.py2 renamed from tests/avocado/acpi-bits/bits-tests/testacpi.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testacpi.py2
rename to tests/functional/acpi-bits/bits-tests/testacpi.py2
tests/functional/acpi-bits/bits-tests/testcpuid.py2 renamed from tests/avocado/acpi-bits/bits-tests/testcpuid.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testcpuid.py2
rename to tests/functional/acpi-bits/bits-tests/testcpuid.py2
tests/functional/meson.build +2 -0
index 109e0408e4..52328f44b6 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'acpi_bits' : 240,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
   'ppc64_powernv' : 120,
@@ -96,6 +97,7 @@ tests_x86_64_quick = [
 ]
 
 tests_x86_64_thorough = [
+  'acpi_bits',
   'netdev_ethtool',
   'virtio_gpu',
 ]
tests/functional/test_acpi_bits.py renamed from tests/avocado/acpi-bits.py +12 -17
old mode 100644
new mode 100755
similarity index 95%
rename from tests/avocado/acpi-bits.py
rename to tests/functional/test_acpi_bits.py
index efe4f52ee0..697fa4e768
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# group: rw quick
+#
 # Exercise QEMU generated ACPI/SMBIOS tables using biosbits,
 # https://biosbits.org/
 #
@@ -41,15 +41,16 @@
 import tempfile
 import time
 import zipfile
+
+from pathlib import Path
 from typing import (
     List,
     Optional,
     Sequence,
 )
 from qemu.machine import QEMUMachine
-from avocado import skipIf
-from avocado.utils import datadrainer as drainer
-from avocado_qemu import QemuBaseTest
+from unittest import skipIf
+from qemu_test import QemuBaseTest
 
 deps = ["xorriso", "mformat"] # dependent tools needed in the test setup/box.
 supported_platforms = ['x86_64'] # supported test platforms.
@@ -129,10 +130,6 @@ def base_args(self):
 class AcpiBitsTest(QemuBaseTest): #pylint: disable=too-many-instance-attributes
     """
     ACPI and SMBIOS tests using biosbits.
-
-    :avocado: tags=arch:x86_64
-    :avocado: tags=acpi
-
     """
     # in slower systems the test can take as long as 3 minutes to complete.
     timeout = BITS_TIMEOUT
@@ -155,8 +152,7 @@ def __init__(self, *args, **kwargs):
                             "download?job=qemu-bits-build" %self._bitsTag)
         self._debugcon_addr = '0x403'
         self._debugcon_log = 'debugcon-log.txt'
-        logging.basicConfig(level=logging.INFO)
-        self.logger = logging.getLogger('acpi-bits')
+        self.logger = self.log
 
     def _print_log(self, log):
         self.logger.info('\nlogs from biosbits follows:')
@@ -282,8 +278,9 @@ def generate_bits_iso(self):
 
     def setUp(self): # pylint: disable=arguments-differ
         super().setUp('qemu-system-')
+        self.logger = self.log
 
-        self._baseDir = os.getenv('AVOCADO_TEST_BASEDIR')
+        self._baseDir = Path(__file__).parent
 
         # workdir could also be avocado's own workdir in self.workdir.
         # At present, I prefer to maintain my own temporary working
@@ -393,12 +390,6 @@ def test_acpi_smbios_bits(self):
         self._vm.set_console()
         self._vm.launch()
 
-        self.logger.debug("Console output from bits VM follows ...")
-        c_drainer = drainer.LineLogger(self._vm.console_socket.fileno(),
-                                       logger=self.logger.getChild("console"),
-                                       stop_check=(lambda :
-                                                   not self._vm.is_running()))
-        c_drainer.start()
 
         # biosbits has been configured to run all the specified test suites
         # in batch mode and then automatically initiate a vm shutdown.
@@ -406,4 +397,8 @@ def test_acpi_smbios_bits(self):
         # with the avocado test timeout.
         self._vm.event_wait('SHUTDOWN', timeout=BITS_TIMEOUT)
         self._vm.wait(timeout=None)
+        self.logger.debug("Checking console output ...")
         self.parse_log()
+
+if __name__ == '__main__':
+    QemuBaseTest.main()
Signature · 1 line
2.45.2
#38 lore raw

PATCH v2 22/23 tests/functional: Convert the rx_gdbsim avocado test into a standalone test

In reply to Thomas Huth

Provide a "gzip_uncompress" function based on the standard "gzip" module to avoid the usage of avocado.utils here.

---
Note: Make sure to apply this fix first before running this test:
https://lore.kernel.org/qemu-devel/[email protected]/

 tests/functional/meson.build                  |  4 +++
 tests/functional/qemu_test/utils.py           | 12 +++++++
 .../test_rx_gdbsim.py}                        | 34 +++++++++----------
 3 files changed, 33 insertions(+), 17 deletions(-)
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (78%)
 mode change 100644 => 100755
tests/functional/meson.build +4 -0
index 52328f44b6..3ae169897f 100644
@@ -79,6 +79,10 @@ tests_riscv64_quick = [
   'riscv_opensbi',
 ]
 
+tests_rx_thorough = [
+  'rx_gdbsim',
+]
+
 tests_s390x_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/qemu_test/utils.py +12 -0
index a12dac51b6..99eae5fc45 100644
@@ -8,6 +8,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import gzip
 import lzma
 import os
 import shutil
@@ -23,6 +24,17 @@ def archive_extract(archive, dest_dir, member=None):
         else:
             tf.extractall(path=dest_dir)
 
+def gzip_uncompress(gz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with gzip.open(gz_path, 'rb') as gz_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(gz_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
+
 def lzma_uncompress(xz_path, output_path):
     if os.path.exists(output_path):
         return
tests/functional/test_rx_gdbsim.py renamed from tests/avocado/machine_rx_gdbsim.py +17 -17
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/machine_rx_gdbsim.py
rename to tests/functional/test_rx_gdbsim.py
index 412a7a5089..ebb8bd5631
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -10,11 +12,11 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import gzip_uncompress
 
 
 class RxGdbSimMachine(QemuSystemTest):
@@ -27,16 +29,14 @@ class RxGdbSimMachine(QemuSystemTest):
     def test_uboot(self):
         """
         U-Boot and checks that the console is operational.
-
-        :avocado: tags=arch:rx
-        :avocado: tags=machine:gdbsim-r5f562n8
-        :avocado: tags=endian:little
-        :avocado: tags=flaky
         """
+        self.set_machine('gdbsim-r5f562n8')
+
         uboot_url = ('https://acc.dl.osdn.jp/users/23/23888/u-boot.bin.gz')
         uboot_hash = '9b78dbd43b40b2526848c0b1ce9de02c24f4dcdb'
-        uboot_path = self.fetch_asset(uboot_url, asset_hash=uboot_hash)
-        uboot_path = archive.uncompress(uboot_path, self.workdir)
+        uboot_path_gz = self.fetch_asset(uboot_url, asset_hash=uboot_hash)
+        uboot_path = os.path.join(self.workdir, 'u-boot.bin')
+        gzip_uncompress(uboot_path_gz, uboot_path)
 
         self.vm.set_console()
         self.vm.add_args('-bios', uboot_path,
@@ -53,12 +53,9 @@ def test_uboot(self):
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
-
-        :avocado: tags=arch:rx
-        :avocado: tags=machine:gdbsim-r5f562n7
-        :avocado: tags=endian:little
-        :avocado: tags=flaky
         """
+        self.set_machine('gdbsim-r5f562n7')
+
         dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
         dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
         dtb_path = self.fetch_asset(dtb_url, asset_hash=dtb_hash)
@@ -75,3 +72,6 @@ def test_linux_sash(self):
         wait_for_console_pattern(self, 'Sash command shell (version 1.1.1)',
                                  failure_message='Kernel panic - not syncing')
         exec_command_and_wait_for_pattern(self, 'printenv', 'TERM=linux')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#39 lore raw

PATCH v2 23/23 gitlab-ci: Add "check-functional" to the build tests

In reply to Thomas Huth

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

---
TODO: We should also make sure of cachine the downloaded assets here...

 .gitlab-ci.d/buildtest-template.yml |  3 +-
 .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
 2 files changed, 32 insertions(+), 31 deletions(-)
.gitlab-ci.d/buildtest-template.yml +2 -1
index 8f7ebfaed8..54cae74a8e 100644
@@ -71,7 +71,7 @@
     reports:
       junit: build/meson-logs/testlog.junit.xml
 
-.avocado_test_job_template:
+.functional_test_job_template:
   extends: .common_test_job_template
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -98,6 +98,7 @@
         du -chs ${CI_PROJECT_DIR}/avocado-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+    - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
.gitlab-ci.d/buildtest.yml +30 -30

171 lines not on this page · raw message

Signature · 1 line
2.45.2
#40 lore raw
In reply to Thomas Huth
Quoted 52 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central Brisbane, and have had multiple downloads be canceled by the timeout. Since the download isn't clever enough to pick up where it left off, it will never succeed.

r~

#41 lore raw
Quoted 59 lines · Richard Henderson
Quoted 52 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

Quoted 7 lines

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central Brisbane, and have had multiple downloads be canceled by the timeout. Since the download isn't clever enough to pick up where it left off, it will never succeed.

This is a tricky problem the way the tests are currently written, given the desire for a minimal-change from the old avocado impl.

IIUC, avocado already had a per-test timeout, so would suffer the same problem with downloads exploding the "normal" running time when cached.

To address this we'll need a refactoring to enable us to declare the required "assets" externally from the test code.

Taking one simple example

class LinuxInitrd(QemuSystemTest):

      def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):

          kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora/li'
                        'nux/releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz')
          kernel_hash = '41464f68efe42b9991250bed86c7081d2ccdbb21'
          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
          ...snip...

if __name__ == '__main__':
QemuSystemTest.main()

Consider if we declared all required assets as class level variable

class LinuxInitrd(QemuSystemTest):

      ASSETS = {
         "fedora18": {
	    "url": ('https://archives.fedoraproject.org/pub/archive/fedora/li'
                     'nux/releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz'),
            "hash": "'41464f68efe42b9991250bed86c7081d2ccdbb21'"
	 }
      }

Then, we change the 'fetch_asset' method to take an asset name, not a URL+hash:

def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
kernel_path = self.fetch_asset("fedora18")

Now, 'fetch_asset' would lookup the URL + hash in the self.__class__.ASSETS dict, so the test would run exactly as before.

Finally, we modify QemuSystemTest.main() so that knows to look for a '--fetch-assets' parameter in sys.argv. When it see --fetch-assets, instead of running each test, it should download everything found in the ASSETS class variables.

This now gives us the ability to run a separate '--fetch-assets' invokation with elevated timeout, while runing tests with a normal timeout.

This is all a non-trivial amount of work though, so I don't think it is reasonable todo this as part of the immediate conversion in this series.

The only short term option is to configure meson run tests with a massively larger timeout, until we're able to enable some pre-caching mechansim.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#42 lore raw
Quoted 68 lines · Daniel P. Berrangé
Quoted 59 lines · Richard Henderson
Quoted 52 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 export QEMU_BUILD_ROOT=$HOME/qemu/build ~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

Quoted 7 lines

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central Brisbane, and have had multiple downloads be canceled by the timeout. Since the download isn't clever enough to pick up where it left off, it will never succeed.

Quoted 5 lines

This is a tricky problem the way the tests are currently written, given the desire for a minimal-change from the old avocado impl.

IIUC, avocado already had a per-test timeout, so would suffer the same problem with downloads exploding the "normal" running time when cached.

Avocado runs a first pass doing all of the downloads, and only afterward runs the actual timed tests. I don't know the specifics of how, but it certainly obvious in the logging.

Quoted 12 lines

Consider if we declared all required assets as class level variable

   class LinuxInitrd(QemuSystemTest):

       ASSETS = {
          "fedora18": {
	    "url": ('https://archives.fedoraproject.org/pub/archive/fedora/li'
                      'nux/releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz'),
             "hash": "'41464f68efe42b9991250bed86c7081d2ccdbb21'"
	 }
       }
Quoted 8 lines

Then, we change the 'fetch_asset' method to take an asset name, not a URL+hash:

def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
kernel_path = self.fetch_asset("fedora18")

Now, 'fetch_asset' would lookup the URL + hash in the self.__class__.ASSETS dict, so the test would run exactly as before.

Sure, that's one possibility.

Quoted 3 lines

This is all a non-trivial amount of work though, so I don't think it is reasonable todo this as part of the immediate conversion in this series.

I think that if we *don't* do this, then we cannot run in CI *at all* because we will be plagued with false timeouts.

And, frankly, any developer more than a few time zones away from the hosting of the asset will be continually frustrated.

I repeat: this is a show-stopper.

r~

#43 lore raw
Quoted 72 lines · Richard Henderson
Quoted 71 lines · Daniel P. Berrangé
Quoted 59 lines · Richard Henderson
Quoted 52 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 export QEMU_BUILD_ROOT=$HOME/qemu/build ~/qemu/tests/functional/test_virtio_version.py

Quoted 5 lines

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

Quoted 7 lines

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central Brisbane, and have had multiple downloads be canceled by the timeout. Since the download isn't clever enough to pick up where it left off, it will never succeed.

This is a tricky problem the way the tests are currently written, given the desire for a minimal-change from the old avocado impl.

Quoted 6 lines
Quoted 2 lines

IIUC, avocado already had a per-test timeout, so would suffer the same problem with downloads exploding the "normal" running time when cached.

Avocado runs a first pass doing all of the downloads, and only afterward runs the actual timed tests. I don't know the specifics of how, but it certainly obvious in the logging.

Oh interesting, I found how it does it..

The file avocado/plugins/assets.py will build an AST of the python code in a test file, look for all 'fetch_asset' calls, then extract the parameters to these calls, and donwload them. This is clever. Basically avoids the refactoring that I suggested.

So yeah, that is a gap.

Practically speaking, we have a choice of either calling into this avocado python lib as is, or copying tthat python lib into QEMU.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#44 lore raw
Quoted 3 lines · Daniel P. Berrangé
Quoted 2 lines · Richard Henderson
Quoted 1 line · Daniel P. Berrangé

On Thu, Jul 25, 2024 at 09:35:22AM +1000, Richard Henderson wrote:

...

Quoted 11 lines
Quoted 3 lines

Avocado runs a first pass doing all of the downloads, and only afterward runs the actual timed tests. I don't know the specifics of how, but it certainly obvious in the logging.

Oh interesting, I found how it does it..

The file avocado/plugins/assets.py will build an AST of the python code in a test file, look for all 'fetch_asset' calls, then extract the parameters to these calls, and donwload them. This is clever. Basically avoids the refactoring that I suggested.

Quoted 4 lines

So yeah, that is a gap.

Practically speaking, we have a choice of either calling into this avocado python lib as is, or copying tthat python lib into QEMU.

Honestly, I'd prefer to do some refactoring instead, something like you suggested in your earlier mail. Rationale: For the basic tests it would be good if we would not depend on the Avacodo framework anymore, otherwise we likely will continue to run into the situation that our test framework stops working on some random new python versions and nobody within the QEMU community has a clue how to fix the situation since nobody is really familiar with the Avocado framework. Also, while that avocado/plugins/assets.py sounds like a very neat trick done by a skilled Python wizard, the average QEMU developer (like me) is just a skilled C coder with only basic Python knowledge, so I'd prefer if we could use a simpler mechanism instead that is easier to understand and to debug for everybody once we run into problems with it.

Thus, I'd suggest to bite the bullet and refactor the tests that download assets. I can look into this after my summer vacation - but if somebody feels interested and wants to look into this during the next two weeks already, that would be very welcome, too, of course!

Thomas

#45 lore raw
Quoted 22 lines · Thomas Huth
Quoted 3 lines · Daniel P. Berrangé
Quoted 2 lines · Richard Henderson
Quoted 1 line · Daniel P. Berrangé

On Thu, Jul 25, 2024 at 09:35:22AM +1000, Richard Henderson wrote:

...

Quoted 11 lines
Quoted 3 lines

Avocado runs a first pass doing all of the downloads, and only afterward runs the actual timed tests. I don't know the specifics of how, but it certainly obvious in the logging.

Oh interesting, I found how it does it..

The file avocado/plugins/assets.py will build an AST of the python code in a test file, look for all 'fetch_asset' calls, then extract the parameters to these calls, and donwload them. This is clever. Basically avoids the refactoring that I suggested.

Quoted 4 lines

So yeah, that is a gap.

Practically speaking, we have a choice of either calling into this avocado python lib as is, or copying tthat python lib into QEMU.

Quoted 13 lines

Honestly, I'd prefer to do some refactoring instead, something like you suggested in your earlier mail. Rationale: For the basic tests it would be good if we would not depend on the Avacodo framework anymore, otherwise we likely will continue to run into the situation that our test framework stops working on some random new python versions and nobody within the QEMU community has a clue how to fix the situation since nobody is really familiar with the Avocado framework. Also, while that avocado/plugins/assets.py sounds like a very neat trick done by a skilled Python wizard, the average QEMU developer (like me) is just a skilled C coder with only basic Python knowledge, so I'd prefer if we could use a simpler mechanism instead that is easier to understand and to debug for everybody once we run into problems with it.

Hi Thomas,

That wizardry is indeed not nice, and has limitations. It was replaced in recent Avocado versions for the dependencies mechanism:

https://avocado-framework.readthedocs.io/en/103.0/guides/user/chapters/dependencies.html

Specifically for the assets (downloadable files), you can find the documentation here:

https://avocado-framework.readthedocs.io/en/103.0/guides/user/chapters/dependencies.html#asset

Those are superior to the previous implementation because they compute a dependency graph that works on the resolution while tests with dependencies met (or no deps) start running right away.

Regards,
- Cleber.

#46 lore raw
In reply to Richard Henderson
Quoted 57 lines · Richard Henderson
Quoted 56 lines · Thomas Huth

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

  make check-functional

You can also run single targets e.g. with:

  make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

  export PYTHONPATH=$HOME/qemu/python
  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
  export QEMU_BUILD_ROOT=$HOME/qemu/build
  ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Quoted 5 lines
Quoted 1 line

Still to be done: Update the documentation for this new test framework.

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central Brisbane, and have had multiple downloads be canceled by the timeout.  Since the download isn't clever enough to pick up where it left off, it will never succeed.

Hi Richard,

just for my understanding, did you try to run the tests in parallel (i.e. something like "make -j$(nproc)")? ... I think in that case you can easily clog your internet connection even on modern systems if a lot of tests are trying to download the assets in parallel.

For me, it works fine if I use normal serial testing with "-j" (btw. Avocado v88 is doing serial testing, too, so you won't lose much time during the first run here). But if downloading fails for you without "-j", too, I agree, we need to tackle that problem first, e.g. by implementing what Daniel suggested. That will take a little bit longer, of course, so I hope you meanwhile found a work-around for the problem with the missing "imp" package on your system?

Thomas

#47 lore raw
Quoted 3 lines · Thomas Huth

 Hi Richard,

just for my understanding, did you try to run the tests in parallel (i.e. something like "make -j$(nproc)")?

No, I ran "make check-functional" with zero parallelism.

Quoted 1 line

For me, it works fine if I use normal serial testing with "-j" (btw. Avocado v88 is doing serial testing, too, so you won't lose much time during the first run here). But if downloading fails for you without "-j", too, I agree, we need to tackle that problem first, e.g. by implementing what Daniel suggested. That will take a little bit longer, of course, so I hope you meanwhile found a work-around for the problem with the missing "imp" package on your system?

Not so far. I'm using VMs for avocado testing at present.

r~

#48 lore raw
In reply to Richard Henderson
Quoted 3 lines · Richard Henderson

On 7/25/24 03:52, Thomas Huth wrote:

I'll say again that the download *must* be handled separately from the test with timeout. This is an absolute show-stopper.

Queuing patches not related to assets downloading
(1-6, 13, 15 and 23).