Skip to contentSkip to conversation

Convert avocado tests to normal Python unittests

PATCH In review Daniel P. Berrangé 6 participants 69 messages Jul 30, 2024 – Aug 13, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover Convert avocado tests to normal Python unittests

This series is an update to Thomas' v2:

https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg05805.html

wherein Thomas suggested someone could do the asset caching updates while he is on vacation:

https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg06228.html

hence this posting what I'm calling a v3.

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

v3:
- Split tests/functional/__init__.py into multiple files
- Introduce an 'Asset' class to handle downloading & caching
- Convert all tests to use new Asset class instead of 'fetch_asset'
- Ensure 'make check-functional' pre-caches all assets before running the tests, to avoid meson test timeouts due to slow downloads
- Fix bios bits test logging in verbose mode to not print subprocess output to stdout, since it confuses TAP parsing

v2:
- Addressed review feedback from v1
- Add pycotap as a wheel instead of trying to install it on demand when running "make check-functional" (works much better now!)
- Converted much more tests
- Lots of other small improvements here and there

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Daniel P. Berrangé (2):
tests/functional: add a module for handling asset download & caching tests/functional: enable pre-emptive caching of assets

Thomas Huth (22):
python: Install pycotap in our venv if necessary tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Set up logging
tests/Makefile.include: Increase the level of indentation in the help
text
tests/functional: Prepare the meson build system for the functional
tests
tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Convert the s390x avocado tests into standalone
tests
tests/functional: Convert the x86_cpu_model_versions test tests/functional: Convert the microblaze avocado tests into standalone
tests
tests/functional: Convert the riscv_opensbi avocado test into a
standalone test
tests/functional: Convert the virtio_gpu avocado test into a
standalone test
tests/functional: Convert most ppc avocado tests into standalone tests tests/functional: Convert the ppc_amiga avocado test into a standalone
test
tests/functional: Convert the ppc_hv avocado test into a standalone
test
tests/functional: Convert the m68k nextcube test with tesseract tests/functional: Convert the acpi-bits test into a standalone test tests/functional: Convert the rx_gdbsim avocado test into a standalone
test
gitlab-ci: Add "check-functional" to the build tests

 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +++---
 MAINTAINERS                                   |  32 +--
 python/wheels/pycotap-1.3.1-py3-none-any.whl  | Bin 0 -> 5119 bytes
 pythondeps.toml                               |   1 +
 tests/Makefile.include                        |  42 ++--
 tests/avocado/machine_microblaze.py           |  61 ------
 tests/avocado/machine_mips_loongson3v.py      |  39 ----
 tests/avocado/machine_sparc64_sun4u.py        |  36 ----
 tests/avocado/ppc_amiga.py                    |  38 ----
 tests/avocado/riscv_opensbi.py                |  63 ------
 tests/avocado/tesseract_utils.py              |  46 -----
 .../acpi-bits/bits-config/bits-cfg.txt        |   0
 .../acpi-bits/bits-tests/smbios.py2           |   0
 .../acpi-bits/bits-tests/smilatency.py2       |   0
 .../acpi-bits/bits-tests/testacpi.py2         |   0
 .../acpi-bits/bits-tests/testcpuid.py2        |   0
 tests/functional/meson.build                  | 180 +++++++++++++++++
 tests/functional/qemu_test/__init__.py        |  14 ++
 tests/functional/qemu_test/asset.py           | 130 ++++++++++++
 tests/functional/qemu_test/cmd.py             | 171 ++++++++++++++++
 tests/functional/qemu_test/config.py          |  36 ++++
 tests/functional/qemu_test/tesseract.py       |  35 ++++
 tests/functional/qemu_test/testcase.py        | 187 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  47 +++++
 .../test_acpi_bits.py}                        |  81 ++++----
 .../test_arm_canona1100.py}                   |  30 +--
 .../test_arm_n8x0.py}                         |  36 ++--
 .../test_avr_mega2560.py}                     |  22 ++-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  46 +++--
 .../test_m68k_nextcube.py}                    |  29 +--
 .../test_mem_addr_space.py}                   |  52 +----
 .../functional/test_microblaze_s3adsp1800.py  |  39 ++++
 .../test_microblazeel_s3adsp1800.py           |  42 ++++
 tests/functional/test_mips64el_loongson3v.py  |  39 ++++
 .../test_netdev_ethtool.py}                   |  50 ++---
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../test_ppc64_hv.py}                         |  48 ++---
 .../test_ppc64_powernv.py}                    |  56 ++----
 .../test_ppc64_pseries.py}                    |  56 ++----
 .../ppc_405.py => functional/test_ppc_405.py} |  28 +--
 .../test_ppc_40p.py}                          |  63 +++---
 .../test_ppc_74xx.py}                         |  74 +++----
 tests/functional/test_ppc_amiga.py            |  42 ++++
 .../test_ppc_bamboo.py}                       |  32 +--
 .../test_ppc_mpc8544ds.py}                    |  28 +--
 .../test_ppc_virtex_ml507.py}                 |  28 +--
 tests/functional/test_riscv_opensbi.py        |  36 ++++
 .../test_rx_gdbsim.py}                        |  57 +++---
 .../test_s390x_ccw_virtio.py}                 |  79 ++++----
 .../test_s390x_topology.py}                   |  86 ++++----
 tests/functional/test_sparc64_sun4u.py        |  40 ++++
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_gpu.py}                       |  64 +++---
 .../test_virtio_version.py}                   |   8 +-
 .../test_x86_cpu_model_versions.py}           |  63 ++----
 tests/meson.build                             |   1 +
 60 files changed, 1631 insertions(+), 994 deletions(-)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
 delete mode 100644 tests/avocado/machine_microblaze.py
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 delete mode 100644 tests/avocado/ppc_amiga.py
 delete mode 100644 tests/avocado/riscv_opensbi.py
 delete mode 100644 tests/avocado/tesseract_utils.py
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/asset.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 create mode 100644 tests/functional/qemu_test/testcase.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (52%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (66%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (88%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_ppc_amiga.py
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_riscv_opensbi.py
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
Signature · 1 line
2.45.2
#1 lore raw

PATCH v3 1/24 python: Install pycotap in our venv if necessary

From: Thomas Huth <[email protected]>

The upcoming functional tests will require pycotap for providing TAP output from the python-based tests. Since we want to be able to run some of the tests offline by default, too, let's install it along with meson in our venv if necessary (it's size is only 5 kB, so adding the wheel here should not really be a problem).

The wheel file has been obtained with:

pip download --only-binary :all: --dest . --no-cache pycotap

---
 python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes
 pythondeps.toml                              |   1 +
 2 files changed, 1 insertion(+)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
python/wheels/pycotap-1.3.1-py3-none-any.whl new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9c2c7d259365c8a0487228c6878e2c0b10ba6bdd
GIT binary patch
pythondeps.toml +1 -0
index f6e590fdd8..0bc4144d4b 100644
@@ -20,6 +20,7 @@
 [meson]
 # The install key should match the version in python/wheels/
 meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
+pycotap = { accepted = ">=1.1.0", installed = "1.3.1" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
Signature · 1 line
2.45.2
Tested-by on v3 1/24
#2 lore raw

Re: [PATCH v3 01/24] python: Install pycotap in our venv if necessary

Quoted 10 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The upcoming functional tests will require pycotap for providing TAP output from the python-based tests. Since we want to be able to run some of the tests offline by default, too, let's install it along with meson in our venv if necessary (it's size is only 5 kB, so adding the wheel here should not really be a problem).

The wheel file has been obtained with:

Quoted 8 lines
  pip download --only-binary :all: --dest . --no-cache pycotap
---
  python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes
  pythondeps.toml                              |   1 +
  2 files changed, 1 insertion(+)

create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl

#3 lore raw

PATCH v3 2/24 tests/functional: Add base classes for the upcoming pytest-based tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

[DB: split __init__.py into multiple files]
---
 tests/functional/qemu_test/__init__.py |  13 ++
 tests/functional/qemu_test/cmd.py      | 171 +++++++++++++++++++++++++
 tests/functional/qemu_test/config.py   |  36 ++++++
 tests/functional/qemu_test/testcase.py | 154 ++++++++++++++++++++++
 4 files changed, 374 insertions(+)
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/testcase.py
tests/functional/qemu_test/__init__.py new file +13 -0
new file mode 100644
index 0000000000..2f1e0bc70d
@@ -0,0 +1,13 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+
+from .config import BUILD_DIR
+from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
+    interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
+    exec_command, exec_command_and_wait_for_pattern
+from .testcase import QemuSystemTest, QemuBaseTest
tests/functional/qemu_test/cmd.py new file +171 -0

171 lines not on this page · raw message

tests/functional/qemu_test/config.py new file +36 -0
new file mode 100644
index 0000000000..edd75b7fd0
@@ -0,0 +1,36 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+from pathlib import Path
+
+
+def _source_dir():
+    # Determine top-level directory of the QEMU sources
+    return Path(__file__).parent.parent.parent.parent
+
+def _build_dir():
+    root = os.getenv('QEMU_BUILD_ROOT')
+    if root is not None:
+        return Path(root)
+    # Makefile.mtest only exists in build dir, so if it is available, use CWD
+    if os.path.exists('Makefile.mtest'):
+        return Path(os.getcwd())
+
+    root = os.path.join(_source_dir(), 'build')
+    if os.path.exists(root):
+        return Path(root)
+
+    raise Exception("Cannot identify build dir, set QEMU_BUILD_ROOT")
+
+BUILD_DIR = _build_dir()
tests/functional/qemu_test/testcase.py new file +154 -0

154 lines not on this page · raw message

Signature · 1 line
2.45.2
#4 lore raw

Re: [PATCH v3 02/24] tests/functional: Add base classes for the upcoming pytest-based tests

Quoted 18 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

[DB: split __init__.py into multiple files]
---
  tests/functional/qemu_test/__init__.py |  13 ++
  tests/functional/qemu_test/cmd.py      | 171 +++++++++++++++++++++++++
  tests/functional/qemu_test/config.py   |  36 ++++++
  tests/functional/qemu_test/testcase.py | 154 ++++++++++++++++++++++
  4 files changed, 374 insertions(+)
Quoted 4 lines

create mode 100644 tests/functional/qemu_test/__init__.py create mode 100644 tests/functional/qemu_test/cmd.py create mode 100644 tests/functional/qemu_test/config.py create mode 100644 tests/functional/qemu_test/testcase.py

Please squash:

-- >8 --

MAINTAINERS +1 -0
index 98eddf7ae1..a906218f9d 100644
@@ -4127,6 +4137,7 @@ F: .travis.yml
 F: docs/devel/ci*
 F: scripts/ci/
 F: tests/docker/
+F: tests/functional/
 F: tests/vm/
 F: tests/lcitool/
 F: tests/avocado/tuxrun_baselines.py

---

#5 lore raw

Re: [PATCH v3 02/24] tests/functional: Add base classes for the upcoming pytest-based tests

In reply to Daniel P. Berrangé
Quoted 8 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

<snip>

Quoted 39 lines
tests/functional/qemu_test/testcase.py new file +33 -0
new file mode 100644
index 0000000000..82cc1d454f
@@ -0,0 +1,154 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import sys
+import unittest
+import uuid
+
+from qemu.machine import QEMUMachine
+from qemu.utils import kvm_available, tcg_available
+
+from .cmd import run_cmd
+from .config import BUILD_DIR
+
+
+class QemuBaseTest(unittest.TestCase):
+
+    qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+    arch = None
+
+    workdir = None

tests/functional/qemu_test/testcase.py line 43

Quoted 10 lines
tests/functional/qemu_test/testcase.py new file +10 -0
+    log = logging.getLogger('qemu-test')
+
+    def setUp(self, bin_prefix):
+        self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+        self.arch = self.qemu_bin.split('-')[-1]
+
+        self.workdir = os.path.join(BUILD_DIR, 'tests/functional', self.arch,
+                                    self.id())
+        if not os.path.exists(self.workdir):
+            os.makedirs(self.workdir)

This is racy under --repeat:

==================================== 1/4 =====================================

  test:         qemu:func-quick+func-riscv64 / func-riscv64-riscv_opensbi
  start time:   14:16:52
  duration:     0.06s
  result:       exit status 1
  command:      PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional QEMU_BUILD_ROOT=/home/alex/lsrc/qemu.git/builds/all QEMU_TEST_QEMU_BINARY=

/home/alex/lsrc/qemu.git/builds/all/qemu-system-riscv64 MALLOC_PERTURB_=71 QEMU_TEST_QEMU_IMG=/home/alex/lsrc/qemu.git/builds/all/qemu-img /home/alex/lsrc/qemu.git/builds/al l/pyvenv/bin/python3 /home/alex/lsrc/qemu.git/tests/functional/test_riscv_opensbi.py
----------------------------------- stdout -----------------------------------
TAP version 13
not ok 1 test_riscv_opensbi.RiscvOpenSBI.test_riscv_sifive_u
not ok 2 test_riscv_opensbi.RiscvOpenSBI.test_riscv_spike
not ok 3 test_riscv_opensbi.RiscvOpenSBI.test_riscv_virt
1..3
----------------------------------- stderr -----------------------------------
Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_sifive_u'

Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_spike'

Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_virt'

(test program exited with status code 1)

We could just:

os.makedirs(self.workdir, exist_ok = True)

<snip>

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#6 lore raw

Re: [PATCH v3 02/24] tests/functional: Add base classes for the upcoming pytest-based tests

Quoted 103 lines · Alex Bennée
Quoted 8 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

<snip>

Quoted 39 lines
tests/functional/qemu_test/testcase.py new file +33 -0
new file mode 100644
index 0000000000..82cc1d454f
@@ -0,0 +1,154 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import sys
+import unittest
+import uuid
+
+from qemu.machine import QEMUMachine
+from qemu.utils import kvm_available, tcg_available
+
+from .cmd import run_cmd
+from .config import BUILD_DIR
+
+
+class QemuBaseTest(unittest.TestCase):
+
+    qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+    arch = None
+
+    workdir = None

tests/functional/qemu_test/testcase.py line 43

Quoted 10 lines
tests/functional/qemu_test/testcase.py new file +10 -0
+    log = logging.getLogger('qemu-test')
+
+    def setUp(self, bin_prefix):
+        self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+        self.arch = self.qemu_bin.split('-')[-1]
+
+        self.workdir = os.path.join(BUILD_DIR, 'tests/functional', self.arch,
+                                    self.id())
+        if not os.path.exists(self.workdir):
+            os.makedirs(self.workdir)

This is racy under --repeat:

==================================== 1/4 =====================================

  test:         qemu:func-quick+func-riscv64 / func-riscv64-riscv_opensbi
  start time:   14:16:52
  duration:     0.06s
  result:       exit status 1
  command:      PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional QEMU_BUILD_ROOT=/home/alex/lsrc/qemu.git/builds/all QEMU_TEST_QEMU_BINARY=

/home/alex/lsrc/qemu.git/builds/all/qemu-system-riscv64 MALLOC_PERTURB_=71 QEMU_TEST_QEMU_IMG=/home/alex/lsrc/qemu.git/builds/all/qemu-img /home/alex/lsrc/qemu.git/builds/al l/pyvenv/bin/python3 /home/alex/lsrc/qemu.git/tests/functional/test_riscv_opensbi.py
----------------------------------- stdout -----------------------------------
TAP version 13
not ok 1 test_riscv_opensbi.RiscvOpenSBI.test_riscv_sifive_u
not ok 2 test_riscv_opensbi.RiscvOpenSBI.test_riscv_spike
not ok 3 test_riscv_opensbi.RiscvOpenSBI.test_riscv_virt
1..3
----------------------------------- stderr -----------------------------------
Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_sifive_u'

Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_spike'

Traceback (most recent call last):
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 85, in setUp
super().setUp('qemu-system-')
File "/home/alex/lsrc/qemu.git/tests/functional/qemu_test/testcase.py", line 45, in setUp
os.makedirs(self.workdir)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/alex/lsrc/qemu.git/builds/all/tests/functional/riscv64/test_riscv_opensbi.RiscvOpenSBI.test_riscv_virt'

Quoted 5 lines

(test program exited with status code 1)

We could just:

os.makedirs(self.workdir, exist_ok = True)

Yeah, that's a better approach.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

Re: [PATCH v3 02/24] tests/functional: Add base classes for the upcoming pytest-based tests

In reply to Daniel P. Berrangé
Quoted 65 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

[DB: split __init__.py into multiple files]
---
  tests/functional/qemu_test/__init__.py |  13 ++
  tests/functional/qemu_test/cmd.py      | 171 +++++++++++++++++++++++++
  tests/functional/qemu_test/config.py   |  36 ++++++
  tests/functional/qemu_test/testcase.py | 154 ++++++++++++++++++++++
  4 files changed, 374 insertions(+)

create mode 100644 tests/functional/qemu_test/__init__.py create mode 100644 tests/functional/qemu_test/cmd.py create mode 100644 tests/functional/qemu_test/config.py create mode 100644 tests/functional/qemu_test/testcase.py

tests/functional/qemu_test/testcase.py new file +35 -0
new file mode 100644
index 0000000000..82cc1d454f
@@ -0,0 +1,154 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import sys
+import unittest
+import uuid
+
+from qemu.machine import QEMUMachine
+from qemu.utils import kvm_available, tcg_available
+
+from .cmd import run_cmd
+from .config import BUILD_DIR
+
+
+class QemuBaseTest(unittest.TestCase):
+
+    qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+    arch = None
+
+    workdir = None
+    log = logging.getLogger('qemu-test')
+

tests/functional/qemu_test/testcase.py line 45

Quoted 10 lines
tests/functional/qemu_test/testcase.py new file +10 -0
+    def setUp(self, bin_prefix):
+        self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+        self.arch = self.qemu_bin.split('-')[-1]
+
+        self.workdir = os.path.join(BUILD_DIR, 'tests/functional', self.arch,
+                                    self.id())
+        if not os.path.exists(self.workdir):
+            os.makedirs(self.workdir)
+
+    def main():

[*]

tests/functional/qemu_test/testcase.py line 48

Quoted 3 lines
tests/functional/qemu_test/testcase.py new file +3 -0
+        tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
+                                   test_output_log = pycotap.LogMode.LogToError)
+        path = os.path.basename(sys.argv[0])[:-3]

Moving the 'path' line in [*] simplifies patch #9 (no duplication).

Quoted 1 line
tests/functional/qemu_test/testcase.py new file +1 -0
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
#8 lore raw

PATCH v3 3/24 tests/functional: Set up logging

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/testcase.py | 27 +++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +26 -1
index 82cc1d454f..27bbf4a0af 100644
@@ -31,7 +31,8 @@ class QemuBaseTest(unittest.TestCase):
     arch = None
 
     workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
+    logdir = None
 
     def setUp(self, bin_prefix):
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -42,6 +43,20 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 
+        self.logdir = self.workdir
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.logdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)
+
+    def tearDown(self):
+        self.log.removeHandler(self._log_fh)
+
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
@@ -61,6 +76,15 @@ def setUp(self):
 
         super().setUp('qemu-system-')
 
+        console_log = logging.getLogger('console')
+        console_log.setLevel(logging.DEBUG)
+        self._console_log_fh = logging.FileHandler(os.path.join(self.workdir,
+                                                   'console.log'), mode='w')
+        self._console_log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
+        self._console_log_fh.setFormatter(fileFormatter)
+        console_log.addHandler(self._console_log_fh)
+
     def set_machine(self, machinename):
         # TODO: We should use QMP to get the list of available machines
         if not self._machinehelp:
@@ -151,4 +175,5 @@ def set_vm_arg(self, arg, value):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
+        logging.getLogger('console').removeHandler(self._console_log_fh)
         super().tearDown()
Signature · 1 line
2.45.2
#9 lore raw

Re: [PATCH v3 03/24] tests/functional: Set up logging

Quoted 29 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/testcase.py | 27 +++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +2 -1
index 82cc1d454f..27bbf4a0af 100644
@@ -31,7 +31,8 @@ class QemuBaseTest(unittest.TestCase):
     arch = None
 
     workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
+    logdir = None
 
     def setUp(self, bin_prefix):
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -42,6 +43,20 @@ def setUp(self, bin_prefix):
         if not os.path.exists(self.workdir):
             os.makedirs(self.workdir)
 

tests/functional/qemu_test/testcase.py def setUp(self, bin_prefix): · line 55

Quoted 10 lines
tests/functional/qemu_test/testcase.py +10 -0
+        self.logdir = self.workdir
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.logdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)

This is in the wrong place, we have a unique machine directory for each run, the console.log should go in there.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#10 lore raw

Re: [PATCH v3 03/24] tests/functional: Set up logging

Quoted 31 lines · Alex Bennée
Quoted 29 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
  tests/functional/qemu_test/testcase.py | 27 +++++++++++++++++++++++++-
  1 file changed, 26 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +2 -1
index 82cc1d454f..27bbf4a0af 100644
@@ -31,7 +31,8 @@ class QemuBaseTest(unittest.TestCase):
      arch = None
 
      workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
+    logdir = None
 
      def setUp(self, bin_prefix):
          self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -42,6 +43,20 @@ def setUp(self, bin_prefix):
          if not os.path.exists(self.workdir):
              os.makedirs(self.workdir)
 
Quoted 13 lines

tests/functional/qemu_test/testcase.py def setUp(self, bin_prefix): · line 55

Quoted 10 lines
tests/functional/qemu_test/testcase.py +10 -0
+        self.logdir = self.workdir
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.logdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)

This is in the wrong place, we have a unique machine directory for each run, the console.log should go in there.

self.workdir already points to the unique directory of the test (see second patch), so setting logdir = workdir should be fine, I think?

Thomas

#11 lore raw

PATCH v3 4/24 tests/Makefile.include: Increase the level of indentation in the help text

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

---
 tests/Makefile.include | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
tests/Makefile.include +15 -15
index 6618bfed70..167d8f29a4 100644
@@ -3,28 +3,28 @@
 .PHONY: check-help
 check-help:
 	@echo "Regression testing targets:"
-	@echo " $(MAKE) check                  Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
-	@echo " $(MAKE) bench                  Run speed tests"
+	@echo " $(MAKE) check                    Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
+	@echo " $(MAKE) bench                    Run speed tests"
 	@echo
 	@echo "Individual test suites:"
-	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
-	@echo " $(MAKE) check-qtest            Run qtest tests"
-	@echo " $(MAKE) check-unit             Run qobject tests"
-	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
-	@echo " $(MAKE) check-block            Run block tests"
+	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
+	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-unit               Run qobject tests"
+	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
+	@echo " $(MAKE) check-block              Run block tests"
 ifneq ($(filter $(all-check-targets), check-softfloat),)
-	@echo " $(MAKE) check-tcg              Run TCG tests"
-	@echo " $(MAKE) check-softfloat        Run FPU emulation tests"
+	@echo " $(MAKE) check-tcg                Run TCG tests"
+	@echo " $(MAKE) check-softfloat          Run FPU emulation tests"
 endif
-	@echo " $(MAKE) check-avocado          Run avocado (integration) tests for currently configured targets"
+	@echo " $(MAKE) check-avocado            Run avocado (integration) tests for currently configured targets"
 	@echo
-	@echo " $(MAKE) check-report.junit.xml Generates an aggregated XML test report"
-	@echo " $(MAKE) check-venv             Creates a Python venv for tests"
-	@echo " $(MAKE) check-clean            Clean the tests and related data"
+	@echo " $(MAKE) check-report.junit.xml   Generates an aggregated XML test report"
+	@echo " $(MAKE) check-venv               Creates a Python venv for tests"
+	@echo " $(MAKE) check-clean              Clean the tests and related data"
 	@echo
 	@echo "The following are useful for CI builds"
-	@echo " $(MAKE) check-build            Build most test binaries"
-	@echo " $(MAKE) get-vm-images          Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
+	@echo " $(MAKE) check-build              Build most test binaries"
+	@echo " $(MAKE) get-vm-images            Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
 	@echo
 	@echo
 	@echo "The variable SPEED can be set to control the gtester speed setting."
Signature · 1 line
2.45.2
Tested-by on v3 4/24
#12 lore raw

Re: [PATCH v3 04/24] tests/Makefile.include: Increase the level of indentation in the help text

Quoted 11 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

---
  tests/Makefile.include | 30 +++++++++++++++---------------
  1 file changed, 15 insertions(+), 15 deletions(-)
Reviewed-by on v3 4/24
#13 lore raw

Re: [PATCH v3 04/24] tests/Makefile.include: Increase the level of indentation in the help text

In reply to Daniel P. Berrangé
Quoted 8 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#14 lore raw

PATCH v3 5/24 tests/functional: Prepare the meson build system for the functional tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

---
 tests/Makefile.include       | 11 ++++++
 tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 3 files changed, 78 insertions(+)
 create mode 100644 tests/functional/meson.build
tests/Makefile.include +11 -0
index 167d8f29a4..d0c2ae30b7 100644
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-functional         Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARGET  Run functional tests for a given target"
 	@echo " $(MAKE) check-unit               Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
 	@echo " $(MAKE) check-block              Run block tests"
@@ -152,6 +154,15 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@$(MAKE) SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@$(MAKE) SPEED=thorough check-func check-func-quick
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
tests/functional/meson.build new file +66 -0

66 lines not on this page · raw message

tests/meson.build +1 -0
index acb6807094..3345ad2098 100644
@@ -85,3 +85,4 @@ subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')
Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 5/24
#15 lore raw

Re: [PATCH v3 05/24] tests/functional: Prepare the meson build system for the functional tests

Hi,

Quoted 54 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

---
  tests/Makefile.include       | 11 ++++++
  tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
  tests/meson.build            |  1 +
  3 files changed, 78 insertions(+)

create mode 100644 tests/functional/meson.build

tests/functional/meson.build new file +32 -0
+foreach speed : ['quick', 'thorough']
+  foreach dir : target_dirs
+    if not dir.endswith('-softmmu')
+      continue
+    endif
+
+    target_base = dir.split('-')[0]
+    test_emulator = emulators['qemu-system-' + target_base]
+
+    if speed == 'quick'
+      suites = ['func-quick', 'func-' + target_base]
+      target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
+    else
+      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
+      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
+    endif
+
+    test_deps = roms
+    test_env = environment()
+    if have_tools
+      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
+      test_deps += [qemu_img]
+    endif
+    test_env.set('QEMU_TEST_QEMU_BINARY',
+                 meson.global_build_root() / 'qemu-system-' + target_base)
+    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
+    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+                               meson.current_source_dir())
+
+    foreach test : target_tests
+      test('func-@0@/@1@'.format(target_base, test),
+           python,

tests/functional/meson.build line 66

Quoted 10 lines
tests/functional/meson.build new file +10 -0
+           depends: [test_deps, test_emulator, emulator_modules],
+           env: test_env,
+           args: [meson.current_source_dir() / 'test_' + test + '.py'],
+           protocol: 'tap',
+           timeout: test_timeouts.get(test, 60),
+           priority: test_timeouts.get(test, 60),
+           suite: suites)
+    endforeach
+  endforeach
+endforeach

Can we have meson report why a test is skipped? All Python unittest @skip* decorators provide a 'reason' argument, so we already have that information.

Not much help on meson doc:
https://mesonbuild.com/Unit-tests.html#skipped-tests-and-hard-errors

If I run a test manually I see the reason in the tap output:

$ FOO=bar tests/functional/test_mips64el_loongson3v.py
TAP version 13
ok 1 test_mips64el_loongson3v.MipsLoongson3v.test_pmon_serial_console # SKIP untrusted code
1..1
$
#16 lore raw

Re: [PATCH v3 05/24] tests/functional: Prepare the meson build system for the functional tests

Quoted 65 lines · Philippe Mathieu-Daudé

Hi,

Quoted 62 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

---
  tests/Makefile.include       | 11 ++++++ tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++ tests/meson.build            |  1 +

  3 files changed, 78 insertions(+)

  create mode 100644 tests/functional/meson.build

tests/functional/meson.build new file +22 -0
+foreach speed : ['quick', 'thorough']
+  foreach dir : target_dirs
+    if not dir.endswith('-softmmu')
+      continue
+    endif
+
+    target_base = dir.split('-')[0]
+    test_emulator = emulators['qemu-system-' + target_base]
+
+    if speed == 'quick'
+      suites = ['func-quick', 'func-' + target_base]
+      target_tests = get_variable('tests_' + target_base + '_quick', []) 
+ tests_generic
+    else
+      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
+      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
+    endif
+
+    test_deps = roms
+    test_env = environment()
+    if have_tools
+      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 

'qemu-img')
+      test_deps += [qemu_img]
+    endif
+    test_env.set('QEMU_TEST_QEMU_BINARY',
+                 meson.global_build_root() / 'qemu-system-' + target_base) +    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root()) +    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' + +                               meson.current_source_dir())
+
+    foreach test : target_tests
+      test('func-@0@/@1@'.format(target_base, test),
+           python,
+           depends: [test_deps, test_emulator, emulator_modules], +           env: test_env,
+           args: [meson.current_source_dir() / 'test_' + test + '.py'], +           protocol: 'tap',
+           timeout: test_timeouts.get(test, 60),
+           priority: test_timeouts.get(test, 60),

Quoted 8 lines
Quoted 4 lines

+           suite: suites)
+    endforeach
+  endforeach
+endforeach

Can we have meson report why a test is skipped? All Python unittest @skip* decorators provide a 'reason' argument, so we already have that information.

I guess you'd need to extend the meson test runner for this...

Quoted 9 lines

Not much help on meson doc:
https://mesonbuild.com/Unit-tests.html#skipped-tests-and-hard-errors

If I run a test manually I see the reason in the tap output:

$ FOO=bar tests/functional/test_mips64el_loongson3v.py
TAP version 13
ok 1 test_mips64el_loongson3v.MipsLoongson3v.test_pmon_serial_console # SKIP untrusted code
1..1

You can also see this information in the txt file in the meson-logs folder, so it is certainly not completely lost.

Thomas

#17 lore raw

PATCH v3 6/24 tests/functional: Convert simple avocado tests into standalone python tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
 tests/functional/meson.build                  |  5 ++
 .../test_cpu_queries.py}                      |  7 ++-
 .../test_empty_cpu_model.py}                  |  7 ++-
 .../test_mem_addr_space.py}                   | 52 +++----------------
 .../test_pc_cpu_hotplug_props.py}             | 11 ++--
 .../test_virtio_version.py}                   |  8 +--
 6 files changed, 34 insertions(+), 56 deletions(-)
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
tests/functional/meson.build +5 -0
index 7fc9c4e513..a8cc0e6330 100644
@@ -14,9 +14,14 @@ test_timeouts = {
 }
 
 tests_generic = [
+  'empty_cpu_model',
 ]
 
 tests_x86_64_quick = [
+  'cpu_queries',
+  'mem_addr_space',
+  'pc_cpu_hotplug_props',
+  'virtio_version',
 ]
 
 tests_x86_64_thorough = [
tests/functional/test_cpu_queries.py renamed from tests/avocado/cpu_queries.py +6 -1
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/cpu_queries.py
rename to tests/functional/test_cpu_queries.py
index d3faa14720..97dd600c01
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Sanity check of query-cpu-* results
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -8,7 +10,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class QueryCPUModelExpansion(QemuSystemTest):
     """
@@ -33,3 +35,6 @@ def test(self):
             e = self.vm.cmd('query-cpu-model-expansion', model=model,
                             type='full')
             self.assertEqual(e['model']['name'], c['name'])
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_empty_cpu_model.py renamed from tests/avocado/empty_cpu_model.py +6 -1
old mode 100644
new mode 100755
similarity index 84%
rename from tests/avocado/empty_cpu_model.py
rename to tests/functional/test_empty_cpu_model.py
index d906ef3d3c..0081b06d85
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Check for crash when using empty -cpu option
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -7,7 +9,7 @@
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class EmptyCPUModel(QemuSystemTest):
     def test(self):
@@ -17,3 +19,6 @@ def test(self):
         self.vm.wait()
         self.assertEqual(self.vm.exitcode(), 1, "QEMU exit code should be 1")
         self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +6 -46

154 lines not on this page · raw message

tests/functional/test_pc_cpu_hotplug_props.py renamed from tests/avocado/pc_cpu_hotplug_props.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/pc_cpu_hotplug_props.py
rename to tests/functional/test_pc_cpu_hotplug_props.py
index 4bd3e02665..9d5a37cb17
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # Ensure CPU die-id can be omitted on -device
 #
@@ -20,16 +21,16 @@
 # License along with this library; if not, see <http://www.gnu.org/licenses/>.
 #
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class OmittedCPUProps(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=cpu:qemu64
-    """
+
     def test_no_die_id(self):
         self.vm.add_args('-nodefaults', '-S')
         self.vm.add_args('-smp', '1,sockets=2,cores=2,threads=2,maxcpus=8')
         self.vm.add_args('-device', 'qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0')
         self.vm.launch()
         self.assertEqual(len(self.vm.cmd('query-cpus-fast')), 2)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_virtio_version.py renamed from tests/avocado/virtio_version.py +5 -3
old mode 100644
new mode 100755
similarity index 98%
rename from tests/avocado/virtio_version.py
rename to tests/functional/test_virtio_version.py
index afe5e828b5..eb23060564
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 """
 Check compatibility of virtio device types
 """
@@ -12,7 +13,7 @@
 import os
 
 from qemu.machine import QEMUMachine
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 # Virtio Device IDs:
 VIRTIO_NET = 1
@@ -60,8 +61,6 @@ class VirtioVersionCheck(QemuSystemTest):
     Check if virtio-version-specific device types result in the
     same device tree created by `disable-modern` and
     `disable-legacy`.
-
-    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
@@ -173,3 +172,6 @@ def test_modern_only_devs(self):
         self.check_modern_only('virtio-mouse-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-tablet-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-keyboard-pci', VIRTIO_INPUT)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Tested-by on v3 6/24
#18 lore raw

Re: [PATCH v3 06/24] tests/functional: Convert simple avocado tests into standalone python tests

Quoted 19 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
  tests/functional/meson.build                  |  5 ++
  .../test_cpu_queries.py}                      |  7 ++-
  .../test_empty_cpu_model.py}                  |  7 ++-
  .../test_mem_addr_space.py}                   | 52 +++----------------
  .../test_pc_cpu_hotplug_props.py}             | 11 ++--
  .../test_virtio_version.py}                   |  8 +--
  6 files changed, 34 insertions(+), 56 deletions(-)
Quoted 10 lines

rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755
rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755
rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%) mode change 100644 => 100755
rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755
rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755

Please squash:

-- >8 --

MAINTAINERS +6 -0
index 98eddf7ae1..a906218f9d 100644
@@ -1830,6 +1832,9 @@ F: hw/isa/apm.c
 F: include/hw/isa/apm.h
 F: tests/unit/test-x86-topo.c
 F: tests/qtest/test-x86-cpuid-compat.c
+F: tests/functional/test_mem_addr_space.py
+F: tests/functional/test_pc_cpu_hotplug_props.py
+F: tests/functional/test_x86_cpu_model_versions.py
 
 PC Chipset
 M: Michael S. Tsirkin <[email protected]>
@@ -1896,6 +1901,8 @@ F: include/hw/boards.h
 F: include/hw/core/cpu.h
 F: include/hw/cpu/cluster.h
 F: include/sysemu/numa.h
+F: tests/functional/test_cpu_queries.py
+F: tests/functional/test_empty_cpu_model.py
 F: tests/unit/test-smp-parse.c
 T: git https://gitlab.com/ehabkost/qemu.git machine-next
 
@@ -2236,6 +2244,7 @@ F: net/vhost-user.c
 F: include/hw/virtio/
 F: docs/devel/virtio*
 F: docs/devel/migration/virtio.rst
+F: tests/functional/test_virtio_version.py
 
 virtio-balloon
 M: Michael S. Tsirkin <[email protected]>
tests/functional/test_x86_cpu_model_versions.py +0 -0
index a5f27c737d..a7294b4b92 100755
@@ -217,7 +217,7 @@ def test_none_alias(self):
         """
         Check if unversioned CPU model is an alias pointing to some 

version

         """
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-S')
         self.vm.launch()

---

#19 lore raw

PATCH v3 7/24 tests/functional: Convert avocado tests that just need a small adjustment

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
 tests/functional/meson.build                  |  6 ++
 .../test_info_usernet.py}                     | 11 ++-
 .../test_ppc_74xx.py}                         | 74 ++++++++-----------
 .../version.py => functional/test_version.py} | 13 ++--
 4 files changed, 53 insertions(+), 51 deletions(-)
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
tests/functional/meson.build +6 -0
index a8cc0e6330..8a8fa0ab99 100644
@@ -15,6 +15,12 @@ test_timeouts = {
 
 tests_generic = [
   'empty_cpu_model',
+  'info_usernet',
+  'version',
+]
+
+tests_ppc_quick = [
+  'ppc_74xx',
 ]
 
 tests_x86_64_quick = [
tests/functional/test_info_usernet.py renamed from tests/avocado/info_usernet.py +7 -4
old mode 100644
new mode 100755
similarity index 87%
rename from tests/avocado/info_usernet.py
rename to tests/functional/test_info_usernet.py
index e1aa7a6e0a..ded973253a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test for the hmp command "info usernet"
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -8,18 +10,16 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 from qemu.utils import get_info_usernet_hostfwd_port
 
 
 class InfoUsernet(QemuSystemTest):
-    """
-    :avocado: tags=machine:none
-    """
 
     def test_hostfwd(self):
         self.require_netdev('user')
+        self.machine = 'none'
         self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
@@ -31,3 +31,6 @@ def test_hostfwd(self):
         self.assertGreater(port, 0,
                            ('Found a redirected port that is not greater than'
                             ' zero'))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_74xx.py renamed from tests/avocado/ppc_74xx.py +32 -42

167 lines not on this page · raw message

tests/functional/test_version.py renamed from tests/avocado/version.py +8 -5
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..5e566d76b1
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Version check example test
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
 class Version(QemuSystemTest):
-    """
-    :avocado: tags=quick
-    :avocado: tags=machine:none
-    """
+
     def test_qmp_human_info_version(self):
+        self.machine = 'none'
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
                           command_line='info version')
         self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Tested-by on v3 7/24
#20 lore raw

Re: [PATCH v3 07/24] tests/functional: Convert avocado tests that just need a small adjustment

Quoted 15 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
  tests/functional/meson.build                  |  6 ++
  .../test_info_usernet.py}                     | 11 ++-
  .../test_ppc_74xx.py}                         | 74 ++++++++-----------
  .../version.py => functional/test_version.py} | 13 ++--
  4 files changed, 53 insertions(+), 51 deletions(-)
Quoted 6 lines

rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755
rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755
rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755

Please squash:

-- >8 --

MAINTAINERS +3 -0
index 98eddf7ae1..a906218f9d 100644
@@ -317,6 +317,7 @@ F: configs/devices/ppc*
 F: docs/system/ppc/embedded.rst
 F: docs/system/target-ppc.rst
 F: tests/tcg/ppc*/*
+F: tests/functional/test_ppc_74xx.py
 
 RISC-V TCG CPUs
 M: Palmer Dabbelt <[email protected]>
@@ -2101,6 +2108,7 @@ S: Odd Fixes
 F: hw/net/
 F: include/hw/net/
 F: tests/qtest/virtio-net-test.c
+F: tests/avocado/info_usernet.py
 F: docs/virtio-net-failover.rst
 T: git https://github.com/jasowang/qemu.git net
 
@@ -2969,6 +2978,7 @@ S: Supported
 F: include/qemu/option.h
 F: tests/unit/test-keyval.c
 F: tests/unit/test-qemu-opts.c
+F: tests/functional/test_version.py
 F: util/keyval.c
 F: util/qemu-option.c
 
tests/functional/test_info_usernet.py +1 -1
index ded973253a..cd37524d94 100755
@@ -19,7 +19,7 @@ class InfoUsernet(QemuSystemTest):
 
     def test_hostfwd(self):
         self.require_netdev('user')
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-netdev', 

'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
self.vm.launch()
res = self.vm.cmd('human-monitor-command',

tests/functional/test_version.py +1 -1
index 5e566d76b1..3ab3b67f7e 100755
@@ -17,7 +17,7 @@
 class Version(QemuSystemTest):
 
     def test_qmp_human_info_version(self):
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',

---

#21 lore raw

Re: [PATCH v3 07/24] tests/functional: Convert avocado tests that just need a small adjustment

Quoted 23 lines · Philippe Mathieu-Daudé
Quoted 14 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
  tests/functional/meson.build                  |  6 ++ .../test_info_usernet.py}                     | 11 ++- .../test_ppc_74xx.py}                         | 74 ++++++++----------- .../version.py => functional/test_version.py} | 13 ++--

Quoted 7 lines
  4 files changed, 53 insertions(+), 51 deletions(-)

  rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755
  rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755
  rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755

Quoted 2 lines
#22 lore raw

PATCH v3 8/24 tests/functional: add a module for handling asset download & caching

In reply to Daniel P. Berrangé

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

---
 tests/functional/qemu_test/__init__.py |  1 +
 tests/functional/qemu_test/asset.py    | 96 ++++++++++++++++++++++++++
 2 files changed, 97 insertions(+)
 create mode 100644 tests/functional/qemu_test/asset.py
tests/functional/qemu_test/__init__.py +1 -0
index 2f1e0bc70d..db05c8f412 100644
@@ -6,6 +6,7 @@
 # later.  See the COPYING file in the top-level directory.
 
 
+from .asset import Asset
 from .config import BUILD_DIR
 from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
     interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
tests/functional/qemu_test/asset.py new file +96 -0

96 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 8/24
#23 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

Quoted 16 lines · Daniel P. Berrangé

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

---
  tests/functional/qemu_test/__init__.py |  1 +
  tests/functional/qemu_test/asset.py    | 96 ++++++++++++++++++++++++++
  2 files changed, 97 insertions(+)

create mode 100644 tests/functional/qemu_test/asset.py

tests/functional/qemu_test/__init__.py +0 -0
index 2f1e0bc70d..db05c8f412 100644

tests/functional/qemu_test/__init__.py line 9

Quoted 4 lines
tests/functional/qemu_test/__init__.py +1 -0
@@ -6,6 +6,7 @@
  # later.  See the COPYING file in the top-level directory.
 
 
+from .asset import Asset

For next patch?

Quoted 41 lines
tests/functional/qemu_test/__init__.py +0 -0
  from .config import BUILD_DIR
  from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
      interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
tests/functional/qemu_test/asset.py new file +32 -0
new file mode 100644
index 0000000000..6432da2e0b
@@ -0,0 +1,96 @@
+# Test utilities for fetching & caching assets
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import hashlib
+import logging
+import os
+import subprocess
+import urllib.request
+from pathlib import Path
+from shutil import copyfileobj
+
+
+# Instances of this class must be declared as class level variables
+# starting with a name "ASSET_". This enables the pre-caching logic
+# to easily find all referenced assets and download them prior to
+# execution of the tests.
+class Asset:
+
+    def __init__(self, url, hash):
+        self.url = url
+        self.hash = hash
+        self.cache_dir = Path(Path("~").expanduser(),
+                              ".cache", "qemu", "download")
+        self.cache_file = Path(self.cache_dir,
+                               hashlib.sha256(url.encode("utf-8")).hexdigest())
+        self.log = logging.getLogger('qemu-test')
+
+    def __repr__(self):

tests/functional/qemu_test/asset.py line 42

Quoted 10 lines
tests/functional/qemu_test/asset.py new file +10 -0
+        return "Asset: url=%s hash=%s cache=%s" % (
+            self.url, self.hash, self.cache_file)
+
+    def _check(self, cache_file):
+        if self.hash is None:
+            return True
+        if len(self.hash) == 40:
+            sum_prog = 'sha1sum'
+        elif len(self.hash) == 64:
+            sum_prog = 'sha256sum'

Do we want to support these? Should we declare them deprecated and emit a warning?

tests/functional/qemu_test/asset.py line 46

Quoted 4 lines
tests/functional/qemu_test/asset.py new file +4 -0
+        elif len(self.hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")

Very nice :)

#24 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

Quoted 68 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Daniel P. Berrangé

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

---
  tests/functional/qemu_test/__init__.py |  1 +
  tests/functional/qemu_test/asset.py    | 96 ++++++++++++++++++++++++++
  2 files changed, 97 insertions(+)

create mode 100644 tests/functional/qemu_test/asset.py

tests/functional/qemu_test/__init__.py +0 -0
index 2f1e0bc70d..db05c8f412 100644

tests/functional/qemu_test/__init__.py line 7

Quoted 2 lines
tests/functional/qemu_test/__init__.py +1 -0
@@ -6,6 +6,7 @@
  # later.  See the COPYING file in the top-level directory.
+from .asset import Asset

For next patch?

Quoted 46 lines
tests/functional/qemu_test/__init__.py +0 -0
  from .config import BUILD_DIR
  from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
      interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
tests/functional/qemu_test/asset.py new file +37 -0
new file mode 100644
index 0000000000..6432da2e0b
@@ -0,0 +1,96 @@
+# Test utilities for fetching & caching assets
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import hashlib
+import logging
+import os
+import subprocess
+import urllib.request
+from pathlib import Path
+from shutil import copyfileobj
+
+
+# Instances of this class must be declared as class level variables
+# starting with a name "ASSET_". This enables the pre-caching logic
+# to easily find all referenced assets and download them prior to
+# execution of the tests.
+class Asset:
+
+    def __init__(self, url, hash):
+        self.url = url
+        self.hash = hash
+        self.cache_dir = Path(Path("~").expanduser(),
+                              ".cache", "qemu", "download")
+        self.cache_file = Path(self.cache_dir,
+                               hashlib.sha256(url.encode("utf-8")).hexdigest())
+        self.log = logging.getLogger('qemu-test')
+
+    def __repr__(self):
+        return "Asset: url=%s hash=%s cache=%s" % (
+            self.url, self.hash, self.cache_file)
+
+    def _check(self, cache_file):
+        if self.hash is None:
Quoted 8 lines

tests/functional/qemu_test/asset.py line 42

Quoted 5 lines
tests/functional/qemu_test/asset.py new file +5 -0
+            return True
+        if len(self.hash) == 40:
+            sum_prog = 'sha1sum'
+        elif len(self.hash) == 64:
+            sum_prog = 'sha256sum'

Do we want to support these? Should we declare them deprecated and emit a warning?

Oh thanks for the reminder. I wanted to standardize on sha256 ,since sha1 is broken and sha512 is overkill.

Since I've run the tests once I've got everything download and can now trivially generate the sha256 for everything we need.

Quoted 11 lines

tests/functional/qemu_test/asset.py line 46

Quoted 5 lines
tests/functional/qemu_test/asset.py new file +4 -0
+        elif len(self.hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")

Very nice :)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#25 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

In reply to Philippe Mathieu-Daudé
Quoted 12 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

---
  tests/functional/qemu_test/__init__.py |  1 +
  tests/functional/qemu_test/asset.py    | 96 ++++++++++++++++++++++++++
  2 files changed, 97 insertions(+)

create mode 100644 tests/functional/qemu_test/asset.py

Quoted 9 lines

tests/functional/qemu_test/__init__.py line 7

Quoted 7 lines
tests/functional/qemu_test/__init__.py +1 -0
index 2f1e0bc70d..db05c8f412 100644
@@ -6,6 +6,7 @@
  # later.  See the COPYING file in the top-level directory.
+from .asset import Asset

For next patch?

No, the idea is that a test class can just do 'from qemu_test import Asset', so they don't need to be aware of the fact that we've split up the code into separate files beneath the qemu_test/ directory.

Quoted 4 lines

tests/functional/qemu_test/__init__.py line 12

Quoted 4 lines
tests/functional/qemu_test/__init__.py +0 -0
  from .config import BUILD_DIR
  from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
      interrupt_interactive_console_until_pattern, wait_for_console_pattern, \

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#26 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

Quoted 23 lines · Daniel P. Berrangé
Quoted 12 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

---
   tests/functional/qemu_test/__init__.py |  1 +
   tests/functional/qemu_test/asset.py    | 96 ++++++++++++++++++++++++++
   2 files changed, 97 insertions(+)

create mode 100644 tests/functional/qemu_test/asset.py

Quoted 9 lines

tests/functional/qemu_test/__init__.py line 7

Quoted 7 lines
tests/functional/qemu_test/__init__.py +1 -0
index 2f1e0bc70d..db05c8f412 100644
@@ -6,6 +6,7 @@
   # later.  See the COPYING file in the top-level directory.
+from .asset import Asset

For next patch?

Quoted 3 lines

No, the idea is that a test class can just do 'from qemu_test import Asset', so they don't need to be aware of the fact that we've split up the code into separate files beneath the qemu_test/ directory.

Oh indeed I missed that, thanks for clarifying.

#27 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

In reply to Daniel P. Berrangé

tests/functional/qemu_test/asset.py line 29

Quoted 7 lines · Daniel P. Berrangé
tests/functional/qemu_test/asset.py new file +7 -0
+    def __init__(self, url, hash):
+        self.url = url
+        self.hash = hash
+        self.cache_dir = Path(Path("~").expanduser(),
+                              ".cache", "qemu", "download")
+        self.cache_file = Path(self.cache_dir,
+                               hashlib.sha256(url.encode("utf-8")).hexdigest())

Out of curiosity, why are we hashing the url from which we downloaded? Why not use the known hash of the contents?

If a url goes stale, and we adjust the url to a new host, we shouldn't need to re-download just because the url changed.

r~

#28 lore raw

Re: [PATCH v3 08/24] tests/functional: add a module for handling asset download & caching

Quoted 9 lines · Richard Henderson

tests/functional/qemu_test/asset.py line 29

Quoted 7 lines · Daniel P. Berrangé
tests/functional/qemu_test/asset.py new file +7 -0
+    def __init__(self, url, hash):
+        self.url = url
+        self.hash = hash
+        self.cache_dir = Path(Path("~").expanduser(),
+                              ".cache", "qemu", "download")
+        self.cache_file = Path(self.cache_dir,
+                               hashlib.sha256(url.encode("utf-8")).hexdigest())
Quoted 5 lines

Out of curiosity, why are we hashing the url from which we downloaded? Why not use the known hash of the contents?

If a url goes stale, and we adjust the url to a new host, we shouldn't need to re-download just because the url changed.

Hmmm, good point, will change it.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#29 lore raw

PATCH v3 9/24 tests/functional: enable pre-emptive caching of assets

In reply to Daniel P. Berrangé

Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

   ninja test --suite functional

will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier.

   QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

nina check-func-precache

will download and cache all assets required by the functional tests

At the make level, precaching is always done by

make check-functional

---
 tests/Makefile.include                 |  3 ++-
 tests/functional/meson.build           | 33 +++++++++++++++++++++++--
 tests/functional/qemu_test/asset.py    | 34 ++++++++++++++++++++++++++
 tests/functional/qemu_test/testcase.py |  8 ++++++
 4 files changed, 75 insertions(+), 3 deletions(-)
tests/Makefile.include +2 -1
index d0c2ae30b7..dbc8fc4678 100644
@@ -161,7 +161,8 @@ $(FUNCTIONAL_TARGETS):
 
 .PHONY: check-functional
 check-functional:
-	@$(MAKE) SPEED=thorough check-func check-func-quick
+	@$(MAKE) precache-functional
+	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick
 
 # Consolidated targets
 
tests/functional/meson.build +31 -2
index 8a8fa0ab99..cef74b82a9 100644
@@ -33,6 +33,7 @@ tests_x86_64_quick = [
 tests_x86_64_thorough = [
 ]
 
+precache_all = []
 foreach speed : ['quick', 'thorough']
   foreach dir : target_dirs
     if not dir.endswith('-softmmu')
@@ -63,11 +64,35 @@ foreach speed : ['quick', 'thorough']
                                meson.current_source_dir())
 
     foreach test : target_tests
-      test('func-@0@/@1@'.format(target_base, test),
+      testname = '@0@-@1@'.format(target_base, test)
+      testfile = 'test_' + test + '.py'
+      testpath = meson.current_source_dir() / testfile
+      teststamp = testname + '.tstamp'
+      test_precache_env = environment()
+      test_precache_env.set('QEMU_TEST_PRECACHE', meson.current_build_dir() / teststamp)
+      test_precache_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+                                          meson.current_source_dir())
+      precache = custom_target('func-precache-' + testname,
+                               output: teststamp,
+                               command: [python, testpath],
+                               depend_files: files(testpath),
+                               build_by_default: false,
+                               env: test_precache_env)
+      precache_all += precache
+
+      # Ideally we would add 'precache' to 'depends' here, such that
+      # 'build_by_default: false' lets the pre-caching automatically
+      # run immediately before the test runs. In practice this is
+      # broken in meson, with it running the pre-caching in the normal
+      # compile phase https://github.com/mesonbuild/meson/issues/2518
+      # If the above bug ever gets fixed, when QEMU changes the min
+      # meson version, add the 'depends' and remove the custom
+      # 'run_target' logic below & in Makefile.include
+      test('func-' + testname,
            python,
            depends: [test_deps, test_emulator, emulator_modules],
            env: test_env,
-           args: [meson.current_source_dir() / 'test_' + test + '.py'],
+           args: [testpath],
            protocol: 'tap',
            timeout: test_timeouts.get(test, 60),
            priority: test_timeouts.get(test, 60),
@@ -75,3 +100,7 @@ foreach speed : ['quick', 'thorough']
     endforeach
   endforeach
 endforeach
+
+run_target('precache-functional',
+           depends: precache_all,
+           command: ['true'])
tests/functional/qemu_test/asset.py +34 -0
index 6432da2e0b..6197d907eb 100644
@@ -9,6 +9,8 @@
 import logging
 import os
 import subprocess
+import sys
+import unittest
 import urllib.request
 from pathlib import Path
 from shutil import copyfileobj
@@ -61,6 +63,9 @@ def fetch(self):
                            self.cache_file, self.url)
             return str(self.cache_file)
 
+        if os.environ.get("QEMU_TEST_NO_DOWNLOAD", False):
+            raise Exception("Asset cache is invalid and downloads disabled")
+
         self.log.info("Downloading %s to %s...", self.url, self.cache_file)
         tmp_cache_file = self.cache_file.with_suffix(".download")
 
@@ -94,3 +99,32 @@ def fetch(self):
 
         self.log.info("Cached %s at %s" % (self.url, self.cache_file))
         return str(self.cache_file)
+
+    def precache_test(test):
+        log = logging.getLogger('qemu-test')
+        log.setLevel(logging.DEBUG)
+        handler = logging.StreamHandler(sys.stdout)
+        handler.setLevel(logging.DEBUG)
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+        handler.setFormatter(formatter)
+        log.addHandler(handler)
+        for name, asset in vars(test.__class__).items():
+            if name.startswith("ASSET_") and type(asset) == Asset:
+                log.info("Attempting to cache '%s'" % asset)
+                asset.fetch()
+        log.removeHandler(handler)
+
+    def precache_suite(suite):
+        for test in suite:
+            if isinstance(test, unittest.TestSuite):
+                Asset.precache_suite(test)
+            elif isinstance(test, unittest.TestCase):
+                Asset.precache_test(test)
+
+    def precache_suites(path, cacheTstamp):
+        loader = unittest.loader.defaultTestLoader
+        tests = loader.loadTestsFromNames([path], None)
+
+        with open(cacheTstamp, "w") as fh:
+            Asset.precache_suite(tests)
tests/functional/qemu_test/testcase.py +8 -0
index 27bbf4a0af..21dbd90613 100644
@@ -21,6 +21,7 @@
 from qemu.machine import QEMUMachine
 from qemu.utils import kvm_available, tcg_available
 
+from .asset import Asset
 from .cmd import run_cmd
 from .config import BUILD_DIR
 
@@ -58,6 +59,13 @@ def tearDown(self):
         self.log.removeHandler(self._log_fh)
 
     def main():
+        path = os.path.basename(sys.argv[0])[:-3]
+
+        cache = os.environ.get("QEMU_TEST_PRECACHE", None)
+        if cache is not None:
+            Asset.precache_suites(path, cache)
+            return
+
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
         path = os.path.basename(sys.argv[0])[:-3]
Signature · 1 line
2.45.2
Tested-by on v3 9/24
#30 lore raw

Re: [PATCH v3 09/24] tests/functional: enable pre-emptive caching of assets

Quoted 19 lines · Daniel P. Berrangé

Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

    ninja test --suite functional
Quoted 8 lines

will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier.

    QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

nina check-func-precache

ninja or make, precache-functional.

Anyway, thanks for this. I can confirm that it works, and that it takes ~50m to finish here.

r~

#31 lore raw

Re: [PATCH v3 09/24] tests/functional: enable pre-emptive caching of assets

In reply to Daniel P. Berrangé
Quoted 57 lines · Daniel P. Berrangé

Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

    ninja test --suite functional

will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier.

    QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

nina check-func-precache

will download and cache all assets required by the functional tests

At the make level, precaching is always done by

make check-functional

---
  tests/Makefile.include                 |  3 ++-
  tests/functional/meson.build           | 33 +++++++++++++++++++++++--
  tests/functional/qemu_test/asset.py    | 34 ++++++++++++++++++++++++++
  tests/functional/qemu_test/testcase.py |  8 ++++++
  4 files changed, 75 insertions(+), 3 deletions(-)
tests/functional/qemu_test/testcase.py +1 -0
index 27bbf4a0af..21dbd90613 100644
@@ -21,6 +21,7 @@
  from qemu.machine import QEMUMachine
  from qemu.utils import kvm_available, tcg_available
 
+from .asset import Asset
  from .cmd import run_cmd
  from .config import BUILD_DIR
 

tests/functional/qemu_test/testcase.py def tearDown(self): · line 62

Quoted 4 lines
tests/functional/qemu_test/testcase.py +1 -0
@@ -58,6 +59,13 @@ def tearDown(self):
          self.log.removeHandler(self._log_fh)
 
      def main():
+        path = os.path.basename(sys.argv[0])[:-3]

This line should be added here in patch #2 in order to avoid ...

tests/functional/qemu_test/testcase.py def tearDown(self): · line 71

Quoted 9 lines
tests/functional/qemu_test/testcase.py +6 -0
+
+        cache = os.environ.get("QEMU_TEST_PRECACHE", None)
+        if cache is not None:
+            Asset.precache_suites(path, cache)
+            return
+
          tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                     test_output_log = pycotap.LogMode.LogToError)
          path = os.path.basename(sys.argv[0])[:-3]

.... this duplication.

#32 lore raw

PATCH v3 10/24 tests/functional: Convert some tests that download files via fetch_asset()

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Now that we've got the Asset class with pre-caching, we can convert some Avocado tests that use fetch_asset() for downloading their required files.

---
 MAINTAINERS                                   | 12 ++---
 tests/avocado/machine_mips_loongson3v.py      | 39 ---------------
 tests/functional/meson.build                  | 22 ++++++++
 .../test_arm_n8x0.py}                         | 36 ++++++-------
 .../test_avr_mega2560.py}                     | 22 ++++----
 .../test_loongarch64_virt.py}                 | 46 +++++++++--------
 tests/functional/test_mips64el_loongson3v.py  | 39 +++++++++++++++
 .../test_netdev_ethtool.py}                   | 50 +++++++------------
 .../ppc_405.py => functional/test_ppc_405.py} | 28 +++++------
 9 files changed, 154 insertions(+), 140 deletions(-)
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (66%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
MAINTAINERS +6 -6
index 98eddf7ae1..c2b4352edd 100644
@@ -222,7 +222,7 @@ S: Maintained
 F: docs/system/target-avr.rst
 F: gdb-xml/avr-cpu.xml
 F: target/avr/
-F: tests/avocado/machine_avr6.py
+F: tests/functional/test_avr_mega2560.py
 
 CRIS TCG CPUs
 M: Edgar E. Iglesias <[email protected]>
@@ -265,7 +265,7 @@ M: Song Gao <[email protected]>
 S: Maintained
 F: target/loongarch/
 F: tests/tcg/loongarch64/
-F: tests/avocado/machine_loongarch.py
+F: tests/functional/test_loongarch64_virt.py
 
 M68K TCG CPUs
 M: Laurent Vivier <[email protected]>
@@ -884,7 +884,7 @@ F: include/hw/display/blizzard.h
 F: include/hw/input/lm832x.h
 F: include/hw/input/tsc2xxx.h
 F: include/hw/misc/cbus.h
-F: tests/avocado/machine_arm_n8x0.py
+F: tests/functional/test_arm_n8x0.py
 F: docs/system/arm/nseries.rst
 
 Palm
@@ -1393,7 +1393,7 @@ F: hw/mips/loongson3_bootp.h
 F: hw/mips/loongson3_virt.c
 F: include/hw/intc/loongson_ipi.h
 F: include/hw/intc/loongson_liointc.h
-F: tests/avocado/machine_mips_loongson3v.py
+F: tests/functional/test_mips64el_loongson3v.py
 
 Boston
 M: Paul Burton <[email protected]>
@@ -1419,7 +1419,7 @@ PowerPC Machines
 L: [email protected]
 S: Orphan
 F: hw/ppc/ppc405*
-F: tests/avocado/ppc_405.py
+F: tests/functional/test_ppc_405.py
 
 Bamboo
 L: [email protected]
@@ -2486,7 +2486,7 @@ R: Sriram Yagnaraman <[email protected]>
 S: Maintained
 F: docs/system/devices/igb.rst
 F: hw/net/igb*
-F: tests/avocado/netdev-ethtool.py
+F: tests/functional/test_netdev_ethtool.py
 F: tests/qtest/igb-test.c
 F: tests/qtest/libqos/igb.c
 
tests/avocado/machine_mips_loongson3v.py deleted +0 -39
deleted file mode 100644
index 5194cf18c9..0000000000
@@ -1,39 +0,0 @@
-# Functional tests for the Generic Loongson-3 Platform.
-#
-# Copyright (c) 2021 Jiaxun Yang <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-#
-# SPDX-License-Identifier: GPL-2.0-or-later
-
-import os
-import time
-
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class MipsLoongson3v(QemuSystemTest):
-    timeout = 60
-
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
-    def test_pmon_serial_console(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=endian:little
-        :avocado: tags=machine:loongson3-virt
-        :avocado: tags=cpu:Loongson-3A1000
-        :avocado: tags=device:liointc
-        :avocado: tags=device:goldfish_rtc
-        """
-
-        pmon_hash = '7c8b45dd81ccfc55ff28f5aa267a41c3'
-        pmon_path = self.fetch_asset('https://github.com/loongson-community/pmon/'
-                                    'releases/download/20210112/pmon-3avirt.bin',
-                                     asset_hash=pmon_hash, algorithm='md5')
-
-        self.vm.set_console()
-        self.vm.add_args('-bios', pmon_path)
-        self.vm.launch()
-        wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
tests/functional/meson.build +22 -0
index cef74b82a9..32c7cc2d3c 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'netdev_ethtool' : 180,
 }
 
 tests_generic = [
@@ -19,10 +20,30 @@ tests_generic = [
   'version',
 ]
 
+tests_arm_thorough = [
+  'arm_n8x0',
+]
+
+tests_avr_thorough = [
+  'avr_mega2560',
+]
+
+tests_loongarch64_thorough = [
+  'loongarch64_virt',
+]
+
+tests_mips64el_thorough = [
+  'mips64el_loongson3v',
+]
+
 tests_ppc_quick = [
   'ppc_74xx',
 ]
 
+tests_ppc_thorough = [
+  'ppc_405',
+]
+
 tests_x86_64_quick = [
   'cpu_queries',
   'mem_addr_space',
@@ -31,6 +52,7 @@ tests_x86_64_quick = [
 ]
 
 tests_x86_64_thorough = [
+  'netdev_ethtool',
 ]
 
 precache_all = []
tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +18 -18
old mode 100644
new mode 100755
similarity index 51%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..5dc473f66f
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,21 +12,22 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class N8x0Machine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
 
     timeout = 90
 
+    ASSET_KERNEL = Asset(('http://stskeeps.subnetmask.net/meego-n8x0/'
+                          'meego-arm-n8x0-1.0.80.20100712.1431-'
+                          'vmlinuz-2.6.35~rc4-129.1-n8x0'),
+                         'e9d5ab8d7548923a0061b6fbf601465e479ed269')
+
     def __do_test_n8x0(self):
-        kernel_url = ('http://stskeeps.subnetmask.net/meego-n8x0/'
-                      'meego-arm-n8x0-1.0.80.20100712.1431-'
-                      'vmlinuz-2.6.35~rc4-129.1-n8x0')
-        kernel_hash = 'e9d5ab8d7548923a0061b6fbf601465e479ed269'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        kernel_path = self.ASSET_KERNEL.fetch()
 
         self.vm.set_console(console_index=1)
         self.vm.add_args('-kernel', kernel_path,
@@ -32,18 +35,15 @@ def __do_test_n8x0(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'TSC2005 driver initializing')
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n800(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n800
-        """
+        self.set_machine('n800')
         self.__do_test_n8x0()
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n810(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n810
-        """
+        self.set_machine('n810')
         self.__do_test_n8x0()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_avr_mega2560.py renamed from tests/avocado/machine_avr6.py +12 -10
old mode 100644
new mode 100755
similarity index 73%
rename from tests/avocado/machine_avr6.py
rename to tests/functional/test_avr_mega2560.py
index 5485db79c6..a2d026e03a
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # QEMU AVR integration tests
 #
@@ -19,26 +20,24 @@
 
 import time
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest, Asset
 
 class AVR6Machine(QemuSystemTest):
     timeout = 5
 
+    ASSET_ROM = Asset(('https://github.com/seharris/qemu-avr-tests'
+                       '/raw/36c3e67b8755dcf/free-rtos/Demo'
+                       '/AVR_ATMega2560_GCC/demo.elf'),
+                      '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4')
+
     def test_freertos(self):
-        """
-        :avocado: tags=arch:avr
-        :avocado: tags=machine:arduino-mega-2560-v3
-        """
         """
         https://github.com/seharris/qemu-avr-tests/raw/master/free-rtos/Demo/AVR_ATMega2560_GCC/demo.elf
         constantly prints out 'ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX'
         """
-        rom_url = ('https://github.com/seharris/qemu-avr-tests'
-                   '/raw/36c3e67b8755dcf/free-rtos/Demo'
-                   '/AVR_ATMega2560_GCC/demo.elf')
-        rom_hash = '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4'
-        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+        rom_path = self.ASSET_ROM.fetch()
 
+        self.set_machine('arduino-mega-2560-v3')
         self.vm.add_args('-bios', rom_path)
         self.vm.add_args('-nographic')
         self.vm.launch()
@@ -48,3 +47,6 @@ def test_freertos(self):
 
         self.assertIn('ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX',
                 self.vm.get_log())
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_loongarch64_virt.py renamed from tests/avocado/machine_loongarch.py +25 -21

71 lines not on this page · raw message

tests/functional/test_mips64el_loongson3v.py new file +39 -0
new file mode 100755
index 0000000000..55d62928c7
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+#
+# Functional tests for the Generic Loongson-3 Platform.
+#
+# Copyright (c) 2021 Jiaxun Yang <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import time
+
+from unittest import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+
+class MipsLoongson3v(QemuSystemTest):
+    timeout = 60
+
+    ASSET_PMON = Asset(
+        ('https://github.com/loongson-community/pmon/'
+         'releases/download/20210112/pmon-3avirt.bin'),
+        'fcdf6bb2cb7885a4a62f31fcb0d5e368bac7b6cea28f40c6dfa678af22fea20a')
+
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    def test_pmon_serial_console(self):
+        self.set_machine('loongson3-virt')
+
+        pmon_path = self.ASSET_PMON.fetch()
+
+        self.vm.set_console()
+        self.vm.add_args('-bios', pmon_path)
+        self.vm.launch()
+        wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_netdev_ethtool.py renamed from tests/avocado/netdev-ethtool.py +18 -32

89 lines not on this page · raw message

tests/functional/test_ppc_405.py renamed from tests/avocado/ppc_405.py +14 -14
old mode 100644
new mode 100755
similarity index 53%
rename from tests/avocado/ppc_405.py
rename to tests/functional/test_ppc_405.py
index 4e7e01aa76..5e200eab95
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that the U-Boot firmware boots on ppc 405 machines and check the console
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -5,20 +7,20 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class Ppc405Machine(QemuSystemTest):
 
     timeout = 90
 
+    ASSET_UBOOT = Asset(('https://gitlab.com/huth/u-boot/-/raw/'
+                         'taihu-2021-10-09/u-boot-taihu.bin'),
+                        '3208940e908a5edc7c03eab072c60f0dcfadc2ab')
+
     def do_test_ppc405(self):
-        uboot_url = ('https://gitlab.com/huth/u-boot/-/raw/'
-                     'taihu-2021-10-09/u-boot-taihu.bin')
-        uboot_hash = ('3208940e908a5edc7c03eab072c60f0dcfadc2ab');
-        file_path = self.fetch_asset(uboot_url, asset_hash=uboot_hash)
+        file_path = self.ASSET_UBOOT.fetch()
         self.vm.set_console(console_index=1)
         self.vm.add_args('-bios', file_path)
         self.vm.launch()
@@ -26,11 +28,9 @@ def do_test_ppc405(self):
         exec_command_and_wait_for_pattern(self, 'reset', 'AMCC PowerPC 405EP')
 
     def test_ppc_ref405ep(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:ref405ep
-        :avocado: tags=cpu:405ep
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('ref405ep')
         self.do_test_ppc405()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#33 lore raw

PATCH v3 11/24 tests/functional: Add a function for extracting files from an archive

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

---
 tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/functional/qemu_test/utils.py
tests/functional/qemu_test/utils.py new file +21 -0
new file mode 100644
index 0000000000..4eb5e5d5e5
@@ -0,0 +1,21 @@
+# Utilities for python-based QEMU tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# Authors:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import tarfile
+
+def archive_extract(archive, dest_dir, member=None):
+    with tarfile.open(archive) as tf:
+        if hasattr(tarfile, 'data_filter'):
+            tf.extraction_filter = getattr(tarfile, 'data_filter',
+                                           (lambda member, path: member))
+        if member:
+            tf.extract(member=member, path=dest_dir)
+        else:
+            tf.extractall(path=dest_dir)
Signature · 1 line
2.45.2
Tested-by on v3 11/24
#34 lore raw

Re: [PATCH v3 11/24] tests/functional: Add a function for extracting files from an archive

Quoted 9 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

Quoted 7 lines
---
  tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
  1 file changed, 21 insertions(+)

create mode 100644 tests/functional/qemu_test/utils.py

#35 lore raw

PATCH v3 12/24 tests/functional: Convert some avocado tests that needed avocado.utils.archive

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

---
 MAINTAINERS                                   |  6 +--
 tests/avocado/machine_sparc64_sun4u.py        | 36 -----------------
 tests/functional/meson.build                  |  6 +++
 .../test_arm_canona1100.py}                   | 30 ++++++++------
 .../test_ppc_bamboo.py}                       | 32 +++++++--------
 tests/functional/test_sparc64_sun4u.py        | 40 +++++++++++++++++++
 6 files changed, 82 insertions(+), 68 deletions(-)
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (52%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
MAINTAINERS +3 -3
index c2b4352edd..ff952d683b 100644
@@ -731,7 +731,7 @@ S: Odd Fixes
 F: include/hw/arm/digic.h
 F: hw/*/digic*
 F: include/hw/*/digic*
-F: tests/avocado/machine_arm_canona1100.py
+F: tests/functional/test_arm_canona1100.py
 F: docs/system/arm/digic.rst
 
 Goldfish RTC
@@ -1426,7 +1426,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/ppc440_bamboo.c
 F: hw/pci-host/ppc4xx_pci.c
-F: tests/avocado/ppc_bamboo.py
+F: tests/functional/test_ppc_bamboo.py
 
 e500
 L: [email protected]
@@ -1712,7 +1712,7 @@ F: include/hw/pci-host/sabre.h
 F: hw/pci-bridge/simba.c
 F: include/hw/pci-bridge/simba.h
 F: pc-bios/openbios-sparc64
-F: tests/avocado/machine_sparc64_sun4u.py
+F: tests/functional/test_sparc64_sun4u.py
 
 Sun4v
 M: Artyom Tarasenko <[email protected]>
tests/avocado/machine_sparc64_sun4u.py deleted +0 -36
deleted file mode 100644
index d333c0ae91..0000000000
@@ -1,36 +0,0 @@
-# Functional test that boots a Linux kernel and checks the console
-#
-# Copyright (c) 2020 Red Hat, Inc.
-#
-# Author:
-#  Thomas Huth <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later. See the COPYING file in the top-level directory.
-
-import os
-
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
-from boot_linux_console import LinuxKernelTest
-
-class Sun4uMachine(LinuxKernelTest):
-    """Boots the Linux kernel and checks that the console is operational"""
-
-    timeout = 90
-
-    def test_sparc64_sun4u(self):
-        """
-        :avocado: tags=arch:sparc64
-        :avocado: tags=machine:sun4u
-        """
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day23.tar.xz')
-        tar_hash = '142db83cd974ffadc4f75c8a5cad5bcc5722c240'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
-        self.vm.set_console()
-        self.vm.add_args('-kernel', self.workdir + '/day23/vmlinux',
-                         '-append', self.KERNEL_COMMON_COMMAND_LINE)
-        self.vm.launch()
-        wait_for_console_pattern(self, 'Starting logging: OK')
tests/functional/meson.build +6 -0
index 32c7cc2d3c..91bf80f0f4 100644
@@ -21,6 +21,7 @@ tests_generic = [
 ]
 
 tests_arm_thorough = [
+  'arm_canona1100',
   'arm_n8x0',
 ]
 
@@ -42,6 +43,11 @@ tests_ppc_quick = [
 
 tests_ppc_thorough = [
   'ppc_405',
+  'ppc_bamboo',
+]
+
+tests_sparc64_thorough = [
+  'sparc64_sun4u',
 ]
 
 tests_x86_64_quick = [
tests/functional/test_arm_canona1100.py renamed from tests/avocado/machine_arm_canona1100.py +17 -13
old mode 100644
new mode 100755
similarity index 52%
rename from tests/avocado/machine_arm_canona1100.py
rename to tests/functional/test_arm_canona1100.py
index a42d8b0f2b..57cecbbb36
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots the canon-a1100 machine with firmware
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -8,28 +10,30 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
 class CanonA1100Machine(QemuSystemTest):
     """Boots the barebox firmware and checks that the console is operational"""
 
     timeout = 90
 
+    ASSET_BIOS = Asset(('https://qemu-advcal.gitlab.io'
+                        '/qac-best-of-multiarch/download/day18.tar.xz'),
+                       '068b5fc4242b29381acee94713509f8a876e9db6')
+
     def test_arm_canona1100(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:canon-a1100
-        :avocado: tags=device:pflash_cfi02
-        """
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day18.tar.xz')
-        tar_hash = '068b5fc4242b29381acee94713509f8a876e9db6'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('canon-a1100')
+
+        file_path = self.ASSET_BIOS.fetch()
+        archive_extract(file_path, dest_dir=self.workdir,
+                        member="day18/barebox.canon-a1100.bin")
         self.vm.set_console()
         self.vm.add_args('-bios',
                          self.workdir + '/day18/barebox.canon-a1100.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'running /env/bin/init')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_bamboo.py renamed from tests/avocado/ppc_bamboo.py +16 -16
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/ppc_bamboo.py
rename to tests/functional/test_ppc_bamboo.py
index a81be3d608..8a1f7eca4d
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on the ppc bamboo board and check the console
 #
 # Copyright (c) 2021 Red Hat
@@ -5,30 +7,25 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class BambooMachine(QemuSystemTest):
 
     timeout = 90
 
+    ASSET_IMAGE = Asset(('http://landley.net/aboriginal/downloads/binaries/'
+                         'system-image-powerpc-440fp.tar.gz'),
+                        '53e5f16414b195b82d2c70272f81c2eedb39bad9')
+
     def test_ppc_bamboo(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:bamboo
-        :avocado: tags=cpu:440epb
-        :avocado: tags=device:rtl8139
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('bamboo')
         self.require_accelerator("tcg")
         self.require_netdev('user')
-        tar_url = ('http://landley.net/aboriginal/downloads/binaries/'
-                   'system-image-powerpc-440fp.tar.gz')
-        tar_hash = '53e5f16414b195b82d2c70272f81c2eedb39bad9'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir +
                                    '/system-image-powerpc-440fp/linux',
@@ -40,3 +37,6 @@ def test_ppc_bamboo(self):
         exec_command_and_wait_for_pattern(self, 'ping 10.0.2.2',
                                           '10.0.2.2 is alive!')
         exec_command_and_wait_for_pattern(self, 'halt', 'System Halted')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_sparc64_sun4u.py new file +40 -0
new file mode 100755
index 0000000000..3ac18aebd1
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+#
+# Functional test that boots a Linux kernel and checks the console
+#
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import os
+
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+
+class Sun4uMachine(QemuSystemTest):
+    """Boots the Linux kernel and checks that the console is operational"""
+
+    timeout = 90
+
+    ASSET_IMAGE = Asset(('https://qemu-advcal.gitlab.io'
+                         '/qac-best-of-multiarch/download/day23.tar.xz'),
+                        '142db83cd974ffadc4f75c8a5cad5bcc5722c240')
+
+    def test_sparc64_sun4u(self):
+        self.set_machine('sun4u')
+        file_path = self.ASSET_IMAGE.fetch()
+        kernel_name = 'day23/vmlinux'
+        archive_extract(file_path, self.workdir, kernel_name)
+        self.vm.set_console()
+        self.vm.add_args('-kernel', os.path.join(self.workdir, kernel_name),
+                         '-append', 'printk.time=0')
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Starting logging: OK')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 12/24
#36 lore raw

Re: [PATCH v3 12/24] tests/functional: Convert some avocado tests that needed avocado.utils.archive

Quoted 16 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

---
  MAINTAINERS                                   |  6 +--
  tests/avocado/machine_sparc64_sun4u.py        | 36 -----------------
  tests/functional/meson.build                  |  6 +++
  .../test_arm_canona1100.py}                   | 30 ++++++++------
  .../test_ppc_bamboo.py}                       | 32 +++++++--------
  tests/functional/test_sparc64_sun4u.py        | 40 +++++++++++++++++++
  6 files changed, 82 insertions(+), 68 deletions(-)
Quoted 6 lines

delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (52%) mode change 100644 => 100755
rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%) mode change 100644 => 100755
create mode 100755 tests/functional/test_sparc64_sun4u.py

#37 lore raw

PATCH v3 13/24 tests/functional: Convert the s390x avocado tests into standalone tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
 MAINTAINERS                                   |  4 +-
 tests/functional/meson.build                  |  6 ++
 tests/functional/qemu_test/utils.py           | 14 +++
 .../test_s390x_ccw_virtio.py}                 | 79 ++++++++---------
 .../test_s390x_topology.py}                   | 86 ++++++++-----------
 5 files changed, 94 insertions(+), 95 deletions(-)
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index ff952d683b..c78edb078a 100644
@@ -1739,7 +1739,7 @@ S: Supported
 F: hw/s390x/
 F: include/hw/s390x/
 F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: [email protected]
 
@@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
 F: target/s390x/kvm/stsi-topology.c
 F: docs/devel/s390-cpu-topology.rst
 F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
 
 X86 Machines
 ------------
tests/functional/meson.build +6 -0
index 91bf80f0f4..ce5e5be2f3 100644
@@ -12,6 +12,7 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  's390x_ccw_virtio' : 180,
 }
 
 tests_generic = [
@@ -46,6 +47,11 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_s390x_thorough = [
+  's390x_ccw_virtio',
+  's390x_topology',
+]
+
 tests_sparc64_thorough = [
   'sparc64_sun4u',
 ]
tests/functional/qemu_test/utils.py +14 -0
index 4eb5e5d5e5..a12dac51b6 100644
@@ -8,6 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import lzma
+import os
+import shutil
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):
@@ -19,3 +22,14 @@ def archive_extract(archive, dest_dir, member=None):
             tf.extract(member=member, path=dest_dir)
         else:
             tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with lzma.open(xz_path, 'rb') as lzma_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(lzma_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
tests/functional/test_s390x_ccw_virtio.py renamed from tests/avocado/machine_s390_ccw_virtio.py +38 -41

113 lines not on this page · raw message

tests/functional/test_s390x_topology.py renamed from tests/avocado/s390_topology.py +34 -52

183 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by on v3 13/24
#38 lore raw

Re: [PATCH v3 13/24] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 18 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
  MAINTAINERS                                   |  4 +-
  tests/functional/meson.build                  |  6 ++
  tests/functional/qemu_test/utils.py           | 14 +++
  .../test_s390x_ccw_virtio.py}                 | 79 ++++++++---------
  .../test_s390x_topology.py}                   | 86 ++++++++-----------
  5 files changed, 94 insertions(+), 95 deletions(-)
Quoted 4 lines

rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%) mode change 100644 => 100755
rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%) mode change 100644 => 100755

Conversion LGTM, so:

Now while testing I'm getting errors, but maybe pre-existing...

pyvenv/bin/meson test --no-rebuild -t 1 --setup thorough --num-processes 1 --print-errorlogs --suite func-s390x --suite func-s390x-thorough qemu:func-thorough+func-s390x-thorough+thorough / func-s390x-s390x_ccw_virtio time out (After 180.0 seconds)
1/5 qemu:func-thorough+func-s390x-thorough+thorough / func-s390x-s390x_ccw_virtio TIMEOUT 180.05s killed by signal 15 SIGTERM
▶ 2/5 test_empty_cpu_model.EmptyCPUModel.test
FAIL
2/5 qemu:func-quick+func-s390x / func-s390x-empty_cpu_model
ERROR 0.21s exit status 1
stderr:
Traceback (most recent call last):
File "/home/user/qemu/tests/functional/test_empty_cpu_model.py", line
21, in test
self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
AssertionError: Regex didn't match: '-cpu option cannot be empty' not found in 'Could not access KVM kernel module: Permission denied\nqemu-system-s390x: failed to initialize kvm: Permission denied\n'

(test program exited with status code 1)
▶ 4/5 test_version.Version.test_qmp_human_info_version
FAIL
Traceback (most recent call last):
File "/home/user/qemu/python/qemu/qmp/protocol.py", line 425, in
_session_guard
await coro
File "/home/user/qemu/python/qemu/qmp/qmp_client.py", line 250, in
_establish_session
self._greeting = await self._get_greeting()
File "/home/user/qemu/python/qemu/qmp/qmp_client.py", line 270, in
_get_greeting
msg = await self._recv()
File "/home/user/qemu/python/qemu/qmp/protocol.py", line 1009, in _recv
message = await self._do_recv()
File "/home/user/qemu/python/qemu/qmp/qmp_client.py", line 402, in
_do_recv
msg_bytes = await self._readline()
File "/home/user/qemu/python/qemu/qmp/protocol.py", line 977, in
_readline
raise EOFError
EOFError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/user/qemu/python/qemu/machine/machine.py", line 448, in
launch

    self._launch()
  File "/home/user/qemu/python/qemu/machine/machine.py", line 487, in 
_launch
    self._post_launch()
  File "/home/user/qemu/python/qemu/machine/machine.py", line 381, in 
_post_launch
    self._qmp.connect()
  File "/home/user/qemu/python/qemu/qmp/legacy.py", line 153, in connect
    self._sync(
  File "/home/user/qemu/python/qemu/qmp/legacy.py", line 102, in _sync
    return self._aloop.run_until_complete(

File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in
run_until_complete
return future.result()
File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
return await fut
File "/home/user/qemu/python/qemu/qmp/protocol.py", line 382, in connect

    await self._session_guard(

File "/home/user/qemu/python/qemu/qmp/protocol.py", line 456, in
_session_guard
raise ConnectError(emsg, err) from err
qemu.qmp.protocol.ConnectError: Failed to establish session: EOFError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/user/qemu/tests/functional/test_version.py", line 22, in
test_qmp_human_info_version
self.vm.launch()
File "/home/user/qemu/python/qemu/machine/machine.py", line 461, in
launch

    raise VMLaunchFailure(
qemu.machine.machine.VMLaunchFailure: ConnectError: Failed to establish session: EOFError
	Exit code: 1
	Command: qemu-system-s390x -display none -vga none -chardev 

socket,id=mon,fd=5 -mon chardev=mon,mode=control -machine none -nodefaults
Output: Could not access KVM kernel module: Permission denied
qemu-system-s390x: failed to initialize kvm: Permission denied

5/5 qemu:func-thorough+func-s390x-thorough+thorough / func-s390x-s390x_topology SKIP 1.14s 0 subtests passed

Summary of Failures:

1/5 qemu:func-thorough+func-s390x-thorough+thorough / func-s390x-s390x_ccw_virtio TIMEOUT 180.05s killed by signal 15 SIGTERM
2/5 qemu:func-quick+func-s390x / func-s390x-empty_cpu_model
ERROR 0.21s exit status 1
4/5 qemu:func-quick+func-s390x / func-s390x-version
ERROR 0.25s exit status 1

I'm surprised by "Could not access KVM kernel module" because kvm_available() checks for RW access:

def kvm_available(target_arch: Optional[str] = None,

                    qemu_bin: Optional[str] = None) -> bool:
      if not os.access("/dev/kvm", os.R_OK | os.W_OK):
          return False

Maybe missing require_accelerator() in tests?

#39 lore raw

Re: [PATCH v3 13/24] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
  MAINTAINERS                                   |  4 +-
  tests/functional/meson.build                  |  6 ++
  tests/functional/qemu_test/utils.py           | 14 +++
  .../test_s390x_ccw_virtio.py}                 | 79 ++++++++---------
  .../test_s390x_topology.py}                   | 86 ++++++++-----------
  5 files changed, 94 insertions(+), 95 deletions(-)
Quoted 4 lines

rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%) mode change 100644 => 100755
rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%) mode change 100644 => 100755

Quoted 5 lines

Conversion LGTM, so:

Now while testing I'm getting errors, but maybe pre-existing...

What host arch / OS are you running tests from ?

I got through the full set of testing on Fedora x86_64, so wondering if something differs.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#40 lore raw

Re: [PATCH v3 13/24] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 27 lines · Daniel P. Berrangé
Quoted 26 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
   MAINTAINERS                                   |  4 +-
   tests/functional/meson.build                  |  6 ++
   tests/functional/qemu_test/utils.py           | 14 +++
   .../test_s390x_ccw_virtio.py}                 | 79 ++++++++---------
   .../test_s390x_topology.py}                   | 86 ++++++++-----------
   5 files changed, 94 insertions(+), 95 deletions(-)
Quoted 4 lines

rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%) mode change 100644 => 100755
rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%) mode change 100644 => 100755

Conversion LGTM, so:

Quoted 8 lines
Quoted 3 lines

Now while testing I'm getting errors, but maybe pre-existing...

What host arch / OS are you running tests from ?

I got through the full set of testing on Fedora x86_64, so wondering if something differs.

Linux s390x but I'm not in the KVM group, I need to find another place to run these (or ask Alex :P).

#41 lore raw

Re: [PATCH v3 13/24] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 32 lines · Philippe Mathieu-Daudé
Quoted 31 lines · Daniel P. Berrangé
Quoted 29 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
   MAINTAINERS                                   |  4 +- tests/functional/meson.build                  |  6 ++ tests/functional/qemu_test/utils.py           | 14 +++ .../test_s390x_ccw_virtio.py}                 | 79 ++++++++--------- .../test_s390x_topology.py}                   | 86 ++++++++-----------

Quoted 5 lines
   5 files changed, 94 insertions(+), 95 deletions(-)

   rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%) mode change 100644 => 100755
   rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%) mode change 100644 => 100755

Conversion LGTM, so:

Now while testing I'm getting errors, but maybe pre-existing...

Quoted 7 lines
Quoted 4 lines

What host arch / OS are you running tests from ?

I got through the full set of testing on Fedora x86_64, so wondering if something differs.

Linux s390x but I'm not in the KVM group, I need to find another place to run these (or ask Alex :P).

Sounds like a pre-existing problem ... I guess the test should check whether it can access /dev/kvm before trying to use it...?

Thomas

#42 lore raw

Re: [PATCH v3 13/24] tests/functional: Convert the s390x avocado tests into standalone tests

Quoted 35 lines · Thomas Huth
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 33 lines · Daniel P. Berrangé
Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
   MAINTAINERS                                   |  4 +- tests/functional/meson.build                  |  6 ++ tests/functional/qemu_test/utils.py           | 14 +++ .../test_s390x_ccw_virtio.py}                 | 79 ++++++++--------- .../test_s390x_topology.py}                   | 86 ++++++++-----------

Quoted 5 lines
   5 files changed, 94 insertions(+), 95 deletions(-)

   rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%) mode change 100644 => 100755
   rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%) mode change 100644 => 100755

Quoted 5 lines

Conversion LGTM, so:

Now while testing I'm getting errors, but maybe pre-existing...

What host arch / OS are you running tests from ?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I got through the full set of testing on Fedora x86_64, so wondering if something differs.

Linux s390x but I'm not in the KVM group, I need to find another place to run these (or ask Alex :P).

Sounds like a pre-existing problem ... I guess the test should check whether it can access /dev/kvm before trying to use it...?

Ah, no, I missed your comment in your earlier patch that kvm_available() already checks this ... so the problem must be something else?

Thomas

#43 lore raw

PATCH v3 14/24 tests/functional: Convert the x86_cpu_model_versions test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                  |  1 +
 .../test_x86_cpu_model_versions.py}           | 63 ++++++-------------
 2 files changed, 20 insertions(+), 44 deletions(-)
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index ce5e5be2f3..e85d869948 100644
@@ -61,6 +61,7 @@ tests_x86_64_quick = [
   'mem_addr_space',
   'pc_cpu_hotplug_props',
   'virtio_version',
+  'x86_cpu_model_versions',
 ]
 
 tests_x86_64_thorough = [
tests/functional/test_x86_cpu_model_versions.py renamed from tests/avocado/x86_cpu_model_versions.py +19 -44

154 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 14/24
#44 lore raw

Re: [PATCH v3 14/24] tests/functional: Convert the x86_cpu_model_versions test

Quoted 14 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                  |  1 +
  .../test_x86_cpu_model_versions.py}           | 63 ++++++-------------
  2 files changed, 20 insertions(+), 44 deletions(-)

rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%) mode change 100644 => 100755

tests/functional/test_x86_cpu_model_versions.py def test_4_1_alias(self): · line 220

Quoted 7 lines
tests/functional/test_x86_cpu_model_versions.py renamed from tests/avocado/x86_cpu_model_versions.py +1 -2
@@ -217,9 +216,8 @@ def test_4_1_alias(self):
      def test_none_alias(self):
          """
          Check if unversioned CPU model is an alias pointing to some version
-
-        :avocado: tags=machine:none
          """
+        self.machine = 'none'

self.set_machine('none')

Review def test_4_1_alias(self): · line 222

Quoted 2 lines
(quoted diff) +0 -0
          self.vm.add_args('-S')
          self.vm.launch()
#45 lore raw

PATCH v3 15/24 tests/functional: Convert the microblaze avocado tests into standalone tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The machine_microblaze.py file contained two tests, one for each endianess. Since we only support one QEMU target binary per file in the new functional test environment, we have to split this file up into two files now.

---
 MAINTAINERS                                   |  2 +-
 tests/avocado/machine_microblaze.py           | 61 -------------------
 tests/functional/meson.build                  |  8 +++
 .../functional/test_microblaze_s3adsp1800.py  | 39 ++++++++++++
 .../test_microblazeel_s3adsp1800.py           | 42 +++++++++++++
 5 files changed, 90 insertions(+), 62 deletions(-)
 delete mode 100644 tests/avocado/machine_microblaze.py
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
MAINTAINERS +1 -1
index c78edb078a..e1d15c9696 100644
@@ -1327,7 +1327,7 @@ M: Edgar E. Iglesias <[email protected]>
 S: Maintained
 F: hw/microblaze/petalogix_s3adsp1800_mmu.c
 F: include/hw/char/xilinx_uartlite.h
-F: tests/avocado/machine_microblaze.py
+F: tests/functional/test_microblaze*.py
 
 petalogix_ml605
 M: Edgar E. Iglesias <[email protected]>
tests/avocado/machine_microblaze.py deleted +0 -61

61 lines not on this page · raw message

tests/functional/meson.build +8 -0
index e85d869948..d6a1a3a116 100644
@@ -34,6 +34,14 @@ tests_loongarch64_thorough = [
   'loongarch64_virt',
 ]
 
+tests_microblaze_thorough = [
+  'microblaze_s3adsp1800'
+]
+
+tests_microblazeel_thorough = [
+  'microblazeel_s3adsp1800'
+]
+
 tests_mips64el_thorough = [
   'mips64el_loongson3v',
 ]
tests/functional/test_microblaze_s3adsp1800.py new file +39 -0
new file mode 100755
index 0000000000..7618531b8c
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+#
+# Functional test that boots a microblaze Linux kernel and checks the console
+#
+# Copyright (c) 2018, 2021 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import time
+from qemu_test import exec_command, exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+
+class MicroblazeMachine(QemuSystemTest):
+
+    timeout = 90
+
+    ASSET_IMAGE = Asset(('https://qemu-advcal.gitlab.io'
+                         '/qac-best-of-multiarch/download/day17.tar.xz'),
+                        '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f')
+
+    def test_microblaze_s3adsp1800(self):
+        self.set_machine('petalogix-s3adsp1800')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
+        self.vm.set_console()
+        self.vm.add_args('-kernel', self.workdir + '/day17/ballerina.bin')
+        self.vm.launch()
+        wait_for_console_pattern(self, 'This architecture does not have '
+                                       'kernel memory protection')
+        # Note:
+        # The kernel sometimes gets stuck after the "This architecture ..."
+        # message, that's why we don't test for a later string here. This
+        # needs some investigation by a microblaze wizard one day...
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_microblazeel_s3adsp1800.py new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 15/24
#46 lore raw

Re: [PATCH v3 15/24] tests/functional: Convert the microblaze avocado tests into standalone tests

Quoted 15 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The machine_microblaze.py file contained two tests, one for each endianess. Since we only support one QEMU target binary per file in the new functional test environment, we have to split this file up into two files now.

---
  MAINTAINERS                                   |  2 +-
  tests/avocado/machine_microblaze.py           | 61 -------------------
  tests/functional/meson.build                  |  8 +++
  .../functional/test_microblaze_s3adsp1800.py  | 39 ++++++++++++
  .../test_microblazeel_s3adsp1800.py           | 42 +++++++++++++
  5 files changed, 90 insertions(+), 62 deletions(-)
Quoted 3 lines

delete mode 100644 tests/avocado/machine_microblaze.py create mode 100755 tests/functional/test_microblaze_s3adsp1800.py create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py

#47 lore raw

PATCH v3 16/24 tests/functional: Convert the riscv_opensbi avocado test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

---
 tests/avocado/riscv_opensbi.py         | 63 --------------------------
 tests/functional/meson.build           |  8 ++++
 tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
 3 files changed, 44 insertions(+), 63 deletions(-)
 delete mode 100644 tests/avocado/riscv_opensbi.py
 create mode 100755 tests/functional/test_riscv_opensbi.py
tests/avocado/riscv_opensbi.py deleted +0 -63
deleted file mode 100644
index bfff9cc3c3..0000000000
@@ -1,63 +0,0 @@
-# OpenSBI boot test for RISC-V machines
-#
-# Copyright (c) 2022, Ventana Micro
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class RiscvOpenSBI(QemuSystemTest):
-    """
-    :avocado: tags=accel:tcg
-    """
-    timeout = 5
-
-    def boot_opensbi(self):
-        self.vm.set_console()
-        self.vm.launch()
-        wait_for_console_pattern(self, 'Platform Name')
-        wait_for_console_pattern(self, 'Boot HART MEDELEG')
-
-    def test_riscv32_spike(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_spike(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:spike
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_sifive_u(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:sifive_u
-        """
-        self.boot_opensbi()
-
-    def test_riscv32_virt(self):
-        """
-        :avocado: tags=arch:riscv32
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
-
-    def test_riscv64_virt(self):
-        """
-        :avocado: tags=arch:riscv64
-        :avocado: tags=machine:virt
-        """
-        self.boot_opensbi()
tests/functional/meson.build +8 -0
index d6a1a3a116..55021c4e4d 100644
@@ -55,6 +55,14 @@ tests_ppc_thorough = [
   'ppc_bamboo',
 ]
 
+tests_riscv32_quick = [
+  'riscv_opensbi',
+]
+
+tests_riscv64_quick = [
+  'riscv_opensbi',
+]
+
 tests_s390x_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/test_riscv_opensbi.py new file +36 -0
new file mode 100755
index 0000000000..d077e40f42
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+#
+# OpenSBI boot test for RISC-V machines
+#
+# Copyright (c) 2022, Ventana Micro
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+
+class RiscvOpenSBI(QemuSystemTest):
+
+    timeout = 5
+
+    def boot_opensbi(self):
+        self.vm.set_console()
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Platform Name')
+        wait_for_console_pattern(self, 'Boot HART MEDELEG')
+
+    def test_riscv_spike(self):
+        self.set_machine('spike')
+        self.boot_opensbi()
+
+    def test_riscv_sifive_u(self):
+        self.set_machine('sifive_u')
+        self.boot_opensbi()
+
+    def test_riscv_virt(self):
+        self.set_machine('virt')
+        self.boot_opensbi()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Tested-by on v3 16/24
#48 lore raw

Re: [PATCH v3 16/24] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 9 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Quoted 7 lines
---
  tests/avocado/riscv_opensbi.py         | 63 --------------------------
  tests/functional/meson.build           |  8 ++++
  tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
  3 files changed, 44 insertions(+), 63 deletions(-)

delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

Please squash:

-- >8 --

MAINTAINERS +1 -0
index 98eddf7ae1..a906218f9d 100644
@@ -336,6 +337,7 @@ F: include/hw/riscv/
 F: linux-user/host/riscv32/
 F: linux-user/host/riscv64/
 F: tests/tcg/riscv64/
+F: tests/functional/test_riscv_opensbi.py
 
 RISC-V XThead* extensions
 M: Christoph Muellner <[email protected]>

---

#49 lore raw

Re: [PATCH v3 16/24] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 26 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Quoted 7 lines

---
  tests/avocado/riscv_opensbi.py         | 63 -------------------------- tests/functional/meson.build           |  8 ++++ tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++

  3 files changed, 44 insertions(+), 63 deletions(-)

  delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

Please squash:

-- >8 --

MAINTAINERS +0 -0
index 98eddf7ae1..a906218f9d 100644

MAINTAINERS F: include/hw/riscv/ · line 340

Quoted 4 lines
MAINTAINERS +1 -0
@@ -336,6 +337,7 @@ F: include/hw/riscv/
  F: linux-user/host/riscv32/
  F: linux-user/host/riscv64/
  F: tests/tcg/riscv64/
+F: tests/functional/test_riscv_opensbi.py

It seems worth taking this opportunity to add per-target subdirs of tests/functional/, so that we don't have to list every individual test in MAINTAINERS.

Surely one

F: tests/functional/riscv/

etc, is actually maintainable...

r~

#50 lore raw

Re: [PATCH v3 16/24] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 36 lines · Richard Henderson
Quoted 26 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Quoted 7 lines

---
  tests/avocado/riscv_opensbi.py         | 63 -------------------------- tests/functional/meson.build           |  8 ++++ tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++

  3 files changed, 44 insertions(+), 63 deletions(-)

  delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

Please squash:

-- >8 --

MAINTAINERS +0 -0
index 98eddf7ae1..a906218f9d 100644

MAINTAINERS F: include/hw/riscv/ · line 340

Quoted 4 lines
MAINTAINERS +1 -0
@@ -336,6 +337,7 @@ F: include/hw/riscv/
  F: linux-user/host/riscv32/
  F: linux-user/host/riscv64/
  F: tests/tcg/riscv64/
+F: tests/functional/test_riscv_opensbi.py

It seems worth taking this opportunity to add per-target subdirs of tests/functional/, so that we don't have to list every individual test in MAINTAINERS.

Quoted 5 lines

Surely one

F: tests/functional/riscv/

etc, is actually maintainable...

Either that or wildcard the filenames

F: tests/functional/test_riscv_*

but subdirs matching the target is probably a little nicer to really encourage people to follow conventions more strictly. We would also need a "tests/functional/common/" for tests which aren't target specific

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#51 lore raw

Re: [PATCH v3 16/24] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 43 lines · Daniel P. Berrangé
Quoted 36 lines · Richard Henderson
Quoted 26 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

Quoted 7 lines
---
   tests/avocado/riscv_opensbi.py         | 63 --------------------------
   tests/functional/meson.build           |  8 ++++
   tests/functional/test_riscv_opensbi.py | 36 +++++++++++++++
   3 files changed, 44 insertions(+), 63 deletions(-)

  delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

Please squash:

-- >8 --

MAINTAINERS +0 -0
index 98eddf7ae1..a906218f9d 100644

MAINTAINERS F: include/hw/riscv/ · line 340

Quoted 4 lines
MAINTAINERS +1 -0
@@ -336,6 +337,7 @@ F: include/hw/riscv/
   F: linux-user/host/riscv32/
   F: linux-user/host/riscv64/
   F: tests/tcg/riscv64/
+F: tests/functional/test_riscv_opensbi.py

It seems worth taking this opportunity to add per-target subdirs of tests/functional/, so that we don't have to list every individual test in MAINTAINERS.

Quoted 5 lines

Surely one

F: tests/functional/riscv/

etc, is actually maintainable...

Quoted 8 lines

Either that or wildcard the filenames

F: tests/functional/test_riscv_*

but subdirs matching the target is probably a little nicer to really encourage people to follow conventions more strictly. We would also need a "tests/functional/common/" for tests which aren't target specific

Hmm I'm a bit reluctant to both suggestions. Various files are added to cover a feature which ends being tested on various archs.

Once I wanted to use Avocado tags like GitLab labels, and keep a file where developers subscribe to tags; doing the parsing and label to developer conversion in Python (or Rust), and having a ./get_maintainer.sh calling both the Perl one and anything we can add in Python/Rust.

That said, wildcard pattern is a simpler start.

#52 lore raw

PATCH v3 17/24 tests/functional: Convert the virtio_gpu avocado test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                  |  1 +
 .../test_virtio_gpu.py}                       | 64 +++++++++----------
 2 files changed, 30 insertions(+), 35 deletions(-)
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index 55021c4e4d..8fbd7c4129 100644
@@ -82,6 +82,7 @@ tests_x86_64_quick = [
 
 tests_x86_64_thorough = [
   'netdev_ethtool',
+  'virtio_gpu',
 ]
 
 precache_all = []
tests/functional/test_virtio_gpu.py renamed from tests/avocado/virtio-gpu.py +29 -35

125 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by on v3 17/24
#53 lore raw

Re: [PATCH v3 17/24] tests/functional: Convert the virtio_gpu avocado test into a standalone test

Quoted 11 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                  |  1 +
  .../test_virtio_gpu.py}                       | 64 +++++++++----------
  2 files changed, 30 insertions(+), 35 deletions(-)

rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%) mode change 100644 => 100755

#54 lore raw

PATCH v3 18/24 tests/functional: Convert most ppc avocado tests into standalone tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Nothing thrilling in here, just straight forward conversions.

---
 MAINTAINERS                                   |  8 +--
 tests/functional/meson.build                  | 11 ++++
 .../test_ppc64_powernv.py}                    | 56 ++++++-----------
 .../test_ppc64_pseries.py}                    | 56 ++++++-----------
 .../test_ppc_40p.py}                          | 63 +++++++++----------
 .../test_ppc_mpc8544ds.py}                    | 28 +++++----
 .../test_ppc_virtex_ml507.py}                 | 28 +++++----
 7 files changed, 109 insertions(+), 141 deletions(-)
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
MAINTAINERS +4 -4
index e1d15c9696..758daad29e 100644
@@ -1449,7 +1449,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/mpc8544ds.c
 F: hw/ppc/mpc8544_guts.c
-F: tests/avocado/ppc_mpc8544ds.py
+F: tests/functional/test_ppc_mpc8544ds.py
 
 New World (mac99)
 M: Mark Cave-Ayland <[email protected]>
@@ -1502,7 +1502,7 @@ F: hw/dma/i82374.c
 F: hw/rtc/m48t59-isa.c
 F: include/hw/isa/pc87312.h
 F: include/hw/rtc/m48t59.h
-F: tests/avocado/ppc_prep_40p.py
+F: tests/functional/test_ppc_40p.py
 
 sPAPR (pseries)
 M: Nicholas Piggin <[email protected]>
@@ -1526,7 +1526,7 @@ F: tests/qtest/spapr*
 F: tests/qtest/libqos/*spapr*
 F: tests/qtest/rtas*
 F: tests/qtest/libqos/rtas*
-F: tests/avocado/ppc_pseries.py
+F: tests/functional/test_ppc64_pseries.py
 F: tests/avocado/ppc_hv_tests.py
 
 PowerNV (Non-Virtualized)
@@ -1558,7 +1558,7 @@ M: Edgar E. Iglesias <[email protected]>
 L: [email protected]
 S: Odd Fixes
 F: hw/ppc/virtex_ml507.c
-F: tests/avocado/ppc_virtex_ml507.py
+F: tests/functional/test_ppc_virtex_ml507.py
 
 sam460ex
 M: BALATON Zoltan <[email protected]>
tests/functional/meson.build +11 -0
index 8fbd7c4129..b4f5c9e38e 100644
@@ -12,6 +12,9 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  'ppc_40p' : 240,
+  'ppc64_powernv' : 120,
+  'ppc64_pseries' : 120,
   's390x_ccw_virtio' : 180,
 }
 
@@ -52,7 +55,15 @@ tests_ppc_quick = [
 
 tests_ppc_thorough = [
   'ppc_405',
+  'ppc_40p',
   'ppc_bamboo',
+  'ppc_mpc8544ds',
+  'ppc_virtex_ml507',
+]
+
+tests_ppc64_thorough = [
+  'ppc64_powernv',
+  'ppc64_pseries',
 ]
 
 tests_riscv32_quick = [
tests/functional/test_ppc64_powernv.py renamed from tests/avocado/ppc_powernv.py +18 -38

105 lines not on this page · raw message

tests/functional/test_ppc64_pseries.py renamed from tests/avocado/ppc_pseries.py +18 -38

110 lines not on this page · raw message

tests/functional/test_ppc_40p.py renamed from tests/avocado/ppc_prep_40p.py +28 -35

101 lines not on this page · raw message

tests/functional/test_ppc_mpc8544ds.py renamed from tests/avocado/ppc_mpc8544ds.py +15 -13
old mode 100644
new mode 100755
similarity index 55%
rename from tests/avocado/ppc_mpc8544ds.py
rename to tests/functional/test_ppc_mpc8544ds.py
index b599fb1cc9..859cdcac6c
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class Mpc8544dsMachine(QemuSystemTest):
 
@@ -15,20 +17,20 @@ class Mpc8544dsMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
     panic_message = 'Kernel panic - not syncing'
 
+    ASSET_IMAGE = Asset(('https://qemu-advcal.gitlab.io'
+                         '/qac-best-of-multiarch/download/day04.tar.xz'),
+                        'f46724d281a9f30fa892d458be7beb7d34dc25f9')
+
     def test_ppc_mpc8544ds(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:mpc8544ds
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day04.tar.xz')
-        tar_hash = 'f46724d281a9f30fa892d458be7beb7d34dc25f9'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('mpc8544ds')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir, member='creek/creek.bin')
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/creek/creek.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_virtex_ml507.py renamed from tests/avocado/ppc_virtex_ml507.py +15 -13
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/ppc_virtex_ml507.py
rename to tests/functional/test_ppc_virtex_ml507.py
index a73f8ae396..b567f7e4e8
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class VirtexMl507Machine(QemuSystemTest):
 
@@ -15,18 +17,15 @@ class VirtexMl507Machine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
     panic_message = 'Kernel panic - not syncing'
 
+    ASSET_IMAGE = Asset(('https://qemu-advcal.gitlab.io'
+                         '/qac-best-of-multiarch/download/day08.tar.xz'),
+                        '74c68f5af7a7b8f21c03097b298f3bb77ff52c1f')
+
     def test_ppc_virtex_ml507(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:virtex-ml507
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day08.tar.xz')
-        tar_hash = '74c68f5af7a7b8f21c03097b298f3bb77ff52c1f'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('virtex-ml507')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/hippo/hippo.linux',
                          '-dtb', self.workdir + '/hippo/virtex440-ml507.dtb',
@@ -34,3 +33,6 @@ def test_ppc_virtex_ml507(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#55 lore raw

PATCH v3 19/24 tests/functional: Convert the ppc_amiga avocado test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Use the Python standard zipfile module instead of avocado.utils for extracting the ZIP file that we download here, and use the standard subprocess module for running the "tail" command.

---
 tests/avocado/ppc_amiga.py         | 38 ---------------------------
 tests/functional/meson.build       |  1 +
 tests/functional/test_ppc_amiga.py | 42 ++++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 38 deletions(-)
 delete mode 100644 tests/avocado/ppc_amiga.py
 create mode 100755 tests/functional/test_ppc_amiga.py
tests/avocado/ppc_amiga.py deleted +0 -38
deleted file mode 100644
index b6f866f91d..0000000000
@@ -1,38 +0,0 @@
-# Test AmigaNG boards
-#
-# Copyright (c) 2023 BALATON Zoltan
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-from avocado.utils import archive
-from avocado.utils import process
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class AmigaOneMachine(QemuSystemTest):
-
-    timeout = 90
-
-    def test_ppc_amigaone(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:amigaone
-        :avocado: tags=device:articia
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        tar_name = 'A1Firmware_Floppy_05-Mar-2005.zip'
-        tar_url = ('https://www.hyperion-entertainment.com/index.php/'
-                   'downloads?view=download&format=raw&file=25')
-        tar_hash = 'c52e59bc73e31d8bcc3cc2106778f7ac84f6c755'
-        zip_file = self.fetch_asset(tar_name, locations=tar_url,
-                                    asset_hash=tar_hash)
-        archive.extract(zip_file, self.workdir)
-        cmd = f"tail -c 524288 {self.workdir}/floppy_edition/updater.image >{self.workdir}/u-boot-amigaone.bin"
-        process.run(cmd, shell=True)
-
-        self.vm.set_console()
-        self.vm.add_args('-bios', self.workdir + '/u-boot-amigaone.bin')
-        self.vm.launch()
-        wait_for_console_pattern(self, 'FLASH:')
tests/functional/meson.build +1 -0
index b4f5c9e38e..fba3891e16 100644
@@ -56,6 +56,7 @@ tests_ppc_quick = [
 tests_ppc_thorough = [
   'ppc_405',
   'ppc_40p',
+  'ppc_amiga',
   'ppc_bamboo',
   'ppc_mpc8544ds',
   'ppc_virtex_ml507',
tests/functional/test_ppc_amiga.py new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 19/24
#56 lore raw

Re: [PATCH v3 19/24] tests/functional: Convert the ppc_amiga avocado test into a standalone test

Quoted 6 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Use the Python standard zipfile module instead of avocado.utils for extracting the ZIP file that we download here, and use the standard subprocess module for running the "tail" command.

Quoted 8 lines
---
  tests/avocado/ppc_amiga.py         | 38 ---------------------------
  tests/functional/meson.build       |  1 +
  tests/functional/test_ppc_amiga.py | 42 ++++++++++++++++++++++++++++++
  3 files changed, 43 insertions(+), 38 deletions(-)

delete mode 100644 tests/avocado/ppc_amiga.py create mode 100755 tests/functional/test_ppc_amiga.py

#57 lore raw

PATCH v3 20/24 tests/functional: Convert the ppc_hv avocado test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Note: The original Avocado test seems currently to be broken, it hangs when the guest is trying to install additional packages. So mark it as broken for now until it gets fixed.

---
 .../test_ppc64_hv.py}                         | 48 ++++++++-----------
 1 file changed, 20 insertions(+), 28 deletions(-)
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (88%)
 mode change 100644 => 100755
tests/functional/test_ppc64_hv.py renamed from tests/avocado/ppc_hv_tests.py +20 -28
old mode 100644
new mode 100755
similarity index 88%
rename from tests/avocado/ppc_hv_tests.py
rename to tests/functional/test_ppc64_hv.py
index bf8822bb97..a45657c87e
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Tests that specifically try to exercise hypervisor features of the
 # target machines. powernv supports the Power hypervisor ISA, and
 # pseries supports the nested-HV hypervisor spec.
@@ -7,10 +9,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado import skipIf, skipUnless
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern, exec_command
+from unittest import skipIf, skipUnless
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern, exec_command
 import os
 import time
 import subprocess
@@ -47,7 +49,7 @@ def missing_deps():
 @skipIf(missing_deps(), 'dependencies (%s) not installed' % ','.join(deps))
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck due to console handling problem')
 @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
-@skipUnless(os.getenv('SPEED') == 'slow', 'runtime limited')
+@skip('broken test - guest fails to install packages')
 class HypervisorTest(QemuSystemTest):
 
     timeout = 1000
@@ -55,6 +57,12 @@ class HypervisorTest(QemuSystemTest):
     panic_message = 'Kernel panic - not syncing'
     good_message = 'VFS: Cannot open root device'
 
+    # Alpine use sha256 so I recalculated this myself
+    ASSET_ISO = Asset(
+        ('https://dl-cdn.alpinelinux.org/alpine/v3.18/'
+         'releases/ppc64le/alpine-standard-3.18.4-ppc64le.iso'),
+        'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff')
+
     def extract_from_iso(self, iso, path):
         """
         Extracts a file from an iso file into the test workdir
@@ -84,14 +92,7 @@ def extract_from_iso(self, iso, path):
     def setUp(self):
         super().setUp()
 
-        iso_url = ('https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/ppc64le/alpine-standard-3.18.4-ppc64le.iso')
-
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha256 = 'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha256,
-                                    algorithm = "sha256")
-
-        self.iso_path = iso_path
+        self.iso_path = self.ASSET_ISO.fetch()
         self.vmlinuz = self.extract_from_iso(iso_path, '/boot/vmlinuz-lts')
         self.initramfs = self.extract_from_iso(iso_path, '/boot/initramfs-lts')
 
@@ -159,12 +160,8 @@ def do_test_kvm(self, hpt=False):
         wait_for_console_pattern(self, 'alpine:~#')
 
     def test_hv_pseries(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on")
@@ -174,12 +171,8 @@ def test_hv_pseries(self):
         self.do_stop_alpine()
 
     def test_hv_pseries_kvm(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:kvm
-        """
         self.require_accelerator("kvm")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "kvm")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on,cap-ccf-assist=off")
@@ -189,12 +182,8 @@ def test_hv_pseries_kvm(self):
         self.do_stop_alpine()
 
     def test_hv_powernv(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:powernv
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('powernv')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,bus=pcie.2,addr=0x0,serial=1234,drive=drive0',
                          '-device', 'e1000e,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pcie.0,addr=0x0',
@@ -204,3 +193,6 @@ def test_hv_powernv(self):
         self.do_test_kvm()
         self.do_test_kvm(True)
         self.do_stop_alpine()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
#58 lore raw

Re: [PATCH v3 20/24] tests/functional: Convert the ppc_hv avocado test into a standalone test

Thanks for doing this.

I have a fix for this problem, it's a bug with the test harness code but was not merged yet. I'll re-send it.

Thanks,
Nick

Quoted 130 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Note: The original Avocado test seems currently to be broken, it hangs when the guest is trying to install additional packages. So mark it as broken for now until it gets fixed.

---
 .../test_ppc64_hv.py}                         | 48 ++++++++-----------
 1 file changed, 20 insertions(+), 28 deletions(-)
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (88%)
 mode change 100644 => 100755
tests/functional/test_ppc64_hv.py renamed from tests/avocado/ppc_hv_tests.py +20 -28
old mode 100644
new mode 100755
similarity index 88%
rename from tests/avocado/ppc_hv_tests.py
rename to tests/functional/test_ppc64_hv.py
index bf8822bb97..a45657c87e
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Tests that specifically try to exercise hypervisor features of the
 # target machines. powernv supports the Power hypervisor ISA, and
 # pseries supports the nested-HV hypervisor spec.
@@ -7,10 +9,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado import skipIf, skipUnless
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern, exec_command
+from unittest import skipIf, skipUnless
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern, exec_command
 import os
 import time
 import subprocess
@@ -47,7 +49,7 @@ def missing_deps():
 @skipIf(missing_deps(), 'dependencies (%s) not installed' % ','.join(deps))
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck due to console handling problem')
 @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
-@skipUnless(os.getenv('SPEED') == 'slow', 'runtime limited')
+@skip('broken test - guest fails to install packages')
 class HypervisorTest(QemuSystemTest):
 
     timeout = 1000
@@ -55,6 +57,12 @@ class HypervisorTest(QemuSystemTest):
     panic_message = 'Kernel panic - not syncing'
     good_message = 'VFS: Cannot open root device'
 
+    # Alpine use sha256 so I recalculated this myself
+    ASSET_ISO = Asset(
+        ('https://dl-cdn.alpinelinux.org/alpine/v3.18/'
+         'releases/ppc64le/alpine-standard-3.18.4-ppc64le.iso'),
+        'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff')
+
     def extract_from_iso(self, iso, path):
         """
         Extracts a file from an iso file into the test workdir
@@ -84,14 +92,7 @@ def extract_from_iso(self, iso, path):
     def setUp(self):
         super().setUp()
 
-        iso_url = ('https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/ppc64le/alpine-standard-3.18.4-ppc64le.iso')
-
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha256 = 'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha256,
-                                    algorithm = "sha256")
-
-        self.iso_path = iso_path
+        self.iso_path = self.ASSET_ISO.fetch()
         self.vmlinuz = self.extract_from_iso(iso_path, '/boot/vmlinuz-lts')
         self.initramfs = self.extract_from_iso(iso_path, '/boot/initramfs-lts')
 
@@ -159,12 +160,8 @@ def do_test_kvm(self, hpt=False):
         wait_for_console_pattern(self, 'alpine:~#')
 
     def test_hv_pseries(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on")
@@ -174,12 +171,8 @@ def test_hv_pseries(self):
         self.do_stop_alpine()
 
     def test_hv_pseries_kvm(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:kvm
-        """
         self.require_accelerator("kvm")
+        self.set_machine('pseries')
         self.vm.add_args("-accel", "kvm")
         self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
         self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on,cap-ccf-assist=off")
@@ -189,12 +182,8 @@ def test_hv_pseries_kvm(self):
         self.do_stop_alpine()
 
     def test_hv_powernv(self):
-        """
-        :avocado: tags=arch:ppc64
-        :avocado: tags=machine:powernv
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('powernv')
         self.vm.add_args("-accel", "tcg,thread=multi")
         self.vm.add_args('-device', 'nvme,bus=pcie.2,addr=0x0,serial=1234,drive=drive0',
                          '-device', 'e1000e,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pcie.0,addr=0x0',
@@ -204,3 +193,6 @@ def test_hv_powernv(self):
         self.do_test_kvm()
         self.do_test_kvm(True)
         self.do_stop_alpine()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
#59 lore raw

PATCH v3 21/24 tests/functional: Convert the m68k nextcube test with tesseract

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The code that handles running of tesseract needs to be tweaked a little bit to be able to run without the functions from avocado.utils, and while we're at it, drop some legacy stuff that was still there due to Tesseract 3 support that we already dropped a while ago.

---
 tests/avocado/tesseract_utils.py              | 46 -------------------
 tests/functional/meson.build                  |  4 ++
 tests/functional/qemu_test/tesseract.py       | 35 ++++++++++++++
 .../test_m68k_nextcube.py}                    | 29 ++++++------
 4 files changed, 55 insertions(+), 59 deletions(-)
 delete mode 100644 tests/avocado/tesseract_utils.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (76%)
 mode change 100644 => 100755
tests/avocado/tesseract_utils.py deleted +0 -46
deleted file mode 100644
index 476f528147..0000000000
@@ -1,46 +0,0 @@
-# ...
-#
-# Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later. See the COPYING file in the top-level directory.
-
-import re
-import logging
-
-from avocado.utils import process
-from avocado.utils.path import find_command, CmdNotFoundError
-
-def tesseract_available(expected_version):
-    try:
-        find_command('tesseract')
-    except CmdNotFoundError:
-        return False
-    res = process.run('tesseract --version')
-    try:
-        version = res.stdout_text.split()[1]
-    except IndexError:
-        version = res.stderr_text.split()[1]
-    return int(version.split('.')[0]) >= expected_version
-
-    match = re.match(r'tesseract\s(\d)', res)
-    if match is None:
-        return False
-    # now this is guaranteed to be a digit
-    return int(match.groups()[0]) >= expected_version
-
-
-def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
-    console_logger = logging.getLogger('tesseract')
-    console_logger.debug(image_path)
-    if tesseract_version == 4:
-        tesseract_args += ' --oem 1'
-    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
-                                                       image_path))
-    lines = []
-    for line in proc.stdout_text.split('\n'):
-        sline = line.strip()
-        if len(sline):
-            console_logger.debug(sline)
-            lines += [sline]
-    return lines
tests/functional/meson.build +4 -0
index fba3891e16..46b6405a58 100644
@@ -37,6 +37,10 @@ tests_loongarch64_thorough = [
   'loongarch64_virt',
 ]
 
+tests_m68k_thorough = [
+  'm68k_nextcube'
+]
+
 tests_microblaze_thorough = [
   'microblaze_s3adsp1800'
 ]
tests/functional/qemu_test/tesseract.py new file +35 -0
new file mode 100644
index 0000000000..c4087b7c11
@@ -0,0 +1,35 @@
+# ...
+#
+# Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import re
+import logging
+
+from . import has_cmd, run_cmd
+
+def tesseract_available(expected_version):
+    if not has_cmd('tesseract'):
+        return False
+    (stdout, stderr, ret) = run_cmd([ 'tesseract', '--version'])
+    if ret:
+        return False
+    version = stdout.split()[1]
+    return int(version.split('.')[0]) >= expected_version
+
+def tesseract_ocr(image_path, tesseract_args=''):
+    console_logger = logging.getLogger('console')
+    console_logger.debug(image_path)
+    (stdout, stderr, ret) = run_cmd(['tesseract', image_path,
+                                     'stdout'])
+    if ret:
+        return None
+    lines = []
+    for line in stdout.split('\n'):
+        sline = line.strip()
+        if len(sline):
+            console_logger.debug(sline)
+            lines += [sline]
+    return lines
tests/functional/test_m68k_nextcube.py renamed from tests/avocado/machine_m68k_nextcube.py +16 -13
old mode 100644
new mode 100755
similarity index 76%
rename from tests/avocado/machine_m68k_nextcube.py
rename to tests/functional/test_m68k_nextcube.py
index 1f3c883910..c394499d8c
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a VM and run OCR on the framebuffer
 #
 # Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
@@ -8,10 +10,10 @@
 import os
 import time
 
-from avocado_qemu import QemuSystemTest
-from avocado import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from unittest import skipUnless
 
-from tesseract_utils import tesseract_available, tesseract_ocr
+from qemu_test.tesseract import tesseract_available, tesseract_ocr
 
 PIL_AVAILABLE = True
 try:
@@ -21,19 +23,15 @@
 
 
 class NextCubeMachine(QemuSystemTest):
-    """
-    :avocado: tags=arch:m68k
-    :avocado: tags=machine:next-cube
-    :avocado: tags=device:framebuffer
-    """
 
     timeout = 15
 
+    ASSET_ROM = Asset(('https://sourceforge.net/p/previous/code/1350/tree/'
+                       'trunk/src/Rev_2.5_v66.BIN?format=raw'),
+                      'b3534796abae238a0111299fc406a9349f7fee24')
+
     def check_bootrom_framebuffer(self, screenshot_path):
-        rom_url = ('https://sourceforge.net/p/previous/code/1350/tree/'
-                   'trunk/src/Rev_2.5_v66.BIN?format=raw')
-        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
-        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+        rom_path = self.ASSET_ROM.fetch()
 
         self.vm.add_args('-bios', rom_path)
         self.vm.launch()
@@ -48,6 +46,7 @@ def check_bootrom_framebuffer(self, screenshot_path):
 
     @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
     def test_bootrom_framebuffer_size(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
@@ -60,11 +59,15 @@ def test_bootrom_framebuffer_size(self):
     # that it is still alpha-level software.
     @skipUnless(tesseract_available(4), 'tesseract OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
-        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
+        lines = tesseract_ocr(screenshot_path)
         text = '\n'.join(lines)
         self.assertIn('Testing the FPU', text)
         self.assertIn('System test failed. Error code', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 21/24
#60 lore raw

Re: [PATCH v3 21/24] tests/functional: Convert the m68k nextcube test with tesseract

Quoted 14 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

The code that handles running of tesseract needs to be tweaked a little bit to be able to run without the functions from avocado.utils, and while we're at it, drop some legacy stuff that was still there due to Tesseract 3 support that we already dropped a while ago.

---
  tests/avocado/tesseract_utils.py              | 46 -------------------
  tests/functional/meson.build                  |  4 ++
  tests/functional/qemu_test/tesseract.py       | 35 ++++++++++++++
  .../test_m68k_nextcube.py}                    | 29 ++++++------
  4 files changed, 55 insertions(+), 59 deletions(-)
Quoted 4 lines

delete mode 100644 tests/avocado/tesseract_utils.py
create mode 100644 tests/functional/qemu_test/tesseract.py
rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (76%) mode change 100644 => 100755

#61 lore raw

PATCH v3 22/24 tests/functional: Convert the acpi-bits test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore.

---
 .../acpi-bits/bits-config/bits-cfg.txt        |  0
 .../acpi-bits/bits-tests/smbios.py2           |  0
 .../acpi-bits/bits-tests/smilatency.py2       |  0
 .../acpi-bits/bits-tests/testacpi.py2         |  0
 .../acpi-bits/bits-tests/testcpuid.py2        |  0
 tests/functional/meson.build                  |  2 +
 .../test_acpi_bits.py}                        | 81 ++++++++++---------
 7 files changed, 43 insertions(+), 40 deletions(-)
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
tests/functional/acpi-bits/bits-config/bits-cfg.txt renamed from tests/avocado/acpi-bits/bits-config/bits-cfg.txt +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-config/bits-cfg.txt
rename to tests/functional/acpi-bits/bits-config/bits-cfg.txt
tests/functional/acpi-bits/bits-tests/smbios.py2 renamed from tests/avocado/acpi-bits/bits-tests/smbios.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smbios.py2
rename to tests/functional/acpi-bits/bits-tests/smbios.py2
tests/functional/acpi-bits/bits-tests/smilatency.py2 renamed from tests/avocado/acpi-bits/bits-tests/smilatency.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smilatency.py2
rename to tests/functional/acpi-bits/bits-tests/smilatency.py2
tests/functional/acpi-bits/bits-tests/testacpi.py2 renamed from tests/avocado/acpi-bits/bits-tests/testacpi.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testacpi.py2
rename to tests/functional/acpi-bits/bits-tests/testacpi.py2
tests/functional/acpi-bits/bits-tests/testcpuid.py2 renamed from tests/avocado/acpi-bits/bits-tests/testcpuid.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testcpuid.py2
rename to tests/functional/acpi-bits/bits-tests/testcpuid.py2
tests/functional/meson.build +2 -0
index 46b6405a58..439997a85d 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'acpi_bits' : 240,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
   'ppc64_powernv' : 120,
@@ -97,6 +98,7 @@ tests_x86_64_quick = [
 ]
 
 tests_x86_64_thorough = [
+  'acpi_bits',
   'netdev_ethtool',
   'virtio_gpu',
 ]
tests/functional/test_acpi_bits.py renamed from tests/avocado/acpi-bits.py +41 -40

172 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 22/24
#62 lore raw

Re: [PATCH v3 22/24] tests/functional: Convert the acpi-bits test into a standalone test

Quoted 16 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore.

---
  .../acpi-bits/bits-config/bits-cfg.txt        |  0
  .../acpi-bits/bits-tests/smbios.py2           |  0
  .../acpi-bits/bits-tests/smilatency.py2       |  0
  .../acpi-bits/bits-tests/testacpi.py2         |  0
  .../acpi-bits/bits-tests/testcpuid.py2        |  0
  tests/functional/meson.build                  |  2 +
  .../test_acpi_bits.py}                        | 81 ++++++++++---------
  7 files changed, 43 insertions(+), 40 deletions(-)
Quoted 7 lines

rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%) rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%) mode change 100644 => 100755

#63 lore raw

Re: [PATCH v3 22/24] tests/functional: Convert the acpi-bits test into a standalone test

Quoted 25 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore.

---
  .../acpi-bits/bits-config/bits-cfg.txt        |  0 .../acpi-bits/bits-tests/smbios.py2           |  0 .../acpi-bits/bits-tests/smilatency.py2       |  0 .../acpi-bits/bits-tests/testacpi.py2         |  0 .../acpi-bits/bits-tests/testcpuid.py2        |  0 tests/functional/meson.build                  |  2 + .../test_acpi_bits.py}                        | 81 ++++++++++---------

Quoted 8 lines
  7 files changed, 43 insertions(+), 40 deletions(-)

  rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%) rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%) rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%) mode change 100644 => 100755

Quoted 2 lines

(missing MAINTAINERS update)

#64 lore raw

PATCH v3 23/24 tests/functional: Convert the rx_gdbsim avocado test into a standalone test

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Provide a "gzip_uncompress" function based on the standard "gzip" module to avoid the usage of avocado.utils here.

---
 tests/functional/meson.build                  |  4 ++
 tests/functional/qemu_test/utils.py           | 12 ++++
 .../test_rx_gdbsim.py}                        | 57 ++++++++++---------
 3 files changed, 45 insertions(+), 28 deletions(-)
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (54%)
 mode change 100644 => 100755
tests/functional/meson.build +4 -0
index 439997a85d..b4889d8d7a 100644
@@ -80,6 +80,10 @@ tests_riscv64_quick = [
   'riscv_opensbi',
 ]
 
+tests_rx_thorough = [
+  'rx_gdbsim',
+]
+
 tests_s390x_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/qemu_test/utils.py +12 -0
index a12dac51b6..99eae5fc45 100644
@@ -8,6 +8,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import gzip
 import lzma
 import os
 import shutil
@@ -23,6 +24,17 @@ def archive_extract(archive, dest_dir, member=None):
         else:
             tf.extractall(path=dest_dir)
 
+def gzip_uncompress(gz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with gzip.open(gz_path, 'rb') as gz_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(gz_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
+
 def lzma_uncompress(xz_path, output_path):
     if os.path.exists(output_path):
         return
tests/functional/test_rx_gdbsim.py renamed from tests/avocado/machine_rx_gdbsim.py +29 -28

90 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 23/24
#65 lore raw

Re: [PATCH v3 23/24] tests/functional: Convert the rx_gdbsim avocado test into a standalone test

Quoted 5 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Provide a "gzip_uncompress" function based on the standard "gzip" module to avoid the usage of avocado.utils here.

Quoted 8 lines
---
  tests/functional/meson.build                  |  4 ++
  tests/functional/qemu_test/utils.py           | 12 ++++
  .../test_rx_gdbsim.py}                        | 57 ++++++++++---------
  3 files changed, 45 insertions(+), 28 deletions(-)

rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (54%) mode change 100644 => 100755

#66 lore raw

PATCH v3 24/24 gitlab-ci: Add "check-functional" to the build tests

In reply to Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

---
 .gitlab-ci.d/buildtest-template.yml |  3 +-
 .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
 2 files changed, 32 insertions(+), 31 deletions(-)
.gitlab-ci.d/buildtest-template.yml +2 -1
index 8f7ebfaed8..54cae74a8e 100644
@@ -71,7 +71,7 @@
     reports:
       junit: build/meson-logs/testlog.junit.xml
 
-.avocado_test_job_template:
+.functional_test_job_template:
   extends: .common_test_job_template
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -98,6 +98,7 @@
         du -chs ${CI_PROJECT_DIR}/avocado-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+    - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
.gitlab-ci.d/buildtest.yml +30 -30

171 lines not on this page · raw message

Signature · 1 line
2.45.2
Reviewed-by Tested-by on v3 24/24
#67 lore raw

Re: [PATCH v3 24/24] gitlab-ci: Add "check-functional" to the build tests

Quoted 6 lines · Daniel P. Berrangé

From: Thomas Huth <[email protected]>

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

Quoted 6 lines
---
  .gitlab-ci.d/buildtest-template.yml |  3 +-
  .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
  2 files changed, 32 insertions(+), 31 deletions(-)
#68 lore raw
In reply to Daniel P. Berrangé
Quoted 10 lines · Daniel P. Berrangé

This series is an update to Thomas' v2:

   https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg05805.html

wherein Thomas suggested someone could do the asset caching updates while he is on vacation:

   https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg06228.html

hence this posting what I'm calling a v3.

Changes included in my PR (can be considered as v2.5):
https://lore.kernel.org/qemu-devel/[email protected]/

-- >8 --

MAINTAINERS +11 -0
index 98eddf7ae1..a906218f9d 100644
@@ -317,6 +317,7 @@ F: configs/devices/ppc*
 F: docs/system/ppc/embedded.rst
 F: docs/system/target-ppc.rst
 F: tests/tcg/ppc*/*
+F: tests/functional/test_ppc_74xx.py
 
 RISC-V TCG CPUs
 M: Palmer Dabbelt <[email protected]>
@@ -336,6 +337,7 @@ F: include/hw/riscv/
 F: linux-user/host/riscv32/
 F: linux-user/host/riscv64/
 F: tests/tcg/riscv64/
+F: tests/functional/test_riscv_opensbi.py
 
 RISC-V XThead* extensions
 M: Christoph Muellner <[email protected]>
@@ -1830,6 +1832,9 @@ F: hw/isa/apm.c
 F: include/hw/isa/apm.h
 F: tests/unit/test-x86-topo.c
 F: tests/qtest/test-x86-cpuid-compat.c
+F: tests/functional/test_mem_addr_space.py
+F: tests/functional/test_pc_cpu_hotplug_props.py
+F: tests/functional/test_x86_cpu_model_versions.py
 
 PC Chipset
 M: Michael S. Tsirkin <[email protected]>
@@ -1896,6 +1901,8 @@ F: include/hw/boards.h
 F: include/hw/core/cpu.h
 F: include/hw/cpu/cluster.h
 F: include/sysemu/numa.h
+F: tests/functional/test_cpu_queries.py
+F: tests/functional/test_empty_cpu_model.py
 F: tests/unit/test-smp-parse.c
 T: git https://gitlab.com/ehabkost/qemu.git machine-next
 
@@ -2101,6 +2108,7 @@ S: Odd Fixes
 F: hw/net/
 F: include/hw/net/
 F: tests/qtest/virtio-net-test.c
+F: tests/avocado/info_usernet.py
 F: docs/virtio-net-failover.rst
 T: git https://github.com/jasowang/qemu.git net
 
@@ -2236,6 +2244,7 @@ F: net/vhost-user.c
 F: include/hw/virtio/
 F: docs/devel/virtio*
 F: docs/devel/migration/virtio.rst
+F: tests/functional/test_virtio_version.py
 
 virtio-balloon
 M: Michael S. Tsirkin <[email protected]>
@@ -2969,6 +2978,7 @@ S: Supported
 F: include/qemu/option.h
 F: tests/unit/test-keyval.c
 F: tests/unit/test-qemu-opts.c
+F: tests/functional/test_version.py
 F: util/keyval.c
 F: util/qemu-option.c
 
@@ -4127,6 +4137,7 @@ F: .travis.yml
 F: docs/devel/ci*
 F: scripts/ci/
 F: tests/docker/
+F: tests/functional/
 F: tests/vm/
 F: tests/lcitool/
 F: tests/avocado/tuxrun_baselines.py
tests/functional/test_info_usernet.py +1 -1
index ded973253a..cd37524d94 100755
@@ -19,7 +19,7 @@ class InfoUsernet(QemuSystemTest):
 
     def test_hostfwd(self):
         self.require_netdev('user')
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-netdev', 

'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
self.vm.launch()
res = self.vm.cmd('human-monitor-command',

tests/functional/test_version.py +1 -1
index 5e566d76b1..3ab3b67f7e 100755
@@ -17,7 +17,7 @@
 class Version(QemuSystemTest):
 
     def test_qmp_human_info_version(self):
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
tests/functional/test_x86_cpu_model_versions.py +0 -0
index a5f27c737d..a7294b4b92 100755
@@ -217,7 +217,7 @@ def test_none_alias(self):
         """
         Check if unversioned CPU model is an alias pointing to some 

version

         """
-        self.machine = 'none'
+        self.set_machine('none')
         self.vm.add_args('-S')
         self.vm.launch()

---