Skip to contentSkip to conversation

Convert avocado tests to normal Python unittests

PATCH In review Thomas Huth 2 participants 54 messages Aug 30, 2024 – Sep 4, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v5 cover Convert avocado tests to normal Python unittests

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet. There's now a new attempt available (I've included the relevant patches from Cleber at the beginning of this series now), but still, it would be nicer to finally get away from the Avocado test runner nowadays: The whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So this series now starts integrating the python-based tests with the test runner of meson instead. Indeed most tests work quite nicely without the Avocado framework already. It does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to convert them, too. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

*** Note: The naming in the cache changed between v3 and v4 of this patch *** series. If you already used v3 of this patch series and want to avoid *** re-downloading all assets, you should rename the files in the cache *** like this before running the tests again:

    cd ~/.cache/qemu/download/
    for i in * ; do mv -v $i $(sha256sum $i | cut -d " " -f 1);  done

v5:
- Updated the documentation patch with information about the asset handling
- Added "Allow asset downloading with concurrent threads" patch
- Improved the meson.build file to be able to handle linux-user tests, too
- Included many of Philippe's test convertions
- Converted the vnc test

v4:
- Addressed review comments from v3 (e.g. update to MAINTAINERS file)
- Included Cleber's patches to update Avocado tests (to avoid conflicts)
- Drop sha1 support, use sha256 in all tests instead
- The files in the cache are now named after the sha256 of the file contents instead of the sha256 of the URL (to avoid redownloading if just the URL changed)
- Allow to set the cache location via environment variable and make sure that the cache gets saved in the gitlab-CI
- Run pre-caching as a separate command in the gitlab-CI since Ninja is disabled during "make check-*" in the functional-* CI jobs
- Convert the linux_initrd test to the new framework, too
- Fix documentation of the acpi-bits test
- Add documentation for the functional testing framework

v3:
- Split tests/functional/__init__.py into multiple files
- Introduce an 'Asset' class to handle downloading & caching
- Convert all tests to use new Asset class instead of 'fetch_asset'
- Ensure 'make check-functional' pre-caches all assets before running the tests, to avoid meson test timeouts due to slow downloads
- Fix bios bits test logging in verbose mode to not print subprocess output to stdout, since it confuses TAP parsing

v2:
- Addressed review feedback from v1
- Add pycotap as a wheel instead of trying to install it on demand when running "make check-functional" (works much better now!)
- Converted much more tests
- Lots of other small improvements here and there

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Cleber Rosa (4):
tests/avocado: machine aarch64: standardize location and RO access tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0

Daniel P. Berrangé (2):
tests/functional: add a module for handling asset download & caching tests/functional: enable pre-emptive caching of assets

Philippe Mathieu-Daudé (7):
tests/functional: Convert ARM Integrator/CP avocado tests tests/functional: Convert Aarch64 SBSA-Ref avocado tests tests/functional: Convert Aarch64 Virt machine avocado tests tests/functional: Convert mips64el Fuloong2e avocado test (1/2) tests/functional: Add QemuUserTest class tests/functional: Convert ARM bFLT linux-user avocado test tests/avocado: Remove unused QemuUserTest class

Thomas Huth (31):
tests/avocado/avocado_qemu: Fix the "from" statements in linuxtest.py tests/avocado/boot_linux_console: Remove the s390x subtest
python: Install pycotap in our venv if necessary tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Set up logging
tests/Makefile.include: Increase the level of indentation in the help
text
tests/functional: Prepare the meson build system for the functional
tests
tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Allow asset downloading with concurrent threads tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Convert the s390x avocado tests into standalone
tests
tests/functional: Convert the x86_cpu_model_versions test tests/functional: Convert the microblaze avocado tests into standalone
tests
tests/functional: Convert the riscv_opensbi avocado test into a
standalone test
tests/functional: Convert the virtio_gpu avocado test into a
standalone test
tests/functional: Convert most ppc avocado tests into standalone tests tests/functional: Convert the ppc_amiga avocado test into a standalone
test
tests/functional: Convert the ppc_hv avocado test into a standalone
test
tests/functional: Convert the m68k nextcube test with tesseract tests/functional: Convert the acpi-bits test into a standalone test tests/functional: Convert the rx_gdbsim avocado test into a standalone
test
tests/functional: Convert the linux_initrd avocado test into a
standalone test
tests/functional: Convert the vnc test
gitlab-ci: Add "check-functional" to the build tests docs/devel: Split testing docs from the build docs and move to
separate folder
docs/devel/testing: Split the Avocado documentation into a separate
file
docs/devel/testing: Rename avocado_qemu.Test class docs/devel/testing: Add documentation for functional tests

 MAINTAINERS                                   |  66 +-
 docs/devel/index-build.rst                    |  14 +-
 docs/devel/index.rst                          |   1 +
 docs/devel/{ => testing}/acpi-bits.rst        |  62 +-
 docs/devel/testing/avocado.rst                | 581 ++++++++++++++++++
 .../{ => testing}/ci-definitions.rst.inc      |   0
 docs/devel/{ => testing}/ci-jobs.rst.inc      |   0
 docs/devel/{ => testing}/ci-runners.rst.inc   |   0
 docs/devel/{ => testing}/ci.rst               |   0
 docs/devel/testing/functional.rst             | 338 ++++++++++
 docs/devel/{ => testing}/fuzzing.rst          |   0
 docs/devel/testing/index.rst                  |  16 +
 docs/devel/{testing.rst => testing/main.rst}  | 577 +----------------
 docs/devel/{ => testing}/qgraph.rst           |   0
 docs/devel/{ => testing}/qtest.rst            |   0
 .gitlab-ci.d/buildtest-template.yml           |  14 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +-
 python/wheels/pycotap-1.3.1-py3-none-any.whl  | Bin 0 -> 5119 bytes
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  44 +-
 tests/avocado/avocado_qemu/__init__.py        |  17 -
 tests/avocado/avocado_qemu/linuxtest.py       |   4 +-
 tests/avocado/boot_linux_console.py           |  20 -
 tests/avocado/boot_xen.py                     |  13 +-
 tests/avocado/load_bflt.py                    |  54 --
 tests/avocado/machine_microblaze.py           |  61 --
 tests/avocado/machine_mips_loongson3v.py      |  39 --
 tests/avocado/machine_sparc64_sun4u.py        |  36 --
 tests/avocado/ppc_amiga.py                    |  38 --
 tests/avocado/riscv_opensbi.py                |  63 --
 tests/avocado/tesseract_utils.py              |  46 --
 .../acpi-bits/bits-config/bits-cfg.txt        |   0
 .../acpi-bits/bits-tests/smbios.py2           |   0
 .../acpi-bits/bits-tests/smilatency.py2       |   0
 .../acpi-bits/bits-tests/testacpi.py2         |   0
 .../acpi-bits/bits-tests/testcpuid.py2        |   0
 tests/functional/meson.build                  | 215 +++++++
 tests/functional/qemu_test/__init__.py        |  14 +
 tests/functional/qemu_test/asset.py           | 171 ++++++
 tests/functional/qemu_test/cmd.py             | 193 ++++++
 tests/functional/qemu_test/config.py          |  36 ++
 tests/functional/qemu_test/tesseract.py       |  35 ++
 tests/functional/qemu_test/testcase.py        | 202 ++++++
 tests/functional/qemu_test/utils.py           |  56 ++
 .../test_aarch64_sbsaref.py}                  | 144 ++---
 .../test_aarch64_virt.py}                     |  71 +--
 .../test_acpi_bits.py}                        |  83 +--
 tests/functional/test_arm_bflt.py             |  44 ++
 .../test_arm_canona1100.py}                   |  30 +-
 .../test_arm_integratorcp.py}                 |  56 +-
 .../test_arm_n8x0.py}                         |  36 +-
 .../test_avr_mega2560.py}                     |  22 +-
 .../test_cpu_queries.py}                      |  12 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_linux_initrd.py}                     |  37 +-
 .../test_loongarch64_virt.py}                 |  46 +-
 .../test_m68k_nextcube.py}                    |  29 +-
 .../test_mem_addr_space.py}                   |  52 +-
 .../functional/test_microblaze_s3adsp1800.py  |  40 ++
 .../test_microblazeel_s3adsp1800.py           |  42 ++
 .../test_mips64el_fuloong2e.py}               |  33 +-
 tests/functional/test_mips64el_loongson3v.py  |  39 ++
 .../test_netdev_ethtool.py}                   |  51 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../test_ppc64_hv.py}                         |  52 +-
 .../test_ppc64_powernv.py}                    |  56 +-
 .../test_ppc64_pseries.py}                    |  56 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  29 +-
 .../test_ppc_40p.py}                          |  65 +-
 .../test_ppc_74xx.py}                         |  74 +--
 tests/functional/test_ppc_amiga.py            |  43 ++
 .../test_ppc_bamboo.py}                       |  33 +-
 .../test_ppc_mpc8544ds.py}                    |  29 +-
 .../test_ppc_virtex_ml507.py}                 |  29 +-
 tests/functional/test_riscv_opensbi.py        |  40 ++
 .../test_rx_gdbsim.py}                        |  54 +-
 .../test_s390x_ccw_virtio.py}                 |  81 ++-
 .../test_s390x_topology.py}                   |  86 +--
 tests/functional/test_sparc64_sun4u.py        |  41 ++
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_gpu.py}                       |  64 +-
 .../test_virtio_version.py}                   |   8 +-
 .../vnc.py => functional/test_vnc.py}         |  12 +-
 .../test_x86_cpu_model_versions.py}           |  65 +-
 tests/meson.build                             |   1 +
 86 files changed, 2992 insertions(+), 1924 deletions(-)
 rename docs/devel/{ => testing}/acpi-bits.rst (78%)
 create mode 100644 docs/devel/testing/avocado.rst
 rename docs/devel/{ => testing}/ci-definitions.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-jobs.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-runners.rst.inc (100%)
 rename docs/devel/{ => testing}/ci.rst (100%)
 create mode 100644 docs/devel/testing/functional.rst
 rename docs/devel/{ => testing}/fuzzing.rst (100%)
 create mode 100644 docs/devel/testing/index.rst
 rename docs/devel/{testing.rst => testing/main.rst} (67%)
 rename docs/devel/{ => testing}/qgraph.rst (100%)
 rename docs/devel/{ => testing}/qtest.rst (100%)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
 delete mode 100644 tests/avocado/load_bflt.py
 delete mode 100644 tests/avocado/machine_microblaze.py
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 delete mode 100644 tests/avocado/ppc_amiga.py
 delete mode 100644 tests/avocado/riscv_opensbi.py
 delete mode 100644 tests/avocado/tesseract_utils.py
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/asset.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 create mode 100644 tests/functional/qemu_test/testcase.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_aarch64_sbsaref.py => functional/test_aarch64_sbsaref.py} (56%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_aarch64_virt.py => functional/test_aarch64_virt.py} (68%)
 mode change 100644 => 100755
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_arm_bflt.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_integratorcp.py => functional/test_arm_integratorcp.py} (63%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (72%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/linux_initrd.py => functional/test_linux_initrd.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
 rename tests/{avocado/machine_mips_fuloong2e.py => functional/test_mips64el_fuloong2e.py} (58%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (66%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_ppc_amiga.py
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_riscv_opensbi.py
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (58%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
 rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%)
 mode change 100644 => 100755
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
Signature · 1 line
2.46.0
#1 lore raw

PATCH v5 1/44 tests/avocado: machine aarch64: standardize location and RO access

From: Cleber Rosa <[email protected]>

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is automatically set.

While at it, let's use a single code style and hash for the ISO url.

Message-ID: <[email protected]>

---
 tests/avocado/machine_aarch64_sbsaref.py |  2 +-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
index e920bbf08c..756f316ac9 100644
@@ -129,7 +129,7 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
         )
 
         self.vm.launch()
tests/avocado/machine_aarch64_virt.py +7 -7
index a90dc6ff4b..6831d2c0ed 100644
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
Signature · 1 line
2.46.0
#2 lore raw

PATCH v5 2/44 tests/avocado/boot_xen.py: fetch kernel during test setUp()

In reply to Thomas Huth

From: Cleber Rosa <[email protected]>

The kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.

Message-ID: <[email protected]>

---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
index f29bc58b9e..490a127a3e 100644
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
Signature · 1 line
2.46.0
#3 lore raw

PATCH v5 3/44 tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

In reply to Thomas Huth

From: Cleber Rosa <[email protected]>

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.

Message-ID: <[email protected]>

---
 tests/avocado/machine_aarch64_sbsaref.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
index 756f316ac9..f8bf40c192 100644
@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={img_path},format=raw",
+            f"file={img_path},format=raw,snapshot=on",
         )
 
         self.vm.launch()
Signature · 1 line
2.46.0
#4 lore raw

PATCH v5 4/44 Bump avocado to 103.0

In reply to Thomas Huth

From: Cleber Rosa <[email protected]>

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.

To keep behavior of jobs as close as possible with previous version, this version bump keeps the execution serial (maximum of one task at a time being run).

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

Message-ID: <[email protected]>

---
 pythondeps.toml        | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
pythondeps.toml +1 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
tests/Makefile.include +1 -1
index 6618bfed70..537804d101 100644
@@ -141,7 +141,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
Signature · 1 line
2.46.0
#5 lore raw

PATCH v5 5/44 tests/avocado/avocado_qemu: Fix the "from" statements in linuxtest.py

In reply to Thomas Huth

Without this change, the new Avocado v103 fails to find the tests that are based on the LinuxTest class.

---
 tests/avocado/avocado_qemu/linuxtest.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/avocado/avocado_qemu/linuxtest.py +2 -2
index e1dc838b1c..66fb9f1507 100644
@@ -13,8 +13,8 @@
 
 from avocado.utils import cloudinit, datadrainer, process, vmimage
 
-from . import LinuxSSHMixIn
-from . import QemuSystemTest
+from avocado_qemu import LinuxSSHMixIn
+from avocado_qemu import QemuSystemTest
 
 if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
     # The link to the avocado tests dir in the source code directory
Signature · 1 line
2.46.0
#6 lore raw

PATCH v5 6/44 tests/avocado/boot_linux_console: Remove the s390x subtest

In reply to Thomas Huth

We've got a much more sophisticated, Fedora-based test for s390x ("test_s390x_fedora" in another file) already, so the test in boot_linux_console.py seems to be rather a waste of precious test cycles. Thus move the command line check and delete the s390x test in boot_linux_console.py.

---
 tests/avocado/boot_linux_console.py      | 20 --------------------
 tests/avocado/machine_s390_ccw_virtio.py |  2 ++
 2 files changed, 2 insertions(+), 20 deletions(-)
tests/avocado/boot_linux_console.py +0 -20
index cffdd6b5a2..18c69d6acc 100644
@@ -1304,26 +1304,6 @@ def test_aarch64_raspi3_atf(self):
         self.vm.launch()
         self.wait_for_console_pattern('version UEFI Firmware v1.15')
 
-    def test_s390x_s390_ccw_virtio(self):
-        """
-        :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390-ccw-virtio
-        """
-        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
-                      '/fedora-secondary/releases/29/Everything/s390x/os/images'
-                      '/kernel.img')
-        kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
-
-        self.vm.set_console()
-        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
-        self.vm.add_args('-nodefaults',
-                         '-kernel', kernel_path,
-                         '-append', kernel_command_line)
-        self.vm.launch()
-        console_pattern = 'Kernel command line: %s' % kernel_command_line
-        self.wait_for_console_pattern(console_pattern)
-
     def test_alpha_clipper(self):
         """
         :avocado: tags=arch:alpha
tests/avocado/machine_s390_ccw_virtio.py +2 -0
index 26e938c9e9..7a214110fc 100644
@@ -200,6 +200,8 @@ def test_s390x_fedora(self):
                          '-device', 'virtio-rng-ccw,devno=fe.1.9876',
                          '-device', 'virtio-gpu-ccw,devno=fe.2.5432')
         self.vm.launch()
+        self.wait_for_console_pattern('Kernel command line: %s'
+                                      % kernel_command_line)
         self.wait_for_console_pattern('Entering emergency mode')
 
         # Some tests to see whether the CLI options have been considered:
Signature · 1 line
2.46.0
#7 lore raw

PATCH v5 7/44 python: Install pycotap in our venv if necessary

In reply to Thomas Huth

The upcoming functional tests will require pycotap for providing TAP output from the python-based tests. Since we want to be able to run some of the tests offline by default, too, let's install it along with meson in our venv if necessary (it's size is only 5 kB, so adding the wheel here should not really be a problem).

The wheel file has been obtained with:

pip download --only-binary :all: --dest . --no-cache pycotap

---
 python/wheels/pycotap-1.3.1-py3-none-any.whl | Bin 0 -> 5119 bytes
 pythondeps.toml                              |   1 +
 2 files changed, 1 insertion(+)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
python/wheels/pycotap-1.3.1-py3-none-any.whl new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9c2c7d259365c8a0487228c6878e2c0b10ba6bdd
GIT binary patch
pythondeps.toml +1 -0
index 175cf99241..0a7f69b196 100644
@@ -20,6 +20,7 @@
 [meson]
 # The install key should match the version in python/wheels/
 meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
+pycotap = { accepted = ">=1.1.0", installed = "1.3.1" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
Signature · 1 line
2.46.0
#8 lore raw

PATCH v5 8/44 tests/functional: Add base classes for the upcoming pytest-based tests

In reply to Thomas Huth

The files are mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via environment variables that will be set via meson.build later.

---
 MAINTAINERS                            |   5 +
 tests/functional/qemu_test/__init__.py |  13 ++
 tests/functional/qemu_test/cmd.py      | 178 +++++++++++++++++++++++++
 tests/functional/qemu_test/config.py   |  36 +++++
 tests/functional/qemu_test/testcase.py | 153 +++++++++++++++++++++
 5 files changed, 385 insertions(+)
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/testcase.py
MAINTAINERS +5 -0
index 3584d6a6c6..80cc84dd99 100644
@@ -4146,6 +4146,11 @@ F: .gitlab-ci.d/cirrus/freebsd*
 F: tests/vm/freebsd
 W: https://cirrus-ci.com/github/qemu/qemu
 
+Functional testing framework
+M: Thomas Huth <[email protected]>
+R: Philippe Mathieu-Daudé <[email protected]>
+F: tests/functional/qemu_test/
+
 Windows Hosted Continuous Integration
 M: Yonggang Luo <[email protected]>
 S: Maintained
tests/functional/qemu_test/__init__.py new file +13 -0
new file mode 100644
index 0000000000..2f1e0bc70d
@@ -0,0 +1,13 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+
+from .config import BUILD_DIR
+from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
+    interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
+    exec_command, exec_command_and_wait_for_pattern
+from .testcase import QemuSystemTest, QemuBaseTest
tests/functional/qemu_test/cmd.py new file +178 -0

178 lines not on this page · raw message

tests/functional/qemu_test/config.py new file +36 -0
new file mode 100644
index 0000000000..edd75b7fd0
@@ -0,0 +1,36 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+#  Cleber Rosa <[email protected]>
+#
+# Adaption for standalone version:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+from pathlib import Path
+
+
+def _source_dir():
+    # Determine top-level directory of the QEMU sources
+    return Path(__file__).parent.parent.parent.parent
+
+def _build_dir():
+    root = os.getenv('QEMU_BUILD_ROOT')
+    if root is not None:
+        return Path(root)
+    # Makefile.mtest only exists in build dir, so if it is available, use CWD
+    if os.path.exists('Makefile.mtest'):
+        return Path(os.getcwd())
+
+    root = os.path.join(_source_dir(), 'build')
+    if os.path.exists(root):
+        return Path(root)
+
+    raise Exception("Cannot identify build dir, set QEMU_BUILD_ROOT")
+
+BUILD_DIR = _build_dir()
tests/functional/qemu_test/testcase.py new file +153 -0

153 lines not on this page · raw message

Signature · 1 line
2.46.0
#9 lore raw

PATCH v5 9/44 tests/functional: Set up logging

In reply to Thomas Huth

Create log files for each test separately, one file that contains the basic logging and one that contains the console output.

---
 tests/functional/qemu_test/testcase.py | 27 +++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +26 -1
index 6905675778..b2dd863c6e 100644
@@ -31,7 +31,8 @@ class QemuBaseTest(unittest.TestCase):
     arch = None
 
     workdir = None
-    log = logging.getLogger('qemu-test')
+    log = None
+    logdir = None
 
     def setUp(self, bin_prefix):
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -41,6 +42,20 @@ def setUp(self, bin_prefix):
                                     self.id())
         os.makedirs(self.workdir, exist_ok=True)
 
+        self.logdir = self.workdir
+        self.log = logging.getLogger('qemu-test')
+        self.log.setLevel(logging.DEBUG)
+        self._log_fh = logging.FileHandler(os.path.join(self.logdir,
+                                                        'base.log'), mode='w')
+        self._log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter(
+            '%(asctime)s - %(levelname)s: %(message)s')
+        self._log_fh.setFormatter(fileFormatter)
+        self.log.addHandler(self._log_fh)
+
+    def tearDown(self):
+        self.log.removeHandler(self._log_fh)
+
     def main():
         path = os.path.basename(sys.argv[0])[:-3]
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
@@ -60,6 +75,15 @@ def setUp(self):
 
         super().setUp('qemu-system-')
 
+        console_log = logging.getLogger('console')
+        console_log.setLevel(logging.DEBUG)
+        self._console_log_fh = logging.FileHandler(os.path.join(self.workdir,
+                                                   'console.log'), mode='w')
+        self._console_log_fh.setLevel(logging.DEBUG)
+        fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
+        self._console_log_fh.setFormatter(fileFormatter)
+        console_log.addHandler(self._console_log_fh)
+
     def set_machine(self, machinename):
         # TODO: We should use QMP to get the list of available machines
         if not self._machinehelp:
@@ -150,4 +174,5 @@ def set_vm_arg(self, arg, value):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
+        logging.getLogger('console').removeHandler(self._console_log_fh)
         super().tearDown()
Signature · 1 line
2.46.0
#10 lore raw

PATCH v5 10/44 tests/Makefile.include: Increase the level of indentation in the help text

In reply to Thomas Huth

The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first.

---
 tests/Makefile.include | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
tests/Makefile.include +15 -15
index 537804d101..fbb1a4b211 100644
@@ -3,28 +3,28 @@
 .PHONY: check-help
 check-help:
 	@echo "Regression testing targets:"
-	@echo " $(MAKE) check                  Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
-	@echo " $(MAKE) bench                  Run speed tests"
+	@echo " $(MAKE) check                    Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
+	@echo " $(MAKE) bench                    Run speed tests"
 	@echo
 	@echo "Individual test suites:"
-	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
-	@echo " $(MAKE) check-qtest            Run qtest tests"
-	@echo " $(MAKE) check-unit             Run qobject tests"
-	@echo " $(MAKE) check-qapi-schema      Run QAPI schema tests"
-	@echo " $(MAKE) check-block            Run block tests"
+	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
+	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-unit               Run qobject tests"
+	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
+	@echo " $(MAKE) check-block              Run block tests"
 ifneq ($(filter $(all-check-targets), check-softfloat),)
-	@echo " $(MAKE) check-tcg              Run TCG tests"
-	@echo " $(MAKE) check-softfloat        Run FPU emulation tests"
+	@echo " $(MAKE) check-tcg                Run TCG tests"
+	@echo " $(MAKE) check-softfloat          Run FPU emulation tests"
 endif
-	@echo " $(MAKE) check-avocado          Run avocado (integration) tests for currently configured targets"
+	@echo " $(MAKE) check-avocado            Run avocado (integration) tests for currently configured targets"
 	@echo
-	@echo " $(MAKE) check-report.junit.xml Generates an aggregated XML test report"
-	@echo " $(MAKE) check-venv             Creates a Python venv for tests"
-	@echo " $(MAKE) check-clean            Clean the tests and related data"
+	@echo " $(MAKE) check-report.junit.xml   Generates an aggregated XML test report"
+	@echo " $(MAKE) check-venv               Creates a Python venv for tests"
+	@echo " $(MAKE) check-clean              Clean the tests and related data"
 	@echo
 	@echo "The following are useful for CI builds"
-	@echo " $(MAKE) check-build            Build most test binaries"
-	@echo " $(MAKE) get-vm-images          Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
+	@echo " $(MAKE) check-build              Build most test binaries"
+	@echo " $(MAKE) get-vm-images            Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
 	@echo
 	@echo
 	@echo "The variable SPEED can be set to control the gtester speed setting."
Signature · 1 line
2.46.0
#11 lore raw

PATCH v5 11/44 tests/functional: Prepare the meson build system for the functional tests

In reply to Thomas Huth

Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet).

The changes to the meson.build files are partly based on an earlier patch by Ani Sinha.

---
 tests/Makefile.include       | 11 +++++
 tests/functional/meson.build | 81 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 3 files changed, 93 insertions(+)
 create mode 100644 tests/functional/meson.build
tests/Makefile.include +11 -0
index fbb1a4b211..66c8cc3123 100644
@@ -9,6 +9,8 @@ check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-functional         Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARGET  Run functional tests for a given target"
 	@echo " $(MAKE) check-unit               Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
 	@echo " $(MAKE) check-block              Run block tests"
@@ -152,6 +154,15 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@$(MAKE) SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@$(MAKE) SPEED=thorough check-func check-func-quick
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
tests/functional/meson.build new file +81 -0

81 lines not on this page · raw message

tests/meson.build +1 -0
index 80dd3029cf..907a4c1c98 100644
@@ -85,3 +85,4 @@ subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')
Signature · 1 line
2.46.0
#12 lore raw

PATCH v5 12/44 tests/functional: Convert simple avocado tests into standalone python tests

In reply to Thomas Huth

These test are rather simple and don't need any modifications apart from adjusting the "from avocado_qemu" line. To ease debugging, make the files executable and add a shebang line and Python '__main__' handling, too, so that these tests can now be run by executing them directly.

---
 MAINTAINERS                                   |  5 ++
 tests/functional/meson.build                  |  5 ++
 .../test_cpu_queries.py}                      | 12 +++--
 .../test_empty_cpu_model.py}                  |  7 ++-
 .../test_mem_addr_space.py}                   | 52 +++----------------
 .../test_pc_cpu_hotplug_props.py}             | 11 ++--
 .../test_virtio_version.py}                   |  8 +--
 7 files changed, 40 insertions(+), 60 deletions(-)
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
MAINTAINERS +5 -0
index 80cc84dd99..b3a0ac0e8b 100644
@@ -1833,6 +1833,8 @@ F: hw/isa/apm.c
 F: include/hw/isa/apm.h
 F: tests/unit/test-x86-topo.c
 F: tests/qtest/test-x86-cpuid-compat.c
+F: tests/functional/test_mem_addr_space.py
+F: tests/functional/test_pc_cpu_hotplug_props.py
 
 PC Chipset
 M: Michael S. Tsirkin <[email protected]>
@@ -1899,6 +1901,8 @@ F: include/hw/boards.h
 F: include/hw/core/cpu.h
 F: include/hw/cpu/cluster.h
 F: include/sysemu/numa.h
+F: tests/functional/test_cpu_queries.py
+F: tests/functional/test_empty_cpu_model.py
 F: tests/unit/test-smp-parse.c
 T: git https://gitlab.com/ehabkost/qemu.git machine-next
 
@@ -2238,6 +2242,7 @@ F: net/vhost-user.c
 F: include/hw/virtio/
 F: docs/devel/virtio*
 F: docs/devel/migration/virtio.rst
+F: tests/functional/test_virtio_version.py
 
 virtio-balloon
 M: Michael S. Tsirkin <[email protected]>
tests/functional/meson.build +5 -0
index 052d2e8781..91201a2e26 100644
@@ -14,6 +14,7 @@ test_timeouts = {
 }
 
 tests_generic_system = [
+  'empty_cpu_model',
 ]
 
 tests_generic_linuxuser = [
@@ -23,6 +24,10 @@ tests_generic_bsduser = [
 ]
 
 tests_x86_64_system_quick = [
+  'cpu_queries',
+  'mem_addr_space',
+  'pc_cpu_hotplug_props',
+  'virtio_version',
 ]
 
 tests_x86_64_system_thorough = [
tests/functional/test_cpu_queries.py renamed from tests/avocado/cpu_queries.py +7 -5
old mode 100644
new mode 100755
similarity index 85%
rename from tests/avocado/cpu_queries.py
rename to tests/functional/test_cpu_queries.py
index d3faa14720..b1122a0e8f
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Sanity check of query-cpu-* results
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -8,7 +10,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class QueryCPUModelExpansion(QemuSystemTest):
     """
@@ -16,10 +18,7 @@ class QueryCPUModelExpansion(QemuSystemTest):
     """
 
     def test(self):
-        """
-        :avocado: tags=arch:x86_64
-        :avocado: tags=machine:none
-        """
+        self.set_machine('none')
         self.vm.add_args('-S')
         self.vm.launch()
 
@@ -33,3 +32,6 @@ def test(self):
             e = self.vm.cmd('query-cpu-model-expansion', model=model,
                             type='full')
             self.assertEqual(e['model']['name'], c['name'])
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_empty_cpu_model.py renamed from tests/avocado/empty_cpu_model.py +6 -1
old mode 100644
new mode 100755
similarity index 84%
rename from tests/avocado/empty_cpu_model.py
rename to tests/functional/test_empty_cpu_model.py
index d906ef3d3c..0081b06d85
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Check for crash when using empty -cpu option
 #
 # Copyright (c) 2019 Red Hat, Inc.
@@ -7,7 +9,7 @@
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class EmptyCPUModel(QemuSystemTest):
     def test(self):
@@ -17,3 +19,6 @@ def test(self):
         self.vm.wait()
         self.assertEqual(self.vm.exitcode(), 1, "QEMU exit code should be 1")
         self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_mem_addr_space.py renamed from tests/avocado/mem-addr-space-check.py +6 -46

154 lines not on this page · raw message

tests/functional/test_pc_cpu_hotplug_props.py renamed from tests/avocado/pc_cpu_hotplug_props.py +6 -5
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/pc_cpu_hotplug_props.py
rename to tests/functional/test_pc_cpu_hotplug_props.py
index 4bd3e02665..9d5a37cb17
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # Ensure CPU die-id can be omitted on -device
 #
@@ -20,16 +21,16 @@
 # License along with this library; if not, see <http://www.gnu.org/licenses/>.
 #
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 class OmittedCPUProps(QemuSystemTest):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=cpu:qemu64
-    """
+
     def test_no_die_id(self):
         self.vm.add_args('-nodefaults', '-S')
         self.vm.add_args('-smp', '1,sockets=2,cores=2,threads=2,maxcpus=8')
         self.vm.add_args('-device', 'qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0')
         self.vm.launch()
         self.assertEqual(len(self.vm.cmd('query-cpus-fast')), 2)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_virtio_version.py renamed from tests/avocado/virtio_version.py +5 -3
old mode 100644
new mode 100755
similarity index 98%
rename from tests/avocado/virtio_version.py
rename to tests/functional/test_virtio_version.py
index afe5e828b5..eb23060564
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 """
 Check compatibility of virtio device types
 """
@@ -12,7 +13,7 @@
 import os
 
 from qemu.machine import QEMUMachine
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 # Virtio Device IDs:
 VIRTIO_NET = 1
@@ -60,8 +61,6 @@ class VirtioVersionCheck(QemuSystemTest):
     Check if virtio-version-specific device types result in the
     same device tree created by `disable-modern` and
     `disable-legacy`.
-
-    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
@@ -173,3 +172,6 @@ def test_modern_only_devs(self):
         self.check_modern_only('virtio-mouse-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-tablet-pci', VIRTIO_INPUT)
         self.check_modern_only('virtio-keyboard-pci', VIRTIO_INPUT)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#13 lore raw

PATCH v5 13/44 tests/functional: Convert avocado tests that just need a small adjustment

In reply to Thomas Huth

These simple tests can be converted to stand-alone tests quite easily, e.g. by just setting the machine to 'none' now manually or by adding "-cpu" command line parameters, since we don't support the corresponding avocado tags in the new python test framework.

---
 MAINTAINERS                                   |  3 +
 tests/functional/meson.build                  |  6 ++
 .../test_info_usernet.py}                     | 11 ++-
 .../test_ppc_74xx.py}                         | 74 ++++++++-----------
 .../version.py => functional/test_version.py} | 13 ++--
 5 files changed, 56 insertions(+), 51 deletions(-)
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
MAINTAINERS +3 -0
index b3a0ac0e8b..4912f8d0da 100644
@@ -318,6 +318,7 @@ F: configs/devices/ppc*
 F: docs/system/ppc/embedded.rst
 F: docs/system/target-ppc.rst
 F: tests/tcg/ppc*/*
+F: tests/functional/test_ppc_74xx.py
 
 RISC-V TCG CPUs
 M: Palmer Dabbelt <[email protected]>
@@ -2107,6 +2108,7 @@ S: Odd Fixes
 F: hw/net/
 F: include/hw/net/
 F: tests/qtest/virtio-net-test.c
+F: tests/functional/test_info_usernet.py
 F: docs/virtio-net-failover.rst
 T: git https://github.com/jasowang/qemu.git net
 
@@ -2976,6 +2978,7 @@ S: Supported
 F: include/qemu/option.h
 F: tests/unit/test-keyval.c
 F: tests/unit/test-qemu-opts.c
+F: tests/functional/test_version.py
 F: util/keyval.c
 F: util/qemu-option.c
 
tests/functional/meson.build +6 -0
index 91201a2e26..f1f344f860 100644
@@ -15,6 +15,12 @@ test_timeouts = {
 
 tests_generic_system = [
   'empty_cpu_model',
+  'info_usernet',
+  'version',
+]
+
+tests_ppc_quick = [
+  'ppc_74xx',
 ]
 
 tests_generic_linuxuser = [
tests/functional/test_info_usernet.py renamed from tests/avocado/info_usernet.py +7 -4
old mode 100644
new mode 100755
similarity index 87%
rename from tests/avocado/info_usernet.py
rename to tests/functional/test_info_usernet.py
index e1aa7a6e0a..cd37524d94
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test for the hmp command "info usernet"
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -8,18 +10,16 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 from qemu.utils import get_info_usernet_hostfwd_port
 
 
 class InfoUsernet(QemuSystemTest):
-    """
-    :avocado: tags=machine:none
-    """
 
     def test_hostfwd(self):
         self.require_netdev('user')
+        self.set_machine('none')
         self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
@@ -31,3 +31,6 @@ def test_hostfwd(self):
         self.assertGreater(port, 0,
                            ('Found a redirected port that is not greater than'
                             ' zero'))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_74xx.py renamed from tests/avocado/ppc_74xx.py +32 -42

167 lines not on this page · raw message

tests/functional/test_version.py renamed from tests/avocado/version.py +8 -5
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..3ab3b67f7e
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Version check example test
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
 class Version(QemuSystemTest):
-    """
-    :avocado: tags=quick
-    :avocado: tags=machine:none
-    """
+
     def test_qmp_human_info_version(self):
+        self.set_machine('none')
         self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.cmd('human-monitor-command',
                           command_line='info version')
         self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#14 lore raw

PATCH v5 14/44 tests/functional: add a module for handling asset download & caching

In reply to Thomas Huth

From: Daniel P. Berrangé <[email protected]>

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

[thuth: Drop sha1 support, use hash on file content for naming instead of URL,
add the possibility to specify the cache dir via environment variable]

---
 tests/functional/qemu_test/__init__.py |  1 +
 tests/functional/qemu_test/asset.py    | 97 ++++++++++++++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100644 tests/functional/qemu_test/asset.py
tests/functional/qemu_test/__init__.py +1 -0
index 2f1e0bc70d..db05c8f412 100644
@@ -6,6 +6,7 @@
 # later.  See the COPYING file in the top-level directory.
 
 
+from .asset import Asset
 from .config import BUILD_DIR
 from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
     interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
tests/functional/qemu_test/asset.py new file +97 -0

97 lines not on this page · raw message

Signature · 1 line
2.46.0
#15 lore raw

Re: [PATCH v5 14/44] tests/functional: add a module for handling asset download & caching

Quoted 57 lines · Thomas Huth

From: Daniel P. Berrangé <[email protected]>

The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.

[thuth: Drop sha1 support, use hash on file content for naming instead of URL,
add the possibility to specify the cache dir via environment variable]

---
  tests/functional/qemu_test/__init__.py |  1 +
  tests/functional/qemu_test/asset.py    | 97 ++++++++++++++++++++++++++
  2 files changed, 98 insertions(+)

create mode 100644 tests/functional/qemu_test/asset.py

tests/functional/qemu_test/__init__.py +1 -0
index 2f1e0bc70d..db05c8f412 100644
@@ -6,6 +6,7 @@
  # later.  See the COPYING file in the top-level directory.
 
 
+from .asset import Asset
  from .config import BUILD_DIR
  from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
      interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
tests/functional/qemu_test/asset.py new file +21 -0
new file mode 100644
index 0000000000..c0e675d847
@@ -0,0 +1,97 @@
+# Test utilities for fetching & caching assets
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import hashlib
+import logging
+import os
+import subprocess
+import urllib.request
+from pathlib import Path
+from shutil import copyfileobj
+
+
+# Instances of this class must be declared as class level variables
+# starting with a name "ASSET_". This enables the pre-caching logic
+# to easily find all referenced assets and download them prior to
+# execution of the tests.
+class Asset:

tests/functional/qemu_test/asset.py line 31

Quoted 10 lines
tests/functional/qemu_test/asset.py new file +10 -0
+
+    def __init__(self, url, hashsum):
+        self.url = url
+        self.hash = hashsum
+        cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR')
+        if cache_dir_env:
+            self.cache_dir = Path(cache_dir_env, "download")
+        else:
+            self.cache_dir = Path(Path("~").expanduser(),
+                                  ".cache", "qemu", "download")

Maybe we should use hashsum[:2] as intermediate bucket directory. (Suggestion now to not have to move the files again later)

Quoted 66 lines
tests/functional/qemu_test/asset.py new file +66 -0
+        self.cache_file = Path(self.cache_dir, hashsum)
+        self.log = logging.getLogger('qemu-test')
+
+    def __repr__(self):
+        return "Asset: url=%s hash=%s cache=%s" % (
+            self.url, self.hash, self.cache_file)
+
+    def _check(self, cache_file):
+        if self.hash is None:
+            return True
+        if len(self.hash) == 64:
+            sum_prog = 'sha256sum'
+        elif len(self.hash) == 128:
+            sum_prog = 'sha512sum'
+        else:
+            raise Exception("unknown hash type")
+
+        checksum = subprocess.check_output(
+            [sum_prog, str(cache_file)]).split()[0]
+        return self.hash == checksum.decode("utf-8")
+
+    def valid(self):
+        return self.cache_file.exists() and self._check(self.cache_file)
+
+    def fetch(self):
+        if not self.cache_dir.exists():
+            self.cache_dir.mkdir(parents=True, exist_ok=True)
+
+        if self.valid():
+            self.log.debug("Using cached asset %s for %s",
+                           self.cache_file, self.url)
+            return str(self.cache_file)
+
+        self.log.info("Downloading %s to %s...", self.url, self.cache_file)
+        tmp_cache_file = self.cache_file.with_suffix(".download")
+
+        try:
+            resp = urllib.request.urlopen(self.url)
+        except Exception as e:
+            self.log.error("Unable to download %s: %s", self.url, e)
+            raise
+
+        try:
+            with tmp_cache_file.open("wb+") as dst:
+                copyfileobj(resp, dst)
+        except:
+            tmp_cache_file.unlink()
+            raise
+        try:
+            # Set these just for informational purposes
+            os.setxattr(str(tmp_cache_file), "user.qemu-asset-url",
+                        self.url.encode('utf8'))
+            os.setxattr(str(tmp_cache_file), "user.qemu-asset-hash",
+                        self.hash.encode('utf8'))
+        except Exception as e:
+            self.log.debug("Unable to set xattr on %s: %s", tmp_cache_file, e)
+            pass
+
+        if not self._check(tmp_cache_file):
+            tmp_cache_file.unlink()
+            raise Exception("Hash of %s does not match %s" %
+                            (self.url, self.hash))
+        tmp_cache_file.replace(self.cache_file)
+
+        self.log.info("Cached %s at %s" % (self.url, self.cache_file))
+        return str(self.cache_file)
#16 lore raw

PATCH v5 15/44 tests/functional: enable pre-emptive caching of assets

In reply to Thomas Huth

From: Daniel P. Berrangé <[email protected]>

Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

   ninja test --suite functional

will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier.

   QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

ninja precache-functional

will download and cache all assets required by the functional tests

At the make level, precaching is always done by

make check-functional

[thuth: Remove the duplicated "path = os.path.basename(...)" line]
---
 tests/Makefile.include                 |  3 ++-
 tests/functional/meson.build           | 33 +++++++++++++++++++++++--
 tests/functional/qemu_test/asset.py    | 34 ++++++++++++++++++++++++++
 tests/functional/qemu_test/testcase.py |  7 ++++++
 4 files changed, 74 insertions(+), 3 deletions(-)
tests/Makefile.include +2 -1
index 66c8cc3123..010369bd3a 100644
@@ -161,7 +161,8 @@ $(FUNCTIONAL_TARGETS):
 
 .PHONY: check-functional
 check-functional:
-	@$(MAKE) SPEED=thorough check-func check-func-quick
+	@$(NINJA) precache-functional
+	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick
 
 # Consolidated targets
 
tests/functional/meson.build +31 -2
index f1f344f860..df79775df3 100644
@@ -39,6 +39,7 @@ tests_x86_64_system_quick = [
 tests_x86_64_system_thorough = [
 ]
 
+precache_all = []
 foreach speed : ['quick', 'thorough']
   foreach dir : target_dirs
 
@@ -78,11 +79,35 @@ foreach speed : ['quick', 'thorough']
                                meson.current_source_dir())
 
     foreach test : target_tests
-      test('func-@0@/@1@'.format(target_base, test),
+      testname = '@0@-@1@'.format(target_base, test)
+      testfile = 'test_' + test + '.py'
+      testpath = meson.current_source_dir() / testfile
+      teststamp = testname + '.tstamp'
+      test_precache_env = environment()
+      test_precache_env.set('QEMU_TEST_PRECACHE', meson.current_build_dir() / teststamp)
+      test_precache_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+                                          meson.current_source_dir())
+      precache = custom_target('func-precache-' + testname,
+                               output: teststamp,
+                               command: [python, testpath],
+                               depend_files: files(testpath),
+                               build_by_default: false,
+                               env: test_precache_env)
+      precache_all += precache
+
+      # Ideally we would add 'precache' to 'depends' here, such that
+      # 'build_by_default: false' lets the pre-caching automatically
+      # run immediately before the test runs. In practice this is
+      # broken in meson, with it running the pre-caching in the normal
+      # compile phase https://github.com/mesonbuild/meson/issues/2518
+      # If the above bug ever gets fixed, when QEMU changes the min
+      # meson version, add the 'depends' and remove the custom
+      # 'run_target' logic below & in Makefile.include
+      test('func-' + testname,
            python,
            depends: [test_deps, test_emulator, emulator_modules],
            env: test_env,
-           args: [meson.current_source_dir() / 'test_' + test + '.py'],
+           args: [testpath],
            protocol: 'tap',
            timeout: test_timeouts.get(test, 60),
            priority: test_timeouts.get(test, 60),
@@ -90,3 +115,7 @@ foreach speed : ['quick', 'thorough']
     endforeach
   endforeach
 endforeach
+
+run_target('precache-functional',
+           depends: precache_all,
+           command: ['true'])
tests/functional/qemu_test/asset.py +34 -0
index c0e675d847..b329ab7dbe 100644
@@ -9,6 +9,8 @@
 import logging
 import os
 import subprocess
+import sys
+import unittest
 import urllib.request
 from pathlib import Path
 from shutil import copyfileobj
@@ -62,6 +64,9 @@ def fetch(self):
                            self.cache_file, self.url)
             return str(self.cache_file)
 
+        if os.environ.get("QEMU_TEST_NO_DOWNLOAD", False):
+            raise Exception("Asset cache is invalid and downloads disabled")
+
         self.log.info("Downloading %s to %s...", self.url, self.cache_file)
         tmp_cache_file = self.cache_file.with_suffix(".download")
 
@@ -95,3 +100,32 @@ def fetch(self):
 
         self.log.info("Cached %s at %s" % (self.url, self.cache_file))
         return str(self.cache_file)
+
+    def precache_test(test):
+        log = logging.getLogger('qemu-test')
+        log.setLevel(logging.DEBUG)
+        handler = logging.StreamHandler(sys.stdout)
+        handler.setLevel(logging.DEBUG)
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+        handler.setFormatter(formatter)
+        log.addHandler(handler)
+        for name, asset in vars(test.__class__).items():
+            if name.startswith("ASSET_") and type(asset) == Asset:
+                log.info("Attempting to cache '%s'" % asset)
+                asset.fetch()
+        log.removeHandler(handler)
+
+    def precache_suite(suite):
+        for test in suite:
+            if isinstance(test, unittest.TestSuite):
+                Asset.precache_suite(test)
+            elif isinstance(test, unittest.TestCase):
+                Asset.precache_test(test)
+
+    def precache_suites(path, cacheTstamp):
+        loader = unittest.loader.defaultTestLoader
+        tests = loader.loadTestsFromNames([path], None)
+
+        with open(cacheTstamp, "w") as fh:
+            Asset.precache_suite(tests)
tests/functional/qemu_test/testcase.py +7 -0
index b2dd863c6e..18314be9d1 100644
@@ -21,6 +21,7 @@
 from qemu.machine import QEMUMachine
 from qemu.utils import kvm_available, tcg_available
 
+from .asset import Asset
 from .cmd import run_cmd
 from .config import BUILD_DIR
 
@@ -58,6 +59,12 @@ def tearDown(self):
 
     def main():
         path = os.path.basename(sys.argv[0])[:-3]
+
+        cache = os.environ.get("QEMU_TEST_PRECACHE", None)
+        if cache is not None:
+            Asset.precache_suites(path, cache)
+            return
+
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
         unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
Signature · 1 line
2.46.0
#17 lore raw

PATCH v5 16/44 tests/functional: Allow asset downloading with concurrent threads

In reply to Thomas Huth

When running "make -j$(nproc) check-functional", tests that use the same asset might be running in parallel. Improve the downloading to detect this situation and wait for the other thread to finish the download.

---
 tests/functional/qemu_test/asset.py | 62 ++++++++++++++++++++++++-----
 1 file changed, 51 insertions(+), 11 deletions(-)
tests/functional/qemu_test/asset.py +51 -11
index b329ab7dbe..d3be2aff82 100644
@@ -12,6 +12,7 @@
 import sys
 import unittest
 import urllib.request
+from time import sleep
 from pathlib import Path
 from shutil import copyfileobj
 
@@ -55,6 +56,35 @@ def _check(self, cache_file):
     def valid(self):
         return self.cache_file.exists() and self._check(self.cache_file)
 
+    def _wait_for_other_download(self, tmp_cache_file):
+        # Another thread already seems to download the asset, so wait until
+        # it is done, while also checking the size to see whether it is stuck
+        try:
+            current_size = tmp_cache_file.stat().st_size
+            new_size = current_size
+        except:
+            if os.path.exists(self.cache_file):
+                return True
+            raise
+        waittime = lastchange = 600
+        while waittime > 0:
+            sleep(1)
+            waittime -= 1
+            try:
+                new_size = tmp_cache_file.stat().st_size
+            except:
+                if os.path.exists(self.cache_file):
+                    return True
+                raise
+            if new_size != current_size:
+                lastchange = waittime
+                current_size = new_size
+            elif lastchange - waittime > 90:
+                return False
+
+        self.log.debug("Time out while waiting for %s!", tmp_cache_file)
+        raise
+
     def fetch(self):
         if not self.cache_dir.exists():
             self.cache_dir.mkdir(parents=True, exist_ok=True)
@@ -70,18 +100,28 @@ def fetch(self):
         self.log.info("Downloading %s to %s...", self.url, self.cache_file)
         tmp_cache_file = self.cache_file.with_suffix(".download")
 
-        try:
-            resp = urllib.request.urlopen(self.url)
-        except Exception as e:
-            self.log.error("Unable to download %s: %s", self.url, e)
-            raise
+        for retries in range(3):
+            try:
+                with tmp_cache_file.open("xb") as dst:
+                    with urllib.request.urlopen(self.url) as resp:
+                        copyfileobj(resp, dst)
+                break
+            except FileExistsError:
+                self.log.debug("%s already exists, "
+                               "waiting for other thread to finish...",
+                               tmp_cache_file)
+                if self._wait_for_other_download(tmp_cache_file):
+                    return str(self.cache_file)
+                self.log.debug("%s seems to be stale, "
+                               "deleting and retrying download...",
+                               tmp_cache_file)
+                tmp_cache_file.unlink()
+                continue
+            except Exception as e:
+                self.log.error("Unable to download %s: %s", self.url, e)
+                tmp_cache_file.unlink()
+                raise
 
-        try:
-            with tmp_cache_file.open("wb+") as dst:
-                copyfileobj(resp, dst)
-        except:
-            tmp_cache_file.unlink()
-            raise
         try:
             # Set these just for informational purposes
             os.setxattr(str(tmp_cache_file), "user.qemu-asset-url",
Signature · 1 line
2.46.0
#18 lore raw

PATCH v5 17/44 tests/functional: Convert some tests that download files via fetch_asset()

In reply to Thomas Huth

Now that we've got the Asset class with pre-caching, we can convert some Avocado tests that use fetch_asset() for downloading their required files.

---
 MAINTAINERS                                   | 12 ++---
 tests/avocado/machine_mips_loongson3v.py      | 39 --------------
 tests/functional/meson.build                  | 30 +++++++++--
 .../test_arm_n8x0.py}                         | 36 ++++++-------
 .../test_avr_mega2560.py}                     | 22 ++++----
 .../test_loongarch64_virt.py}                 | 46 +++++++++--------
 tests/functional/test_mips64el_loongson3v.py  | 39 ++++++++++++++
 .../test_netdev_ethtool.py}                   | 51 +++++++------------
 .../ppc_405.py => functional/test_ppc_405.py} | 29 ++++++-----
 9 files changed, 160 insertions(+), 144 deletions(-)
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (72%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (66%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
MAINTAINERS +6 -6
index 4912f8d0da..6545662eaf 100644
@@ -222,7 +222,7 @@ S: Maintained
 F: docs/system/target-avr.rst
 F: gdb-xml/avr-cpu.xml
 F: target/avr/
-F: tests/avocado/machine_avr6.py
+F: tests/functional/test_avr_mega2560.py
 
 CRIS TCG CPUs
 M: Edgar E. Iglesias <[email protected]>
@@ -266,7 +266,7 @@ M: Song Gao <[email protected]>
 S: Maintained
 F: target/loongarch/
 F: tests/tcg/loongarch64/
-F: tests/avocado/machine_loongarch.py
+F: tests/functional/test_loongarch64_virt.py
 
 M68K TCG CPUs
 M: Laurent Vivier <[email protected]>
@@ -886,7 +886,7 @@ F: include/hw/display/blizzard.h
 F: include/hw/input/lm832x.h
 F: include/hw/input/tsc2xxx.h
 F: include/hw/misc/cbus.h
-F: tests/avocado/machine_arm_n8x0.py
+F: tests/functional/test_arm_n8x0.py
 F: docs/system/arm/nseries.rst
 
 Palm
@@ -1397,7 +1397,7 @@ F: hw/mips/loongson3_virt.c
 F: include/hw/intc/loongson_ipi_common.h
 F: include/hw/intc/loongson_ipi.h
 F: include/hw/intc/loongson_liointc.h
-F: tests/avocado/machine_mips_loongson3v.py
+F: tests/functional/test_mips64el_loongson3v.py
 
 Boston
 M: Paul Burton <[email protected]>
@@ -1423,7 +1423,7 @@ PowerPC Machines
 L: [email protected]
 S: Orphan
 F: hw/ppc/ppc405*
-F: tests/avocado/ppc_405.py
+F: tests/functional/test_ppc_405.py
 
 Bamboo
 L: [email protected]
@@ -2495,7 +2495,7 @@ R: Sriram Yagnaraman <[email protected]>
 S: Maintained
 F: docs/system/devices/igb.rst
 F: hw/net/igb*
-F: tests/avocado/netdev-ethtool.py
+F: tests/functional/test_netdev_ethtool.py
 F: tests/qtest/igb-test.c
 F: tests/qtest/libqos/igb.c
 
tests/avocado/machine_mips_loongson3v.py deleted +0 -39
deleted file mode 100644
index 5194cf18c9..0000000000
@@ -1,39 +0,0 @@
-# Functional tests for the Generic Loongson-3 Platform.
-#
-# Copyright (c) 2021 Jiaxun Yang <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-#
-# SPDX-License-Identifier: GPL-2.0-or-later
-
-import os
-import time
-
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class MipsLoongson3v(QemuSystemTest):
-    timeout = 60
-
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
-    def test_pmon_serial_console(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=endian:little
-        :avocado: tags=machine:loongson3-virt
-        :avocado: tags=cpu:Loongson-3A1000
-        :avocado: tags=device:liointc
-        :avocado: tags=device:goldfish_rtc
-        """
-
-        pmon_hash = '7c8b45dd81ccfc55ff28f5aa267a41c3'
-        pmon_path = self.fetch_asset('https://github.com/loongson-community/pmon/'
-                                    'releases/download/20210112/pmon-3avirt.bin',
-                                     asset_hash=pmon_hash, algorithm='md5')
-
-        self.vm.set_console()
-        self.vm.add_args('-bios', pmon_path)
-        self.vm.launch()
-        wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
tests/functional/meson.build +26 -4
index df79775df3..11ff246cef 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'netdev_ethtool' : 180,
 }
 
 tests_generic_system = [
@@ -19,16 +20,36 @@ tests_generic_system = [
   'version',
 ]
 
-tests_ppc_quick = [
-  'ppc_74xx',
-]
-
 tests_generic_linuxuser = [
 ]
 
 tests_generic_bsduser = [
 ]
 
+tests_arm_system_thorough = [
+  'arm_n8x0',
+]
+
+tests_avr_system_thorough = [
+  'avr_mega2560',
+]
+
+tests_loongarch64_system_thorough = [
+  'loongarch64_virt',
+]
+
+tests_mips64el_system_thorough = [
+  'mips64el_loongson3v',
+]
+
+tests_ppc_system_quick = [
+  'ppc_74xx',
+]
+
+tests_ppc_system_thorough = [
+  'ppc_405',
+]
+
 tests_x86_64_system_quick = [
   'cpu_queries',
   'mem_addr_space',
@@ -37,6 +58,7 @@ tests_x86_64_system_quick = [
 ]
 
 tests_x86_64_system_thorough = [
+  'netdev_ethtool',
 ]
 
 precache_all = []
tests/functional/test_arm_n8x0.py renamed from tests/avocado/machine_arm_n8x0.py +18 -18
old mode 100644
new mode 100755
similarity index 51%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..ffcf468fe8
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a Linux kernel and checks the console
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -10,21 +12,22 @@
 
 import os
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class N8x0Machine(QemuSystemTest):
     """Boots the Linux kernel and checks that the console is operational"""
 
     timeout = 90
 
+    ASSET_KERNEL = Asset(
+        ('http://stskeeps.subnetmask.net/meego-n8x0/'
+         'meego-arm-n8x0-1.0.80.20100712.1431-vmlinuz-2.6.35~rc4-129.1-n8x0'),
+        '629f4616caacacd116d23d7572178f93691b0b918b25e28d7736c24f888d4292')
+
     def __do_test_n8x0(self):
-        kernel_url = ('http://stskeeps.subnetmask.net/meego-n8x0/'
-                      'meego-arm-n8x0-1.0.80.20100712.1431-'
-                      'vmlinuz-2.6.35~rc4-129.1-n8x0')
-        kernel_hash = 'e9d5ab8d7548923a0061b6fbf601465e479ed269'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        kernel_path = self.ASSET_KERNEL.fetch()
 
         self.vm.set_console(console_index=1)
         self.vm.add_args('-kernel', kernel_path,
@@ -32,18 +35,15 @@ def __do_test_n8x0(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'TSC2005 driver initializing')
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n800(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n800
-        """
+        self.set_machine('n800')
         self.__do_test_n8x0()
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_n810(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:n810
-        """
+        self.set_machine('n810')
         self.__do_test_n8x0()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_avr_mega2560.py renamed from tests/avocado/machine_avr6.py +12 -10
old mode 100644
new mode 100755
similarity index 72%
rename from tests/avocado/machine_avr6.py
rename to tests/functional/test_avr_mega2560.py
index 5485db79c6..8e47b4200b
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 #
 # QEMU AVR integration tests
 #
@@ -19,26 +20,24 @@
 
 import time
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest, Asset
 
 class AVR6Machine(QemuSystemTest):
     timeout = 5
 
+    ASSET_ROM = Asset(('https://github.com/seharris/qemu-avr-tests'
+                       '/raw/36c3e67b8755dcf/free-rtos/Demo'
+                       '/AVR_ATMega2560_GCC/demo.elf'),
+                      'ee4833bd65fc69e84a79ed1c608affddbd499a60e63acf87d9113618401904e4')
+
     def test_freertos(self):
-        """
-        :avocado: tags=arch:avr
-        :avocado: tags=machine:arduino-mega-2560-v3
-        """
         """
         https://github.com/seharris/qemu-avr-tests/raw/master/free-rtos/Demo/AVR_ATMega2560_GCC/demo.elf
         constantly prints out 'ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX'
         """
-        rom_url = ('https://github.com/seharris/qemu-avr-tests'
-                   '/raw/36c3e67b8755dcf/free-rtos/Demo'
-                   '/AVR_ATMega2560_GCC/demo.elf')
-        rom_hash = '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4'
-        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+        rom_path = self.ASSET_ROM.fetch()
 
+        self.set_machine('arduino-mega-2560-v3')
         self.vm.add_args('-bios', rom_path)
         self.vm.add_args('-nographic')
         self.vm.launch()
@@ -48,3 +47,6 @@ def test_freertos(self):
 
         self.assertIn('ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX',
                 self.vm.get_log())
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_loongarch64_virt.py renamed from tests/avocado/machine_loongarch.py +25 -21

71 lines not on this page · raw message

tests/functional/test_mips64el_loongson3v.py new file +39 -0
new file mode 100755
index 0000000000..55d62928c7
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+#
+# Functional tests for the Generic Loongson-3 Platform.
+#
+# Copyright (c) 2021 Jiaxun Yang <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import time
+
+from unittest import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+
+class MipsLoongson3v(QemuSystemTest):
+    timeout = 60
+
+    ASSET_PMON = Asset(
+        ('https://github.com/loongson-community/pmon/'
+         'releases/download/20210112/pmon-3avirt.bin'),
+        'fcdf6bb2cb7885a4a62f31fcb0d5e368bac7b6cea28f40c6dfa678af22fea20a')
+
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    def test_pmon_serial_console(self):
+        self.set_machine('loongson3-virt')
+
+        pmon_path = self.ASSET_PMON.fetch()
+
+        self.vm.set_console()
+        self.vm.add_args('-bios', pmon_path)
+        self.vm.launch()
+        wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_netdev_ethtool.py renamed from tests/avocado/netdev-ethtool.py +19 -32

90 lines not on this page · raw message

tests/functional/test_ppc_405.py renamed from tests/avocado/ppc_405.py +15 -14
old mode 100644
new mode 100755
similarity index 53%
rename from tests/avocado/ppc_405.py
rename to tests/functional/test_ppc_405.py
index 4e7e01aa76..9851c03ee9
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that the U-Boot firmware boots on ppc 405 machines and check the console
 #
 # Copyright (c) 2021 Red Hat, Inc.
@@ -5,20 +7,21 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class Ppc405Machine(QemuSystemTest):
 
     timeout = 90
 
+    ASSET_UBOOT = Asset(
+        ('https://gitlab.com/huth/u-boot/-/raw/taihu-2021-10-09/'
+         'u-boot-taihu.bin'),
+        'a076bb6cdeaafa406330e51e074b66d8878d9036d67d4caa0137be03ee4c112c')
+
     def do_test_ppc405(self):
-        uboot_url = ('https://gitlab.com/huth/u-boot/-/raw/'
-                     'taihu-2021-10-09/u-boot-taihu.bin')
-        uboot_hash = ('3208940e908a5edc7c03eab072c60f0dcfadc2ab');
-        file_path = self.fetch_asset(uboot_url, asset_hash=uboot_hash)
+        file_path = self.ASSET_UBOOT.fetch()
         self.vm.set_console(console_index=1)
         self.vm.add_args('-bios', file_path)
         self.vm.launch()
@@ -26,11 +29,9 @@ def do_test_ppc405(self):
         exec_command_and_wait_for_pattern(self, 'reset', 'AMCC PowerPC 405EP')
 
     def test_ppc_ref405ep(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:ref405ep
-        :avocado: tags=cpu:405ep
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
+        self.set_machine('ref405ep')
         self.do_test_ppc405()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#19 lore raw

Re: [PATCH v5 17/44] tests/functional: Convert some tests that download files via fetch_asset()

Quoted 10 lines · Thomas Huth

Now that we've got the Asset class with pre-caching, we can convert some Avocado tests that use fetch_asset() for downloading their required files.

---
  MAINTAINERS                                   | 12 ++---
  tests/avocado/machine_mips_loongson3v.py      | 39 --------------
  tests/functional/meson.build                  | 30 +++++++++--
  .../test_arm_n8x0.py}                         | 36 ++++++-------

FWIW: I'm going to drop the arm_n8x0 part from this patch since Peter is currently preparing for removing the n800 and n810 machines (and his series removes this test:

https://lore.kernel.org/qemu-devel/[email protected]/

Thomas

#20 lore raw

PATCH v5 18/44 tests/functional: Add a function for extracting files from an archive

In reply to Thomas Huth

Some Avocado-based tests use the "archive" module from avocado.utils to extract files from an archive. To be able to use these tests without Avocado, we have to provide our own function for extracting files. Fortunately, there is already the tarfile module that will provide us with this functionality, so let's just add a nice wrapper function around that.

---
 tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/functional/qemu_test/utils.py
tests/functional/qemu_test/utils.py new file +21 -0
new file mode 100644
index 0000000000..4eb5e5d5e5
@@ -0,0 +1,21 @@
+# Utilities for python-based QEMU tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# Authors:
+#  Thomas Huth <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import tarfile
+
+def archive_extract(archive, dest_dir, member=None):
+    with tarfile.open(archive) as tf:
+        if hasattr(tarfile, 'data_filter'):
+            tf.extraction_filter = getattr(tarfile, 'data_filter',
+                                           (lambda member, path: member))
+        if member:
+            tf.extract(member=member, path=dest_dir)
+        else:
+            tf.extractall(path=dest_dir)
Signature · 1 line
2.46.0
#21 lore raw

PATCH v5 19/44 tests/functional: Convert some avocado tests that needed avocado.utils.archive

In reply to Thomas Huth

Instead of using the "archive" module from avocado.utils, switch these tests to use the new wrapper function that is based on the "tarfile" module instead.

---
 MAINTAINERS                                   |  6 +--
 tests/avocado/machine_sparc64_sun4u.py        | 36 ----------------
 tests/functional/meson.build                  |  6 +++
 .../test_arm_canona1100.py}                   | 30 ++++++++------
 .../test_ppc_bamboo.py}                       | 33 +++++++--------
 tests/functional/test_sparc64_sun4u.py        | 41 +++++++++++++++++++
 6 files changed, 84 insertions(+), 68 deletions(-)
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
MAINTAINERS +3 -3
index 6545662eaf..56dcd56375 100644
@@ -733,7 +733,7 @@ S: Odd Fixes
 F: include/hw/arm/digic.h
 F: hw/*/digic*
 F: include/hw/*/digic*
-F: tests/avocado/machine_arm_canona1100.py
+F: tests/functional/test_arm_canona1100.py
 F: docs/system/arm/digic.rst
 
 Goldfish RTC
@@ -1430,7 +1430,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/ppc440_bamboo.c
 F: hw/pci-host/ppc4xx_pci.c
-F: tests/avocado/ppc_bamboo.py
+F: tests/functional/test_ppc_bamboo.py
 
 e500
 L: [email protected]
@@ -1716,7 +1716,7 @@ F: include/hw/pci-host/sabre.h
 F: hw/pci-bridge/simba.c
 F: include/hw/pci-bridge/simba.h
 F: pc-bios/openbios-sparc64
-F: tests/avocado/machine_sparc64_sun4u.py
+F: tests/functional/test_sparc64_sun4u.py
 
 Sun4v
 M: Artyom Tarasenko <[email protected]>
tests/avocado/machine_sparc64_sun4u.py deleted +0 -36
deleted file mode 100644
index d333c0ae91..0000000000
@@ -1,36 +0,0 @@
-# Functional test that boots a Linux kernel and checks the console
-#
-# Copyright (c) 2020 Red Hat, Inc.
-#
-# Author:
-#  Thomas Huth <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later. See the COPYING file in the top-level directory.
-
-import os
-
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
-from boot_linux_console import LinuxKernelTest
-
-class Sun4uMachine(LinuxKernelTest):
-    """Boots the Linux kernel and checks that the console is operational"""
-
-    timeout = 90
-
-    def test_sparc64_sun4u(self):
-        """
-        :avocado: tags=arch:sparc64
-        :avocado: tags=machine:sun4u
-        """
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day23.tar.xz')
-        tar_hash = '142db83cd974ffadc4f75c8a5cad5bcc5722c240'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
-        self.vm.set_console()
-        self.vm.add_args('-kernel', self.workdir + '/day23/vmlinux',
-                         '-append', self.KERNEL_COMMON_COMMAND_LINE)
-        self.vm.launch()
-        wait_for_console_pattern(self, 'Starting logging: OK')
tests/functional/meson.build +6 -0
index 11ff246cef..920c0a3928 100644
@@ -27,6 +27,7 @@ tests_generic_bsduser = [
 ]
 
 tests_arm_system_thorough = [
+  'arm_canona1100',
   'arm_n8x0',
 ]
 
@@ -48,6 +49,11 @@ tests_ppc_system_quick = [
 
 tests_ppc_system_thorough = [
   'ppc_405',
+  'ppc_bamboo',
+]
+
+tests_sparc64_system_thorough = [
+  'sparc64_sun4u',
 ]
 
 tests_x86_64_system_quick = [
tests/functional/test_arm_canona1100.py renamed from tests/avocado/machine_arm_canona1100.py +17 -13
old mode 100644
new mode 100755
similarity index 51%
rename from tests/avocado/machine_arm_canona1100.py
rename to tests/functional/test_arm_canona1100.py
index a42d8b0f2b..65f1228296
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots the canon-a1100 machine with firmware
 #
 # Copyright (c) 2020 Red Hat, Inc.
@@ -8,28 +10,30 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
 
 class CanonA1100Machine(QemuSystemTest):
     """Boots the barebox firmware and checks that the console is operational"""
 
     timeout = 90
 
+    ASSET_BIOS = Asset(('https://qemu-advcal.gitlab.io'
+                        '/qac-best-of-multiarch/download/day18.tar.xz'),
+                       '28e71874ce985be66b7fd1345ed88cb2523b982f899c8d2900d6353054a1be49')
+
     def test_arm_canona1100(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=machine:canon-a1100
-        :avocado: tags=device:pflash_cfi02
-        """
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day18.tar.xz')
-        tar_hash = '068b5fc4242b29381acee94713509f8a876e9db6'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('canon-a1100')
+
+        file_path = self.ASSET_BIOS.fetch()
+        archive_extract(file_path, dest_dir=self.workdir,
+                        member="day18/barebox.canon-a1100.bin")
         self.vm.set_console()
         self.vm.add_args('-bios',
                          self.workdir + '/day18/barebox.canon-a1100.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'running /env/bin/init')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_bamboo.py renamed from tests/avocado/ppc_bamboo.py +17 -16
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/ppc_bamboo.py
rename to tests/functional/test_ppc_bamboo.py
index a81be3d608..e72cbdee12
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on the ppc bamboo board and check the console
 #
 # Copyright (c) 2021 Red Hat
@@ -5,30 +7,26 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
 
 class BambooMachine(QemuSystemTest):
 
     timeout = 90
 
+    ASSET_IMAGE = Asset(
+        ('http://landley.net/aboriginal/downloads/binaries/'
+         'system-image-powerpc-440fp.tar.gz'),
+        'c12b58f841c775a0e6df4832a55afe6b74814d1565d08ddeafc1fb949a075c5e')
+
     def test_ppc_bamboo(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:bamboo
-        :avocado: tags=cpu:440epb
-        :avocado: tags=device:rtl8139
-        :avocado: tags=accel:tcg
-        """
+        self.set_machine('bamboo')
         self.require_accelerator("tcg")
         self.require_netdev('user')
-        tar_url = ('http://landley.net/aboriginal/downloads/binaries/'
-                   'system-image-powerpc-440fp.tar.gz')
-        tar_hash = '53e5f16414b195b82d2c70272f81c2eedb39bad9'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir +
                                    '/system-image-powerpc-440fp/linux',
@@ -40,3 +38,6 @@ def test_ppc_bamboo(self):
         exec_command_and_wait_for_pattern(self, 'ping 10.0.2.2',
                                           '10.0.2.2 is alive!')
         exec_command_and_wait_for_pattern(self, 'halt', 'System Halted')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_sparc64_sun4u.py new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.46.0
#22 lore raw

PATCH v5 20/44 tests/functional: Convert the s390x avocado tests into standalone tests

In reply to Thomas Huth

These tests use archive.lzma_uncompress() from the Avocado utils, so provide a small helper function for this, based on the standard lzma module from Python instead.

And while we're at it, replace the MD5 hashes in the topology test with proper SHA256 hashes, since MD5 should not be used anymore nowadays.

---
 MAINTAINERS                                   |  4 +-
 tests/functional/meson.build                  |  6 ++
 tests/functional/qemu_test/utils.py           | 14 +++
 .../test_s390x_ccw_virtio.py}                 | 79 ++++++++---------
 .../test_s390x_topology.py}                   | 86 ++++++++-----------
 5 files changed, 94 insertions(+), 95 deletions(-)
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index 56dcd56375..7fad267307 100644
@@ -1743,7 +1743,7 @@ S: Supported
 F: hw/s390x/
 F: include/hw/s390x/
 F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
 T: git https://github.com/borntraeger/qemu.git s390-next
 L: [email protected]
 
@@ -1806,7 +1806,7 @@ F: hw/s390x/cpu-topology.c
 F: target/s390x/kvm/stsi-topology.c
 F: docs/devel/s390-cpu-topology.rst
 F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
 
 X86 Machines
 ------------
tests/functional/meson.build +6 -0
index 920c0a3928..16aa304c24 100644
@@ -12,6 +12,7 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  's390x_ccw_virtio' : 180,
 }
 
 tests_generic_system = [
@@ -52,6 +53,11 @@ tests_ppc_system_thorough = [
   'ppc_bamboo',
 ]
 
+tests_s390x_system_thorough = [
+  's390x_ccw_virtio',
+  's390x_topology',
+]
+
 tests_sparc64_system_thorough = [
   'sparc64_sun4u',
 ]
tests/functional/qemu_test/utils.py +14 -0
index 4eb5e5d5e5..a12dac51b6 100644
@@ -8,6 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import lzma
+import os
+import shutil
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):
@@ -19,3 +22,14 @@ def archive_extract(archive, dest_dir, member=None):
             tf.extract(member=member, path=dest_dir)
         else:
             tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with lzma.open(xz_path, 'rb') as lzma_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(lzma_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
tests/functional/test_s390x_ccw_virtio.py renamed from tests/avocado/machine_s390_ccw_virtio.py +38 -41

113 lines not on this page · raw message

tests/functional/test_s390x_topology.py renamed from tests/avocado/s390_topology.py +34 -52

183 lines not on this page · raw message

Signature · 1 line
2.46.0
#23 lore raw

PATCH v5 21/44 tests/functional: Convert the x86_cpu_model_versions test

In reply to Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
 MAINTAINERS                                   |  1 +
 tests/functional/meson.build                  |  1 +
 .../test_x86_cpu_model_versions.py}           | 65 ++++++-------------
 3 files changed, 21 insertions(+), 46 deletions(-)
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
MAINTAINERS +1 -0
index 7fad267307..bbb8fc78c5 100644
@@ -1836,6 +1836,7 @@ F: tests/unit/test-x86-topo.c
 F: tests/qtest/test-x86-cpuid-compat.c
 F: tests/functional/test_mem_addr_space.py
 F: tests/functional/test_pc_cpu_hotplug_props.py
+F: tests/functional/test_x86_cpu_model_versions.py
 
 PC Chipset
 M: Michael S. Tsirkin <[email protected]>
tests/functional/meson.build +1 -0
index 16aa304c24..540e1361f8 100644
@@ -67,6 +67,7 @@ tests_x86_64_system_quick = [
   'mem_addr_space',
   'pc_cpu_hotplug_props',
   'virtio_version',
+  'x86_cpu_model_versions',
 ]
 
 tests_x86_64_system_thorough = [
tests/functional/test_x86_cpu_model_versions.py renamed from tests/avocado/x86_cpu_model_versions.py +19 -46

158 lines not on this page · raw message

Signature · 1 line
2.46.0
#24 lore raw

PATCH v5 22/44 tests/functional: Convert the microblaze avocado tests into standalone tests

In reply to Thomas Huth

The machine_microblaze.py file contained two tests, one for each endianness. Since we only support one QEMU target binary per file in the new functional test environment, we have to split this file up into two files now.

---
 MAINTAINERS                                   |  2 +-
 tests/avocado/machine_microblaze.py           | 61 -------------------
 tests/functional/meson.build                  |  8 +++
 .../functional/test_microblaze_s3adsp1800.py  | 40 ++++++++++++
 .../test_microblazeel_s3adsp1800.py           | 42 +++++++++++++
 5 files changed, 91 insertions(+), 62 deletions(-)
 delete mode 100644 tests/avocado/machine_microblaze.py
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
MAINTAINERS +1 -1
index bbb8fc78c5..91f5a0545f 100644
@@ -1329,7 +1329,7 @@ M: Edgar E. Iglesias <[email protected]>
 S: Maintained
 F: hw/microblaze/petalogix_s3adsp1800_mmu.c
 F: include/hw/char/xilinx_uartlite.h
-F: tests/avocado/machine_microblaze.py
+F: tests/functional/test_microblaze*.py
 
 petalogix_ml605
 M: Edgar E. Iglesias <[email protected]>
tests/avocado/machine_microblaze.py deleted +0 -61

61 lines not on this page · raw message

tests/functional/meson.build +8 -0
index 540e1361f8..f32afed30f 100644
@@ -40,6 +40,14 @@ tests_loongarch64_system_thorough = [
   'loongarch64_virt',
 ]
 
+tests_microblaze_system_thorough = [
+  'microblaze_s3adsp1800'
+]
+
+tests_microblazeel_system_thorough = [
+  'microblazeel_s3adsp1800'
+]
+
 tests_mips64el_system_thorough = [
   'mips64el_loongson3v',
 ]
tests/functional/test_microblaze_s3adsp1800.py new file +40 -0
new file mode 100755
index 0000000000..4f692ffdb1
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+#
+# Functional test that boots a microblaze Linux kernel and checks the console
+#
+# Copyright (c) 2018, 2021 Red Hat, Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import time
+from qemu_test import exec_command, exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+
+class MicroblazeMachine(QemuSystemTest):
+
+    timeout = 90
+
+    ASSET_IMAGE = Asset(
+        ('https://qemu-advcal.gitlab.io/qac-best-of-multiarch/download/'
+         'day17.tar.xz'),
+        '3ba7439dfbea7af4876662c97f8e1f0cdad9231fc166e4861d17042489270057')
+
+    def test_microblaze_s3adsp1800(self):
+        self.set_machine('petalogix-s3adsp1800')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
+        self.vm.set_console()
+        self.vm.add_args('-kernel', self.workdir + '/day17/ballerina.bin')
+        self.vm.launch()
+        wait_for_console_pattern(self, 'This architecture does not have '
+                                       'kernel memory protection')
+        # Note:
+        # The kernel sometimes gets stuck after the "This architecture ..."
+        # message, that's why we don't test for a later string here. This
+        # needs some investigation by a microblaze wizard one day...
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_microblazeel_s3adsp1800.py new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.46.0
#25 lore raw

PATCH v5 23/44 tests/functional: Convert the riscv_opensbi avocado test into a standalone test

In reply to Thomas Huth

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

However, we have to mark the "spike" test as flaky now since it sometimes fails when running the tests in parallel (the failure happens also with the Avocado test when the host is under load, so the problem is not related to this conversion to the functional framework).

---
 MAINTAINERS                            |  1 +
 tests/avocado/riscv_opensbi.py         | 63 --------------------------
 tests/functional/meson.build           |  8 ++++
 tests/functional/test_riscv_opensbi.py | 40 ++++++++++++++++
 4 files changed, 49 insertions(+), 63 deletions(-)
 delete mode 100644 tests/avocado/riscv_opensbi.py
 create mode 100755 tests/functional/test_riscv_opensbi.py
MAINTAINERS +1 -0
index 91f5a0545f..11cefa015e 100644
@@ -337,6 +337,7 @@ F: hw/intc/riscv*
 F: include/hw/riscv/
 F: linux-user/host/riscv32/
 F: linux-user/host/riscv64/
+F: tests/functional/test_riscv*
 F: tests/tcg/riscv64/
 
 RISC-V XThead* extensions
tests/avocado/riscv_opensbi.py deleted +0 -63

63 lines not on this page · raw message

tests/functional/meson.build +8 -0
index f32afed30f..700deed24a 100644
@@ -61,6 +61,14 @@ tests_ppc_system_thorough = [
   'ppc_bamboo',
 ]
 
+tests_riscv32_system_quick = [
+  'riscv_opensbi',
+]
+
+tests_riscv64_system_quick = [
+  'riscv_opensbi',
+]
+
 tests_s390x_system_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/test_riscv_opensbi.py new file +40 -0
new file mode 100755
index 0000000000..de632d05cc
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+#
+# OpenSBI boot test for RISC-V machines
+#
+# Copyright (c) 2022, Ventana Micro
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+
+class RiscvOpenSBI(QemuSystemTest):
+
+    timeout = 5
+
+    def boot_opensbi(self):
+        self.vm.set_console()
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Platform Name')
+        wait_for_console_pattern(self, 'Boot HART MEDELEG')
+
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'),
+                'Test is unstable when system is under load')
+    def test_riscv_spike(self):
+        self.set_machine('spike')
+        self.boot_opensbi()
+
+    def test_riscv_sifive_u(self):
+        self.set_machine('sifive_u')
+        self.boot_opensbi()
+
+    def test_riscv_virt(self):
+        self.set_machine('virt')
+        self.boot_opensbi()
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
Tested-by on v5 23/44
#26 lore raw

Re: [PATCH v5 23/44] tests/functional: Convert the riscv_opensbi avocado test into a standalone test

Quoted 13 lines · Thomas Huth

The avocado test defined test functions for both, riscv32 and riscv64. Since we can run the whole file with multiple targets in the new framework, we can now consolidate the functions so we have to only define one function per machine now.

However, we have to mark the "spike" test as flaky now since it sometimes fails when running the tests in parallel (the failure happens also with the Avocado test when the host is under load, so the problem is not related to this conversion to the functional framework).

Quoted 8 lines
---
  MAINTAINERS                            |  1 +
  tests/avocado/riscv_opensbi.py         | 63 --------------------------
  tests/functional/meson.build           |  8 ++++
  tests/functional/test_riscv_opensbi.py | 40 ++++++++++++++++
  4 files changed, 49 insertions(+), 63 deletions(-)

delete mode 100644 tests/avocado/riscv_opensbi.py create mode 100755 tests/functional/test_riscv_opensbi.py

#27 lore raw

PATCH v5 24/44 tests/functional: Convert the virtio_gpu avocado test into a standalone test

In reply to Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                  |  1 +
 .../test_virtio_gpu.py}                       | 64 +++++++++----------
 2 files changed, 30 insertions(+), 35 deletions(-)
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index 700deed24a..5559d131f3 100644
@@ -88,6 +88,7 @@ tests_x86_64_system_quick = [
 
 tests_x86_64_system_thorough = [
   'netdev_ethtool',
+  'virtio_gpu',
 ]
 
 precache_all = []
tests/functional/test_virtio_gpu.py renamed from tests/avocado/virtio-gpu.py +29 -35

125 lines not on this page · raw message

Signature · 1 line
2.46.0
#28 lore raw

PATCH v5 25/44 tests/functional: Convert most ppc avocado tests into standalone tests

In reply to Thomas Huth

Nothing thrilling in here, just straight forward conversions.

---
 MAINTAINERS                                   |  9 +--
 tests/functional/meson.build                  | 11 ++++
 .../test_ppc64_powernv.py}                    | 56 +++++-----------
 .../test_ppc64_pseries.py}                    | 56 +++++-----------
 .../test_ppc_40p.py}                          | 65 +++++++++----------
 .../test_ppc_mpc8544ds.py}                    | 29 +++++----
 .../test_ppc_virtex_ml507.py}                 | 29 +++++----
 7 files changed, 113 insertions(+), 142 deletions(-)
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
MAINTAINERS +5 -4
index 11cefa015e..26c9fba7c5 100644
@@ -1454,7 +1454,7 @@ L: [email protected]
 S: Orphan
 F: hw/ppc/mpc8544ds.c
 F: hw/ppc/mpc8544_guts.c
-F: tests/avocado/ppc_mpc8544ds.py
+F: tests/functional/test_ppc_mpc8544ds.py
 
 New World (mac99)
 M: Mark Cave-Ayland <[email protected]>
@@ -1507,7 +1507,7 @@ F: hw/dma/i82374.c
 F: hw/rtc/m48t59-isa.c
 F: include/hw/isa/pc87312.h
 F: include/hw/rtc/m48t59.h
-F: tests/avocado/ppc_prep_40p.py
+F: tests/functional/test_ppc_40p.py
 
 sPAPR (pseries)
 M: Nicholas Piggin <[email protected]>
@@ -1531,7 +1531,7 @@ F: tests/qtest/spapr*
 F: tests/qtest/libqos/*spapr*
 F: tests/qtest/rtas*
 F: tests/qtest/libqos/rtas*
-F: tests/avocado/ppc_pseries.py
+F: tests/functional/test_ppc64_pseries.py
 F: tests/avocado/ppc_hv_tests.py
 
 PowerNV (Non-Virtualized)
@@ -1549,6 +1549,7 @@ F: include/hw/ppc/pnv*
 F: include/hw/pci-host/pnv*
 F: pc-bios/skiboot.lid
 F: tests/qtest/pnv*
+F: tests/functional/test_ppc64_powernv.py
 
 pca955x
 M: Glenn Miles <[email protected]>
@@ -1563,7 +1564,7 @@ M: Edgar E. Iglesias <[email protected]>
 L: [email protected]
 S: Odd Fixes
 F: hw/ppc/virtex_ml507.c
-F: tests/avocado/ppc_virtex_ml507.py
+F: tests/functional/test_ppc_virtex_ml507.py
 
 sam460ex
 M: BALATON Zoltan <[email protected]>
tests/functional/meson.build +11 -0
index 5559d131f3..3726c61ca3 100644
@@ -12,6 +12,9 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'netdev_ethtool' : 180,
+  'ppc_40p' : 240,
+  'ppc64_powernv' : 120,
+  'ppc64_pseries' : 120,
   's390x_ccw_virtio' : 180,
 }
 
@@ -58,7 +61,15 @@ tests_ppc_system_quick = [
 
 tests_ppc_system_thorough = [
   'ppc_405',
+  'ppc_40p',
   'ppc_bamboo',
+  'ppc_mpc8544ds',
+  'ppc_virtex_ml507',
+]
+
+tests_ppc64_system_thorough = [
+  'ppc64_powernv',
+  'ppc64_pseries',
 ]
 
 tests_riscv32_system_quick = [
tests/functional/test_ppc64_powernv.py renamed from tests/avocado/ppc_powernv.py +18 -38

105 lines not on this page · raw message

tests/functional/test_ppc64_pseries.py renamed from tests/avocado/ppc_pseries.py +18 -38

110 lines not on this page · raw message

tests/functional/test_ppc_40p.py renamed from tests/avocado/ppc_prep_40p.py +29 -36

103 lines not on this page · raw message

tests/functional/test_ppc_mpc8544ds.py renamed from tests/avocado/ppc_mpc8544ds.py +16 -13
old mode 100644
new mode 100755
similarity index 55%
rename from tests/avocado/ppc_mpc8544ds.py
rename to tests/functional/test_ppc_mpc8544ds.py
index b599fb1cc9..2b3f0894ae
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class Mpc8544dsMachine(QemuSystemTest):
 
@@ -15,20 +17,21 @@ class Mpc8544dsMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
     panic_message = 'Kernel panic - not syncing'
 
+    ASSET_IMAGE = Asset(
+        ('https://qemu-advcal.gitlab.io/qac-best-of-multiarch/download/'
+         'day04.tar.xz'),
+        '88bc83f3c9f3d633bcfc108a6342d677abca247066a2fb8d4636744a0d319f94')
+
     def test_ppc_mpc8544ds(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:mpc8544ds
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day04.tar.xz')
-        tar_hash = 'f46724d281a9f30fa892d458be7beb7d34dc25f9'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('mpc8544ds')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir, member='creek/creek.bin')
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/creek/creek.bin')
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
tests/functional/test_ppc_virtex_ml507.py renamed from tests/avocado/ppc_virtex_ml507.py +16 -13
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/ppc_virtex_ml507.py
rename to tests/functional/test_ppc_virtex_ml507.py
index a73f8ae396..ffa9a0633e
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Test that Linux kernel boots on ppc machines and check the console
 #
 # Copyright (c) 2018, 2020 Red Hat, Inc.
@@ -5,9 +7,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
 
 class VirtexMl507Machine(QemuSystemTest):
 
@@ -15,18 +17,16 @@ class VirtexMl507Machine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
     panic_message = 'Kernel panic - not syncing'
 
+    ASSET_IMAGE = Asset(
+        ('https://qemu-advcal.gitlab.io/qac-best-of-multiarch/download/'
+         'day08.tar.xz'),
+        'cefe5b8aeb5e9d2d1d4fd22dcf48d917d68d5a765132bf2ddd6332dc393b824c')
+
     def test_ppc_virtex_ml507(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:virtex-ml507
-        :avocado: tags=accel:tcg
-        """
         self.require_accelerator("tcg")
-        tar_url = ('https://qemu-advcal.gitlab.io'
-                   '/qac-best-of-multiarch/download/day08.tar.xz')
-        tar_hash = '74c68f5af7a7b8f21c03097b298f3bb77ff52c1f'
-        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
-        archive.extract(file_path, self.workdir)
+        self.set_machine('virtex-ml507')
+        file_path = self.ASSET_IMAGE.fetch()
+        archive_extract(file_path, self.workdir)
         self.vm.set_console()
         self.vm.add_args('-kernel', self.workdir + '/hippo/hippo.linux',
                          '-dtb', self.workdir + '/hippo/virtex440-ml507.dtb',
@@ -34,3 +34,6 @@ def test_ppc_virtex_ml507(self):
         self.vm.launch()
         wait_for_console_pattern(self, 'QEMU advent calendar 2020',
                                  self.panic_message)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
Tested-by on v5 25/44
#29 lore raw

Re: [PATCH v5 25/44] tests/functional: Convert most ppc avocado tests into standalone tests

Quoted 13 lines · Thomas Huth

Nothing thrilling in here, just straight forward conversions.

---
  MAINTAINERS                                   |  9 +--
  tests/functional/meson.build                  | 11 ++++
  .../test_ppc64_powernv.py}                    | 56 +++++-----------
  .../test_ppc64_pseries.py}                    | 56 +++++-----------
  .../test_ppc_40p.py}                          | 65 +++++++++----------
  .../test_ppc_mpc8544ds.py}                    | 29 +++++----
  .../test_ppc_virtex_ml507.py}                 | 29 +++++----
  7 files changed, 113 insertions(+), 142 deletions(-)
Quoted 10 lines

rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%) mode change 100644 => 100755
rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%) mode change 100644 => 100755
rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (51%) mode change 100644 => 100755
rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%) mode change 100644 => 100755
rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%) mode change 100644 => 100755

#30 lore raw

PATCH v5 26/44 tests/functional: Convert the ppc_amiga avocado test into a standalone test

In reply to Thomas Huth

Use the Python standard zipfile module instead of avocado.utils for extracting the ZIP file that we download here, and use the standard subprocess module for running the "tail" command.

---
 MAINTAINERS                        |  1 +
 tests/avocado/ppc_amiga.py         | 38 --------------------------
 tests/functional/meson.build       |  1 +
 tests/functional/test_ppc_amiga.py | 43 ++++++++++++++++++++++++++++++
 4 files changed, 45 insertions(+), 38 deletions(-)
 delete mode 100644 tests/avocado/ppc_amiga.py
 create mode 100755 tests/functional/test_ppc_amiga.py
MAINTAINERS +1 -0
index 26c9fba7c5..f01e865025 100644
@@ -1597,6 +1597,7 @@ S: Maintained
 F: hw/ppc/amigaone.c
 F: hw/pci-host/articia.c
 F: include/hw/pci-host/articia.h
+F: tests/functional/test_ppc_amiga.py
 
 Virtual Open Firmware (VOF)
 M: Alexey Kardashevskiy <[email protected]>
tests/avocado/ppc_amiga.py deleted +0 -38
deleted file mode 100644
index b6f866f91d..0000000000
@@ -1,38 +0,0 @@
-# Test AmigaNG boards
-#
-# Copyright (c) 2023 BALATON Zoltan
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-from avocado.utils import archive
-from avocado.utils import process
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-
-class AmigaOneMachine(QemuSystemTest):
-
-    timeout = 90
-
-    def test_ppc_amigaone(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:amigaone
-        :avocado: tags=device:articia
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        tar_name = 'A1Firmware_Floppy_05-Mar-2005.zip'
-        tar_url = ('https://www.hyperion-entertainment.com/index.php/'
-                   'downloads?view=download&format=raw&file=25')
-        tar_hash = 'c52e59bc73e31d8bcc3cc2106778f7ac84f6c755'
-        zip_file = self.fetch_asset(tar_name, locations=tar_url,
-                                    asset_hash=tar_hash)
-        archive.extract(zip_file, self.workdir)
-        cmd = f"tail -c 524288 {self.workdir}/floppy_edition/updater.image >{self.workdir}/u-boot-amigaone.bin"
-        process.run(cmd, shell=True)
-
-        self.vm.set_console()
-        self.vm.add_args('-bios', self.workdir + '/u-boot-amigaone.bin')
-        self.vm.launch()
-        wait_for_console_pattern(self, 'FLASH:')
tests/functional/meson.build +1 -0
index 3726c61ca3..8f46a90003 100644
@@ -62,6 +62,7 @@ tests_ppc_system_quick = [
 tests_ppc_system_thorough = [
   'ppc_405',
   'ppc_40p',
+  'ppc_amiga',
   'ppc_bamboo',
   'ppc_mpc8544ds',
   'ppc_virtex_ml507',
tests/functional/test_ppc_amiga.py new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.46.0
#31 lore raw

PATCH v5 27/44 tests/functional: Convert the ppc_hv avocado test into a standalone test

In reply to Thomas Huth

A straight forward conversion, we just also have to remove the decorator @skipUnless(os.getenv('SPEED')) since all non-trivial functional tests are running in SPEED=thorough mode now. Also make sure that the extracted assets are writable, so that the test does not fail if it gets re-run and there are stale read-only files already around.

---
 MAINTAINERS                                   |  2 +-
 tests/functional/meson.build                  |  2 +
 .../test_ppc64_hv.py}                         | 52 ++++++++-----------
 3 files changed, 24 insertions(+), 32 deletions(-)
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (85%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index f01e865025..cca48683ce 100644
@@ -1532,7 +1532,7 @@ F: tests/qtest/libqos/*spapr*
 F: tests/qtest/rtas*
 F: tests/qtest/libqos/rtas*
 F: tests/functional/test_ppc64_pseries.py
-F: tests/avocado/ppc_hv_tests.py
+F: tests/functional/test_ppc64_hv.py
 
 PowerNV (Non-Virtualized)
 M: Cédric Le Goater <[email protected]>
tests/functional/meson.build +2 -0
index 8f46a90003..1b2e6d142c 100644
@@ -13,6 +13,7 @@ endif
 test_timeouts = {
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
+  'ppc64_hv' : 1000,
   'ppc64_powernv' : 120,
   'ppc64_pseries' : 120,
   's390x_ccw_virtio' : 180,
@@ -69,6 +70,7 @@ tests_ppc_system_thorough = [
 ]
 
 tests_ppc64_system_thorough = [
+  'ppc64_hv',
   'ppc64_powernv',
   'ppc64_pseries',
 ]
tests/functional/test_ppc64_hv.py renamed from tests/avocado/ppc_hv_tests.py +21 -31

109 lines not on this page · raw message

Signature · 1 line
2.46.0
#32 lore raw

PATCH v5 28/44 tests/functional: Convert the m68k nextcube test with tesseract

In reply to Thomas Huth

The code that handles running of tesseract needs to be tweaked a little bit to be able to run without the functions from avocado.utils, and while we're at it, drop some legacy stuff that was still there due to Tesseract 3 support that we already dropped a while ago.

---
 MAINTAINERS                                   |  1 +
 tests/avocado/tesseract_utils.py              | 46 -------------------
 tests/functional/meson.build                  |  4 ++
 tests/functional/qemu_test/tesseract.py       | 35 ++++++++++++++
 .../test_m68k_nextcube.py}                    | 29 ++++++------
 5 files changed, 56 insertions(+), 59 deletions(-)
 delete mode 100644 tests/avocado/tesseract_utils.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (75%)
 mode change 100644 => 100755
MAINTAINERS +1 -0
index cca48683ce..394d704515 100644
@@ -1283,6 +1283,7 @@ S: Odd Fixes
 F: hw/m68k/next-*.c
 F: hw/display/next-fb.c
 F: include/hw/m68k/next-cube.h
+F: tests/functional/test_m68k_nextcube.py
 
 q800
 M: Laurent Vivier <[email protected]>
tests/avocado/tesseract_utils.py deleted +0 -46

46 lines not on this page · raw message

tests/functional/meson.build +4 -0
index 1b2e6d142c..3a71e02e74 100644
@@ -44,6 +44,10 @@ tests_loongarch64_system_thorough = [
   'loongarch64_virt',
 ]
 
+tests_m68k_system_thorough = [
+  'm68k_nextcube'
+]
+
 tests_microblaze_system_thorough = [
   'microblaze_s3adsp1800'
 ]
tests/functional/qemu_test/tesseract.py new file +35 -0
new file mode 100644
index 0000000000..c4087b7c11
@@ -0,0 +1,35 @@
+# ...
+#
+# Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import re
+import logging
+
+from . import has_cmd, run_cmd
+
+def tesseract_available(expected_version):
+    if not has_cmd('tesseract'):
+        return False
+    (stdout, stderr, ret) = run_cmd([ 'tesseract', '--version'])
+    if ret:
+        return False
+    version = stdout.split()[1]
+    return int(version.split('.')[0]) >= expected_version
+
+def tesseract_ocr(image_path, tesseract_args=''):
+    console_logger = logging.getLogger('console')
+    console_logger.debug(image_path)
+    (stdout, stderr, ret) = run_cmd(['tesseract', image_path,
+                                     'stdout'])
+    if ret:
+        return None
+    lines = []
+    for line in stdout.split('\n'):
+        sline = line.strip()
+        if len(sline):
+            console_logger.debug(sline)
+            lines += [sline]
+    return lines
tests/functional/test_m68k_nextcube.py renamed from tests/avocado/machine_m68k_nextcube.py +16 -13
old mode 100644
new mode 100755
similarity index 75%
rename from tests/avocado/machine_m68k_nextcube.py
rename to tests/functional/test_m68k_nextcube.py
index 1f3c883910..89385a134a
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional test that boots a VM and run OCR on the framebuffer
 #
 # Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
@@ -8,10 +10,10 @@
 import os
 import time
 
-from avocado_qemu import QemuSystemTest
-from avocado import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from unittest import skipUnless
 
-from tesseract_utils import tesseract_available, tesseract_ocr
+from qemu_test.tesseract import tesseract_available, tesseract_ocr
 
 PIL_AVAILABLE = True
 try:
@@ -21,19 +23,15 @@
 
 
 class NextCubeMachine(QemuSystemTest):
-    """
-    :avocado: tags=arch:m68k
-    :avocado: tags=machine:next-cube
-    :avocado: tags=device:framebuffer
-    """
 
     timeout = 15
 
+    ASSET_ROM = Asset(('https://sourceforge.net/p/previous/code/1350/tree/'
+                       'trunk/src/Rev_2.5_v66.BIN?format=raw'),
+                      '1b753890b67095b73e104c939ddf62eca9e7d0aedde5108e3893b0ed9d8000a4')
+
     def check_bootrom_framebuffer(self, screenshot_path):
-        rom_url = ('https://sourceforge.net/p/previous/code/1350/tree/'
-                   'trunk/src/Rev_2.5_v66.BIN?format=raw')
-        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
-        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+        rom_path = self.ASSET_ROM.fetch()
 
         self.vm.add_args('-bios', rom_path)
         self.vm.launch()
@@ -48,6 +46,7 @@ def check_bootrom_framebuffer(self, screenshot_path):
 
     @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
     def test_bootrom_framebuffer_size(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
@@ -60,11 +59,15 @@ def test_bootrom_framebuffer_size(self):
     # that it is still alpha-level software.
     @skipUnless(tesseract_available(4), 'tesseract OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract(self):
+        self.set_machine('next-cube')
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
-        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
+        lines = tesseract_ocr(screenshot_path)
         text = '\n'.join(lines)
         self.assertIn('Testing the FPU', text)
         self.assertIn('System test failed. Error code', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#33 lore raw

PATCH v5 29/44 tests/functional: Convert the acpi-bits test into a standalone test

In reply to Thomas Huth

Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore.

---
 MAINTAINERS                                   |  4 +-
 docs/devel/acpi-bits.rst                      | 62 ++++++--------
 .../acpi-bits/bits-config/bits-cfg.txt        |  0
 .../acpi-bits/bits-tests/smbios.py2           |  0
 .../acpi-bits/bits-tests/smilatency.py2       |  0
 .../acpi-bits/bits-tests/testacpi.py2         |  0
 .../acpi-bits/bits-tests/testcpuid.py2        |  0
 tests/functional/meson.build                  |  2 +
 .../test_acpi_bits.py}                        | 83 ++++++++++---------
 9 files changed, 71 insertions(+), 80 deletions(-)
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
MAINTAINERS +2 -2
index 394d704515..c2dfb3dad3 100644
@@ -2075,8 +2075,8 @@ ACPI/AVOCADO/BIOSBITS
 M: Ani Sinha <[email protected]>
 M: Michael S. Tsirkin <[email protected]>
 S: Supported
-F: tests/avocado/acpi-bits/*
-F: tests/avocado/acpi-bits.py
+F: tests/functional/acpi-bits/*
+F: tests/functional/test_acpi_bits.py
 F: docs/devel/acpi-bits.rst
 
 ACPI/HEST/GHES
docs/devel/acpi-bits.rst +25 -37

111 lines not on this page · raw message

tests/functional/acpi-bits/bits-config/bits-cfg.txt renamed from tests/avocado/acpi-bits/bits-config/bits-cfg.txt +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-config/bits-cfg.txt
rename to tests/functional/acpi-bits/bits-config/bits-cfg.txt
tests/functional/acpi-bits/bits-tests/smbios.py2 renamed from tests/avocado/acpi-bits/bits-tests/smbios.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smbios.py2
rename to tests/functional/acpi-bits/bits-tests/smbios.py2
tests/functional/acpi-bits/bits-tests/smilatency.py2 renamed from tests/avocado/acpi-bits/bits-tests/smilatency.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/smilatency.py2
rename to tests/functional/acpi-bits/bits-tests/smilatency.py2
tests/functional/acpi-bits/bits-tests/testacpi.py2 renamed from tests/avocado/acpi-bits/bits-tests/testacpi.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testacpi.py2
rename to tests/functional/acpi-bits/bits-tests/testacpi.py2
tests/functional/acpi-bits/bits-tests/testcpuid.py2 renamed from tests/avocado/acpi-bits/bits-tests/testcpuid.py2 +0 -0
similarity index 100%
rename from tests/avocado/acpi-bits/bits-tests/testcpuid.py2
rename to tests/functional/acpi-bits/bits-tests/testcpuid.py2
tests/functional/meson.build +2 -0
index 3a71e02e74..76eb00ef46 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'acpi_bits' : 240,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
   'ppc64_hv' : 1000,
@@ -105,6 +106,7 @@ tests_x86_64_system_quick = [
 ]
 
 tests_x86_64_system_thorough = [
+  'acpi_bits',
   'netdev_ethtool',
   'virtio_gpu',
 ]
tests/functional/test_acpi_bits.py renamed from tests/avocado/acpi-bits.py +42 -41

180 lines not on this page · raw message

Signature · 1 line
2.46.0
#34 lore raw

PATCH v5 30/44 tests/functional: Convert the rx_gdbsim avocado test into a standalone test

In reply to Thomas Huth

Provide a "gzip_uncompress" function based on the standard "gzip" module to avoid the usage of avocado.utils here.

---
 MAINTAINERS                                   |  2 +-
 tests/functional/meson.build                  |  4 ++
 tests/functional/qemu_test/utils.py           | 12 +++++
 .../test_rx_gdbsim.py}                        | 54 ++++++++++---------
 4 files changed, 46 insertions(+), 26 deletions(-)
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (58%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index c2dfb3dad3..25e71ac14c 100644
@@ -1665,7 +1665,7 @@ R: Yoshinori Sato <[email protected]>
 S: Orphan
 F: docs/system/target-rx.rst
 F: hw/rx/rx-gdbsim.c
-F: tests/avocado/machine_rx_gdbsim.py
+F: tests/functional/test_rx_gdbsim.py
 
 SH4 Machines
 ------------
tests/functional/meson.build +4 -0
index 76eb00ef46..bb511482cb 100644
@@ -88,6 +88,10 @@ tests_riscv64_system_quick = [
   'riscv_opensbi',
 ]
 
+tests_rx_system_thorough = [
+  'rx_gdbsim',
+]
+
 tests_s390x_system_thorough = [
   's390x_ccw_virtio',
   's390x_topology',
tests/functional/qemu_test/utils.py +12 -0
index a12dac51b6..99eae5fc45 100644
@@ -8,6 +8,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import gzip
 import lzma
 import os
 import shutil
@@ -23,6 +24,17 @@ def archive_extract(archive, dest_dir, member=None):
         else:
             tf.extractall(path=dest_dir)
 
+def gzip_uncompress(gz_path, output_path):
+    if os.path.exists(output_path):
+        return
+    with gzip.open(gz_path, 'rb') as gz_in:
+        try:
+            with open(output_path, 'wb') as raw_out:
+                shutil.copyfileobj(gz_in, raw_out)
+        except:
+            os.remove(output_path)
+            raise
+
 def lzma_uncompress(xz_path, output_path):
     if os.path.exists(output_path):
         return
tests/functional/test_rx_gdbsim.py renamed from tests/avocado/machine_rx_gdbsim.py +29 -25

83 lines not on this page · raw message

Signature · 1 line
2.46.0
#35 lore raw

PATCH v5 31/44 tests/functional: Convert the linux_initrd avocado test into a standalone test

In reply to Thomas Huth

While we're at it, update the SHA1 hashes to SHA256 hashes since SHA1 should not be used anymore nowadays.

---
 tests/functional/meson.build                  |  1 +
 .../test_linux_initrd.py}                     | 37 +++++++++++--------
 2 files changed, 22 insertions(+), 16 deletions(-)
 rename tests/{avocado/linux_initrd.py => functional/test_linux_initrd.py} (76%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index bb511482cb..34a9b277f8 100644
@@ -111,6 +111,7 @@ tests_x86_64_system_quick = [
 
 tests_x86_64_system_thorough = [
   'acpi_bits',
+  'linux_initrd',
   'netdev_ethtool',
   'virtio_gpu',
 ]
tests/functional/test_linux_initrd.py renamed from tests/avocado/linux_initrd.py +21 -16
old mode 100644
new mode 100755
similarity index 76%
rename from tests/avocado/linux_initrd.py
rename to tests/functional/test_linux_initrd.py
index 7f47b98ae7..c71a59d4c9
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Linux initrd integration test.
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -12,20 +14,27 @@
 import logging
 import tempfile
 
-from avocado_qemu import QemuSystemTest
-from avocado import skipUnless
+from qemu_test import QemuSystemTest, Asset
+from unittest import skipUnless
 
 
 class LinuxInitrd(QemuSystemTest):
     """
     Checks QEMU evaluates correctly the initrd file passed as -initrd option.
-
-    :avocado: tags=arch:x86_64
-    :avocado: tags=machine:pc
     """
 
     timeout = 300
 
+    ASSET_F18_KERNEL = Asset(
+        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
+         'releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz'),
+        '1a27cb42559ce29237ac186699d063556ad69c8349d732bb1bd8d614e5a8cc2e')
+
+    ASSET_F28_KERNEL = Asset(
+        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
+         'releases/28/Everything/x86_64/os/images/pxeboot/vmlinuz'),
+        'd05909c9d4a742a6fcc84dcc0361009e4611769619cc187a07107579a035f24e')
+
     def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
         """
         Pretends to boot QEMU with an initrd file with size of 2GiB
@@ -33,10 +42,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
         Fedora-18 shipped with linux-3.6 which have not supported xloadflags
         cannot support more than 2GiB initrd.
         """
-        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora/li'
-                      'nux/releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz')
-        kernel_hash = '41464f68efe42b9991250bed86c7081d2ccdbb21'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.set_machine('pc')
+        kernel_path = self.ASSET_F18_KERNEL.fetch()
         max_size = 2 * (1024 ** 3) - 1
 
         with tempfile.NamedTemporaryFile() as initrd:
@@ -56,16 +63,11 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
-        :avocado: tags=flaky
-
         QEMU has supported up to 4 GiB initrd for recent kernel
         Expect guest can reach 'Unpacking initramfs...'
         """
-        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
-                      '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
-                      'vmlinuz')
-        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.set_machine('pc')
+        kernel_path = self.ASSET_F28_KERNEL.fetch()
         max_size = 2 * (1024 ** 3) + 1
 
         with tempfile.NamedTemporaryFile() as initrd:
@@ -89,3 +91,6 @@ def test_with_2gib_file_should_work_with_linux_v4_16(self):
                     break
                 if 'Kernel panic - not syncing' in msg:
                     self.fail("Kernel panic reached")
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#36 lore raw

PATCH v5 32/44 tests/functional: Convert the vnc test

In reply to Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
 tests/functional/meson.build                     |  1 +
 tests/{avocado/vnc.py => functional/test_vnc.py} | 12 +++++++-----
 2 files changed, 8 insertions(+), 5 deletions(-)
 rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%)
 mode change 100644 => 100755
tests/functional/meson.build +1 -0
index 34a9b277f8..58f6bcf95d 100644
@@ -107,6 +107,7 @@ tests_x86_64_system_quick = [
   'pc_cpu_hotplug_props',
   'virtio_version',
   'x86_cpu_model_versions',
+  'vnc',
 ]
 
 tests_x86_64_system_thorough = [
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +7 -5
old mode 100644
new mode 100755
similarity index 97%
rename from tests/avocado/vnc.py
rename to tests/functional/test_vnc.py
index 862c8996a8..b769d3b268
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Simple functional tests for VNC functionality
 #
 # Copyright (c) 2018 Red Hat, Inc.
@@ -11,7 +13,7 @@
 import socket
 from typing import List
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
 VNC_ADDR = '127.0.0.1'
@@ -51,10 +53,7 @@ def find_free_ports(count: int) -> List[int]:
 
 
 class Vnc(QemuSystemTest):
-    """
-    :avocado: tags=vnc,quick
-    :avocado: tags=machine:none
-    """
+
     def test_no_vnc(self):
         self.vm.add_args('-nodefaults', '-S')
         self.vm.launch()
@@ -113,3 +112,6 @@ def test_change_listen(self):
         self.assertFalse(check_connect(a))
         self.assertTrue(check_connect(b))
         self.assertTrue(check_connect(c))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
Reviewed-by Tested-by on v5 32/44
#37 lore raw

Re: [PATCH v5 32/44] tests/functional: Convert the vnc test

Quoted 59 lines · Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                     |  1 +
  tests/{avocado/vnc.py => functional/test_vnc.py} | 12 +++++++-----
  2 files changed, 8 insertions(+), 5 deletions(-)

rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%) mode change 100644 => 100755

tests/functional/meson.build +1 -0
index 34a9b277f8..58f6bcf95d 100644
@@ -107,6 +107,7 @@ tests_x86_64_system_quick = [
    'pc_cpu_hotplug_props',
    'virtio_version',
    'x86_cpu_model_versions',
+  'vnc',
  ]
 
  tests_x86_64_system_thorough = [
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +4 -5
old mode 100644
new mode 100755
similarity index 97%
rename from tests/avocado/vnc.py
rename to tests/functional/test_vnc.py
index 862c8996a8..b769d3b268
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
  # Simple functional tests for VNC functionality
  #
  # Copyright (c) 2018 Red Hat, Inc.
@@ -11,7 +13,7 @@
  import socket
  from typing import List
 
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
 
 
  VNC_ADDR = '127.0.0.1'
@@ -51,10 +53,7 @@ def find_free_ports(count: int) -> List[int]:
 
 
  class Vnc(QemuSystemTest):
-    """
-    :avocado: tags=vnc,quick
-    :avocado: tags=machine:none
-    """
+
      def test_no_vnc(self):
          self.vm.add_args('-nodefaults', '-S')
          self.vm.launch()

tests/functional/test_vnc.py def test_change_listen(self): · line 117

Quoted 6 lines
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +3 -0
@@ -113,3 +112,6 @@ def test_change_listen(self):
          self.assertFalse(check_connect(a))
          self.assertTrue(check_connect(b))
          self.assertTrue(check_connect(c))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()

For a faithful conversion we need to squash:

-- >8 --

tests/functional/meson.build +1 -2
index 6b00d240dd..a590646adc 100644
@@ -26,6 +26,7 @@ tests_generic_system = [
   'empty_cpu_model',
   'info_usernet',
   'version',
+  'vnc',
 ]
 
 tests_generic_linuxuser = [
@@ -123,7 +124,6 @@ tests_x86_64_system_quick = [
   'pc_cpu_hotplug_props',
   'virtio_version',
   'x86_cpu_model_versions',
-  'vnc',
 ]
 
---

(each test is <1s so I'd keep it generic).

Regardless the variant kept:

#38 lore raw

Re: [PATCH v5 32/44] tests/functional: Convert the vnc test

Quoted 85 lines · Philippe Mathieu-Daudé
Quoted 53 lines · Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                     |  1 + tests/{avocado/vnc.py => functional/test_vnc.py} | 12 +++++++-----

  2 files changed, 8 insertions(+), 5 deletions(-)

  rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%) mode change 100644 => 100755

tests/functional/meson.build +1 -0
index 34a9b277f8..58f6bcf95d 100644
@@ -107,6 +107,7 @@ tests_x86_64_system_quick = [
     'pc_cpu_hotplug_props',
     'virtio_version',
     'x86_cpu_model_versions',
+  'vnc',

  ]
  tests_x86_64_system_thorough = [

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +2 -0
old mode 100644
new mode 100755
similarity index 97%
rename from tests/avocado/vnc.py
rename to tests/functional/test_vnc.py
index 862c8996a8..b769d3b268
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
   # Simple functional tests for VNC functionality
   #
   # Copyright (c) 2018 Red Hat, Inc.
@@ -11,7 +13,7 @@

  import socket
  from typing import List

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +1 -1
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest

  VNC_ADDR = '127.0.0.1'

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +0 -0
@@ -51,10 +53,7 @@ def find_free_ports(count: int) -> List[int]:

  class Vnc(QemuSystemTest):
-    """
-    :avocado: tags=vnc,quick
-    :avocado: tags=machine:none
-    """
+
      def test_no_vnc(self):
          self.vm.add_args('-nodefaults', '-S')
          self.vm.launch()

tests/functional/test_vnc.py def test_change_listen(self): · line 117

Quoted 6 lines
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +3 -0
@@ -113,3 +112,6 @@ def test_change_listen(self):
           self.assertFalse(check_connect(a))
           self.assertTrue(check_connect(b))
           self.assertTrue(check_connect(c))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()

For a faithful conversion we need to squash:

-- >8 --

tests/functional/meson.build +1 -2
index 6b00d240dd..a590646adc 100644
@@ -26,6 +26,7 @@ tests_generic_system = [
    'empty_cpu_model',
    'info_usernet',
    'version',
+  'vnc',
  ]
 
  tests_generic_linuxuser = [
@@ -123,7 +124,6 @@ tests_x86_64_system_quick = [
    'pc_cpu_hotplug_props',
    'virtio_version',
    'x86_cpu_model_versions',
-  'vnc',
  ]
 
---
Quoted 1 line

(each test is <1s so I'd keep it generic).

Unfortunately that doesn't really work - the test cannot be really be run in parallel since the find_free_ports() stuff here is racy.

Thomas

#39 lore raw

Re: [PATCH v5 32/44] tests/functional: Convert the vnc test

Quoted 84 lines · Thomas Huth
Quoted 83 lines · Philippe Mathieu-Daudé
Quoted 53 lines · Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                     |  1 + tests/{avocado/vnc.py => functional/test_vnc.py} | 12 +++++++-----

  2 files changed, 8 insertions(+), 5 deletions(-)

  rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%) mode change 100644 => 100755

tests/functional/meson.build +1 -0
index 34a9b277f8..58f6bcf95d 100644
@@ -107,6 +107,7 @@ tests_x86_64_system_quick = [
     'pc_cpu_hotplug_props',
     'virtio_version',
     'x86_cpu_model_versions',
+  'vnc',

  ]
  tests_x86_64_system_thorough = [

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +2 -0
old mode 100644
new mode 100755
similarity index 97%
rename from tests/avocado/vnc.py
rename to tests/functional/test_vnc.py
index 862c8996a8..b769d3b268
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
   # Simple functional tests for VNC functionality
   #
   # Copyright (c) 2018 Red Hat, Inc.
@@ -11,7 +13,7 @@

  import socket
  from typing import List

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +1 -1
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest

  VNC_ADDR = '127.0.0.1'

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +0 -0
@@ -51,10 +53,7 @@ def find_free_ports(count: int) -> List[int]:

  class Vnc(QemuSystemTest):
-    """
-    :avocado: tags=vnc,quick
-    :avocado: tags=machine:none
-    """
+
      def test_no_vnc(self):
          self.vm.add_args('-nodefaults', '-S')
          self.vm.launch()

tests/functional/test_vnc.py def test_change_listen(self): · line 117

Quoted 6 lines
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +3 -0
@@ -113,3 +112,6 @@ def test_change_listen(self):
           self.assertFalse(check_connect(a))
           self.assertTrue(check_connect(b))
           self.assertTrue(check_connect(c))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()

For a faithful conversion we need to squash:

-- >8 --

tests/functional/meson.build +1 -0
index 6b00d240dd..a590646adc 100644
@@ -26,6 +26,7 @@ tests_generic_system = [
     'empty_cpu_model',
     'info_usernet',
     'version',
+  'vnc',

  ]

  tests_generic_linuxuser = [

tests/functional/meson.build +0 -1
@@ -123,7 +124,6 @@ tests_x86_64_system_quick = [
     'pc_cpu_hotplug_props',
     'virtio_version',
     'x86_cpu_model_versions',
-  'vnc',

  ]

Quoted 5 lines
Quoted 3 lines

---

(each test is <1s so I'd keep it generic).

Unfortunately that doesn't really work - the test cannot be really be run in parallel since the find_free_ports() stuff here is racy.

Oh right, I ran check-functional-{arm,mips,...} with at most 3 targets but never tried the full 'check-functional'.

Quoted 4 lines

 Thomas

#40 lore raw

Re: [PATCH v5 32/44] tests/functional: Convert the vnc test

Quoted 85 lines · Philippe Mathieu-Daudé
Quoted 84 lines · Thomas Huth
Quoted 83 lines · Philippe Mathieu-Daudé
Quoted 53 lines · Thomas Huth

Nothing thrilling in here, it's just a straight forward conversion.

---
  tests/functional/meson.build                     |  1 + tests/{avocado/vnc.py => functional/test_vnc.py} | 12 +++++++-----

  2 files changed, 8 insertions(+), 5 deletions(-)

  rename tests/{avocado/vnc.py => functional/test_vnc.py} (97%) mode change 100644 => 100755

tests/functional/meson.build +1 -0
index 34a9b277f8..58f6bcf95d 100644
@@ -107,6 +107,7 @@ tests_x86_64_system_quick = [
     'pc_cpu_hotplug_props',
     'virtio_version',
     'x86_cpu_model_versions',
+  'vnc',

  ]
  tests_x86_64_system_thorough = [

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +2 -0
old mode 100644
new mode 100755
similarity index 97%
rename from tests/avocado/vnc.py
rename to tests/functional/test_vnc.py
index 862c8996a8..b769d3b268
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
   # Simple functional tests for VNC functionality
   #
   # Copyright (c) 2018 Red Hat, Inc.
@@ -11,7 +13,7 @@

  import socket
  from typing import List

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +1 -1
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest

  VNC_ADDR = '127.0.0.1'

tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +0 -0
@@ -51,10 +53,7 @@ def find_free_ports(count: int) -> List[int]:

  class Vnc(QemuSystemTest):
-    """
-    :avocado: tags=vnc,quick
-    :avocado: tags=machine:none
-    """
+
      def test_no_vnc(self):
          self.vm.add_args('-nodefaults', '-S')
          self.vm.launch()

tests/functional/test_vnc.py def test_change_listen(self): · line 117

Quoted 6 lines
tests/functional/test_vnc.py renamed from tests/avocado/vnc.py +3 -0
@@ -113,3 +112,6 @@ def test_change_listen(self):
           self.assertFalse(check_connect(a))
           self.assertTrue(check_connect(b))
           self.assertTrue(check_connect(c))
+
+if __name__ == '__main__':
+    QemuSystemTest.main()

For a faithful conversion we need to squash:

-- >8 --

tests/functional/meson.build +1 -0
index 6b00d240dd..a590646adc 100644
@@ -26,6 +26,7 @@ tests_generic_system = [
     'empty_cpu_model',
     'info_usernet',
     'version',
+  'vnc',

  ]

  tests_generic_linuxuser = [

tests/functional/meson.build +0 -1
@@ -123,7 +124,6 @@ tests_x86_64_system_quick = [
     'pc_cpu_hotplug_props',
     'virtio_version',
     'x86_cpu_model_versions',
-  'vnc',

  ]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

---

(each test is <1s so I'd keep it generic).

Unfortunately that doesn't really work - the test cannot be really be run in parallel since the find_free_ports() stuff here is racy.

Oh right, I ran check-functional-{arm,mips,...} with at most 3 targets but never tried the full 'check-functional'.

Also I just noticed this test doesn't explicit machine, so when a target don't have a default we get:

Exit code: 1
Command: qemu-system-arm -display none -vga none -chardev
socket,id=mon,fd=14 -mon chardev=mon,mode=control -nodefaults -S
Output: qemu-system-arm: No machine specified, and there is no default

#41 lore raw

PATCH v5 33/44 tests/functional: Convert ARM Integrator/CP avocado tests

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Update the SHA1 hashes to SHA256 hashes since SHA1 should not be used anymore nowadays.

  $ QEMU_TEST_ALLOW_UNTRUSTED_CODE=1 make check-functional-arm
  ...
  6/6 qemu:func-thorough+func-arm-thorough+thorough / func-arm-arm_integratorcp   OK   3.90s   2 subtests passed

Message-ID: <[email protected]>

---
 MAINTAINERS                                   |  2 +-
 tests/functional/meson.build                  |  1 +
 .../test_arm_integratorcp.py}                 | 56 ++++++++++---------
 3 files changed, 33 insertions(+), 26 deletions(-)
 rename tests/{avocado/machine_arm_integratorcp.py => functional/test_arm_integratorcp.py} (63%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index 25e71ac14c..14500cf8a5 100644
@@ -785,7 +785,7 @@ S: Maintained
 F: hw/arm/integratorcp.c
 F: hw/misc/arm_integrator_debug.c
 F: include/hw/misc/arm_integrator_debug.h
-F: tests/avocado/machine_arm_integratorcp.py
+F: tests/functional/test_arm_integratorcp.py
 F: docs/system/arm/integratorcp.rst
 
 MCIMX6UL EVK / i.MX6ul
tests/functional/meson.build +1 -0
index 58f6bcf95d..182d40e052 100644
@@ -34,6 +34,7 @@ tests_generic_bsduser = [
 
 tests_arm_system_thorough = [
   'arm_canona1100',
+  'arm_integratorcp',
   'arm_n8x0',
 ]
 
tests/functional/test_arm_integratorcp.py renamed from tests/avocado/machine_arm_integratorcp.py +31 -25

100 lines not on this page · raw message

Signature · 1 line
2.46.0
#42 lore raw

PATCH v5 34/44 tests/functional: Convert Aarch64 SBSA-Ref avocado tests

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Since SBSA_FLASH files are not tarballs, use lzma_uncompress() method.

Avocado used to set a timeout of 11 tests * 180s = 1980s. Hopefully 600s should be sufficient.

Running on macOS Sonoma / Apple silicon M1:

  $ QEMU_TEST_TIMEOUT_EXPECTED=1 make check-functional-aarch64 V=1
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max              OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_impdef OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_off    OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1      OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_edk2_firmware                 OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57          OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max                 OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_impdef    OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_off       OK
  ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1         OK
  1/5 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_sbsaref        OK   241.79s  11 subtests passed
  ...

Message-ID: <[email protected]>

---
 MAINTAINERS                                   |   2 +-
 tests/functional/meson.build                  |   5 +
 .../test_aarch64_sbsaref.py}                  | 144 ++++++------------
 3 files changed, 55 insertions(+), 96 deletions(-)
 rename tests/{avocado/machine_aarch64_sbsaref.py => functional/test_aarch64_sbsaref.py} (56%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index 14500cf8a5..e9aa174dc7 100644
@@ -971,7 +971,7 @@ F: hw/misc/sbsa_ec.c
 F: hw/watchdog/sbsa_gwdt.c
 F: include/hw/watchdog/sbsa_gwdt.h
 F: docs/system/arm/sbsa.rst
-F: tests/avocado/machine_aarch64_sbsaref.py
+F: tests/functional/test_aarch64_sbsaref.py
 
 Sharp SL-5500 (Collie) PDA
 M: Peter Maydell <[email protected]>
tests/functional/meson.build +5 -0
index 182d40e052..bfa04818cc 100644
@@ -11,6 +11,7 @@ endif
 
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
+  'aarch64_sbsaref' : 600,
   'acpi_bits' : 240,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
@@ -32,6 +33,10 @@ tests_generic_linuxuser = [
 tests_generic_bsduser = [
 ]
 
+tests_aarch64_system_thorough = [
+  'aarch64_sbsaref',
+]
+
 tests_arm_system_thorough = [
   'arm_canona1100',
   'arm_integratorcp',
tests/functional/test_aarch64_sbsaref.py renamed from tests/avocado/machine_aarch64_sbsaref.py +49 -95

242 lines not on this page · raw message

Signature · 1 line
2.46.0
#43 lore raw

Re: [PATCH v5 34/44] tests/functional: Convert Aarch64 SBSA-Ref avocado tests

Quoted 96 lines · Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Since SBSA_FLASH files are not tarballs, use lzma_uncompress() method.

Avocado used to set a timeout of 11 tests * 180s = 1980s. Hopefully 600s should be sufficient.

Running on macOS Sonoma / Apple silicon M1:

   $ QEMU_TEST_TIMEOUT_EXPECTED=1 make check-functional-aarch64 V=1
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max              OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_impdef OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_off    OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1      OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_edk2_firmware                 OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57          OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max                 OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_impdef    OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_off       OK
   ▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1         OK
   1/5 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_sbsaref        OK   241.79s  11 subtests passed
   ...

Message-ID: <[email protected]>

---
  MAINTAINERS                                   |   2 +-
  tests/functional/meson.build                  |   5 +
  .../test_aarch64_sbsaref.py}                  | 144 ++++++------------
  3 files changed, 55 insertions(+), 96 deletions(-)

rename tests/{avocado/machine_aarch64_sbsaref.py => functional/test_aarch64_sbsaref.py} (56%) mode change 100644 => 100755

MAINTAINERS +1 -1
index 14500cf8a5..e9aa174dc7 100644
@@ -971,7 +971,7 @@ F: hw/misc/sbsa_ec.c
  F: hw/watchdog/sbsa_gwdt.c
  F: include/hw/watchdog/sbsa_gwdt.h
  F: docs/system/arm/sbsa.rst
-F: tests/avocado/machine_aarch64_sbsaref.py
+F: tests/functional/test_aarch64_sbsaref.py
 
  Sharp SL-5500 (Collie) PDA
  M: Peter Maydell <[email protected]>
tests/functional/meson.build +5 -0
index 182d40e052..bfa04818cc 100644
@@ -11,6 +11,7 @@ endif
 
  # Timeouts for individual tests that can be slow e.g. with debugging enabled
  test_timeouts = {
+  'aarch64_sbsaref' : 600,
    'acpi_bits' : 240,
    'netdev_ethtool' : 180,
    'ppc_40p' : 240,
@@ -32,6 +33,10 @@ tests_generic_linuxuser = [
  tests_generic_bsduser = [
  ]
 
+tests_aarch64_system_thorough = [
+  'aarch64_sbsaref',
+]
+
  tests_arm_system_thorough = [
    'arm_canona1100',
    'arm_integratorcp',
tests/functional/test_aarch64_sbsaref.py renamed from tests/avocado/machine_aarch64_sbsaref.py +2 -4
old mode 100644
new mode 100755
similarity index 56%
rename from tests/avocado/machine_aarch64_sbsaref.py
rename to tests/functional/test_aarch64_sbsaref.py
index f8bf40c192..e90566cdb8
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
  # Functional test that boots a Linux kernel and checks the console
  #
  # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
@@ -8,12 +10,11 @@
 
  import os
 
-from avocado import skipUnless
-from avocado.utils import archive
-
-from avocado_qemu import QemuSystemTest

tests/functional/test_aarch64_sbsaref.py line 20

Quoted 10 lines
tests/functional/test_aarch64_sbsaref.py renamed from tests/avocado/machine_aarch64_sbsaref.py +5 -2
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import interrupt_interactive_console_until_pattern
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern
+from qemu_test import interrupt_interactive_console_until_pattern
+from qemu_test.utils import lzma_uncompress
+from unittest import skipUnless
 
 
  class Aarch64SbsarefMachine(QemuSystemTest):

Oops, I forgot to squash:

-- >8 --

tests/functional/test_aarch64_sbsaref.py +0 -4
index e90566cdb8..f31c2a60b6 100755
@@ -20,8 +20,4 @@
 class Aarch64SbsarefMachine(QemuSystemTest):
     """
-    :avocado: tags=arch:aarch64
-    :avocado: tags=machine:sbsa-ref
-    :avocado: tags=accel:tcg
-
     As firmware runs at a higher privilege level than the hypervisor we
     can only run these tests under TCG emulation.

---

Quoted 3 lines
tests/functional/test_aarch64_sbsaref.py renamed from tests/avocado/machine_aarch64_sbsaref.py +0 -0
@@ -28,6 +29,16 @@ class Aarch64SbsarefMachine(QemuSystemTest):
 
      timeout = 180
#44 lore raw

PATCH v5 35/44 tests/functional: Convert Aarch64 Virt machine avocado tests

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Update the SHA1 hash to SHA256 since SHA1 should not be used anymore nowadays.

Provide get_qemu_img() helper in qemu_test.

  $ make check-functional-aarch64 V=1
  ▶ 1/4 test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv2                 OK
  ▶ 1/4 test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv3                 OK
  ▶ 1/4 test_aarch64_virt.Aarch64VirtMachine.test_alpine_virt_tcg_gic_max            OK
  1/4 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_virt  OK   30.44s  3 subtests passed
  ...

Message-ID: <[email protected]>

---
 MAINTAINERS                                   |  2 +-
 tests/functional/meson.build                  |  2 +
 tests/functional/qemu_test/__init__.py        |  2 +-
 tests/functional/qemu_test/cmd.py             | 15 ++++
 .../test_aarch64_virt.py}                     | 69 ++++++++-----------
 5 files changed, 46 insertions(+), 44 deletions(-)
 rename tests/{avocado/machine_aarch64_virt.py => functional/test_aarch64_virt.py} (69%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index e9aa174dc7..cbdf2863fa 100644
@@ -1023,7 +1023,7 @@ S: Maintained
 F: hw/arm/virt*
 F: include/hw/arm/virt.h
 F: docs/system/arm/virt.rst
-F: tests/avocado/machine_aarch64_virt.py
+F: tests/functional/test_aarch64_virt.py
 
 Xilinx Zynq
 M: Edgar E. Iglesias <[email protected]>
tests/functional/meson.build +2 -0
index bfa04818cc..4154b1cbcd 100644
@@ -12,6 +12,7 @@ endif
 # Timeouts for individual tests that can be slow e.g. with debugging enabled
 test_timeouts = {
   'aarch64_sbsaref' : 600,
+  'aarch64_virt' : 360,
   'acpi_bits' : 240,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
@@ -35,6 +36,7 @@ tests_generic_bsduser = [
 
 tests_aarch64_system_thorough = [
   'aarch64_sbsaref',
+  'aarch64_virt',
 ]
 
 tests_arm_system_thorough = [
tests/functional/qemu_test/__init__.py +1 -1
index db05c8f412..1d87d7122f 100644
@@ -10,5 +10,5 @@
 from .config import BUILD_DIR
 from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
     interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
-    exec_command, exec_command_and_wait_for_pattern
+    exec_command, exec_command_and_wait_for_pattern, get_qemu_img
 from .testcase import QemuSystemTest, QemuBaseTest
tests/functional/qemu_test/cmd.py +15 -0
index 8580ee8faf..3acd617324 100644
@@ -16,6 +16,8 @@
 import os.path
 import subprocess
 
+from .config import BUILD_DIR
+
 
 def has_cmd(name, args=None):
     """
@@ -176,3 +178,16 @@ def exec_command_and_wait_for_pattern(test, command,
     :param failure_message: if this message appears, test fails
     """
     _console_interaction(test, success_message, failure_message, command + '\r')
+
+def get_qemu_img(test):
+    test.log.debug('Looking for and selecting a qemu-img binary')
+
+    # If qemu-img has been built, use it, otherwise the system wide one
+    # will be used.
+    qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
+    if os.path.exists(qemu_img):
+        return qemu_img
+    if has_cmd('qemu-img'):
+        return 'qemu-img'
+    test.skipTest('Could not find "qemu-img", which is required to '
+                  'create temporary images')
tests/functional/test_aarch64_virt.py renamed from tests/avocado/machine_aarch64_virt.py +27 -42

115 lines not on this page · raw message

Signature · 1 line
2.46.0
#45 lore raw

PATCH v5 36/44 tests/functional: Convert mips64el Fuloong2e avocado test (1/2)

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Update the SHA1 hashes to SHA256 hashes since SHA1 should not be used anymore nowadays.

Since the asset is expected locally and the test is guarded with RESCUE_YL_PATH, keep it under the 'quick' category.

  $ RESCUE_YL_PATH=/path/to/rescue-yl QEMU_TEST_ALLOW_UNTRUSTED_CODE=1 \
    make check-functional-mips64el
  1/4 qemu:func-quick+func-mips64el / func-mips64el-empty_cpu_model      OK   0.12s   1 subtests passed
  2/4 qemu:func-quick+func-mips64el / func-mips64el-version              OK   0.13s   1 subtests passed
  3/4 qemu:func-quick+func-mips64el / func-mips64el-info_usernet         OK   0.15s   1 subtests passed
  4/4 qemu:func-quick+func-mips64el / func-mips64el-mips64el_fuloong2e   OK   0.19s   1 subtests passed

Message-ID: <[email protected]>

---
 MAINTAINERS                                   |  2 +-
 tests/functional/meson.build                  |  4 +++
 .../test_mips64el_fuloong2e.py}               | 33 ++++++++++---------
 3 files changed, 23 insertions(+), 16 deletions(-)
 rename tests/{avocado/machine_mips_fuloong2e.py => functional/test_mips64el_fuloong2e.py} (58%)
 mode change 100644 => 100755
MAINTAINERS +1 -1
index cbdf2863fa..f6e6e5ebf5 100644
@@ -1384,7 +1384,7 @@ S: Odd Fixes
 F: hw/mips/fuloong2e.c
 F: hw/pci-host/bonito.c
 F: include/hw/pci-host/bonito.h
-F: tests/avocado/machine_mips_fuloong2e.py
+F: tests/functional/test_mips64el_fuloong2e.py
 
 Loongson-3 virtual platforms
 M: Huacai Chen <[email protected]>
tests/functional/meson.build +4 -0
index 4154b1cbcd..dcede9733e 100644
@@ -65,6 +65,10 @@ tests_microblazeel_system_thorough = [
   'microblazeel_s3adsp1800'
 ]
 
+tests_mips64el_system_quick = [
+  'mips64el_fuloong2e',
+]
+
 tests_mips64el_system_thorough = [
   'mips64el_loongson3v',
 ]
tests/functional/test_mips64el_fuloong2e.py renamed from tests/avocado/machine_mips_fuloong2e.py +18 -15
old mode 100644
new mode 100755
similarity index 58%
rename from tests/avocado/machine_mips_fuloong2e.py
rename to tests/functional/test_mips64el_fuloong2e.py
index 89291f47b2..7688a32713
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
 # Functional tests for the Lemote Fuloong-2E machine.
 #
 # Copyright (c) 2019 Philippe Mathieu-Daudé <[email protected]>
@@ -8,35 +10,36 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import subprocess
 
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from unittest import skipUnless
 
 class MipsFuloong2e(QemuSystemTest):
 
     timeout = 60
 
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     @skipUnless(os.getenv('RESCUE_YL_PATH'), 'RESCUE_YL_PATH not available')
-    def test_linux_kernel_isa_serial(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=machine:fuloong2e
-        :avocado: tags=endian:little
-        :avocado: tags=device:bonito64
-        :avocado: tags=device:via686b
-        """
+    def test_linux_kernel_2_6_27_isa_serial(self):
         # Recovery system for the Yeeloong laptop
         # (enough to test the fuloong2e southbridge, accessing its ISA bus)
         # http://dev.lemote.com/files/resource/download/rescue/rescue-yl
-        kernel_hash = 'ec4d1bd89a8439c41033ca63db60160cc6d6f09a'
-        kernel_path = self.fetch_asset('file://' + os.getenv('RESCUE_YL_PATH'),
-                                       asset_hash=kernel_hash)
+        sha = 'ab588d3316777c62cc81baa20ac92e98b01955c244dff3794b711bc34e26e51d'
+        kernel_path = os.getenv('RESCUE_YL_PATH')
+        output = subprocess.check_output(['sha256sum', kernel_path])
+        checksum = output.split()[0]
+        assert checksum.decode("utf-8") == sha
 
+        self.set_machine('fuloong2e')
         self.vm.set_console()
         self.vm.add_args('-kernel', kernel_path)
         self.vm.launch()
         wait_for_console_pattern(self, 'Linux version 2.6.27.7lemote')
         cpu_revision = 'CPU revision is: 00006302 (ICT Loongson-2)'
         wait_for_console_pattern(self, cpu_revision)
+
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
Signature · 1 line
2.46.0
#46 lore raw

PATCH v5 37/44 tests/functional: Add QemuUserTest class

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Per commit 5334df4822 ("tests/avocado: Introduce
QemuUserTest base class"):

Similarly to the 'System' Test base class with methods for testing system emulation, the QemuUserTest class contains methods useful to test user-mode emulation.

Message-ID: <[email protected]>

---
 tests/functional/qemu_test/__init__.py |  2 +-
 tests/functional/qemu_test/testcase.py | 17 +++++++++++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
tests/functional/qemu_test/__init__.py +1 -1
index 1d87d7122f..f33282efe8 100644
@@ -11,4 +11,4 @@
 from .cmd import has_cmd, has_cmds, run_cmd, is_readable_executable_file, \
     interrupt_interactive_console_until_pattern, wait_for_console_pattern, \
     exec_command, exec_command_and_wait_for_pattern, get_qemu_img
-from .testcase import QemuSystemTest, QemuBaseTest
+from .testcase import QemuBaseTest, QemuUserTest, QemuSystemTest
tests/functional/qemu_test/testcase.py +17 -0
index 18314be9d1..aa0146265a 100644
@@ -13,6 +13,7 @@
 
 import logging
 import os
+import subprocess
 import pycotap
 import sys
 import unittest
@@ -70,6 +71,22 @@ def main():
         unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
 
 
+class QemuUserTest(QemuBaseTest):
+
+    def setUp(self):
+        super().setUp('qemu-')
+        self._ldpath = []
+
+    def add_ldpath(self, ldpath):
+        self._ldpath.append(os.path.abspath(ldpath))
+
+    def run_cmd(self, bin_path, args=[]):
+        return subprocess.run([self.qemu_bin]
+                              + ["-L %s" % ldpath for ldpath in self._ldpath]
+                              + [bin_path]
+                              + args,
+                              text=True, capture_output=True)
+
 class QemuSystemTest(QemuBaseTest):
     """Facilitates system emulation tests."""
 
Signature · 1 line
2.46.0
#47 lore raw

PATCH v5 38/44 tests/functional: Convert ARM bFLT linux-user avocado test

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

Straight forward conversion. Update the SHA1 hashes to SHA256 hashes since SHA1 should not be used anymore nowadays. Expose cpio_extract() in qemu_test.utils for possible reuse.

Message-ID: <[email protected]>

[thuth: Add test to meson.build]
---
 tests/avocado/load_bflt.py          | 54 -----------------------------
 tests/functional/meson.build        |  4 +++
 tests/functional/qemu_test/utils.py |  9 +++++
 tests/functional/test_arm_bflt.py   | 44 +++++++++++++++++++++++
 4 files changed, 57 insertions(+), 54 deletions(-)
 delete mode 100644 tests/avocado/load_bflt.py
 create mode 100755 tests/functional/test_arm_bflt.py
tests/avocado/load_bflt.py deleted +0 -54
deleted file mode 100644
index bb50cec1ee..0000000000
@@ -1,54 +0,0 @@
-# Test the bFLT loader format
-#
-# Copyright (C) 2019 Philippe Mathieu-Daudé <[email protected]>
-#
-# SPDX-License-Identifier: GPL-2.0-or-later
-
-import os
-import bz2
-import subprocess
-
-from avocado import skipUnless
-from avocado_qemu import QemuUserTest
-from avocado_qemu import has_cmd
-
-
-class LoadBFLT(QemuUserTest):
-
-    def extract_cpio(self, cpio_path):
-        """
-        Extracts a cpio archive into the test workdir
-
-        :param cpio_path: path to the cpio archive
-        """
-        cwd = os.getcwd()
-        os.chdir(self.workdir)
-        with bz2.open(cpio_path, 'rb') as archive_cpio:
-            subprocess.run(['cpio', '-i'], input=archive_cpio.read(),
-                           stderr=subprocess.DEVNULL)
-        os.chdir(cwd)
-
-    @skipUnless(*has_cmd('cpio'))
-    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
-    def test_stm32(self):
-        """
-        :avocado: tags=arch:arm
-        :avocado: tags=linux_user
-        :avocado: tags=quick
-        """
-        # See https://elinux.org/STM32#User_Space
-        rootfs_url = ('https://elinux.org/images/5/51/'
-                      'Stm32_mini_rootfs.cpio.bz2')
-        rootfs_hash = '9f065e6ba40cce7411ba757f924f30fcc57951e6'
-        rootfs_path_bz2 = self.fetch_asset(rootfs_url, asset_hash=rootfs_hash)
-        busybox_path = os.path.join(self.workdir, "/bin/busybox")
-
-        self.extract_cpio(rootfs_path_bz2)
-
-        res = self.run(busybox_path)
-        ver = 'BusyBox v1.24.0.git (2015-02-03 22:17:13 CET) multi-call binary.'
-        self.assertIn(ver, res.stdout_text)
-
-        res = self.run(busybox_path, ['uname', '-a'])
-        unm = 'armv7l GNU/Linux'
-        self.assertIn(unm, res.stdout_text)
tests/functional/meson.build +4 -0
index dcede9733e..6b00d240dd 100644
@@ -45,6 +45,10 @@ tests_arm_system_thorough = [
   'arm_n8x0',
 ]
 
+tests_arm_linuxuser_thorough = [
+  'arm_bflt',
+]
+
 tests_avr_system_thorough = [
   'avr_mega2560',
 ]
tests/functional/qemu_test/utils.py +9 -0
index 99eae5fc45..2a1cb60d38 100644
@@ -12,6 +12,7 @@
 import lzma
 import os
 import shutil
+import subprocess
 import tarfile
 
 def archive_extract(archive, dest_dir, member=None):
@@ -45,3 +46,11 @@ def lzma_uncompress(xz_path, output_path):
         except:
             os.remove(output_path)
             raise
+
+def cpio_extract(cpio_handle, output_path):
+    cwd = os.getcwd()
+    os.chdir(output_path)
+    subprocess.run(['cpio', '-i'],
+                   input=cpio_handle.read(),
+                   stderr=subprocess.DEVNULL)
+    os.chdir(cwd)
tests/functional/test_arm_bflt.py new file +44 -0

44 lines not on this page · raw message

Signature · 1 line
2.46.0
#48 lore raw

PATCH v5 39/44 tests/avocado: Remove unused QemuUserTest class

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

The single test that was using the QemuUserTest class has been converted to the functional test framework. This class is now unused, remove it.

Message-ID: <[email protected]>

---
 tests/avocado/avocado_qemu/__init__.py | 17 -----------------
 1 file changed, 17 deletions(-)
tests/avocado/avocado_qemu/__init__.py +0 -17
index ef935614cf..0e4ecea7a0 100644
@@ -384,23 +384,6 @@ def tearDown(self):
         super().tearDown()
 
 
-class QemuUserTest(QemuBaseTest):
-    """Facilitates user-mode emulation tests."""
-
-    def setUp(self):
-        self._ldpath = []
-        super().setUp('qemu-')
-
-    def add_ldpath(self, ldpath):
-        self._ldpath.append(os.path.abspath(ldpath))
-
-    def run(self, bin_path, args=[]):
-        qemu_args = " ".join(["-L %s" % ldpath for ldpath in self._ldpath])
-        bin_args = " ".join(args)
-        return process.run("%s %s %s %s" % (self.qemu_bin, qemu_args,
-                                            bin_path, bin_args))
-
-
 class LinuxSSHMixIn:
     """Contains utility methods for interacting with a guest via SSH."""
 
Signature · 1 line
2.46.0
#49 lore raw

PATCH v5 40/44 gitlab-ci: Add "check-functional" to the build tests

In reply to Thomas Huth

Now that we converted many tests from the "check-avocado" test suite to the "check-functional" test suite, we should make sure that these also get tested in the CI.

---
 .gitlab-ci.d/buildtest-template.yml | 14 +++++--
 .gitlab-ci.d/buildtest.yml          | 60 ++++++++++++++---------------
 2 files changed, 41 insertions(+), 33 deletions(-)
.gitlab-ci.d/buildtest-template.yml +11 -3
index 844c26623d..5f2fc7e6f4 100644
@@ -59,6 +59,10 @@
     - cd build
     - find . -type f -exec touch {} +
     # Avoid recompiling by hiding ninja with NINJA=":"
+    # We also have to pre-cache the functional tests manually in this case
+    - if [ "x${QEMU_TEST_CACHE_DIR}" != "x" ]; then
+        $MAKE precache-functional ;
+      fi
     - $MAKE NINJA=":" $MAKE_CHECK_ARGS
 
 .native_test_job_template:
@@ -72,12 +76,13 @@
     reports:
       junit: build/meson-logs/testlog.junit.xml
 
-.avocado_test_job_template:
+.functional_test_job_template:
   extends: .common_test_job_template
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
       - ${CI_PROJECT_DIR}/avocado-cache
+      - ${CI_PROJECT_DIR}/functional-cache
     policy: pull-push
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
@@ -86,6 +91,7 @@
     paths:
       - build/tests/results/latest/results.xml
       - build/tests/results/latest/test-results
+      - build/tests/functional/*/*/*.log
     reports:
       junit: build/tests/results/latest/results.xml
   before_script:
@@ -96,11 +102,13 @@
     - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
            >> ~/.config/avocado/avocado.conf
     - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
-        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
+        du -chs ${CI_PROJECT_DIR}/*-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+    - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
+    - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache
   after_script:
     - cd build
-    - du -chs ${CI_PROJECT_DIR}/avocado-cache
+    - du -chs ${CI_PROJECT_DIR}/*-cache
   variables:
     QEMU_JOB_AVOCADO: 1
.gitlab-ci.d/buildtest.yml +30 -30

171 lines not on this page · raw message

Signature · 1 line
2.46.0
#50 lore raw

PATCH v5 41/44 docs/devel: Split testing docs from the build docs and move to separate folder

In reply to Thomas Huth

Building and testing are two separate topics, so let's split the testing into a separate category and move the related files into a separate folder.

---
 docs/devel/index-build.rst                      | 14 ++++----------
 docs/devel/index.rst                            |  1 +
 docs/devel/{ => testing}/acpi-bits.rst          |  0
 docs/devel/{ => testing}/ci-definitions.rst.inc |  0
 docs/devel/{ => testing}/ci-jobs.rst.inc        |  0
 docs/devel/{ => testing}/ci-runners.rst.inc     |  0
 docs/devel/{ => testing}/ci.rst                 |  0
 docs/devel/{ => testing}/fuzzing.rst            |  0
 docs/devel/testing/index.rst                    | 14 ++++++++++++++
 docs/devel/{testing.rst => testing/main.rst}    |  0
 docs/devel/{ => testing}/qgraph.rst             |  0
 docs/devel/{ => testing}/qtest.rst              |  0
 12 files changed, 19 insertions(+), 10 deletions(-)
 rename docs/devel/{ => testing}/acpi-bits.rst (100%)
 rename docs/devel/{ => testing}/ci-definitions.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-jobs.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-runners.rst.inc (100%)
 rename docs/devel/{ => testing}/ci.rst (100%)
 rename docs/devel/{ => testing}/fuzzing.rst (100%)
 create mode 100644 docs/devel/testing/index.rst
 rename docs/devel/{testing.rst => testing/main.rst} (100%)
 rename docs/devel/{ => testing}/qgraph.rst (100%)
 rename docs/devel/{ => testing}/qtest.rst (100%)
docs/devel/index-build.rst +4 -10
index 90b406ca0e..0023953be3 100644
@@ -1,9 +1,8 @@
-QEMU Build and Test System
---------------------------
+QEMU Build System
+-----------------
 
-Details about how QEMU's build system works and how it is integrated
-into our testing infrastructure. You will need to understand some of
-the basics if you are adding new files and targets to the build.
+Details about how QEMU's build system works. You will need to understand
+some of the basics if you are adding new files and targets to the build.
 
 .. toctree::
    :maxdepth: 3
@@ -11,10 +10,5 @@ the basics if you are adding new files and targets to the build.
    build-system
    kconfig
    docs
-   testing
-   acpi-bits
-   qtest
-   ci
    qapi-code-gen
-   fuzzing
    control-flow-integrity
docs/devel/index.rst +1 -0
index abf60457c2..a53f1bfda5 100644
@@ -31,6 +31,7 @@ the :ref:`tcg_internals`.
 
    index-process
    index-build
+   testing/index
    index-api
    index-internals
    index-tcg
docs/devel/testing/acpi-bits.rst renamed from docs/devel/acpi-bits.rst +0 -0
similarity index 100%
rename from docs/devel/acpi-bits.rst
rename to docs/devel/testing/acpi-bits.rst
docs/devel/testing/ci-definitions.rst.inc renamed from docs/devel/ci-definitions.rst.inc +0 -0
similarity index 100%
rename from docs/devel/ci-definitions.rst.inc
rename to docs/devel/testing/ci-definitions.rst.inc
docs/devel/testing/ci-jobs.rst.inc renamed from docs/devel/ci-jobs.rst.inc +0 -0
similarity index 100%
rename from docs/devel/ci-jobs.rst.inc
rename to docs/devel/testing/ci-jobs.rst.inc
docs/devel/testing/ci-runners.rst.inc renamed from docs/devel/ci-runners.rst.inc +0 -0
similarity index 100%
rename from docs/devel/ci-runners.rst.inc
rename to docs/devel/testing/ci-runners.rst.inc
docs/devel/testing/ci.rst renamed from docs/devel/ci.rst +0 -0
similarity index 100%
rename from docs/devel/ci.rst
rename to docs/devel/testing/ci.rst
docs/devel/testing/fuzzing.rst renamed from docs/devel/fuzzing.rst +0 -0
similarity index 100%
rename from docs/devel/fuzzing.rst
rename to docs/devel/testing/fuzzing.rst
docs/devel/testing/index.rst new file +14 -0
new file mode 100644
index 0000000000..2711fd78b7
@@ -0,0 +1,14 @@
+Testing QEMU
+------------
+
+Details about how to test QEMU and how it is integrated into our CI
+testing infrastructure.
+
+.. toctree::
+   :maxdepth: 3
+
+   main
+   qtest
+   acpi-bits
+   ci
+   fuzzing
docs/devel/testing/main.rst renamed from docs/devel/testing.rst +0 -0
similarity index 100%
rename from docs/devel/testing.rst
rename to docs/devel/testing/main.rst
docs/devel/testing/qgraph.rst renamed from docs/devel/qgraph.rst +0 -0
similarity index 100%
rename from docs/devel/qgraph.rst
rename to docs/devel/testing/qgraph.rst
docs/devel/testing/qtest.rst renamed from docs/devel/qtest.rst +0 -0
similarity index 100%
rename from docs/devel/qtest.rst
rename to docs/devel/testing/qtest.rst
Signature · 1 line
2.46.0
#51 lore raw

PATCH v5 42/44 docs/devel/testing: Split the Avocado documentation into a separate file

In reply to Thomas Huth

The main testing documentation file got very overloaded already. Thus let's split the Avocado information into a separate file.

---
 docs/devel/testing/avocado.rst | 581 +++++++++++++++++++++++++++++++++
 docs/devel/testing/index.rst   |   1 +
 docs/devel/testing/main.rst    | 565 +-------------------------------
 3 files changed, 583 insertions(+), 564 deletions(-)
 create mode 100644 docs/devel/testing/avocado.rst
docs/devel/testing/avocado.rst new file +581 -0
new file mode 100644
index 0000000000..0d207a05c5
@@ -0,0 +1,581 @@
+.. _checkavocado-ref:
+
+
+Integration testing with Avocado
+================================
+
+The ``tests/avocado`` directory hosts integration tests. They're usually
+higher level tests, and may interact with external resources and with
+various guest operating systems.
+
+These tests are written using the Avocado Testing Framework (which must
+be installed separately) in conjunction with a the ``avocado_qemu.Test``
+class, implemented at ``tests/avocado/avocado_qemu``.
+
+Tests based on ``avocado_qemu.Test`` can easily:
+
+ * Customize the command line arguments given to the convenience
+   ``self.vm`` attribute (a QEMUMachine instance)
+
+ * Interact with the QEMU monitor, send QMP commands and check
+   their results
+
+ * Interact with the guest OS, using the convenience console device
+   (which may be useful to assert the effectiveness and correctness of
+   command line arguments or QMP commands)
+
+ * Interact with external data files that accompany the test itself
+   (see ``self.get_data()``)
+
+ * Download (and cache) remote data files, such as firmware and kernel
+   images
+
+ * Have access to a library of guest OS images (by means of the
+   ``avocado.utils.vmimage`` library)
+
+ * Make use of various other test related utilities available at the
+   test class itself and at the utility library:
+
+   - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
+   - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
+
+Running tests
+-------------
+
+You can run the avocado tests simply by executing:
+
+.. code::
+
+  make check-avocado
+
+This involves the automatic installation, from PyPI, of all the
+necessary avocado-framework dependencies into the QEMU venv within the
+build tree (at ``./pyvenv``). Test results are also saved within the
+build tree (at ``tests/results``).
+
+Note: the build environment must be using a Python 3 stack, and have
+the ``venv`` and ``pip`` packages installed.  If necessary, make sure
+``configure`` is called with ``--python=`` and that those modules are
+available.  On Debian and Ubuntu based systems, depending on the
+specific version, they may be on packages named ``python3-venv`` and
+``python3-pip``.
+
+It is also possible to run tests based on tags using the
+``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
+variable:
+
+.. code::
+
+   make check-avocado AVOCADO_TAGS=quick
+
+Note that tags separated with commas have an AND behavior, while tags
+separated by spaces have an OR behavior. For more information on Avocado
+tags, see:
+
+ https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
+
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+  make check-avocado AVOCADO_TESTS=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+  make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+  make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+  make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
+The scripts installed inside the virtual environment may be used
+without an "activation".  For instance, the Avocado test runner
+may be invoked by running:
+
+ .. code::
+
+  pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
+
+Note that if ``make check-avocado`` was not executed before, it is
+possible to create the Python virtual environment with the dependencies
+needed running:
+
+ .. code::
+
+  make check-venv
+
+It is also possible to run tests from a single file or a single test within
+a test file. To run tests from a single file within the build tree, use:
+
+ .. code::
+
+  pyvenv/bin/avocado run tests/avocado/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+  pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
+
+Valid test names are visible in the output from any previous execution
+of Avocado or ``make check-avocado``, and can also be queried using:
+
+ .. code::
+
+  pyvenv/bin/avocado list tests/avocado
+
+Manual Installation
+-------------------
+
+To manually install Avocado and its dependencies, run:
+
+.. code::
+
+  pip install --user avocado-framework
+
+Alternatively, follow the instructions on this link:
+
+  https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
+
+Overview
+--------
+
+The ``tests/avocado/avocado_qemu`` directory provides the
+``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
+class.  Here's a simple usage example:
+
+.. code::
+
+  from avocado_qemu import QemuSystemTest
+
+
+  class Version(QemuSystemTest):
+      """
+      :avocado: tags=quick
+      """
+      def test_qmp_human_info_version(self):
+          self.vm.launch()
+          res = self.vm.cmd('human-monitor-command',
+                            command_line='info version')
+          self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+To execute your test, run:
+
+.. code::
+
+  avocado run version.py
+
+Tests may be classified according to a convention by using docstring
+directives such as ``:avocado: tags=TAG1,TAG2``.  To run all tests
+in the current directory, tagged as "quick", run:
+
+.. code::
+
+  avocado run -t quick .
+
+The ``avocado_qemu.Test`` base test class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``avocado_qemu.Test`` class has a number of characteristics that
+are worth being mentioned right away.
+
+First of all, it attempts to give each test a ready to use QEMUMachine
+instance, available at ``self.vm``.  Because many tests will tweak the
+QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
+is left to the test writer.
+
+The base test class has also support for tests with more than one
+QEMUMachine. The way to get machines is through the ``self.get_vm()``
+method which will return a QEMUMachine instance. The ``self.get_vm()``
+method accepts arguments that will be passed to the QEMUMachine creation
+and also an optional ``name`` attribute so you can identify a specific
+machine and get it more than once through the tests methods. A simple
+and hypothetical example follows:
+
+.. code::
+
+  from avocado_qemu import QemuSystemTest
+
+
+  class MultipleMachines(QemuSystemTest):
+      def test_multiple_machines(self):
+          first_machine = self.get_vm()
+          second_machine = self.get_vm()
+          self.get_vm(name='third_machine').launch()
+
+          first_machine.launch()
+          second_machine.launch()
+
+          first_res = first_machine.cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          second_res = second_machine.cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          third_res = self.get_vm(name='third_machine').cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          self.assertEqual(first_res, second_res, third_res)
+
+At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
+shutdown.
+
+The ``avocado_qemu.LinuxTest`` base test class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``avocado_qemu.LinuxTest`` is further specialization of the
+``avocado_qemu.Test`` class, so it contains all the characteristics of
+the later plus some extra features.
+
+First of all, this base class is intended for tests that need to
+interact with a fully booted and operational Linux guest.  At this
+time, it uses a Fedora 31 guest image.  The most basic example looks
+like this:
+
+.. code::
+
+  from avocado_qemu import LinuxTest
+
+
+  class SomeTest(LinuxTest):
+
+      def test(self):
+          self.launch_and_wait()
+          self.ssh_command('some_command_to_be_run_in_the_guest')
+
+Please refer to tests that use ``avocado_qemu.LinuxTest`` under
+``tests/avocado`` for more examples.
+
+QEMUMachine
+-----------
+
+The QEMUMachine API is already widely used in the Python iotests,
+device-crash-test and other Python scripts.  It's a wrapper around the
+execution of a QEMU binary, giving its users:
+
+ * the ability to set command line arguments to be given to the QEMU
+   binary
+
+ * a ready to use QMP connection and interface, which can be used to
+   send commands and inspect its results, as well as asynchronous
+   events
+
+ * convenience methods to set commonly used command line arguments in
+   a more succinct and intuitive way
+
+QEMU binary selection
+^^^^^^^^^^^^^^^^^^^^^
+
+The QEMU binary used for the ``self.vm`` QEMUMachine instance will
+primarily depend on the value of the ``qemu_bin`` parameter.  If it's
+not explicitly set, its default value will be the result of a dynamic
+probe in the same source tree.  A suitable binary will be one that
+targets the architecture matching host machine.
+
+Based on this description, test writers will usually rely on one of
+the following approaches:
+
+1) Set ``qemu_bin``, and use the given binary
+
+2) Do not set ``qemu_bin``, and use a QEMU binary named like
+   "qemu-system-${arch}", either in the current
+   working directory, or in the current source tree.
+
+The resulting ``qemu_bin`` value will be preserved in the
+``avocado_qemu.Test`` as an attribute with the same name.
+
+Attribute reference
+-------------------
+
+Test
+^^^^
+
+Besides the attributes and methods that are part of the base
+``avocado.Test`` class, the following attributes are available on any
+``avocado_qemu.Test`` instance.
+
+vm
+""
+
+A QEMUMachine instance, initially configured according to the given
+``qemu_bin`` parameter.
+
+arch
+""""
+
+The architecture can be used on different levels of the stack, e.g. by
+the framework or by the test itself.  At the framework level, it will
+currently influence the selection of a QEMU binary (when one is not
+explicitly given).
+
+Tests are also free to use this attribute value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+The ``arch`` attribute will be set to the test parameter of the same
+name.  If one is not given explicitly, it will either be set to
+``None``, or, if the test is tagged with one (and only one)
+``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
+
+cpu
+"""
+
+The cpu model that will be set to all QEMUMachine instances created
+by the test.
+
+The ``cpu`` attribute will be set to the test parameter of the same
+name. If one is not given explicitly, it will either be set to
+``None ``, or, if the test is tagged with one (and only one)
+``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
+
+machine
+"""""""
+
+The machine type that will be set to all QEMUMachine instances created
+by the test.
+
+The ``machine`` attribute will be set to the test parameter of the same
+name.  If one is not given explicitly, it will either be set to
+``None``, or, if the test is tagged with one (and only one)
+``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
+
+qemu_bin
+""""""""
+
+The preserved value of the ``qemu_bin`` parameter or the result of the
+dynamic probe for a QEMU binary in the current working directory or
+source tree.
+
+LinuxTest
+^^^^^^^^^
+
+Besides the attributes present on the ``avocado_qemu.Test`` base
+class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
+
+distro
+""""""
+
+The name of the Linux distribution used as the guest image for the
+test.  The name should match the **Provider** column on the list
+of images supported by the avocado.utils.vmimage library:
+
+https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
+
+distro_version
+""""""""""""""
+
+The version of the Linux distribution as the guest image for the
+test.  The name should match the **Version** column on the list
+of images supported by the avocado.utils.vmimage library:
+
+https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
+
+distro_checksum
+"""""""""""""""
+
+The sha256 hash of the guest image file used for the test.
+
+If this value is not set in the code or by a test parameter (with the
+same name), no validation on the integrity of the image will be
+performed.
+
+Parameter reference
+-------------------
+
+To understand how Avocado parameters are accessed by tests, and how
+they can be passed to tests, please refer to::
+
+  https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
+
+Parameter values can be easily seen in the log files, and will look
+like the following:
+
+.. code::
+
+  PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
+
+Test
+^^^^
+
+arch
+""""
+
+The architecture that will influence the selection of a QEMU binary
+(when one is not explicitly given).
+
+Tests are also free to use this parameter value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+This parameter has a direct relation with the ``arch`` attribute.  If
+not given, it will default to None.
+
+cpu
+"""
+
+The cpu model that will be set to all QEMUMachine instances created
+by the test.
+
+machine
+"""""""
+
+The machine type that will be set to all QEMUMachine instances created
+by the test.
+
+qemu_bin
+""""""""
+
+The exact QEMU binary to be used on QEMUMachine.
+
+LinuxTest
+^^^^^^^^^
+
+Besides the parameters present on the ``avocado_qemu.Test`` base
+class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
+
+distro
+""""""
+
+The name of the Linux distribution used as the guest image for the
+test.  The name should match the **Provider** column on the list
+of images supported by the avocado.utils.vmimage library:
+
+https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
+
+distro_version
+""""""""""""""
+
+The version of the Linux distribution as the guest image for the
+test.  The name should match the **Version** column on the list
+of images supported by the avocado.utils.vmimage library:
+
+https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
+
+distro_checksum
+"""""""""""""""
+
+The sha256 hash of the guest image file used for the test.
+
+If this value is not set in the code or by this parameter no
+validation on the integrity of the image will be performed.
+
+Skipping tests
+--------------
+
+The Avocado framework provides Python decorators which allow for easily skip
+tests running under certain conditions. For example, on the lack of a binary
+on the test system or when the running environment is a CI system. For further
+information about those decorators, please refer to::
+
+  https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
+
+While the conditions for skipping tests are often specifics of each one, there
+are recurring scenarios identified by the QEMU developers and the use of
+environment variables became a kind of standard way to enable/disable tests.
+
+Here is a list of the most used variables:
+
+AVOCADO_ALLOW_LARGE_STORAGE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Tests which are going to fetch or produce assets considered *large* are not
+going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
+the environment.
+
+The definition of *large* is a bit arbitrary here, but it usually means an
+asset which occupies at least 1GB of size on disk when uncompressed.
+
+SPEED
+^^^^^
+Tests which have a long runtime will not be run unless ``SPEED=slow`` is
+exported on the environment.
+
+The definition of *long* is a bit arbitrary here, and it depends on the
+usefulness of the test too. A unique test is worth spending more time on,
+small variations on existing tests perhaps less so. As a rough guide,
+a test or set of similar tests which take more than 100 seconds to
+complete.
+
+AVOCADO_ALLOW_UNTRUSTED_CODE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+There are tests which will boot a kernel image or firmware that can be
+considered not safe to run on the developer's workstation, thus they are
+skipped by default. The definition of *not safe* is also arbitrary but
+usually it means a blob which either its source or build process aren't
+public available.
+
+You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
+order to allow tests which make use of those kind of assets.
+
+AVOCADO_TIMEOUT_EXPECTED
+^^^^^^^^^^^^^^^^^^^^^^^^
+The Avocado framework has a timeout mechanism which interrupts tests to avoid the
+test suite of getting stuck. The timeout value can be set via test parameter or
+property defined in the test class, for further details::
+
+  https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
+
+Even though the timeout can be set by the test developer, there are some tests
+that may not have a well-defined limit of time to finish under certain
+conditions. For example, tests that take longer to execute when QEMU is
+compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
+has been used to determine whether those tests should run or not.
+
+QEMU_TEST_FLAKY_TESTS
+^^^^^^^^^^^^^^^^^^^^^
+Some tests are not working reliably and thus are disabled by default.
+This includes tests that don't run reliably on GitLab's CI which
+usually expose real issues that are rarely seen on developer machines
+due to the constraints of the CI environment. If you encounter a
+similar situation then raise a bug and then mark the test as shown on
+the code snippet below:
+
+.. code::
+
+  # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
+  @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+  def test(self):
+      do_something()
+
+You can also add ``:avocado: tags=flaky`` to the test meta-data so
+only the flaky tests can be run as a group:
+
+.. code::
+
+   env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
+      run tests/avocado -filter-by-tags=flaky
+
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
+
+Uninstalling Avocado
+--------------------
+
+If you've followed the manual installation instructions above, you can
+easily uninstall Avocado.  Start by listing the packages you have
+installed::
+
+  pip list --user
+
+And remove any package you want with::
+
+  pip uninstall <package_name>
+
+If you've used ``make check-avocado``, the Python virtual environment where
+Avocado is installed will be cleaned up as part of ``make check-clean``.
docs/devel/testing/index.rst +1 -0
index 2711fd78b7..cdf7ba1f8b 100644
@@ -9,6 +9,7 @@ testing infrastructure.
 
    main
    qtest
+   avocado
    acpi-bits
    ci
    fuzzing
docs/devel/testing/main.rst +1 -564

578 lines not on this page · raw message

Signature · 1 line
2.46.0
#52 lore raw

PATCH v5 43/44 docs/devel/testing: Rename avocado_qemu.Test class

In reply to Thomas Huth

The avocado_qemu.Test class has been renamed a while back in commit 2283b627bc ("tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest"), so we should reflect this now in the documentation, too.

---
 docs/devel/testing/avocado.rst | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
docs/devel/testing/avocado.rst +16 -16
index 0d207a05c5..eda76fe2db 100644
@@ -8,11 +8,11 @@ The ``tests/avocado`` directory hosts integration tests. They're usually
 higher level tests, and may interact with external resources and with
 various guest operating systems.
 
-These tests are written using the Avocado Testing Framework (which must
-be installed separately) in conjunction with a the ``avocado_qemu.Test``
+These tests are written using the Avocado Testing Framework (which must be
+installed separately) in conjunction with a the ``avocado_qemu.QemuSystemTest``
 class, implemented at ``tests/avocado/avocado_qemu``.
 
-Tests based on ``avocado_qemu.Test`` can easily:
+Tests based on ``avocado_qemu.QemuSystemTest`` can easily:
 
  * Customize the command line arguments given to the convenience
    ``self.vm`` attribute (a QEMUMachine instance)
@@ -154,7 +154,7 @@ Overview
 --------
 
 The ``tests/avocado/avocado_qemu`` directory provides the
-``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
+``avocado_qemu`` Python module, containing the ``avocado_qemu.QemuSystemTest``
 class.  Here's a simple usage example:
 
 .. code::
@@ -186,11 +186,11 @@ in the current directory, tagged as "quick", run:
 
   avocado run -t quick .
 
-The ``avocado_qemu.Test`` base test class
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The ``avocado_qemu.QemuSystemTest`` base test class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The ``avocado_qemu.Test`` class has a number of characteristics that
-are worth being mentioned right away.
+The ``avocado_qemu.QemuSystemTest`` class has a number of characteristics
+that are worth being mentioned right away.
 
 First of all, it attempts to give each test a ready to use QEMUMachine
 instance, available at ``self.vm``.  Because many tests will tweak the
@@ -233,15 +233,15 @@ and hypothetical example follows:
 
           self.assertEqual(first_res, second_res, third_res)
 
-At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
-shutdown.
+At test "tear down", ``avocado_qemu.QemuSystemTest`` handles all the
+QEMUMachines shutdown.
 
 The ``avocado_qemu.LinuxTest`` base test class
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The ``avocado_qemu.LinuxTest`` is further specialization of the
-``avocado_qemu.Test`` class, so it contains all the characteristics of
-the later plus some extra features.
+``avocado_qemu.QemuSystemTest`` class, so it contains all the characteristics
+of the later plus some extra features.
 
 First of all, this base class is intended for tests that need to
 interact with a fully booted and operational Linux guest.  At this
@@ -298,7 +298,7 @@ the following approaches:
    working directory, or in the current source tree.
 
 The resulting ``qemu_bin`` value will be preserved in the
-``avocado_qemu.Test`` as an attribute with the same name.
+``avocado_qemu.QemuSystemTest`` as an attribute with the same name.
 
 Attribute reference
 -------------------
@@ -308,7 +308,7 @@ Test
 
 Besides the attributes and methods that are part of the base
 ``avocado.Test`` class, the following attributes are available on any
-``avocado_qemu.Test`` instance.
+``avocado_qemu.QemuSystemTest`` instance.
 
 vm
 ""
@@ -365,7 +365,7 @@ source tree.
 LinuxTest
 ^^^^^^^^^
 
-Besides the attributes present on the ``avocado_qemu.Test`` base
+Besides the attributes present on the ``avocado_qemu.QemuSystemTest`` base
 class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
 
 distro
@@ -446,7 +446,7 @@ The exact QEMU binary to be used on QEMUMachine.
 LinuxTest
 ^^^^^^^^^
 
-Besides the parameters present on the ``avocado_qemu.Test`` base
+Besides the parameters present on the ``avocado_qemu.QemuSystemTest`` base
 class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
 
 distro
Signature · 1 line
2.46.0
#53 lore raw

PATCH v5 44/44 docs/devel/testing: Add documentation for functional tests

In reply to Thomas Huth

Document the new functional testing framework. The text is originally based on the Avocado documentation, but heavily modified to match the new framework.

---
 docs/devel/testing/functional.rst | 338 ++++++++++++++++++++++++++++++
 docs/devel/testing/index.rst      |   1 +
 docs/devel/testing/main.rst       |  12 ++
 3 files changed, 351 insertions(+)
 create mode 100644 docs/devel/testing/functional.rst
docs/devel/testing/functional.rst new file +338 -0
new file mode 100644
index 0000000000..bf6f1bb81e
@@ -0,0 +1,338 @@
+.. _checkfunctional-ref:
+
+Functional testing with Python
+==============================
+
+The ``tests/functional`` directory hosts functional tests written in
+Python. They are usually higher level tests, and may interact with
+external resources and with various guest operating systems.
+The functional tests have initially evolved from the Avocado tests, so there
+is a lot of similarity to those tests here (see :ref:`checkavocado-ref` for
+details about the Avocado tests).
+
+The tests should be written in the style of the Python `unittest`_ framework,
+using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
+provides classes (e.g. the ``QemuBaseTest``, ``QemuUserTest`` and the
+``QemuSystemTest`` classes) and utility functions that help to get your test
+into the right shape, e.g. by replacing the 'stdout' python object to redirect
+the normal output of your test to stderr instead.
+
+Note that if you don't use one of the QemuBaseTest based classes for your
+test, or if you spawn subprocesses from your test, you have to make sure
+that there is no TAP-incompatible output written to stdio, e.g. either by
+prefixing every line with a "# " to mark the output as a TAP comment, or
+e.g. by capturing the stdout output of subprocesses (redirecting it to
+stderr is OK).
+
+Tests based on ``qemu_test.QemuSystemTest`` can easily:
+
+ * Customize the command line arguments given to the convenience
+   ``self.vm`` attribute (a QEMUMachine instance)
+
+ * Interact with the QEMU monitor, send QMP commands and check
+   their results
+
+ * Interact with the guest OS, using the convenience console device
+   (which may be useful to assert the effectiveness and correctness of
+   command line arguments or QMP commands)
+
+ * Download (and cache) remote data files, such as firmware and kernel
+   images
+
+Running tests
+-------------
+
+You can run the functional tests simply by executing:
+
+.. code::
+
+  make check-functional
+
+It is also possible to run tests for a certain target only, for example
+the following line will only run the tests for the x86_64 target:
+
+.. code::
+
+  make check-functional-x86_64
+
+To run a single test file without the meson test runner, you can also
+execute the file directly by specifying two environment variables first,
+the PYTHONPATH that has to include the python folder and the tests/functional
+folder of the source tree, and QEMU_TEST_QEMU_BINARY that has to point
+to the QEMU binary that should be used for the test, for example::
+
+  $ export PYTHONPATH=../python:../tests/functional
+  $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
+  $ python3 ../tests/functional/test_file.py
+
+Overview
+--------
+
+The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
+Python module, containing the ``qemu_test.QemuSystemTest`` class.
+Here is a simple usage example:
+
+.. code::
+
+  #!/usr/bin/env python3
+
+  from qemu_test import QemuSystemTest
+
+  class Version(QemuSystemTest):
+
+      def test_qmp_human_info_version(self):
+          self.vm.launch()
+          res = self.vm.cmd('human-monitor-command',
+                            command_line='info version')
+          self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+  if __name__ == '__main__':
+      QemuSystemTest.main()
+
+By providing the "hash bang" line at the beginning of the script, marking
+the file as executable and by calling into QemuSystemTest.main(), the test
+can also be run stand-alone, without a test runner. OTOH when run via a test
+runner, the QemuSystemTest.main() function takes care of running the test
+functions in the right fassion (e.g. with TAP output that is required by the
+meson test runner).
+
+The ``qemu_test.QemuSystemTest`` base test class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``qemu_test.QemuSystemTest`` class has a number of characteristics
+that are worth being mentioned.
+
+First of all, it attempts to give each test a ready to use QEMUMachine
+instance, available at ``self.vm``.  Because many tests will tweak the
+QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
+is left to the test writer.
+
+The base test class has also support for tests with more than one
+QEMUMachine. The way to get machines is through the ``self.get_vm()``
+method which will return a QEMUMachine instance. The ``self.get_vm()``
+method accepts arguments that will be passed to the QEMUMachine creation
+and also an optional ``name`` attribute so you can identify a specific
+machine and get it more than once through the tests methods. A simple
+and hypothetical example follows:
+
+.. code::
+
+  from qemu_test import QemuSystemTest
+
+  class MultipleMachines(QemuSystemTest):
+      def test_multiple_machines(self):
+          first_machine = self.get_vm()
+          second_machine = self.get_vm()
+          self.get_vm(name='third_machine').launch()
+
+          first_machine.launch()
+          second_machine.launch()
+
+          first_res = first_machine.cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          second_res = second_machine.cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          third_res = self.get_vm(name='third_machine').cmd(
+              'human-monitor-command',
+              command_line='info version')
+
+          self.assertEqual(first_res, second_res, third_res)
+
+At test "tear down", ``qemu_test.QemuSystemTest`` handles all the QEMUMachines
+shutdown.
+
+QEMUMachine
+-----------
+
+The QEMUMachine API is already widely used in the Python iotests,
+device-crash-test and other Python scripts.  It's a wrapper around the
+execution of a QEMU binary, giving its users:
+
+ * the ability to set command line arguments to be given to the QEMU
+   binary
+
+ * a ready to use QMP connection and interface, which can be used to
+   send commands and inspect its results, as well as asynchronous
+   events
+
+ * convenience methods to set commonly used command line arguments in
+   a more succinct and intuitive way
+
+QEMU binary selection
+^^^^^^^^^^^^^^^^^^^^^
+
+The QEMU binary used for the ``self.vm`` QEMUMachine instance will
+primarily depend on the value of the ``qemu_bin`` class attribute.
+If it is not explicitly set by the test code, its default value will
+be the result the QEMU_TEST_QEMU_BINARY environment variable.
+
+Attribute reference
+-------------------
+
+QemuBaseTest
+^^^^^^^^^^^^
+
+The following attributes are available on any ``qemu_test.QemuBaseTest``
+instance.
+
+arch
+""""
+
+The target architecture of the QEMU binary.
+
+Tests are also free to use this attribute value, for their own needs.
+A test may, for instance, use this value when selecting the architecture
+of a kernel or disk image to boot a VM with.
+
+qemu_bin
+""""""""
+
+The preserved value of the ``QEMU_TEST_QEMU_BINARY`` environment
+variable.
+
+QemuUserTest
+^^^^^^^^^^^^
+
+The QemuUserTest class can be used for running an executable via the
+usermode emulation binaries.
+
+QemuSystemTest
+^^^^^^^^^^^^^^
+
+The QemuSystemTest class can be used for running tests via one of the
+qemu-system-* binaries.
+
+vm
+""
+
+A QEMUMachine instance, initially configured according to the given
+``qemu_bin`` parameter.
+
+cpu
+"""
+
+The cpu model that will be set to all QEMUMachine instances created
+by the test.
+
+machine
+"""""""
+
+The machine type that will be set to all QEMUMachine instances created
+by the test. By using the set_machine() function of the QemuSystemTest
+class to set this attribute, you can automatically check whether the
+machine is available to skip the test in case it is not built into the
+QEMU binary.
+
+Asset handling
+--------------
+
+Many functional tests download assets (e.g. Linux kernels, initrds,
+firmware images, etc.) from the internet to be able to run tests with
+them. This imposes additional challenges to the test framework.
+
+First there is the the problem that some people might not have an
+unconstrained internet connection, so such tests should not be run by
+default when running ``make check``. To accomplish this situation,
+the tests that download files should only be added to the "thorough"
+speed mode in the meson.build file, while the "quick" speed mode is
+fine for functional tests that can be run without downloading files.
+``make check`` then only runs the quick functional tests along with
+the other quick tests from the other test suites. If you choose to
+run only run ``make check-functional``, the "thorough" tests will be
+executed, too. And to run all functional tests along with the others,
+you can use something like::
+
+  make -j$(nproc) check SPEED=thorough
+
+The second problem with downloading files from the internet are time
+constraints. The time for downloading files should not be taken into
+account when the test is running and the timeout of the test is ticking
+(since downloading can be very slow, depending on the network bandwidth).
+This problem is solved by downloading the assets ahead of time, before
+the tests are run. This pre-caching is done with the qemu_test.Asset
+class. To use it in your test, declare an asset in your test class with
+its URL and SHA256 checksum like this::
+
+    ASSET_somename = (
+        ('https://www.qemu.org/assets/images/qemu_head_200.png'),
+        '34b74cad46ea28a2966c1d04e102510daf1fd73e6582b6b74523940d5da029dd')
+
+In your test function, you can then get the file name of the cached
+asset like this::
+
+    def test_function(self):
+        file_path = self.ASSET_somename.fetch()
+
+The pre-caching will be done automatically when running
+``make check-functional`` (but not when running e.g.
+``make check-functional-<target>``). In case you just want to download
+the assets without running the tests, you can do so by running::
+
+    make precache-functional
+
+The cache is populated in the ``~/.cache/qemu/download`` directory by
+default, but the location can be changed by setting the
+``QEMU_TEST_CACHE_DIR`` environment variable.
+
+Skipping tests
+--------------
+
+Since the test framework is based on the common Python unittest framework,
+you can use the usual Python decorators which allow for easily skipping
+tests running under certain conditions, for example, on the lack of a binary
+on the test system or when the running environment is a CI system. For further
+information about those decorators, please refer to:
+
+  https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures
+
+While the conditions for skipping tests are often specifics of each one, there
+are recurring scenarios identified by the QEMU developers and the use of
+environment variables became a kind of standard way to enable/disable tests.
+
+Here is a list of the most used variables:
+
+QEMU_TEST_ALLOW_LARGE_STORAGE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Tests which are going to fetch or produce assets considered *large* are not
+going to run unless that ``QEMU_TEST_ALLOW_LARGE_STORAGE=1`` is exported on
+the environment.
+
+The definition of *large* is a bit arbitrary here, but it usually means an
+asset which occupies at least 1GB of size on disk when uncompressed.
+
+QEMU_TEST_ALLOW_UNTRUSTED_CODE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+There are tests which will boot a kernel image or firmware that can be
+considered not safe to run on the developer's workstation, thus they are
+skipped by default. The definition of *not safe* is also arbitrary but
+usually it means a blob which either its source or build process aren't
+public available.
+
+You should export ``QEMU_TEST_ALLOW_UNTRUSTED_CODE=1`` on the environment in
+order to allow tests which make use of those kind of assets.
+
+QEMU_TEST_FLAKY_TESTS
+^^^^^^^^^^^^^^^^^^^^^
+Some tests are not working reliably and thus are disabled by default.
+This includes tests that don't run reliably on GitLab's CI which
+usually expose real issues that are rarely seen on developer machines
+due to the constraints of the CI environment. If you encounter a
+similar situation then raise a bug and then mark the test as shown on
+the code snippet below:
+
+.. code::
+
+  # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
+  @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+  def test(self):
+      do_something()
+
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
+
+.. _unittest: https://docs.python.org/3/library/unittest.html
docs/devel/testing/index.rst +1 -0
index cdf7ba1f8b..45eb4a7181 100644
@@ -9,6 +9,7 @@ testing infrastructure.
 
    main
    qtest
+   functional
    avocado
    acpi-bits
    ci
docs/devel/testing/main.rst +12 -0
index 39b965ecf6..e9921a4b10 100644
@@ -862,6 +862,18 @@ supported. To start the fuzzer, run
 Alternatively, some command different from ``qemu-img info`` can be tested, by
 changing the ``-c`` option.
 
+Functional tests using Python
+-----------------------------
+
+The ``tests/functional`` directory hosts functional tests written in
+Python. You can run the functional tests simply by executing:
+
+.. code::
+
+  make check-functional
+
+See :ref:`checkfunctional-ref` for more details.
+
 Integration tests using the Avocado Framework
 ---------------------------------------------
 
Signature · 1 line
2.46.0