Skip to content

Convert avocado tests to normal Python unittests

Daniel P. Berrangé qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v3 Latest24 patches 44 repliesIn review

Changes in v3 from the cover letter

- Split tests/functional/__init__.py into multiple files
- Introduce an 'Asset' class to handle downloading & caching
- Convert all tests to use new Asset class instead of 'fetch_asset'
- Ensure 'make check-functional' pre-caches all assets before running the tests, to avoid meson test timeouts due to slow downloads
- Fix bios bits test logging in verbose mode to not print subprocess output to stdout, since it confuses TAP parsing

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/24python: Install pycotap in our venv if necessarypython/wheels/pycotap-1.3.1-py3-none-any.whl pythondeps.toml+1 -01 reply
2/24tests/functional: Add base classes for the upcoming pytest-based teststests/functional/qemu_test/__init__.py tests/functional/qemu_test/cmd.py tests/functional/qemu_test/config.py +1 more+374 -04 replies
3/24tests/functional: Set up loggingtests/functional/qemu_test/testcase.py+26 -12 replies
4/24tests/Makefile.include: Increase the level of indentation in the help texttests/Makefile.include+15 -152 replies
5/24tests/functional: Prepare the meson build system for the functional teststests/Makefile.include tests/functional/meson.build tests/meson.build+78 -02 replies
6/24tests/functional: Convert simple avocado tests into standalone python teststests/functional/meson.build tests/functional/test_cpu_queries.py tests/functional/test_empty_cpu_model.py +3 more+34 -561 reply
7/24tests/functional: Convert avocado tests that just need a small adjustmenttests/functional/meson.build tests/functional/test_info_usernet.py tests/functional/test_ppc_74xx.py +1 more+53 -512 replies
8/24tests/functional: add a module for handling asset download & cachingtests/functional/qemu_test/__init__.py tests/functional/qemu_test/asset.py+97 -06 replies
9/24tests/functional: enable pre-emptive caching of assetstests/Makefile.include tests/functional/meson.build tests/functional/qemu_test/asset.py +1 more+75 -32 replies
10/24tests/functional: Convert some tests that download files via fetch_asset()MAINTAINERS tests/avocado/machine_mips_loongson3v.py tests/functional/meson.build +6 more+154 -1400 replies
11/24tests/functional: Add a function for extracting files from an archivetests/functional/qemu_test/utils.py+21 -01 reply
12/24tests/functional: Convert some avocado tests that needed avocado.utils.archiveMAINTAINERS tests/avocado/machine_sparc64_sun4u.py tests/functional/meson.build +3 more+82 -681 reply
13/24tests/functional: Convert the s390x avocado tests into standalone testsMAINTAINERS tests/functional/meson.build tests/functional/qemu_test/utils.py +2 more+94 -955 replies
14/24tests/functional: Convert the x86_cpu_model_versions testtests/functional/meson.build tests/functional/test_x86_cpu_model_versions.py+20 -441 reply
15/24tests/functional: Convert the microblaze avocado tests into standalone testsMAINTAINERS tests/avocado/machine_microblaze.py tests/functional/meson.build +2 more+90 -621 reply
16/24tests/functional: Convert the riscv_opensbi avocado test into a standalone testtests/avocado/riscv_opensbi.py tests/functional/meson.build tests/functional/test_riscv_opensbi.py+44 -634 replies
17/24tests/functional: Convert the virtio_gpu avocado test into a standalone testtests/functional/meson.build tests/functional/test_virtio_gpu.py+30 -351 reply
18/24tests/functional: Convert most ppc avocado tests into standalone testsMAINTAINERS tests/functional/meson.build tests/functional/test_ppc64_powernv.py +4 more+109 -1410 replies
19/24tests/functional: Convert the ppc_amiga avocado test into a standalone testtests/avocado/ppc_amiga.py tests/functional/meson.build tests/functional/test_ppc_amiga.py+43 -381 reply
20/24tests/functional: Convert the ppc_hv avocado test into a standalone testtests/functional/test_ppc64_hv.py+20 -281 reply
21/24tests/functional: Convert the m68k nextcube test with tesseracttests/avocado/tesseract_utils.py tests/functional/meson.build tests/functional/qemu_test/tesseract.py +1 more+55 -591 reply
22/24tests/functional: Convert the acpi-bits test into a standalone testtests/functional/acpi-bits/bits-config/bits-cfg.txt tests/functional/acpi-bits/bits-tests/smbios.py2 tests/functional/acpi-bits/bits-tests/smilatency.py2 +4 more+43 -402 replies
23/24tests/functional: Convert the rx_gdbsim avocado test into a standalone testtests/functional/meson.build tests/functional/qemu_test/utils.py tests/functional/test_rx_gdbsim.py+45 -281 reply
24/24gitlab-ci: Add "check-functional" to the build tests.gitlab-ci.d/buildtest-template.yml .gitlab-ci.d/buildtest.yml+32 -311 reply

Reviews trailers per patch

19 of 24 patches have a Reviewed-by or Acked-by.

PatchDaniel P. BerrangéPhilippe Mathieu-DaudéAlistair FrancisRichard HendersonAlex Bennée
1/24T-b
2/24
3/24R-b
4/24R-b T-b R-b
5/24R-b T-b
6/24R-b R-b T-b
7/24R-b T-b R-b
8/24R-b T-b
9/24T-b
10/24
11/24R-b R-b T-b
12/24R-b R-b T-b
13/24R-b
14/24R-b T-b
15/24R-b T-b
16/24T-b R-b
17/24R-b
18/24R-b
19/24R-b T-b
20/24
21/24R-b T-b
22/24R-b T-b
23/24R-b T-b
24/24R-b R-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v3 thread

Cover letter [PATCH v3 00/24] Convert avocado tests to normal Python unittests

This series is an update to Thomas' v2:

https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg05805.html

wherein Thomas suggested someone could do the asset caching updates while he is on vacation:

https://lists.nongnu.org/archive/html/qemu-devel/2024-07/msg06228.html

hence this posting what I'm calling a v3.

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later.

Now if you want to try out these patches: Apply the patches, then recompile and then run:

make check-functional

You can also run single targets e.g. with:

make check-functional-ppc

You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export QEMU_BUILD_ROOT=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under tests/functional/<arch>/<testname> - console log and general logs will be put in separate files there.

Still to be done: Update the documentation for this new test framework.

v3:
- Split tests/functional/__init__.py into multiple files
- Introduce an 'Asset' class to handle downloading & caching
- Convert all tests to use new Asset class instead of 'fetch_asset'
- Ensure 'make check-functional' pre-caches all assets before running the tests, to avoid meson test timeouts due to slow downloads
- Fix bios bits test logging in verbose mode to not print subprocess output to stdout, since it confuses TAP parsing

v2:
- Addressed review feedback from v1
- Add pycotap as a wheel instead of trying to install it on demand when running "make check-functional" (works much better now!)
- Converted much more tests
- Lots of other small improvements here and there

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC

Daniel P. Berrangé (2):
tests/functional: add a module for handling asset download & caching tests/functional: enable pre-emptive caching of assets

Thomas Huth (22):
python: Install pycotap in our venv if necessary tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Set up logging
tests/Makefile.include: Increase the level of indentation in the help
text
tests/functional: Prepare the meson build system for the functional
tests
tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Convert the s390x avocado tests into standalone
tests
tests/functional: Convert the x86_cpu_model_versions test tests/functional: Convert the microblaze avocado tests into standalone
tests
tests/functional: Convert the riscv_opensbi avocado test into a
standalone test
tests/functional: Convert the virtio_gpu avocado test into a
standalone test
tests/functional: Convert most ppc avocado tests into standalone tests tests/functional: Convert the ppc_amiga avocado test into a standalone
test
tests/functional: Convert the ppc_hv avocado test into a standalone
test
tests/functional: Convert the m68k nextcube test with tesseract tests/functional: Convert the acpi-bits test into a standalone test tests/functional: Convert the rx_gdbsim avocado test into a standalone
test
gitlab-ci: Add "check-functional" to the build tests

 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +++---
 MAINTAINERS                                   |  32 +--
 python/wheels/pycotap-1.3.1-py3-none-any.whl  | Bin 0 -> 5119 bytes
 pythondeps.toml                               |   1 +
 tests/Makefile.include                        |  42 ++--
 tests/avocado/machine_microblaze.py           |  61 ------
 tests/avocado/machine_mips_loongson3v.py      |  39 ----
 tests/avocado/machine_sparc64_sun4u.py        |  36 ----
 tests/avocado/ppc_amiga.py                    |  38 ----
 tests/avocado/riscv_opensbi.py                |  63 ------
 tests/avocado/tesseract_utils.py              |  46 -----
 .../acpi-bits/bits-config/bits-cfg.txt        |   0
 .../acpi-bits/bits-tests/smbios.py2           |   0
 .../acpi-bits/bits-tests/smilatency.py2       |   0
 .../acpi-bits/bits-tests/testacpi.py2         |   0
 .../acpi-bits/bits-tests/testcpuid.py2        |   0
 tests/functional/meson.build                  | 180 +++++++++++++++++
 tests/functional/qemu_test/__init__.py        |  14 ++
 tests/functional/qemu_test/asset.py           | 130 ++++++++++++
 tests/functional/qemu_test/cmd.py             | 171 ++++++++++++++++
 tests/functional/qemu_test/config.py          |  36 ++++
 tests/functional/qemu_test/tesseract.py       |  35 ++++
 tests/functional/qemu_test/testcase.py        | 187 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  47 +++++
 .../test_acpi_bits.py}                        |  81 ++++----
 .../test_arm_canona1100.py}                   |  30 +--
 .../test_arm_n8x0.py}                         |  36 ++--
 .../test_avr_mega2560.py}                     |  22 ++-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  46 +++--
 .../test_m68k_nextcube.py}                    |  29 +--
 .../test_mem_addr_space.py}                   |  52 +----
 .../functional/test_microblaze_s3adsp1800.py  |  39 ++++
 .../test_microblazeel_s3adsp1800.py           |  42 ++++
 tests/functional/test_mips64el_loongson3v.py  |  39 ++++
 .../test_netdev_ethtool.py}                   |  50 ++---
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../test_ppc64_hv.py}                         |  48 ++---
 .../test_ppc64_powernv.py}                    |  56 ++----
 .../test_ppc64_pseries.py}                    |  56 ++----
 .../ppc_405.py => functional/test_ppc_405.py} |  28 +--
 .../test_ppc_40p.py}                          |  63 +++---
 .../test_ppc_74xx.py}                         |  74 +++----
 tests/functional/test_ppc_amiga.py            |  42 ++++
 .../test_ppc_bamboo.py}                       |  32 +--
 .../test_ppc_mpc8544ds.py}                    |  28 +--
 .../test_ppc_virtex_ml507.py}                 |  28 +--
 tests/functional/test_riscv_opensbi.py        |  36 ++++
 .../test_rx_gdbsim.py}                        |  57 +++---
 .../test_s390x_ccw_virtio.py}                 |  79 ++++----
 .../test_s390x_topology.py}                   |  86 ++++----
 tests/functional/test_sparc64_sun4u.py        |  40 ++++
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_gpu.py}                       |  64 +++---
 .../test_virtio_version.py}                   |   8 +-
 .../test_x86_cpu_model_versions.py}           |  63 ++----
 tests/meson.build                             |   1 +
 60 files changed, 1631 insertions(+), 994 deletions(-)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
 delete mode 100644 tests/avocado/machine_microblaze.py
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 delete mode 100644 tests/avocado/ppc_amiga.py
 delete mode 100644 tests/avocado/riscv_opensbi.py
 delete mode 100644 tests/avocado/tesseract_utils.py
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/asset.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 create mode 100644 tests/functional/qemu_test/testcase.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (52%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_m68k_nextcube.py => functional/test_m68k_nextcube.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (66%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (88%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_ppc_amiga.py
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_riscv_opensbi.py
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (88%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755
 rename tests/{avocado/x86_cpu_model_versions.py => functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755
Signature · 1 line
2.45.2