Bump Avocado to 103.0 LTS and update tests for compatibility and new features
Subsystems (2)
v2 cover Bump Avocado to 103.0 LTS and update tests for compatibility and new features
This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.
These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.
Nevertheless, as pointed out, on the ML, these changes are needed NOW.
Some examples of runs in the CI can be seen below:
* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s
* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 s
Changes from v1 (references are from v1):
This version first applies the Avocado version bump (now PATCH 1), adding to that patch the serial execution behavior. Avocado 103.0 LTS can run all existing tests in QEMU without any changes, so it's safe to pick PATCH 1 *only* from this series if needed. A GitLab CI job that does only that can be seen here:
https://gitlab.com/cleber.gnu/qemu/-/pipelines/1402633650
Details on changes:
* Moved "Bump avocado to 103.0" to first patch
* Patches already applied:
- [PATCH 1/13] tests/avocado: mips: fallback to HTTP given certificate expiration
- [PATCH 2/13] tests/avocado: mips: add hint for fetchasset plugin
- [PATCH 8/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
* Dropped patches:
- [PATCH 3/13] tests/avocado/intel_iommu.py: increase timeout
- [PATCH 6/13] tests/avocado: use more distinct names for assets
- Replaced with "tests/avocado: simplify parameters on fetch_asset with name only"
- [PATCH 7/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
* [PATCH 5/13] tests/avocado: machine aarch64: standardize location and RO access:
- Fixed rebase mistake
* [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
- Use "snapshot=on" instead of new copy of file
* New patches:
- tests/avocado: apply proper skipUnless decorator
- tests/avocado: simplify parameters on fetch_asset with name only
Cleber Rosa (9):
Bump avocado to 103.0
tests/avocado: apply proper skipUnless decorator tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: simplify parameters on fetch_asset with name only tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Avocado tests: allow for parallel execution of tests
docs/devel/testing.rst | 12 +++++++ pythondeps.toml | 2 +- tests/Makefile.include | 6 +++- tests/avocado/boot_linux_console.py | 1 - tests/avocado/boot_xen.py | 13 ++++---- tests/avocado/cdrom.py | 41 ++++++++++++++++++++++++ tests/avocado/intel_iommu.py | 1 - tests/avocado/kvm_xen_guest.py | 2 +- tests/avocado/linux_initrd.py | 1 - tests/avocado/machine_aarch64_sbsaref.py | 4 +-- tests/avocado/machine_aarch64_virt.py | 14 ++++---- tests/avocado/machine_aspeed.py | 2 -- tests/avocado/machine_mips_malta.py | 2 -- tests/avocado/machine_rx_gdbsim.py | 2 -- tests/avocado/netdev-ethtool.py | 2 +- tests/avocado/reverse_debugging.py | 4 --- tests/avocado/smmu.py | 1 - tests/avocado/tuxrun_baselines.py | 16 ++++----- 18 files changed, 82 insertions(+), 44 deletions(-) create mode 100644 tests/avocado/cdrom.py
Signature · 1 line
2.45.2
PATCH v2 1/9 Bump avocado to 103.0
This bumps Avocado to latest the LTS release.
An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.
Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.
To keep behavior of jobs as close as possible with previous version, this version bump keeps the execution serial (maximum of one task at a time being run).
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
--- pythondeps.toml | 2 +- tests/Makefile.include | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
pythondeps.toml +1 -1
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" } # Note that qemu.git/python/ is always implicitly installed. # Prefer an LTS version when updating the accepted versions of # avocado-framework, for example right now the limit is 92.x. -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" } pycdlib = { accepted = ">=1.11.0" }
tests/Makefile.include +1 -1
@@ -141,7 +141,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")
Signature · 1 line
2.45.2
PATCH v2 2/9 tests/avocado: apply proper skipUnless decorator
In reply to Cleber RosaCommit 9b45cc993 added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what they decorate, so the newlines added should *not* exist there.
--- tests/avocado/boot_linux_console.py | 1 - tests/avocado/intel_iommu.py | 1 - tests/avocado/linux_initrd.py | 1 - tests/avocado/machine_aspeed.py | 2 -- tests/avocado/machine_mips_malta.py | 2 -- tests/avocado/machine_rx_gdbsim.py | 2 -- tests/avocado/reverse_debugging.py | 4 ---- tests/avocado/smmu.py | 1 - 8 files changed, 14 deletions(-)
tests/avocado/boot_linux_console.py +0 -1
@@ -1522,7 +1522,6 @@ def test_ppc_mac99(self): # like issues with a buggy kernel. As a result we don't want it # gating releases on Gitlab. @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_sh4_r2d(self): """ :avocado: tags=arch:sh4
tests/avocado/intel_iommu.py +0 -1
@@ -13,7 +13,6 @@ from avocado_qemu.linuxtest import LinuxTest @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - class IntelIOMMU(LinuxTest): """ :avocado: tags=arch:x86_64
tests/avocado/linux_initrd.py +0 -1
@@ -54,7 +54,6 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self): self.assertRegex(self.vm.get_log(), expected_msg) @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_with_2gib_file_should_work_with_linux_v4_16(self): """ :avocado: tags=flaky
tests/avocado/machine_aspeed.py +0 -2
@@ -323,7 +323,6 @@ def do_test_aarch64_aspeed_sdk_start(self, image): self.wait_for_console_pattern('Starting kernel ...') @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_arm_ast2500_evb_sdk(self): """ :avocado: tags=arch:arm
@@ -343,7 +342,6 @@ def test_arm_ast2500_evb_sdk(self): self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4') @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_arm_ast2600_evb_sdk(self): """ :avocado: tags=arch:arm
tests/avocado/machine_mips_malta.py +0 -2
@@ -102,7 +102,6 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self): self.do_test_i6400_framebuffer_logo(1) @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_mips_malta_i6400_framebuffer_logo_7cores(self): """ :avocado: tags=arch:mips64el
@@ -114,7 +113,6 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self): self.do_test_i6400_framebuffer_logo(7) @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_mips_malta_i6400_framebuffer_logo_8cores(self): """ :avocado: tags=arch:mips64el
tests/avocado/machine_rx_gdbsim.py +0 -2
@@ -23,7 +23,6 @@ class RxGdbSimMachine(QemuSystemTest): KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_uboot(self): """ U-Boot and checks that the console is operational.
@@ -49,7 +48,6 @@ def test_uboot(self): #exec_command_and_wait_for_pattern(self, 'version', gcc_version) @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_linux_sash(self): """ Boots a Linux kernel and checks that the console is operational.
tests/avocado/reverse_debugging.py +0 -4
@@ -207,7 +207,6 @@ def get_pc(self, g): # unidentified gitlab timeout problem @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_x86_64_pc(self): """ :avocado: tags=arch:x86_64
@@ -225,7 +224,6 @@ class ReverseDebugging_AArch64(ReverseDebugging): # unidentified gitlab timeout problem @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_aarch64_virt(self): """ :avocado: tags=arch:aarch64
@@ -250,7 +248,6 @@ class ReverseDebugging_ppc64(ReverseDebugging): # unidentified gitlab timeout problem @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_ppc64_pseries(self): """ :avocado: tags=arch:ppc64
@@ -265,7 +262,6 @@ def test_ppc64_pseries(self): # See https://gitlab.com/qemu-project/qemu/-/issues/1992 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test_ppc64_powernv(self): """ :avocado: tags=arch:ppc64
tests/avocado/smmu.py +0 -1
@@ -14,7 +14,6 @@ from avocado_qemu.linuxtest import LinuxTest @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - class SMMU(LinuxTest): """ :avocado: tags=accel:kvm
Signature · 1 line
2.45.2
Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator
Quoted 5 lines · Cleber Rosa
Commit
9b45cc993added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what they decorate, so the newlines added should *not* exist there.Signed-off-by Cleber Rosa <[email protected]>
Quoted 10 lines
--- tests/avocado/boot_linux_console.py | 1 - tests/avocado/intel_iommu.py | 1 - tests/avocado/linux_initrd.py | 1 - tests/avocado/machine_aspeed.py | 2 -- tests/avocado/machine_mips_malta.py | 2 -- tests/avocado/machine_rx_gdbsim.py | 2 -- tests/avocado/reverse_debugging.py | 4 ---- tests/avocado/smmu.py | 1 - 8 files changed, 14 deletions(-)
Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator
In reply to Cleber RosaQuoted 2 lines · Cleber Rosa
Commit
9b45cc993added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what
s/follow/directly precede/ ?
Apart from that:
Quoted 13 lines
they decorate, so the newlines added should *not* exist there.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/boot_linux_console.py | 1 - tests/avocado/intel_iommu.py | 1 - tests/avocado/linux_initrd.py | 1 - tests/avocado/machine_aspeed.py | 2 -- tests/avocado/machine_mips_malta.py | 2 -- tests/avocado/machine_rx_gdbsim.py | 2 -- tests/avocado/reverse_debugging.py | 4 ---- tests/avocado/smmu.py | 1 - 8 files changed, 14 deletions(-)
PATCH v2 3/9 tests/avocado: add cdrom permission related tests
In reply to Cleber RosaThe "media=cdrom" parameter is also used on some Avocado tests as a way to protect files from being written. The tests here bring a more fundamental check that this behavior can be trusted.
--- tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 tests/avocado/cdrom.py
tests/avocado/cdrom.py new file +41 -0
@@ -0,0 +1,41 @@ +# Simple functional tests for cdrom devices +# +# Copyright (c) 2023 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado.utils.iso9660 import iso9660 +from avocado_qemu import QemuSystemTest + + +class Cdrom(QemuSystemTest): + """ + :avocado: tags=block,cdrom,quick + :avocado: tags=machine:none + """ + def setUp(self): + super().setUp() + self.iso_path = os.path.join(self.workdir, "cdrom.iso") + iso = iso9660(self.iso_path) + iso.create() + iso.close() + + def test_plain_iso_rw(self): + self.vm.add_args('-drive', f'file={self.iso_path}') + self.vm.launch() + query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertFalse(query_block_result[0]["inserted"]["ro"]) + + def test_media_cdrom_ro(self): + self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom') + self.vm.launch() + query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertTrue(query_block_result[0]["inserted"]["ro"])
Signature · 1 line
2.45.2
Re: [PATCH v2 3/9] tests/avocado: add cdrom permission related tests
Quoted 47 lines · Cleber Rosa
The "media=cdrom" parameter is also used on some Avocado tests as a way to protect files from being written. The tests here bring a more fundamental check that this behavior can be trusted.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)create mode 100644
tests/avocado/cdrom.pytests/avocado/cdrom.py new file +31
-0@@ -0,0 +1,41 @@ +# Simple functional tests for cdrom devices +# +# Copyright (c) 2023 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado.utils.iso9660 import iso9660 +from avocado_qemu import QemuSystemTest + + +class Cdrom(QemuSystemTest): + """ + :avocado: tags=block,cdrom,quick + :avocado: tags=machine:none + """ + def setUp(self): + super().setUp() + self.iso_path = os.path.join(self.workdir, "cdrom.iso") + iso = iso9660(self.iso_path) + iso.create() + iso.close() + + def test_plain_iso_rw(self): + self.vm.add_args('-drive', f'file={self.iso_path}') + self.vm.launch()
tests/avocado/cdrom.py line 41
Quoted 10 lines
tests/avocado/cdrom.py new file +10
-0+ query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertFalse(query_block_result[0]["inserted"]["ro"]) + + def test_media_cdrom_ro(self): + self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom') + self.vm.launch() + query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertTrue(query_block_result[0]["inserted"]["ro"])
I think such a test would be a better fit for the qemu-iotests, e.g. tests/qemu-iotests/118 looks pretty similar already, maybe this could be added there instead?
Thomas
PATCH v2 4/9 tests/avocado: machine aarch64: standardize location and RO access
In reply to Cleber RosaThe tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.
While at it, let's use a single code style and hash for the ISO url.
--- tests/avocado/machine_aarch64_sbsaref.py | 2 +- tests/avocado/machine_aarch64_virt.py | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
@@ -129,7 +129,7 @@ def boot_alpine_linux(self, cpu): "-cpu", cpu, "-drive", - f"file={iso_path},format=raw", + f"file={iso_path},media=cdrom,format=raw", ) self.vm.launch()
tests/avocado/machine_aarch64_virt.py +7 -7
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self): :avocado: tags=machine:virt :avocado: tags=accel:tcg """ - iso_url = ('https://dl-cdn.alpinelinux.org/' - 'alpine/v3.17/releases/aarch64/' - 'alpine-standard-3.17.2-aarch64.iso') + iso_url = ( + "https://dl-cdn.alpinelinux.org/" + "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso" + ) - # Alpine use sha256 so I recalculated this myself - iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839' - iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1) + iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027" + iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash) self.vm.set_console() kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self): self.vm.add_args("-smp", "2", "-m", "1024") self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', 'edk2-aarch64-code.fd')) - self.vm.add_args("-drive", f"file={iso_path},format=raw") + self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw") self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
Signature · 1 line
2.45.2
PATCH v2 5/9 tests/avocado: simplify parameters on fetch_asset with name only
In reply to Cleber RosaWhen an asset has a single location, it's possible to use that URI as the name of the asset.
Reference: https://avocado-framework.readthedocs.io/en/103.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset
Reference: https://avocado-framework.readthedocs.io/en/103.0/api/test/avocado.html#avocado.Test.fetch_asset
--- tests/avocado/kvm_xen_guest.py | 2 +- tests/avocado/netdev-ethtool.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
@@ -40,7 +40,7 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=url, asset_hash=sha1) def common_vm_setup(self): # We also catch lack of KVM_XEN support if we fail to launch
tests/avocado/netdev-ethtool.py +1 -1
@@ -27,7 +27,7 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=url, asset_hash=sha1) def common_test_code(self, netdev, extra_args=None):
Signature · 1 line
2.45.2
PATCH v2 6/9 tests/avocado/boot_xen.py: fetch kernel during test setUp()
In reply to Cleber RosaThe kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.
--- tests/avocado/boot_xen.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest): timeout = 90 XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all' - def fetch_guest_kernel(self): + def setUp(self): + super(BootXen, self).setUp() + # Using my own built kernel - which works kernel_url = ('https://fileserver.linaro.org/' 's/JSsewXGZ6mqxPr5/download?path=%2F&files=' 'linux-5.9.9-arm64-ajb') kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83' - kernel_path = self.fetch_asset(kernel_url, - asset_hash=kernel_sha1) - - return kernel_path + self.kernel_path = self.fetch_asset(kernel_url, + asset_hash=kernel_sha1) def launch_xen(self, xen_path): """ Launch Xen with a dom0 guest kernel """ self.log.info("launch with xen_path: %s", xen_path) - kernel_path = self.fetch_guest_kernel() self.vm.set_console()
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path): '-append', self.XEN_COMMON_COMMAND_LINE, '-device', 'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0' - % (kernel_path)) + % (self.kernel_path)) self.vm.launch()
Signature · 1 line
2.45.2
PATCH v2 7/9 tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
In reply to Cleber RosaThis makes use of the avocado.utils.archive support for zstd.
In order to not duplicate code, the skip condition uses a private utility from the module which is going to become public in Avocado versions 103.1 LTS (and also in versions >= 107.0).
Reference: https://github.com/avocado-framework/avocado/pull/5996
Reference: https://github.com/avocado-framework/avocado/pull/5953
--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +6 -10
@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_command
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(), + 'Could not find "zstd", or it is not able to properly ' + 'decompress decompress the rootfs') def setUp(self): super().setUp() - # We need zstd for all the tuxrun tests - # See https://github.com/avocado-framework/avocado/issues/5609 - zstd = find_command('zstd', False) - if zstd is False: - self.cancel('Could not find "zstd", which is required to ' - 'decompress rootfs') - self.zstd = zstd - # Process the TuxRun specific tags, most machines work with # reasonable defaults but we sometimes need to tweak the # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None): asset_hash = isum, algorithm = "sha256") - cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4" - process.run(cmd) + archive.extract(disk_image_zst, os.path.join(self.workdir, + "rootfs.ext4")) if dt: dsum = csums.get(dt, None)
Signature · 1 line
2.45.2
Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 26 lines · Cleber Rosa
This makes use of the avocado.utils.archive support for zstd.
In order to not duplicate code, the skip condition uses a private utility from the module which is going to become public in Avocado versions 103.1 LTS (and also in versions >= 107.0).
Reference: https://github.com/avocado-framework/avocado/pull/5996
Reference: https://github.com/avocado-framework/avocado/pull/5953Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)tests/avocado/tuxrun_baselines.py +1
-0@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_command
tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 46
Quoted 6 lines
tests/avocado/tuxrun_baselines.py +3
-0@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(), + 'Could not find "zstd", or it is not able to properly ' + 'decompress decompress the rootfs')
One "decompress" would be enough.
Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 8 lines · Marcin Juszkiewicz
tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 46
Quoted 3 lines · Cleber Rosa
tests/avocado/tuxrun_baselines.py +3
-0+ @skipUnless(archive._probe_zstd_cmd(), + 'Could not find "zstd", or it is not able to properly ' + 'decompress decompress the rootfs')One "decompress" would be enough.
True! Thanks for spotting that.
Regards,
- Cleber.
PATCH v2 8/9 tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
In reply to Cleber RosaWhen the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.
--- tests/avocado/machine_aarch64_sbsaref.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu): "-cpu", cpu, "-drive", - f"file={img_path},format=raw", + f"file={img_path},format=raw,snapshot=on", ) self.vm.launch()
Signature · 1 line
2.45.2
Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
Quoted 4 lines · Cleber Rosa
When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.
?
Quoted 18 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/machine_aarch64_sbsaref.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/avocado/machine_aarch64_sbsaref.py +1
-1@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu): "-cpu", cpu, "-drive", - f"file={img_path},format=raw", + f"file={img_path},format=raw,snapshot=on", ) self.vm.launch()
Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
Quoted 9 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Cleber Rosa
When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.
Suggested-by Alex Bennée <[email protected]>?
Yes, sorry about missing that.
Quoted 3 lines
Reviewed-by Philippe Mathieu-Daudé <[email protected]>
Thanks for the review.
Regards,
- Cleber
PATCH v2 9/9 Avocado tests: allow for parallel execution of tests
In reply to Cleber RosaThe updated Avocado version allows for the execution of tests in parallel.
While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.
This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.
Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.
--- docs/devel/testing.rst | 12 ++++++++++++ tests/Makefile.include | 6 +++++- 2 files changed, 17 insertions(+), 1 deletion(-)
docs/devel/testing.rst +12 -0
@@ -998,6 +998,18 @@ of Avocado or ``make check-avocado``, and can also be queried using: pyvenv/bin/avocado list tests/avocado +To run tests in parallel, the ``AVOCADO_PARALLEL`` environment +variable can be defined with a value different than ``1`` (its default +value). Example: + + .. code:: + + make check-avocado AVOCADO_PARALLEL=4 + +Please exercise care when using parallel execution with the QEMU +Avocado tests as a higher system load can cause time sensitive tests +to timeout and be interrupted. + Manual Installation ~~~~~~~~~~~~~~~~~~~
tests/Makefile.include +5 -1
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results ifndef AVOCADO_TESTS AVOCADO_TESTS=tests/avocado endif +ifndef AVOCADO_PARALLEL + AVOCADO_PARALLEL=1 +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \ + -p timeout_factor=$(AVOCADO_PARALLEL) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")
Signature · 1 line
2.45.2
Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests
Quoted 20 lines · Cleber Rosa
The updated Avocado version allows for the execution of tests in parallel.
While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.
This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.
Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.
Signed-off-by Cleber Rosa <[email protected]>---
...
Quoted 15 lines
tests/Makefile.include +3
-0@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results ifndef AVOCADO_TESTS AVOCADO_TESTS=tests/avocado endif +ifndef AVOCADO_PARALLEL + AVOCADO_PARALLEL=1 +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help".
tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150
Quoted 8 lines
tests/Makefile.include +2
-1@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \ + -p timeout_factor=$(AVOCADO_PARALLEL) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")
I think it was nicer in the previous attempt to bump the avocado version:
https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783
This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.
Thomas
Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests
Quoted 27 lines · Thomas Huth
...
Quoted 15 lines
tests/Makefile.include +3
-0@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results ifndef AVOCADO_TESTS AVOCADO_TESTS=tests/avocado endif +ifndef AVOCADO_PARALLEL + AVOCADO_PARALLEL=1 +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help".tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150
Quoted 8 lines
tests/Makefile.include +2
-1@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \ + -p timeout_factor=$(AVOCADO_PARALLEL) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")I think it was nicer in the previous attempt to bump the avocado version:
Quoted 5 lines
https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783
This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.
Hi Thomas,
I can see why it looks better, but in practice, I'm not getting the best behavior with such a change.
First, the fact that it enables the parallelization by default, while there still seems to be issues with test timeout issues, and even existing races between tests (which this series tried to address as much as possible) will not result in the best experience IMO. On my 12 core machine, and also on GitLab CI, having 4 tests running in parallel gets a nice speed up (as others have reported) while still being very stable.
I'd say making the number of parallel tests equal to nproc is best kept for a future round.
Let me know if this sounds reasonable to you.
Regards,
- Cleber.
Quoted 2 lines
Thomas
Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests
Quoted 47 lines · Cleber Rosa
Quoted 27 lines · Thomas Huth
...
Quoted 15 lines
tests/Makefile.include +3
-0@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results ifndef AVOCADO_TESTS AVOCADO_TESTS=tests/avocado endif +ifndef AVOCADO_PARALLEL + AVOCADO_PARALLEL=1 +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help".tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150
Quoted 8 lines
tests/Makefile.include +2
-1@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \ + -p timeout_factor=$(AVOCADO_PARALLEL) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")I think it was nicer in the previous attempt to bump the avocado version:
Quoted 5 lines
https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783
This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.
Hi Thomas,
I can see why it looks better, but in practice, I'm not getting the best behavior with such a change.
First, the fact that it enables the parallelization by default, while there still seems to be issues with test timeout issues, and even existing races between tests (which this series tried to address as much as possible) will not result in the best experience IMO. On my 12 core machine, and also on GitLab CI, having 4 tests running in parallel gets a nice speed up (as others have reported) while still being very stable.
Quoted 4 lines
I'd say making the number of parallel tests equal to
nprocis best kept for a future round.Let me know if this sounds reasonable to you.
Hi Cleber,
that patch that I linked did not set the default number of parallel tests to $(nproc), it just used the value of the "-j" option of make. So if you just run "make check-avocado" there, you only get single threaded execution as before. You explicitely have to run "make -jX check-avocado" to get X parallel threads. IMHO using "-j" is more intuitive than using yet another environment variable.
Thomas
Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests
Quoted 3 lines · Thomas Huth
that patch that I linked did not set the default number of parallel tests to $(nproc), it just used the value of the "-j" option of make. So if you just run "make check-avocado" there, you only get single threaded execution as before. You explicitely have to run "make -jX check-avocado" to get X parallel threads. IMHO using "-j" is more intuitive than using
yet another environment variable.
Seconded.
r~
Quoted 9 lines · Cleber Rosa
This is along overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.
These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.
Nevertheless, as pointed out, on the ML, these changes are needed NOW.
SBSA Reference Platform tests can be done in 1/3rd of time is a nice update.
Serial run:
real 6m20,324s user 12m18,446s sys 0m36,686s
Parallel (4) run:
real 2m22,658s user 11m50,514s sys 0m26,088s