Skip to contentSkip to conversation

Bump Avocado to 103.0 LTS and update tests for compatibility and new features

PATCH In review Cleber Rosa 5 participants 22 messages Aug 6, 2024 – Aug 16, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover Bump Avocado to 103.0 LTS and update tests for compatibility and new features

This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.

These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 s

Changes from v1 (references are from v1):

This version first applies the Avocado version bump (now PATCH 1), adding to that patch the serial execution behavior. Avocado 103.0 LTS can run all existing tests in QEMU without any changes, so it's safe to pick PATCH 1 *only* from this series if needed. A GitLab CI job that does only that can be seen here:

https://gitlab.com/cleber.gnu/qemu/-/pipelines/1402633650

Details on changes:

* Moved "Bump avocado to 103.0" to first patch

* Patches already applied:
- [PATCH 1/13] tests/avocado: mips: fallback to HTTP given certificate expiration
- [PATCH 2/13] tests/avocado: mips: add hint for fetchasset plugin
- [PATCH 8/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

* Dropped patches:
- [PATCH 3/13] tests/avocado/intel_iommu.py: increase timeout
- [PATCH 6/13] tests/avocado: use more distinct names for assets
- Replaced with "tests/avocado: simplify parameters on fetch_asset with name only"
- [PATCH 7/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

* [PATCH 5/13] tests/avocado: machine aarch64: standardize location and RO access:
- Fixed rebase mistake

* [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
- Use "snapshot=on" instead of new copy of file

* New patches:
- tests/avocado: apply proper skipUnless decorator
- tests/avocado: simplify parameters on fetch_asset with name only

Cleber Rosa (9):
Bump avocado to 103.0
tests/avocado: apply proper skipUnless decorator tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: simplify parameters on fetch_asset with name only tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Avocado tests: allow for parallel execution of tests

 docs/devel/testing.rst                   | 12 +++++++
 pythondeps.toml                          |  2 +-
 tests/Makefile.include                   |  6 +++-
 tests/avocado/boot_linux_console.py      |  1 -
 tests/avocado/boot_xen.py                | 13 ++++----
 tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
 tests/avocado/intel_iommu.py             |  1 -
 tests/avocado/kvm_xen_guest.py           |  2 +-
 tests/avocado/linux_initrd.py            |  1 -
 tests/avocado/machine_aarch64_sbsaref.py |  4 +--
 tests/avocado/machine_aarch64_virt.py    | 14 ++++----
 tests/avocado/machine_aspeed.py          |  2 --
 tests/avocado/machine_mips_malta.py      |  2 --
 tests/avocado/machine_rx_gdbsim.py       |  2 --
 tests/avocado/netdev-ethtool.py          |  2 +-
 tests/avocado/reverse_debugging.py       |  4 ---
 tests/avocado/smmu.py                    |  1 -
 tests/avocado/tuxrun_baselines.py        | 16 ++++-----
 18 files changed, 82 insertions(+), 44 deletions(-)
 create mode 100644 tests/avocado/cdrom.py
Signature · 1 line
2.45.2
#1 lore raw

PATCH v2 1/9 Bump avocado to 103.0

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.

To keep behavior of jobs as close as possible with previous version, this version bump keeps the execution serial (maximum of one task at a time being run).

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

---
 pythondeps.toml        | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
pythondeps.toml +1 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
tests/Makefile.include +1 -1
index 6618bfed70..537804d101 100644
@@ -141,7 +141,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
Signature · 1 line
2.45.2
#2 lore raw

PATCH v2 2/9 tests/avocado: apply proper skipUnless decorator

In reply to Cleber Rosa

Commit 9b45cc993 added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what they decorate, so the newlines added should *not* exist there.

---
 tests/avocado/boot_linux_console.py | 1 -
 tests/avocado/intel_iommu.py        | 1 -
 tests/avocado/linux_initrd.py       | 1 -
 tests/avocado/machine_aspeed.py     | 2 --
 tests/avocado/machine_mips_malta.py | 2 --
 tests/avocado/machine_rx_gdbsim.py  | 2 --
 tests/avocado/reverse_debugging.py  | 4 ----
 tests/avocado/smmu.py               | 1 -
 8 files changed, 14 deletions(-)
tests/avocado/boot_linux_console.py +0 -1
index 2929aa042d..cffdd6b5a2 100644
@@ -1522,7 +1522,6 @@ def test_ppc_mac99(self):
     # like issues with a buggy kernel. As a result we don't want it
     # gating releases on Gitlab.
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_sh4_r2d(self):
         """
         :avocado: tags=arch:sh4
tests/avocado/intel_iommu.py +0 -1
index 008f214397..992583fa7d 100644
@@ -13,7 +13,6 @@
 from avocado_qemu.linuxtest import LinuxTest
 
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
 class IntelIOMMU(LinuxTest):
     """
     :avocado: tags=arch:x86_64
tests/avocado/linux_initrd.py +0 -1
index aad5b19bd9..7f47b98ae7 100644
@@ -54,7 +54,6 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
             self.assertRegex(self.vm.get_log(), expected_msg)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
         :avocado: tags=flaky
tests/avocado/machine_aspeed.py +0 -2
index f8e263d37e..c0b01e8f1f 100644
@@ -323,7 +323,6 @@ def do_test_aarch64_aspeed_sdk_start(self, image):
         self.wait_for_console_pattern('Starting kernel ...')
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_arm_ast2500_evb_sdk(self):
         """
         :avocado: tags=arch:arm
@@ -343,7 +342,6 @@ def test_arm_ast2500_evb_sdk(self):
         self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4')
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_arm_ast2600_evb_sdk(self):
         """
         :avocado: tags=arch:arm
tests/avocado/machine_mips_malta.py +0 -2
index 8cf84bd805..07a80633b5 100644
@@ -102,7 +102,6 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self):
         self.do_test_i6400_framebuffer_logo(1)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         """
         :avocado: tags=arch:mips64el
@@ -114,7 +113,6 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         self.do_test_i6400_framebuffer_logo(7)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_mips_malta_i6400_framebuffer_logo_8cores(self):
         """
         :avocado: tags=arch:mips64el
tests/avocado/machine_rx_gdbsim.py +0 -2
index 412a7a5089..72f0296e21 100644
@@ -23,7 +23,6 @@ class RxGdbSimMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_uboot(self):
         """
         U-Boot and checks that the console is operational.
@@ -49,7 +48,6 @@ def test_uboot(self):
         #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
tests/avocado/reverse_debugging.py +0 -4
index 92855a02a5..f24287cd0a 100644
@@ -207,7 +207,6 @@ def get_pc(self, g):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -225,7 +224,6 @@ class ReverseDebugging_AArch64(ReverseDebugging):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_aarch64_virt(self):
         """
         :avocado: tags=arch:aarch64
@@ -250,7 +248,6 @@ class ReverseDebugging_ppc64(ReverseDebugging):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64
@@ -265,7 +262,6 @@ def test_ppc64_pseries(self):
 
     # See https://gitlab.com/qemu-project/qemu/-/issues/1992
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_ppc64_powernv(self):
         """
         :avocado: tags=arch:ppc64
tests/avocado/smmu.py +0 -1
index aadda71e4b..83fd79e922 100644
@@ -14,7 +14,6 @@
 from avocado_qemu.linuxtest import LinuxTest
 
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
 class SMMU(LinuxTest):
     """
     :avocado: tags=accel:kvm
Signature · 1 line
2.45.2
Reviewed-by on v2 2/9
#3 lore raw

Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator

Quoted 5 lines · Cleber Rosa

Commit 9b45cc993 added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what they decorate, so the newlines added should *not* exist there.

Quoted 10 lines
---
  tests/avocado/boot_linux_console.py | 1 -
  tests/avocado/intel_iommu.py        | 1 -
  tests/avocado/linux_initrd.py       | 1 -
  tests/avocado/machine_aspeed.py     | 2 --
  tests/avocado/machine_mips_malta.py | 2 --
  tests/avocado/machine_rx_gdbsim.py  | 2 --
  tests/avocado/reverse_debugging.py  | 4 ----
  tests/avocado/smmu.py               | 1 -
  8 files changed, 14 deletions(-)
Reviewed-by on v2 2/9
#4 lore raw

Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator

In reply to Cleber Rosa
Quoted 2 lines · Cleber Rosa

Commit 9b45cc993 added many cases of skipUnless for the sake of organizing flaky tests. But, Python decorators *must* follow what

s/follow/directly precede/ ?

Apart from that:

Quoted 13 lines

they decorate, so the newlines added should *not* exist there.

---
  tests/avocado/boot_linux_console.py | 1 -
  tests/avocado/intel_iommu.py        | 1 -
  tests/avocado/linux_initrd.py       | 1 -
  tests/avocado/machine_aspeed.py     | 2 --
  tests/avocado/machine_mips_malta.py | 2 --
  tests/avocado/machine_rx_gdbsim.py  | 2 --
  tests/avocado/reverse_debugging.py  | 4 ----
  tests/avocado/smmu.py               | 1 -
  8 files changed, 14 deletions(-)
#5 lore raw

PATCH v2 3/9 tests/avocado: add cdrom permission related tests

In reply to Cleber Rosa

The "media=cdrom" parameter is also used on some Avocado tests as a way to protect files from being written. The tests here bring a more fundamental check that this behavior can be trusted.

---
 tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 tests/avocado/cdrom.py
tests/avocado/cdrom.py new file +41 -0
new file mode 100644
index 0000000000..c9aa5d69cb
@@ -0,0 +1,41 @@
+# Simple functional tests for cdrom devices
+#
+# Copyright (c) 2023 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado.utils.iso9660 import iso9660
+from avocado_qemu import QemuSystemTest
+
+
+class Cdrom(QemuSystemTest):
+    """
+    :avocado: tags=block,cdrom,quick
+    :avocado: tags=machine:none
+    """
+    def setUp(self):
+        super().setUp()
+        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
+        iso = iso9660(self.iso_path)
+        iso.create()
+        iso.close()
+
+    def test_plain_iso_rw(self):
+        self.vm.add_args('-drive', f'file={self.iso_path}')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertFalse(query_block_result[0]["inserted"]["ro"])
+
+    def test_media_cdrom_ro(self):
+        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertTrue(query_block_result[0]["inserted"]["ro"])
Signature · 1 line
2.45.2
#6 lore raw

Re: [PATCH v2 3/9] tests/avocado: add cdrom permission related tests

Quoted 47 lines · Cleber Rosa

The "media=cdrom" parameter is also used on some Avocado tests as a way to protect files from being written. The tests here bring a more fundamental check that this behavior can be trusted.

---
  tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
  1 file changed, 41 insertions(+)

create mode 100644 tests/avocado/cdrom.py

tests/avocado/cdrom.py new file +31 -0
new file mode 100644
index 0000000000..c9aa5d69cb
@@ -0,0 +1,41 @@
+# Simple functional tests for cdrom devices
+#
+# Copyright (c) 2023 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado.utils.iso9660 import iso9660
+from avocado_qemu import QemuSystemTest
+
+
+class Cdrom(QemuSystemTest):
+    """
+    :avocado: tags=block,cdrom,quick
+    :avocado: tags=machine:none
+    """
+    def setUp(self):
+        super().setUp()
+        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
+        iso = iso9660(self.iso_path)
+        iso.create()
+        iso.close()
+
+    def test_plain_iso_rw(self):
+        self.vm.add_args('-drive', f'file={self.iso_path}')
+        self.vm.launch()

tests/avocado/cdrom.py line 41

Quoted 10 lines
tests/avocado/cdrom.py new file +10 -0
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertFalse(query_block_result[0]["inserted"]["ro"])
+
+    def test_media_cdrom_ro(self):
+        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertTrue(query_block_result[0]["inserted"]["ro"])

I think such a test would be a better fit for the qemu-iotests, e.g. tests/qemu-iotests/118 looks pretty similar already, maybe this could be added there instead?

Thomas

#7 lore raw

PATCH v2 4/9 tests/avocado: machine aarch64: standardize location and RO access

In reply to Cleber Rosa

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

---
 tests/avocado/machine_aarch64_sbsaref.py |  2 +-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
index e920bbf08c..756f316ac9 100644
@@ -129,7 +129,7 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
         )
 
         self.vm.launch()
tests/avocado/machine_aarch64_virt.py +7 -7
index a90dc6ff4b..6831d2c0ed 100644
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
Signature · 1 line
2.45.2
#8 lore raw

PATCH v2 5/9 tests/avocado: simplify parameters on fetch_asset with name only

In reply to Cleber Rosa

When an asset has a single location, it's possible to use that URI as the name of the asset.

Reference: https://avocado-framework.readthedocs.io/en/103.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset
Reference: https://avocado-framework.readthedocs.io/en/103.0/api/test/avocado.html#avocado.Test.fetch_asset

---
 tests/avocado/kvm_xen_guest.py  | 2 +-
 tests/avocado/netdev-ethtool.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
index f8cb458d5d..b6ee1ff752 100644
@@ -40,7 +40,7 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=url, asset_hash=sha1)
 
     def common_vm_setup(self):
         # We also catch lack of KVM_XEN support if we fail to launch
tests/avocado/netdev-ethtool.py +1 -1
index 5f33288f81..7345ded28c 100644
@@ -27,7 +27,7 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=url, asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
 
Signature · 1 line
2.45.2
#9 lore raw

PATCH v2 6/9 tests/avocado/boot_xen.py: fetch kernel during test setUp()

In reply to Cleber Rosa

The kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.

---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
index f29bc58b9e..490a127a3e 100644
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
Signature · 1 line
2.45.2
#10 lore raw

PATCH v2 7/9 tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

In reply to Cleber Rosa

This makes use of the avocado.utils.archive support for zstd.

In order to not duplicate code, the skip condition uses a private utility from the module which is going to become public in Avocado versions 103.1 LTS (and also in versions >= 107.0).

Reference: https://github.com/avocado-framework/avocado/pull/5996
Reference: https://github.com/avocado-framework/avocado/pull/5953

---
 tests/avocado/tuxrun_baselines.py | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +6 -10
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
 from avocado.utils import process
 from avocado.utils.path import find_command
 
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
         return default
 
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')
     def setUp(self):
         super().setUp()
 
-        # We need zstd for all the tuxrun tests
-        # See https://github.com/avocado-framework/avocado/issues/5609
-        zstd = find_command('zstd', False)
-        if zstd is False:
-            self.cancel('Could not find "zstd", which is required to '
-                        'decompress rootfs')
-        self.zstd = zstd
-
         # Process the TuxRun specific tags, most machines work with
         # reasonable defaults but we sometimes need to tweak the
         # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
                                          asset_hash = isum,
                                          algorithm = "sha256")
 
-        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
-        process.run(cmd)
+        archive.extract(disk_image_zst, os.path.join(self.workdir,
+                                                     "rootfs.ext4"))
 
         if dt:
             dsum = csums.get(dt, None)
Signature · 1 line
2.45.2
#11 lore raw

Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 26 lines · Cleber Rosa

This makes use of the avocado.utils.archive support for zstd.

In order to not duplicate code, the skip condition uses a private utility from the module which is going to become public in Avocado versions 103.1 LTS (and also in versions >= 107.0).

Reference: https://github.com/avocado-framework/avocado/pull/5996
Reference: https://github.com/avocado-framework/avocado/pull/5953

---
  tests/avocado/tuxrun_baselines.py | 16 ++++++----------
  1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +1 -0
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
  from avocado_qemu import QemuSystemTest
  from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
  from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
  from avocado.utils import process
  from avocado.utils.path import find_command
 

tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 46

Quoted 6 lines
tests/avocado/tuxrun_baselines.py +3 -0
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
          return default
 
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')

One "decompress" would be enough.

#12 lore raw

Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 8 lines · Marcin Juszkiewicz

tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 46

Quoted 3 lines · Cleber Rosa
tests/avocado/tuxrun_baselines.py +3 -0
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')

One "decompress" would be enough.

True! Thanks for spotting that.

Regards,
- Cleber.

#13 lore raw

PATCH v2 8/9 tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

In reply to Cleber Rosa

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.

---
 tests/avocado/machine_aarch64_sbsaref.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
index 756f316ac9..f8bf40c192 100644
@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={img_path},format=raw",
+            f"file={img_path},format=raw,snapshot=on",
         )
 
         self.vm.launch()
Signature · 1 line
2.45.2
Reviewed-by on v2 8/9
#14 lore raw

Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

Quoted 4 lines · Cleber Rosa

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.

?

Quoted 18 lines
---
  tests/avocado/machine_aarch64_sbsaref.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -1
index 756f316ac9..f8bf40c192 100644
@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu):
              "-cpu",
              cpu,
              "-drive",
-            f"file={img_path},format=raw",
+            f"file={img_path},format=raw,snapshot=on",
          )
 
          self.vm.launch()
#15 lore raw

Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

Quoted 9 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Cleber Rosa

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own snapshot.

?

Yes, sorry about missing that.

Quoted 3 lines

Thanks for the review.

Regards,
- Cleber

#16 lore raw

PATCH v2 9/9 Avocado tests: allow for parallel execution of tests

In reply to Cleber Rosa

The updated Avocado version allows for the execution of tests in parallel.

While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.

This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.

Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.

---
 docs/devel/testing.rst | 12 ++++++++++++
 tests/Makefile.include |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
docs/devel/testing.rst +12 -0
index af73d3d64f..97ebc8211f 100644
@@ -998,6 +998,18 @@ of Avocado or ``make check-avocado``, and can also be queried using:
 
   pyvenv/bin/avocado list tests/avocado
 
+To run tests in parallel, the ``AVOCADO_PARALLEL`` environment
+variable can be defined with a value different than ``1`` (its default
+value).  Example:
+
+ .. code::
+
+  make check-avocado AVOCADO_PARALLEL=4
+
+Please exercise care when using parallel execution with the QEMU
+Avocado tests as a higher system load can cause time sensitive tests
+to timeout and be interrupted.
+
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
 
tests/Makefile.include +5 -1
index 537804d101..545b5155f9 100644
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
 endif
+ifndef AVOCADO_PARALLEL
+	AVOCADO_PARALLEL=1
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+			-p timeout_factor=$(AVOCADO_PARALLEL) \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
Signature · 1 line
2.45.2
#17 lore raw

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

Quoted 20 lines · Cleber Rosa

The updated Avocado version allows for the execution of tests in parallel.

While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.

This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.

Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.

---

...

Quoted 15 lines
tests/Makefile.include +3 -0
index 537804d101..545b5155f9 100644
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
  ifndef AVOCADO_TESTS
  	AVOCADO_TESTS=tests/avocado
  endif
+ifndef AVOCADO_PARALLEL
+	AVOCADO_PARALLEL=1
+endif
  # Controls the output generated by Avocado when running tests.
  # Any number of command separated loggers are accepted.  For more
  # information please refer to "avocado --help".

tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150

Quoted 8 lines
tests/Makefile.include +2 -1
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
              $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
  			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+			-p timeout_factor=$(AVOCADO_PARALLEL) \
              $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
              "AVOCADO", "tests/avocado")

I think it was nicer in the previous attempt to bump the avocado version:

https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783

This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.

Thomas

#18 lore raw

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

Quoted 27 lines · Thomas Huth

...

Quoted 15 lines
tests/Makefile.include +3 -0
index 537804d101..545b5155f9 100644
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
  ifndef AVOCADO_TESTS
      AVOCADO_TESTS=tests/avocado
  endif
+ifndef AVOCADO_PARALLEL
+     AVOCADO_PARALLEL=1
+endif
  # Controls the output generated by Avocado when running tests.
  # Any number of command separated loggers are accepted.  For more
  # information please refer to "avocado --help".

tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150

Quoted 8 lines
tests/Makefile.include +2 -1
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
              $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
                      --filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+                     -p timeout_factor=$(AVOCADO_PARALLEL) \
              $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
              "AVOCADO", "tests/avocado")

I think it was nicer in the previous attempt to bump the avocado version:

Quoted 5 lines

https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783

This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.

Hi Thomas,

I can see why it looks better, but in practice, I'm not getting the best behavior with such a change.

First, the fact that it enables the parallelization by default, while there still seems to be issues with test timeout issues, and even existing races between tests (which this series tried to address as much as possible) will not result in the best experience IMO. On my 12 core machine, and also on GitLab CI, having 4 tests running in parallel gets a nice speed up (as others have reported) while still being very stable.

I'd say making the number of parallel tests equal to nproc is best kept for a future round.

Let me know if this sounds reasonable to you.

Regards,
- Cleber.

Quoted 2 lines

Thomas

#19 lore raw

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

Quoted 47 lines · Cleber Rosa
Quoted 27 lines · Thomas Huth

...

Quoted 15 lines
tests/Makefile.include +3 -0
index 537804d101..545b5155f9 100644
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
   ifndef AVOCADO_TESTS
       AVOCADO_TESTS=tests/avocado
   endif
+ifndef AVOCADO_PARALLEL
+     AVOCADO_PARALLEL=1
+endif
   # Controls the output generated by Avocado when running tests.
   # Any number of command separated loggers are accepted.  For more
   # information please refer to "avocado --help".

tests/Makefile.include check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images · line 150

Quoted 8 lines
tests/Makefile.include +2 -1
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
               $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
                       --filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+                     -p timeout_factor=$(AVOCADO_PARALLEL) \
               $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
               "AVOCADO", "tests/avocado")

I think it was nicer in the previous attempt to bump the avocado version:

Quoted 5 lines

https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783

This re-used the "-j" option from "make", so you could do "make -j$(nproc) check-avocado" just like with the other "check" targets.

Hi Thomas,

I can see why it looks better, but in practice, I'm not getting the best behavior with such a change.

First, the fact that it enables the parallelization by default, while there still seems to be issues with test timeout issues, and even existing races between tests (which this series tried to address as much as possible) will not result in the best experience IMO. On my 12 core machine, and also on GitLab CI, having 4 tests running in parallel gets a nice speed up (as others have reported) while still being very stable.

Quoted 4 lines

I'd say making the number of parallel tests equal to nproc is best kept for a future round.

Let me know if this sounds reasonable to you.

Hi Cleber,

that patch that I linked did not set the default number of parallel tests to $(nproc), it just used the value of the "-j" option of make. So if you just run "make check-avocado" there, you only get single threaded execution as before. You explicitely have to run "make -jX check-avocado" to get X parallel threads. IMHO using "-j" is more intuitive than using yet another environment variable.

Thomas

#20 lore raw

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

Quoted 3 lines · Thomas Huth

that patch that I linked did not set the default number of parallel tests to $(nproc), it just used the value of the "-j" option of make. So if you just run "make check-avocado" there, you only get single threaded execution as before. You explicitely have to run "make -jX check-avocado" to get X parallel threads. IMHO using "-j" is more intuitive than using
yet another environment variable.

Seconded.

r~

Tested-by on the v2 series
#21 lore raw
In reply to Cleber Rosa
Quoted 9 lines · Cleber Rosa

This is along overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.

These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

SBSA Reference Platform tests can be done in 1/3rd of time is a nice update.

Serial run:

real    6m20,324s
user    12m18,446s
sys     0m36,686s

Parallel (4) run:

real    2m22,658s
user    11m50,514s
sys     0m26,088s