Skip to contentSkip to conversation

Bump Avocado to 103.0 LTS and update tests for compatibility and new features

PATCH Superseded Cleber Rosa 7 participants 44 messages Jul 26, 2024 – Aug 12, 2024 KVM

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Bump Avocado to 103.0 LTS and update tests for compatibility and new features

This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.

These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 s

Cleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugin tests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: use more distinct names for assets tests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of tests

 docs/devel/testing.rst                   | 12 +++++++
 pythondeps.toml                          |  2 +-
 tests/Makefile.include                   |  6 +++-
 tests/avocado/boot_linux_console.py      | 24 ++++++++------
 tests/avocado/boot_xen.py                | 13 ++++----
 tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
 tests/avocado/intel_iommu.py             |  2 ++
 tests/avocado/kvm_xen_guest.py           | 30 +++++++++++------
 tests/avocado/machine_aarch64_sbsaref.py | 11 +++++--
 tests/avocado/machine_aarch64_virt.py    | 14 ++++----
 tests/avocado/netdev-ethtool.py          |  3 +-
 tests/avocado/tuxrun_baselines.py        | 16 ++++-----
 12 files changed, 125 insertions(+), 49 deletions(-)
 create mode 100644 tests/avocado/cdrom.py
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/13 tests/avocado: mips: fallback to HTTP given certificate expiration

The SSL certificate installed at mipsdistros.mips.com has expired:

0 s:CN = mipsdistros.mips.com
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMT

Because this project has no control over that certificate and host, this falls back to plain HTTP instead. The integrity of the downloaded files can be guaranteed by the existing hashes for those files (which are not modified here).

---
 tests/avocado/boot_linux_console.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tests/avocado/boot_linux_console.py +3 -3
index c35fc5e9ba..450d67be6a 100644
@@ -299,7 +299,7 @@ def test_mips_malta32el_nanomips_4k(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
@@ -312,7 +312,7 @@ def test_mips_malta32el_nanomips_16k_up(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
@@ -325,7 +325,7 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
Signature · 1 line
2.45.2
Tested-by on v1 1/13
#2 lore raw

Re: [PATCH 01/13] tests/avocado: mips: fallback to HTTP given certificate expiration

Quoted 12 lines · Cleber Rosa

The SSL certificate installed at mipsdistros.mips.com has expired:

0 s:CN = mipsdistros.mips.com
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMT

Because this project has no control over that certificate and host, this falls back to plain HTTP instead. The integrity of the downloaded files can be guaranteed by the existing hashes for those files (which are not modified here).

Quoted 5 lines
---
  tests/avocado/boot_linux_console.py | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
#3 lore raw

PATCH 2/13 tests/avocado: mips: add hint for fetchasset plugin

In reply to Cleber Rosa

Avocado's fetchasset plugin runs before the actual Avocado job (and any test). It analyses the test's code looking for occurrences of "self.fetch_asset()" in the either the actual test or setUp() method. It's not able to fully analyze all code, though.

The way these tests are written, make the fetchasset plugin blind to the assets. This adds some more code duplication, true, but it will aid the fetchasset plugin to download or verify the existence of these assets in advance.

---
 tests/avocado/boot_linux_console.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
tests/avocado/boot_linux_console.py +7 -5
index 450d67be6a..b8b0a4df10 100644
@@ -274,8 +274,7 @@ def test_mips64el_malta_5KEc_cpio(self):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
-    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
-        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+    def do_test_mips_malta32el_nanomips(self, kernel_path_xz):
         kernel_path = self.workdir + "kernel"
         with lzma.open(kernel_path_xz, 'rb') as f_in:
             with open(kernel_path, 'wb') as f_out:
@@ -303,7 +302,8 @@ def test_mips_malta32el_nanomips_4k(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_mips_malta32el_nanomips_16k_up(self):
         """
@@ -316,7 +316,8 @@ def test_mips_malta32el_nanomips_16k_up(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_mips_malta32el_nanomips_64k_dbg(self):
         """
@@ -329,7 +330,8 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_aarch64_xlnx_versal_virt(self):
         """
Signature · 1 line
2.45.2
Tested-by on v1 2/13
#4 lore raw

Re: [PATCH 02/13] tests/avocado: mips: add hint for fetchasset plugin

Quoted 10 lines · Cleber Rosa

Avocado's fetchasset plugin runs before the actual Avocado job (and any test). It analyses the test's code looking for occurrences of "self.fetch_asset()" in the either the actual test or setUp() method. It's not able to fully analyze all code, though.

The way these tests are written, make the fetchasset plugin blind to the assets. This adds some more code duplication, true, but it will aid the fetchasset plugin to download or verify the existence of these assets in advance.

Quoted 4 lines
---
  tests/avocado/boot_linux_console.py | 12 +++++++-----
  1 file changed, 7 insertions(+), 5 deletions(-)
#5 lore raw

PATCH 3/13 tests/avocado/intel_iommu.py: increase timeout

In reply to Cleber Rosa

Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience.

Let's increase the timeout so these tests get a chance to completion.

---
 tests/avocado/intel_iommu.py | 2 ++
 1 file changed, 2 insertions(+)
tests/avocado/intel_iommu.py +2 -0
index 008f214397..9e7965c5df 100644
@@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest):
     :avocado: tags=flaky
     """
 
+    timeout = 360
+
     IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
     kernel_path = None
     initrd_path = None
Signature · 1 line
2.45.2
#6 lore raw

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

Quoted 4 lines · Cleber Rosa

Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.

A high watermark of over 5 minutes is pretty long for a test.

Looking at the test I see it runs

self.ssh_command('dnf -y install numactl-devel')

but then never actually uses the installed package.

I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.

If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.

Quoted 22 lines
---
 tests/avocado/intel_iommu.py | 2 ++
 1 file changed, 2 insertions(+)
tests/avocado/intel_iommu.py +2 -0
index 008f214397..9e7965c5df 100644
@@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest):
     :avocado: tags=flaky
     """
 
+    timeout = 360
+
     IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
     kernel_path = None
     initrd_path = None
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

Quoted 9 lines · Daniel P. Berrangé
Quoted 4 lines · Cleber Rosa

Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.

A high watermark of over 5 minutes is pretty long for a test.

I agree.

Quoted 9 lines

Looking at the test I see it runs

self.ssh_command('dnf -y install numactl-devel')

but then never actually uses the installed package.

I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.

Exactly.

Quoted 5 lines

If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.

I was trying not to get in the way of the original test writer.

Eric,

Are you OK with replacing this command for a simpler file transfer? Any suggestions?

Regards,
- Cleber.

#8 lore raw

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

Hi Cleber,

Quoted 26 lines · Cleber Rosa
Quoted 7 lines · Daniel P. Berrangé
Quoted 4 lines · Cleber Rosa

Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.

A high watermark of over 5 minutes is pretty long for a test.

I agree.

Quoted 9 lines

Looking at the test I see it runs

self.ssh_command('dnf -y install numactl-devel')

but then never actually uses the installed package.

I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.

Exactly.

Quoted 5 lines

If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.

Quoted 6 lines

I was trying not to get in the way of the original test writer.

Eric,

Are you OK with replacing this command for a simpler file transfer? Any suggestions?

Sorry I am just returning from PTO. Yes I am OK to replace it with a file transfer if it takes too much time. When the test was written we just found out a bug with that exact dnf install command so at that time this was a reproducer. Either replace it with a dummy file transfer or withdraw it from the automated tests.

Thanks

Eric

Quoted 4 lines

Regards,
- Cleber.

#9 lore raw

PATCH 4/13 tests/avocado: add cdrom permission related tests

In reply to Cleber Rosa
---
 tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 tests/avocado/cdrom.py
tests/avocado/cdrom.py new file +41 -0
new file mode 100644
index 0000000000..c9aa5d69cb
@@ -0,0 +1,41 @@
+# Simple functional tests for cdrom devices
+#
+# Copyright (c) 2023 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado.utils.iso9660 import iso9660
+from avocado_qemu import QemuSystemTest
+
+
+class Cdrom(QemuSystemTest):
+    """
+    :avocado: tags=block,cdrom,quick
+    :avocado: tags=machine:none
+    """
+    def setUp(self):
+        super().setUp()
+        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
+        iso = iso9660(self.iso_path)
+        iso.create()
+        iso.close()
+
+    def test_plain_iso_rw(self):
+        self.vm.add_args('-drive', f'file={self.iso_path}')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertFalse(query_block_result[0]["inserted"]["ro"])
+
+    def test_media_cdrom_ro(self):
+        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertTrue(query_block_result[0]["inserted"]["ro"])
Signature · 1 line
2.45.2
#10 lore raw

PATCH 5/13 tests/avocado: machine aarch64: standardize location and RO access

In reply to Cleber Rosa

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

---
 tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +5 -1
index e920bbf08c..1275f24532 100644
@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
+            "-device",
+            "virtio-rng-pci,rng=rng0",
+            "-object",
+            "rng-random,id=rng0,filename=/dev/urandom",
         )
 
         self.vm.launch()
tests/avocado/machine_aarch64_virt.py +7 -7
index a90dc6ff4b..6831d2c0ed 100644
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
Signature · 1 line
2.45.2
#11 lore raw

Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access

Quoted 17 lines · Cleber Rosa

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

---
 tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +0 -0
index e920bbf08c..1275f24532 100644

tests/avocado/machine_aarch64_sbsaref.py def boot_alpine_linux(self, cpu): · line 137

Quoted 10 lines
tests/avocado/machine_aarch64_sbsaref.py +5 -1
@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
+            "-device",
+            "virtio-rng-pci,rng=rng0",
+            "-object",
+            "rng-random,id=rng0,filename=/dev/urandom",
         )

The commit message doesn't say anything about adding virtio-rng. If that's needed for some reason, do it as a separate commit with an explanation of the bug its fixing.

Quoted 37 lines
tests/avocado/machine_aarch64_virt.py +7 -7
index a90dc6ff4b..6831d2c0ed 100644
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access

Quoted 25 lines · Daniel P. Berrangé
Quoted 23 lines · Cleber Rosa

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

---
 tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +2 -1
index e920bbf08c..1275f24532 100644
@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
+            "-device",
Quoted 9 lines

tests/avocado/machine_aarch64_sbsaref.py def boot_alpine_linux(self, cpu): · line 137

Quoted 4 lines
tests/avocado/machine_aarch64_sbsaref.py +3 -0
+            "virtio-rng-pci,rng=rng0",
+            "-object",
+            "rng-random,id=rng0,filename=/dev/urandom",
         )

The commit message doesn't say anything about adding virtio-rng. If that's needed for some reason, do it as a separate commit with an explanation of the bug its fixing.

This is actually a rebase mistake. virtio-rng was removed in 21f123f3c.

I'll fix it in a v2.

Thanks for spotting it,
- Cleber.

#13 lore raw

PATCH 6/13 tests/avocado: use more distinct names for assets

In reply to Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
 tests/avocado/kvm_xen_guest.py  | 3 ++-
 tests/avocado/netdev-ethtool.py | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +2 -1
index f8cb458d5d..318fadebc3 100644
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
+                                locations=(url), asset_hash=sha1)
 
     def common_vm_setup(self):
         # We also catch lack of KVM_XEN support if we fail to launch
tests/avocado/netdev-ethtool.py +2 -1
index 5f33288f81..462cf8de7d 100644
@@ -27,7 +27,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}",
+                                locations=(url), asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
 
Signature · 1 line
2.45.2
#14 lore raw

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

Quoted 21 lines · Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
 tests/avocado/kvm_xen_guest.py  | 3 ++-
 tests/avocado/netdev-ethtool.py | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +0 -0
index f8cb458d5d..318fadebc3 100644

tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44

Quoted 6 lines
tests/avocado/kvm_xen_guest.py +2 -1
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
+                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.

Quoted 21 lines
tests/avocado/kvm_xen_guest.py +0 -0
     def common_vm_setup(self):
         # We also catch lack of KVM_XEN support if we fail to launch
tests/avocado/netdev-ethtool.py +2 -1
index 5f33288f81..462cf8de7d 100644
@@ -27,7 +27,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}",
+                                locations=(url), asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
 
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#15 lore raw

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

Quoted 27 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
  tests/avocado/kvm_xen_guest.py  | 3 ++-
  tests/avocado/netdev-ethtool.py | 3 ++-
  2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
index f8cb458d5d..318fadebc3 100644
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
          url = base_url + name
          # use explicit name rather than failing to neatly parse the
          # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 8 lines

tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44

Quoted 1 line
tests/avocado/kvm_xen_guest.py +1 -0
+                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.

I thought $name was purely for debugging; the file was downloaded in a temporary location, and if the hash matched, it was renamed in the cache as $asset_hash which is unique. This was suggested in order to avoid dealing with URL updates for the same asset. Isn't it the case?

#16 lore raw

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

Quoted 38 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
  tests/avocado/kvm_xen_guest.py  | 3 ++-
  tests/avocado/netdev-ethtool.py | 3 ++-
  2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
index f8cb458d5d..318fadebc3 100644
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
          url = base_url + name
          # use explicit name rather than failing to neatly parse the
          # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 8 lines

tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44

Quoted 1 line
tests/avocado/kvm_xen_guest.py +1 -0
+                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.

Quoted 6 lines

I thought $name was purely for debugging; the file was downloaded in a temporary location, and if the hash matched, it was renamed in the cache as $asset_hash which is unique. This was suggested in order to avoid dealing with URL updates for the same asset. Isn't it the case?

Hi Phillipe,

I've replied to Daniel's question, but let me repeat the relevant parts of the $name behavior here.

---

Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:

avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)

Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.

This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.

I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.

---

Now, an update: instead of dropping the patch, it could be simplified by keeping just the "name" parameter (with the URL) and dropping the "locations" parameter (that has a single location).

Let me know if this makes sense and what you take on it.

Regards,
- Cleber.

#17 lore raw

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

In reply to Daniel P. Berrangé
Quoted 28 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
 tests/avocado/kvm_xen_guest.py  | 3 ++-
 tests/avocado/netdev-ethtool.py | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
index f8cb458d5d..318fadebc3 100644
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 9 lines

tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44

Quoted 1 line
tests/avocado/kvm_xen_guest.py +1 -0
+                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.

Now with your comment, I do agree that this adds some unneeded maintenance burden indeed. Also, this was part of my pre-avocado bump patches that would work around issues present in < 103.0. But let me give the complete answer.

Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:

avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)

Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.

This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.

I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.

#18 lore raw

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

Quoted 47 lines · Cleber Rosa
Quoted 28 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa

Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.

---
 tests/avocado/kvm_xen_guest.py  | 3 ++-
 tests/avocado/netdev-ethtool.py | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +1 -1
index f8cb458d5d..318fadebc3 100644
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 9 lines

tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44

Quoted 1 line
tests/avocado/kvm_xen_guest.py +1 -0
+                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.

Now with your comment, I do agree that this adds some unneeded maintenance burden indeed. Also, this was part of my pre-avocado bump patches that would work around issues present in < 103.0. But let me give the complete answer.

Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:

Quoted 6 lines

avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)

This is mostly a hack to avoid having to tell NextCloud to generate a unique sharing URL for every file.

Quoted 13 lines

Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.

This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.

I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#19 lore raw

PATCH 7/13 tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

In reply to Cleber Rosa

Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.

This makes a distinction between the tests that actually have this requirement and those who don't. The goal is to be as safe as possible, avoiding causing cache misses (because the assets get modified and thus need to be dowloaded again) while avoid copying the root filesystem backing file whenever possible.

This also allow these tests to be run in parallel with newer Avocado versions.

---
 tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)
tests/avocado/kvm_xen_guest.py +18 -9
index 318fadebc3..d73fa888ef 100644
@@ -10,6 +10,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from qemu.machine import machine
 
@@ -43,7 +44,7 @@ def get_asset(self, name, sha1):
         return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
                                 locations=(url), asset_hash=sha1)
 
-    def common_vm_setup(self):
+    def common_vm_setup(self, readwrite=False):
         # We also catch lack of KVM_XEN support if we fail to launch
         self.require_accelerator("kvm")
 
@@ -56,11 +57,19 @@ def common_vm_setup(self):
                                           "367962983d0d32109998a70b45dcee4672d0b045")
         self.rootfs = self.get_asset("rootfs.ext4",
                                      "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
+        if readwrite:
+            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
+            shutil.copy(self.rootfs, dest)
+            self.rootfs = dest
 
-    def run_and_check(self):
+    def run_and_check(self, readwrite=False):
+        if readwrite:
+            drive = f"file={self.rootfs},if=none,format=raw,id=drv0"
+        else:
+            drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0"
         self.vm.add_args('-kernel', self.kernel_path,
                          '-append', self.kernel_params,
-                         '-drive',  f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0",
+                         '-drive',  drive,
                          '-device', 'xen-disk,drive=drv0,vdev=xvda',
                          '-device', 'virtio-net-pci,netdev=unet',
                          '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')
@@ -90,11 +99,11 @@ def test_kvm_xen_guest(self):
         :avocado: tags=kvm_xen_guest
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.*msi /proc/interrupts')
 
     def test_kvm_xen_guest_nomsi(self):
@@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self):
         :avocado: tags=kvm_xen_guest_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.* /proc/interrupts')
 
     def test_kvm_xen_guest_noapic_nomsi(self):
@@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self):
         :avocado: tags=kvm_xen_guest_noapic_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks noapic pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq /proc/interrupts')
 
     def test_kvm_xen_guest_vapic(self):
Signature · 1 line
2.45.2
#20 lore raw

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

Quoted 40 lines · Cleber Rosa

Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.

This makes a distinction between the tests that actually have this requirement and those who don't. The goal is to be as safe as possible, avoiding causing cache misses (because the assets get modified and thus need to be dowloaded again) while avoid copying the root filesystem backing file whenever possible.

This also allow these tests to be run in parallel with newer Avocado versions.

---
 tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)
tests/avocado/kvm_xen_guest.py +2 -1
index 318fadebc3..d73fa888ef 100644
@@ -10,6 +10,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from qemu.machine import machine
 
@@ -43,7 +44,7 @@ def get_asset(self, name, sha1):
         return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
                                 locations=(url), asset_hash=sha1)
 
-    def common_vm_setup(self):
+    def common_vm_setup(self, readwrite=False):
         # We also catch lack of KVM_XEN support if we fail to launch
         self.require_accelerator("kvm")
 

tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63

Quoted 7 lines
tests/avocado/kvm_xen_guest.py +4 -0
@@ -56,11 +57,19 @@ def common_vm_setup(self):
                                           "367962983d0d32109998a70b45dcee4672d0b045")
         self.rootfs = self.get_asset("rootfs.ext4",
                                      "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
+        if readwrite:
+            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
+            shutil.copy(self.rootfs, dest)
+            self.rootfs = dest

This is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.

Quoted 60 lines
tests/avocado/kvm_xen_guest.py +12 -8
-    def run_and_check(self):
+    def run_and_check(self, readwrite=False):
+        if readwrite:
+            drive = f"file={self.rootfs},if=none,format=raw,id=drv0"
+        else:
+            drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0"
         self.vm.add_args('-kernel', self.kernel_path,
                          '-append', self.kernel_params,
-                         '-drive',  f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0",
+                         '-drive',  drive,
                          '-device', 'xen-disk,drive=drv0,vdev=xvda',
                          '-device', 'virtio-net-pci,netdev=unet',
                          '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')
@@ -90,11 +99,11 @@ def test_kvm_xen_guest(self):
         :avocado: tags=kvm_xen_guest
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.*msi /proc/interrupts')
 
     def test_kvm_xen_guest_nomsi(self):
@@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self):
         :avocado: tags=kvm_xen_guest_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.* /proc/interrupts')
 
     def test_kvm_xen_guest_noapic_nomsi(self):
@@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self):
         :avocado: tags=kvm_xen_guest_noapic_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks noapic pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq /proc/interrupts')
 
     def test_kvm_xen_guest_vapic(self):
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

Quoted 4 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa

Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.

Hm, I'm not sure *why* they require a writable image. Mostly they're just testing the interrupt routing. What's the failure mode for a read- only image?

Quoted 5 lines
Quoted 5 lines
tests/avocado/kvm_xen_guest.py +1 -0
@@ -56,11 +57,19 @@ def common_vm_setup(self):
                                            "367962983d0d32109998a70b45dcee4672d0b045")
          self.rootfs = self.get_asset("rootfs.ext4",
                                       "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
+        if readwrite:
Quoted 8 lines

tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63

Quoted 3 lines
tests/avocado/kvm_xen_guest.py +3 -0
+            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
+            shutil.copy(self.rootfs, dest)
+            self.rootfs = dest

This is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.

Or just use -snapshot?

#22 lore raw

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

Quoted 9 lines · David Woodhouse
Quoted 4 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa

Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.

Hm, I'm not sure *why* they require a writable image. Mostly they're just testing the interrupt routing. What's the failure mode for a read- only image?

There are no real failures with readonly=on although you do see init complain a bit about not being able to mount /dev and open log files. snapshot allows that to happen but doesn't change the underlying storage.

Quoted 16 lines
Quoted 14 lines

tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63

Quoted 9 lines
tests/avocado/kvm_xen_guest.py +4 -0
@@ -56,11 +57,19 @@ def common_vm_setup(self):
                                            "367962983d0d32109998a70b45dcee4672d0b045")
          self.rootfs = self.get_asset("rootfs.ext4",
                                       "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
+        if readwrite:
+            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
+            shutil.copy(self.rootfs, dest)
+            self.rootfs = dest

This is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.

Or just use -snapshot?

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#23 lore raw

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

Quoted 5 lines · Alex Bennée

There are no real failures with readonly=on although you do see init complain a bit about not being able to mount /dev and open log files. snapshot allows that to happen but doesn't change the underlying storage.

I could swear I experienced failures when I developed this patch. But like your patch b9371a7b90 already fixes it.

Dropping this one.

Thanks!
- Cleber.

#24 lore raw

PATCH 8/13 testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

In reply to Cleber Rosa

The asset used in the mentioned test gets truncated before it's used in the test. This means that the file gets modified, and thus the asset's expected hash doesn't match anymore. This causes cache misses and re-downloads every time the test is re-run.

Let's make a copy of the asset so that the one in the cache is preserved and the cache sees a hit on re-runs.

---
 tests/avocado/boot_linux_console.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/avocado/boot_linux_console.py +4 -2
index b8b0a4df10..2929aa042d 100644
@@ -401,14 +401,16 @@ def test_arm_emcraft_sf2(self):
                    'fe371d32e50ca682391e1e70ab98c2942aeffb01/spi.bin')
         spi_hash = '65523a1835949b6f4553be96dec1b6a38fb05501'
         spi_path = self.fetch_asset(spi_url, asset_hash=spi_hash)
+        spi_path_rw = os.path.join(self.workdir, os.path.basename(spi_path))
+        shutil.copy(spi_path, spi_path_rw)
 
-        file_truncate(spi_path, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB
+        file_truncate(spi_path_rw, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB
 
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
         self.vm.add_args('-kernel', uboot_path,
                          '-append', kernel_command_line,
-                         '-drive', 'file=' + spi_path + ',if=mtd,format=raw',
+                         '-drive', 'file=' + spi_path_rw + ',if=mtd,format=raw',
                          '-no-reboot')
         self.vm.launch()
         self.wait_for_console_pattern('Enter \'help\' for a list')
Signature · 1 line
2.45.2
Reviewed-by on v1 8/13
#25 lore raw

Re: [PATCH 08/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

Quoted 5 lines · Cleber Rosa

The asset used in the mentioned test gets truncated before it's used in the test. This means that the file gets modified, and thus the asset's expected hash doesn't match anymore. This causes cache misses and re-downloads every time the test is re-run.

Quoted 7 lines

Let's make a copy of the asset so that the one in the cache is preserved and the cache sees a hit on re-runs.

---
 tests/avocado/boot_linux_console.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#26 lore raw

PATCH 9/13 tests/avocado/boot_xen.py: fetch kernel during test setUp()

In reply to Cleber Rosa

The kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.

---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
index f29bc58b9e..490a127a3e 100644
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
Signature · 1 line
2.45.2
#27 lore raw

Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()

Quoted 3 lines · Cleber Rosa

The kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.

The other tests don't call fetch_asset() from their setUp method - what's different about this test that prevents the asset caching working ?

Quoted 52 lines
---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
index f29bc58b9e..490a127a3e 100644
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#28 lore raw

Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()

Quoted 10 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa

The kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.

The other tests don't call fetch_asset() from their setUp method - what's different about this test that prevents the asset caching working ?

The "wizardry" of the "fetch asset" plugin limits itself to analyzing test methods and the setUp() method. This test currently does that in the "fetch_guest_kernel()" method, which makes it blind to the "fetch asset" plugin.

If there are other tests doing that (in custom methods), then a similar change would be desirable.

Regards,
- Cleber.

#29 lore raw

PATCH 10/13 tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

In reply to Cleber Rosa
---
 tests/avocado/tuxrun_baselines.py | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +6 -10
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
 from avocado.utils import process
 from avocado.utils.path import find_command
 
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
         return default
 
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')
     def setUp(self):
         super().setUp()
 
-        # We need zstd for all the tuxrun tests
-        # See https://github.com/avocado-framework/avocado/issues/5609
-        zstd = find_command('zstd', False)
-        if zstd is False:
-            self.cancel('Could not find "zstd", which is required to '
-                        'decompress rootfs')
-        self.zstd = zstd
-
         # Process the TuxRun specific tags, most machines work with
         # reasonable defaults but we sometimes need to tweak the
         # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
                                          asset_hash = isum,
                                          algorithm = "sha256")
 
-        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
-        process.run(cmd)
+        archive.extract(disk_image_zst, os.path.join(self.workdir,
+                                                     "rootfs.ext4"))
 
         if dt:
             dsum = csums.get(dt, None)
Signature · 1 line
2.45.2
Reviewed-by on v1 10/13
#30 lore raw

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 4 lines · Cleber Rosa
---
 tests/avocado/tuxrun_baselines.py | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#31 lore raw

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

In reply to Cleber Rosa
Quoted 18 lines · Cleber Rosa
---
  tests/avocado/tuxrun_baselines.py | 16 ++++++----------
  1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +1 -0
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
  from avocado_qemu import QemuSystemTest
  from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
  from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
  from avocado.utils import process
  from avocado.utils.path import find_command
 

tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44

Quoted 4 lines
tests/avocado/tuxrun_baselines.py +1 -0
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
          return default
 
+    @skipUnless(archive._probe_zstd_cmd(),

_probe_zstd_cmd() isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.

Quoted 27 lines
tests/avocado/tuxrun_baselines.py +4 -10
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')
      def setUp(self):
          super().setUp()
 
-        # We need zstd for all the tuxrun tests
-        # See https://github.com/avocado-framework/avocado/issues/5609
-        zstd = find_command('zstd', False)
-        if zstd is False:
-            self.cancel('Could not find "zstd", which is required to '
-                        'decompress rootfs')
-        self.zstd = zstd
-
          # Process the TuxRun specific tags, most machines work with
          # reasonable defaults but we sometimes need to tweak the
          # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
                                           asset_hash = isum,
                                           algorithm = "sha256")
 
-        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
-        process.run(cmd)
+        archive.extract(disk_image_zst, os.path.join(self.workdir,
+                                                     "rootfs.ext4"))
 
          if dt:
              dsum = csums.get(dt, None)
#32 lore raw

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Cleber Rosa
---
  tests/avocado/tuxrun_baselines.py | 16 ++++++----------
  1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +1 -0
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
  from avocado_qemu import QemuSystemTest
  from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
  from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
  from avocado.utils import process
  from avocado.utils.path import find_command
 
Quoted 9 lines

tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44

Quoted 5 lines
tests/avocado/tuxrun_baselines.py +1 -0
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
          return default
 
+    @skipUnless(archive._probe_zstd_cmd(),

_probe_zstd_cmd() isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.

While it's not the best practice to use private functions, I just couldn't accept rewriting that for the skip condition. I can make sure future versions (including 103.1) make it public.

Also, these patches count on the bump to 103.0 indeed.

#33 lore raw

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 31 lines · Cleber Rosa
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Cleber Rosa
---
   tests/avocado/tuxrun_baselines.py | 16 ++++++----------
   1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +1 -0
index 736e4aa289..bd02e88ed6 100644
@@ -17,6 +17,7 @@
   from avocado_qemu import QemuSystemTest
   from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
   from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
   from avocado.utils import process
   from avocado.utils.path import find_command
 
Quoted 9 lines

tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44

Quoted 5 lines
tests/avocado/tuxrun_baselines.py +1 -0
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
           return default
 
+    @skipUnless(archive._probe_zstd_cmd(),

_probe_zstd_cmd() isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.

Quoted 5 lines

While it's not the best practice to use private functions, I just couldn't accept rewriting that for the skip condition. I can make sure future versions (including 103.1) make it public.

Also, these patches count on the bump to 103.0 indeed.

Then either mention it in the commit description to avoid wasting time to developers cherry-picking / testing this single patch, or move it after the version bump, avoiding bisectability issues.

Thanks,

Phil.

#34 lore raw

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 1 line

Also, these patches count on the bump to 103.0 indeed.

Then either mention it in the commit description to avoid wasting time to developers cherry-picking / testing this single patch, or move it after the version bump, avoiding bisectability issues.

You're right. My bad.

Regards,
- Cleber.

Quoted 4 lines

Thanks,

Phil.

#35 lore raw

PATCH 11/13 tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

In reply to Cleber Rosa

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own copy.

---
 tests/avocado/machine_aarch64_sbsaref.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +4 -1
index 1275f24532..8816308b86 100644
@@ -7,6 +7,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from avocado import skipUnless
 from avocado.utils import archive
@@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu):
         )
 
         img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
-        img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        img_path = os.path.join(self.workdir, os.path.basename(cached_img_path))
+        shutil.copy(cached_img_path, img_path)
 
         self.vm.set_console()
         self.vm.add_args(
Signature · 1 line
2.45.2
#36 lore raw

Re: [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

Quoted 22 lines · Cleber Rosa

When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own copy.

---
 tests/avocado/machine_aarch64_sbsaref.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +1 -0
index 1275f24532..8816308b86 100644
@@ -7,6 +7,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from avocado import skipUnless
 from avocado.utils import archive

tests/avocado/machine_aarch64_sbsaref.py def boot_openbsd73(self, cpu): · line 193

Quoted 7 lines
tests/avocado/machine_aarch64_sbsaref.py +3 -1
@@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu):
         )
 
         img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
-        img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        img_path = os.path.join(self.workdir, os.path.basename(cached_img_path))
+        shutil.copy(cached_img_path, img_path)

Again I think we should be using a throwaway qcow2 overlay rather than copying the full image.

tests/avocado/machine_aarch64_sbsaref.py def boot_openbsd73(self, cpu): · line 196

Quoted 3 lines
tests/avocado/machine_aarch64_sbsaref.py +0 -0
         self.vm.set_console()
         self.vm.add_args(

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#37 lore raw

PATCH 12/13 Bump avocado to 103.0

In reply to Cleber Rosa

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

---
 pythondeps.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
pythondeps.toml +1 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
Signature · 1 line
2.45.2
#38 lore raw

Re: [PATCH 12/13] Bump avocado to 103.0

Quoted 10 lines · Cleber Rosa

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.

Rebasing to a completely re-written test harness while in freeze feels on the risky side to me, despite the known problems we have with the existing release.

Quoted 21 lines

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

---
 pythondeps.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
pythondeps.toml +1 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
Signature · 1 line
2.45.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#39 lore raw

Re: [PATCH 12/13] Bump avocado to 103.0

In reply to Cleber Rosa
Quoted 10 lines · Cleber Rosa

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.

Does that restore feature parity for macOS developers? Because this community has been left behind ignored for over 2 years and already looked at alternatives for functional testing.

Quoted 17 lines

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

---
  pythondeps.toml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
pythondeps.toml +1 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
  # Note that qemu.git/python/ is always implicitly installed.
  # Prefer an LTS version when updating the accepted versions of
  # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
  pycdlib = { accepted = ">=1.11.0" }
#40 lore raw

Re: [PATCH 12/13] Bump avocado to 103.0

Quoted 4 lines · Philippe Mathieu-Daudé

Does that restore feature parity for macOS developers? Because this community has been left behind ignored for over 2 years and already looked at alternatives for functional testing.

Hi Phillipe,

As early as Avocado 102.0,  macOS support is pretty complete. The exact words on the release notes[1] are:

"User of macOS will have a better experience when using Avocado. The full set of Avocado’s selftests are now run under macOS on CI. Please be advised that macOS is not currently supported at the same level of Linux-based operating systems due to the lack of contributors/maintainers with access to the needed hardware. If you are a user/developer and are willing to contribute to this, please let the Avocado team know."

When it comes to the lack of updates, that is a longer discussion indeed.  When it comes to alternatives, I don't expect the QEMU project to do anything else than what it's in its best interest. As late as this can be, please take it for what it's worth.  If it does any good to QEMU, please consider it.

Best,

- Cleber.

[1] - https://avocado-framework.readthedocs.io/en/latest/releases/102_0.html

Quoted 21 lines
Quoted 20 lines

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html

---
  pythondeps.toml | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
pythondeps.toml +0 -1
index f6e590fdd8..175cf99241 100644
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed 
 = "1.1.1" }
   # Note that qemu.git/python/ is always implicitly installed.
   # Prefer an LTS version when updating the accepted versions of
   # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = 

"88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
  pycdlib = { accepted = ">=1.11.0" }

#41 lore raw

PATCH 13/13 Avocado tests: allow for parallel execution of tests

In reply to Cleber Rosa

The updated Avocado version allows for the execution of tests in parallel.

While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.

This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.

Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.

---
 docs/devel/testing.rst | 12 ++++++++++++
 tests/Makefile.include |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
docs/devel/testing.rst +12 -0
index 23d3f44f52..5600123743 100644
@@ -983,6 +983,18 @@ of Avocado or ``make check-avocado``, and can also be queried using:
 
   pyvenv/bin/avocado list tests/avocado
 
+To run tests in parallel, the ``AVOCADO_PARALLEL`` environment
+variable can be defined with a value different than ``1`` (its default
+value).  Example:
+
+ .. code::
+
+  make check-avocado AVOCADO_PARALLEL=4
+
+Please exercise care when using parallel execution with the QEMU
+Avocado tests as a higher system load can cause time sensitive tests
+to timeout and be interrupted.
+
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
 
tests/Makefile.include +5 -1
index 6618bfed70..545b5155f9 100644
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
 endif
+ifndef AVOCADO_PARALLEL
+	AVOCADO_PARALLEL=1
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+			-p timeout_factor=$(AVOCADO_PARALLEL) \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
Signature · 1 line
2.45.2
Reviewed-by on the v1 series
#42 lore raw
In reply to Cleber Rosa
Quoted 48 lines · Cleber Rosa

This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.

These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 s

Cleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugin tests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: use more distinct names for assets tests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of tests

  docs/devel/testing.rst                   | 12 +++++++
  pythondeps.toml                          |  2 +-
  tests/Makefile.include                   |  6 +++-
  tests/avocado/boot_linux_console.py      | 24 ++++++++------
  tests/avocado/boot_xen.py                | 13 ++++----
  tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
  tests/avocado/intel_iommu.py             |  2 ++
  tests/avocado/kvm_xen_guest.py           | 30 +++++++++++------
  tests/avocado/machine_aarch64_sbsaref.py | 11 +++++--
  tests/avocado/machine_aarch64_virt.py    | 14 ++++----
  tests/avocado/netdev-ethtool.py          |  3 +-
  tests/avocado/tuxrun_baselines.py        | 16 ++++-----
  12 files changed, 125 insertions(+), 49 deletions(-)
Quoted 2 lines

create mode 100644 tests/avocado/cdrom.py

#43 lore raw
In reply to Cleber Rosa
Quoted 6 lines · Cleber Rosa

Cleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugin tests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access

Quoted 8 lines

tests/avocado: use more distinct names for assets tests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of tests

Queuing patches #1, #2 and #8. Unfortunately #10 depends on Avocado 101.0 so can't be merged at this point.