Bump Avocado to 103.0 LTS and update tests for compatibility and new features
cover Bump Avocado to 103.0 LTS and update tests for compatibility and new features
This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.
These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.
Nevertheless, as pointed out, on the ML, these changes are needed NOW.
Some examples of runs in the CI can be seen below:
* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s
* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 s
Cleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugin tests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: use more distinct names for assets tests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset tests/avocado/boot_xen.py: fetch kernel during test setUp() tests/avocado/tuxrun_baselines.py: use Avocado's zstd support tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of tests
docs/devel/testing.rst | 12 +++++++ pythondeps.toml | 2 +- tests/Makefile.include | 6 +++- tests/avocado/boot_linux_console.py | 24 ++++++++------ tests/avocado/boot_xen.py | 13 ++++---- tests/avocado/cdrom.py | 41 ++++++++++++++++++++++++ tests/avocado/intel_iommu.py | 2 ++ tests/avocado/kvm_xen_guest.py | 30 +++++++++++------ tests/avocado/machine_aarch64_sbsaref.py | 11 +++++-- tests/avocado/machine_aarch64_virt.py | 14 ++++---- tests/avocado/netdev-ethtool.py | 3 +- tests/avocado/tuxrun_baselines.py | 16 ++++----- 12 files changed, 125 insertions(+), 49 deletions(-) create mode 100644 tests/avocado/cdrom.py
Signature · 1 line
2.45.2
PATCH 1/13 tests/avocado: mips: fallback to HTTP given certificate expiration
The SSL certificate installed at mipsdistros.mips.com has expired:
0 s:CN = mipsdistros.mips.com
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMT
Because this project has no control over that certificate and host, this falls back to plain HTTP instead. The integrity of the downloaded files can be guaranteed by the existing hashes for those files (which are not modified here).
--- tests/avocado/boot_linux_console.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tests/avocado/boot_linux_console.py +3 -3
@@ -299,7 +299,7 @@ def test_mips_malta32el_nanomips_4k(self): :avocado: tags=endian:little :avocado: tags=cpu:I7200 """ - kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/' + kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/' 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page4k.xz') kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
@@ -312,7 +312,7 @@ def test_mips_malta32el_nanomips_16k_up(self): :avocado: tags=endian:little :avocado: tags=cpu:I7200 """ - kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/' + kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/' 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page16k_up.xz') kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
@@ -325,7 +325,7 @@ def test_mips_malta32el_nanomips_64k_dbg(self): :avocado: tags=endian:little :avocado: tags=cpu:I7200 """ - kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/' + kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/' 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page64k_dbg.xz') kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
Signature · 1 line
2.45.2
Re: [PATCH 01/13] tests/avocado: mips: fallback to HTTP given certificate expiration
Quoted 12 lines · Cleber Rosa
The SSL certificate installed at mipsdistros.mips.com has expired:
0 s:CN = mipsdistros.mips.com
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMTBecause this project has no control over that certificate and host, this falls back to plain HTTP instead. The integrity of the downloaded files can be guaranteed by the existing hashes for those files (which are not modified here).
Quoted 5 lines
Reviewed-by Alex Bennée <[email protected]>Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/boot_linux_console.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
PATCH 2/13 tests/avocado: mips: add hint for fetchasset plugin
In reply to Cleber RosaAvocado's fetchasset plugin runs before the actual Avocado job (and any test). It analyses the test's code looking for occurrences of "self.fetch_asset()" in the either the actual test or setUp() method. It's not able to fully analyze all code, though.
The way these tests are written, make the fetchasset plugin blind to the assets. This adds some more code duplication, true, but it will aid the fetchasset plugin to download or verify the existence of these assets in advance.
--- tests/avocado/boot_linux_console.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
tests/avocado/boot_linux_console.py +7 -5
@@ -274,8 +274,7 @@ def test_mips64el_malta_5KEc_cpio(self): # Wait for VM to shut down gracefully self.vm.wait() - def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash): - kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + def do_test_mips_malta32el_nanomips(self, kernel_path_xz): kernel_path = self.workdir + "kernel" with lzma.open(kernel_path_xz, 'rb') as f_in: with open(kernel_path, 'wb') as f_out:
@@ -303,7 +302,8 @@ def test_mips_malta32el_nanomips_4k(self): 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page4k.xz') kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6' - self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash) + kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + self.do_test_mips_malta32el_nanomips(kernel_path_xz) def test_mips_malta32el_nanomips_16k_up(self): """
@@ -316,7 +316,8 @@ def test_mips_malta32el_nanomips_16k_up(self): 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page16k_up.xz') kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc' - self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash) + kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + self.do_test_mips_malta32el_nanomips(kernel_path_xz) def test_mips_malta32el_nanomips_64k_dbg(self): """
@@ -329,7 +330,8 @@ def test_mips_malta32el_nanomips_64k_dbg(self): 'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/' 'generic_nano32r6el_page64k_dbg.xz') kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180' - self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash) + kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + self.do_test_mips_malta32el_nanomips(kernel_path_xz) def test_aarch64_xlnx_versal_virt(self): """
Signature · 1 line
2.45.2
Re: [PATCH 02/13] tests/avocado: mips: add hint for fetchasset plugin
Quoted 10 lines · Cleber Rosa
Avocado's fetchasset plugin runs before the actual Avocado job (and any test). It analyses the test's code looking for occurrences of "
self.fetch_asset()" in the either the actual test orsetUp()method. It's not able to fully analyze all code, though.The way these tests are written, make the fetchasset plugin blind to the assets. This adds some more code duplication, true, but it will aid the fetchasset plugin to download or verify the existence of these assets in advance.
Quoted 4 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/boot_linux_console.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
PATCH 3/13 tests/avocado/intel_iommu.py: increase timeout
In reply to Cleber RosaBased on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience.
Let's increase the timeout so these tests get a chance to completion.
--- tests/avocado/intel_iommu.py | 2 ++ 1 file changed, 2 insertions(+)
tests/avocado/intel_iommu.py +2 -0
@@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest): :avocado: tags=flaky """ + timeout = 360 + IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' kernel_path = None initrd_path = None
Signature · 1 line
2.45.2
Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout
Quoted 4 lines · Cleber Rosa
Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.
A high watermark of over 5 minutes is pretty long for a test.
Looking at the test I see it runs
self.ssh_command('dnf -y install numactl-devel')
but then never actually uses the installed package.
I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.
If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.
Quoted 22 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/intel_iommu.py | 2 ++ 1 file changed, 2 insertions(+)tests/avocado/intel_iommu.py +2
-0@@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest): :avocado: tags=flaky """ + timeout = 360 + IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' kernel_path = None initrd_path = NoneSignature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout
Quoted 9 lines · Daniel P. Berrangé
Quoted 4 lines · Cleber Rosa
Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.
A high watermark of over 5 minutes is pretty long for a test.
I agree.
Quoted 9 lines
Looking at the test I see it runs
self.ssh_command('dnf -y install numactl-devel')but then never actually uses the installed package.
I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.
Exactly.
Quoted 5 lines
If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.
I was trying not to get in the way of the original test writer.
Eric,
Are you OK with replacing this command for a simpler file transfer? Any suggestions?
Regards,
- Cleber.
Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout
Hi Cleber,
Quoted 26 lines · Cleber Rosa
Quoted 7 lines · Daniel P. Berrangé
Quoted 4 lines · Cleber Rosa
Based on many runs, the average run time for these 4 tests is around 250 seconds, with 320 seconds being the ceiling. In any way, the default 120 seconds timeout is inappropriate in my experience. Let's increase the timeout so these tests get a chance to completion.
A high watermark of over 5 minutes is pretty long for a test.
I agree.
Quoted 9 lines
Looking at the test I see it runs
self.ssh_command('dnf -y install numactl-devel')but then never actually uses the installed package.
I expect that most of the wallclock time here is coming from having dnf download all the repodata, 4 times over.
Exactly.
Quoted 5 lines
If the intention was to test networking, then replace this with something that doesn't have to download 100's of MB of data, then see what kind of running time we get before increasing any timeout.
Quoted 6 lines
I was trying not to get in the way of the original test writer.
Eric,
Are you OK with replacing this command for a simpler file transfer? Any suggestions?
Sorry I am just returning from PTO. Yes I am OK to replace it with a file transfer if it takes too much time. When the test was written we just found out a bug with that exact dnf install command so at that time this was a reproducer. Either replace it with a dummy file transfer or withdraw it from the automated tests.
Thanks
Eric
Quoted 4 lines
Regards,
- Cleber.
PATCH 4/13 tests/avocado: add cdrom permission related tests
In reply to Cleber Rosa--- tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 tests/avocado/cdrom.py
tests/avocado/cdrom.py new file +41 -0
@@ -0,0 +1,41 @@ +# Simple functional tests for cdrom devices +# +# Copyright (c) 2023 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado.utils.iso9660 import iso9660 +from avocado_qemu import QemuSystemTest + + +class Cdrom(QemuSystemTest): + """ + :avocado: tags=block,cdrom,quick + :avocado: tags=machine:none + """ + def setUp(self): + super().setUp() + self.iso_path = os.path.join(self.workdir, "cdrom.iso") + iso = iso9660(self.iso_path) + iso.create() + iso.close() + + def test_plain_iso_rw(self): + self.vm.add_args('-drive', f'file={self.iso_path}') + self.vm.launch() + query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertFalse(query_block_result[0]["inserted"]["ro"]) + + def test_media_cdrom_ro(self): + self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom') + self.vm.launch() + query_block_result = self.vm.qmp('query-block')['return'] + self.assertEqual(len(query_block_result), 1) + self.assertTrue(query_block_result[0]["inserted"]["ro"])
Signature · 1 line
2.45.2
PATCH 5/13 tests/avocado: machine aarch64: standardize location and RO access
In reply to Cleber RosaThe tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py should not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.
While at it, let's use a single code style and hash for the ISO url.
--- tests/avocado/machine_aarch64_sbsaref.py | 6 +++++- tests/avocado/machine_aarch64_virt.py | 14 +++++++------- 2 files changed, 12 insertions(+), 8 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +5 -1
@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu): "-cpu", cpu, "-drive", - f"file={iso_path},format=raw", + f"file={iso_path},media=cdrom,format=raw", + "-device", + "virtio-rng-pci,rng=rng0", + "-object", + "rng-random,id=rng0,filename=/dev/urandom", ) self.vm.launch()
tests/avocado/machine_aarch64_virt.py +7 -7
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self): :avocado: tags=machine:virt :avocado: tags=accel:tcg """ - iso_url = ('https://dl-cdn.alpinelinux.org/' - 'alpine/v3.17/releases/aarch64/' - 'alpine-standard-3.17.2-aarch64.iso') + iso_url = ( + "https://dl-cdn.alpinelinux.org/" + "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso" + ) - # Alpine use sha256 so I recalculated this myself - iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839' - iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1) + iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027" + iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash) self.vm.set_console() kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self): self.vm.add_args("-smp", "2", "-m", "1024") self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', 'edk2-aarch64-code.fd')) - self.vm.add_args("-drive", f"file={iso_path},format=raw") + self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw") self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
Signature · 1 line
2.45.2
Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access
Quoted 17 lines · Cleber Rosa
The tests under
machine_aarch64_virt.pyandmachine_aarch64_sbsaref.pyshould not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.While at it, let's use a single code style and hash for the ISO url.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/machine_aarch64_sbsaref.py | 6 +++++- tests/avocado/machine_aarch64_virt.py | 14 +++++++------- 2 files changed, 12 insertions(+), 8 deletions(-)tests/avocado/machine_aarch64_sbsaref.py +0
-0
tests/avocado/machine_aarch64_sbsaref.py def boot_alpine_linux(self, cpu): · line 137
Quoted 10 lines
tests/avocado/machine_aarch64_sbsaref.py +5
-1@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu): "-cpu", cpu, "-drive", - f"file={iso_path},format=raw", + f"file={iso_path},media=cdrom,format=raw", + "-device", + "virtio-rng-pci,rng=rng0", + "-object", + "rng-random,id=rng0,filename=/dev/urandom", )
The commit message doesn't say anything about adding virtio-rng. If that's needed for some reason, do it as a separate commit with an explanation of the bug its fixing.
Quoted 37 lines
tests/avocado/machine_aarch64_virt.py +7
-7@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self): :avocado: tags=machine:virt :avocado: tags=accel:tcg """ - iso_url = ('https://dl-cdn.alpinelinux.org/' - 'alpine/v3.17/releases/aarch64/' - 'alpine-standard-3.17.2-aarch64.iso') + iso_url = ( + "https://dl-cdn.alpinelinux.org/" + "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso" + ) - # Alpine use sha256 so I recalculated this myself - iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839' - iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1) + iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027" + iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash) self.vm.set_console() kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self): self.vm.add_args("-smp", "2", "-m", "1024") self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', 'edk2-aarch64-code.fd')) - self.vm.add_args("-drive", f"file={iso_path},format=raw") + self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw") self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access
Quoted 25 lines · Daniel P. Berrangé
Quoted 23 lines · Cleber Rosa
The tests under
machine_aarch64_virt.pyandmachine_aarch64_sbsaref.pyshould not be writing to the ISO files. By adding "media=cdrom" the "ro" is autmatically set.While at it, let's use a single code style and hash for the ISO url.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/machine_aarch64_sbsaref.py | 6 +++++- tests/avocado/machine_aarch64_virt.py | 14 +++++++------- 2 files changed, 12 insertions(+), 8 deletions(-)tests/avocado/machine_aarch64_sbsaref.py +2
-1@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu): "-cpu", cpu, "-drive", - f"file={iso_path},format=raw", + f"file={iso_path},media=cdrom,format=raw", + "-device",
Quoted 9 lines
tests/avocado/machine_aarch64_sbsaref.py def boot_alpine_linux(self, cpu): · line 137
Quoted 4 lines
tests/avocado/machine_aarch64_sbsaref.py +3
-0+ "virtio-rng-pci,rng=rng0", + "-object", + "rng-random,id=rng0,filename=/dev/urandom", )The commit message doesn't say anything about adding virtio-rng. If that's needed for some reason, do it as a separate commit with an explanation of the bug its fixing.
This is actually a rebase mistake. virtio-rng was removed in 21f123f3c.
I'll fix it in a v2.
Thanks for spotting it,
- Cleber.
PATCH 6/13 tests/avocado: use more distinct names for assets
In reply to Cleber RosaAvocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.
This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
tests/avocado/kvm_xen_guest.py +2 -1
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}", + locations=(url), asset_hash=sha1) def common_vm_setup(self): # We also catch lack of KVM_XEN support if we fail to launch
tests/avocado/netdev-ethtool.py +2 -1
@@ -27,7 +27,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}", + locations=(url), asset_hash=sha1) def common_test_code(self, netdev, extra_args=None):
Signature · 1 line
2.45.2
Re: [PATCH 06/13] tests/avocado: use more distinct names for assets
Quoted 21 lines · Cleber Rosa
Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "
by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)tests/avocado/kvm_xen_guest.py +0
-0
tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44
Quoted 6 lines
tests/avocado/kvm_xen_guest.py +2
-1@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}", + locations=(url), asset_hash=sha1)
Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.
Quoted 21 lines
tests/avocado/kvm_xen_guest.py +0
-0def common_vm_setup(self): # We also catch lack of KVM_XEN support if we fail to launchtests/avocado/netdev-ethtool.py +2
-1@@ -27,7 +27,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}", + locations=(url), asset_hash=sha1) def common_test_code(self, netdev, extra_args=None):Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 06/13] tests/avocado: use more distinct names for assets
Quoted 27 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa
Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "
by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)tests/avocado/kvm_xen_guest.py +1
-1@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 8 lines
tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44
Quoted 1 line
tests/avocado/kvm_xen_guest.py +1
-0+ locations=(url), asset_hash=sha1)Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.
I thought $name was purely for debugging; the file was downloaded in a temporary location, and if the hash matched, it was renamed in the cache as $asset_hash which is unique. This was suggested in order to avoid dealing with URL updates for the same asset. Isn't it the case?
Re: [PATCH 06/13] tests/avocado: use more distinct names for assets
Quoted 38 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa
Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "
by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)tests/avocado/kvm_xen_guest.py +1
-1@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",Quoted 8 lines
tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44
Quoted 1 line
tests/avocado/kvm_xen_guest.py +1
-0+ locations=(url), asset_hash=sha1)Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.
Quoted 6 lines
I thought $name was purely for debugging; the file was downloaded in a temporary location, and if the hash matched, it was renamed in the cache as $asset_hash which is unique. This was suggested in order to avoid dealing with URL updates for the same asset. Isn't it the case?
Hi Phillipe,
I've replied to Daniel's question, but let me repeat the relevant parts of the $name behavior here.
---
Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)
Would save content to the same location:/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.
This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.
I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.
---
Now, an update: instead of dropping the patch, it could be simplified by keeping just the "name" parameter (with the URL) and dropping the "locations" parameter (that has a single location).
Let me know if this makes sense and what you take on it.
Regards,
- Cleber.
Re: [PATCH 06/13] tests/avocado: use more distinct names for assets
In reply to Daniel P. BerrangéQuoted 28 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa
Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "
by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)tests/avocado/kvm_xen_guest.py +1
-1@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
Quoted 9 lines
tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44
Quoted 1 line
tests/avocado/kvm_xen_guest.py +1
-0+ locations=(url), asset_hash=sha1)Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.
Now with your comment, I do agree that this adds some unneeded maintenance burden indeed. Also, this was part of my pre-avocado bump patches that would work around issues present in < 103.0. But let me give the complete answer.
Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)
Would save content to the same location:/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.
This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.
I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.
Re: [PATCH 06/13] tests/avocado: use more distinct names for assets
Quoted 47 lines · Cleber Rosa
Quoted 28 lines · Daniel P. Berrangé
Quoted 26 lines · Cleber Rosa
Avocado's asset system will deposit files in a cache organized either by their original location (the URI) or by their names. Because the cache (and the "
by_name" sub directory) is common across tests, it's a good idea to make these names as distinct as possible.This avoid name clashes, which makes future Avocado runs to attempt to redownload the assets with the same name, but from the different locations they actually are from. This causes cache misses, extra downloads, and possibly canceled tests.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 3 ++- tests/avocado/netdev-ethtool.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)tests/avocado/kvm_xen_guest.py +1
-1@@ -40,7 +40,8 @@ def get_asset(self, name, sha1): url = base_url + name # use explicit name rather than failing to neatly parse the # URL into a unique one - return self.fetch_asset(name=name, locations=(url), asset_hash=sha1) + return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",Quoted 9 lines
tests/avocado/kvm_xen_guest.py def get_asset(self, name, sha1): · line 44
Quoted 1 line
tests/avocado/kvm_xen_guest.py +1
-0+ locations=(url), asset_hash=sha1)Why do we need to pass a name here at all ? I see the comment here but it isn't very clear about what the problem is. It just feels wrong to be creating ourselves uniqueness naming problems, when we have a nicely unique URL, and that cached URL can be shared across tests, where as the custom names added by this patch are forcing no-caching of the same URL between tests.
Now with your comment, I do agree that this adds some unneeded maintenance burden indeed. Also, this was part of my pre-avocado bump patches that would work around issues present in < 103.0. But let me give the complete answer.
Under 88.1 the "uniqueness" of the URL did not consider the query parameters in the URL. So, under 88.1:
Quoted 6 lines
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage', ...)
avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage', ...)
This is mostly a hack to avoid having to tell NextCloud to generate a unique sharing URL for every file.
Quoted 13 lines
Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage' and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.I personally don't like having the files named, although uniquely, after the query parameters. But, If this doesn't bother others more than the maintenance burden, and Avocado version bump is applied, this patch can be dropped.
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
PATCH 7/13 tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
In reply to Cleber RosaSome of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.
This makes a distinction between the tests that actually have this requirement and those who don't. The goal is to be as safe as possible, avoiding causing cache misses (because the assets get modified and thus need to be dowloaded again) while avoid copying the root filesystem backing file whenever possible.
This also allow these tests to be run in parallel with newer Avocado versions.
--- tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)
tests/avocado/kvm_xen_guest.py +18 -9
@@ -10,6 +10,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later import os +import shutil from qemu.machine import machine
@@ -43,7 +44,7 @@ def get_asset(self, name, sha1): return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}", locations=(url), asset_hash=sha1) - def common_vm_setup(self): + def common_vm_setup(self, readwrite=False): # We also catch lack of KVM_XEN support if we fail to launch self.require_accelerator("kvm")
@@ -56,11 +57,19 @@ def common_vm_setup(self): "367962983d0d32109998a70b45dcee4672d0b045") self.rootfs = self.get_asset("rootfs.ext4", "f1478401ea4b3fa2ea196396be44315bab2bb5e4") + if readwrite: + dest = os.path.join(self.workdir, os.path.basename(self.rootfs)) + shutil.copy(self.rootfs, dest) + self.rootfs = dest - def run_and_check(self): + def run_and_check(self, readwrite=False): + if readwrite: + drive = f"file={self.rootfs},if=none,format=raw,id=drv0" + else: + drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0" self.vm.add_args('-kernel', self.kernel_path, '-append', self.kernel_params, - '-drive', f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0", + '-drive', drive, '-device', 'xen-disk,drive=drv0,vdev=xvda', '-device', 'virtio-net-pci,netdev=unet', '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')
@@ -90,11 +99,11 @@ def test_kvm_xen_guest(self): :avocado: tags=kvm_xen_guest """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq.*msi /proc/interrupts') def test_kvm_xen_guest_nomsi(self):
@@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self): :avocado: tags=kvm_xen_guest_nomsi """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks pci=nomsi') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq.* /proc/interrupts') def test_kvm_xen_guest_noapic_nomsi(self):
@@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self): :avocado: tags=kvm_xen_guest_noapic_nomsi """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks noapic pci=nomsi') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq /proc/interrupts') def test_kvm_xen_guest_vapic(self):
Signature · 1 line
2.45.2
Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
Quoted 40 lines · Cleber Rosa
Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.
This makes a distinction between the tests that actually have this requirement and those who don't. The goal is to be as safe as possible, avoiding causing cache misses (because the assets get modified and thus need to be dowloaded again) while avoid copying the root filesystem backing file whenever possible.
This also allow these tests to be run in parallel with newer Avocado versions.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)tests/avocado/kvm_xen_guest.py +2
-1@@ -10,6 +10,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later import os +import shutil from qemu.machine import machine@@ -43,7 +44,7 @@ def get_asset(self, name, sha1): return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}", locations=(url), asset_hash=sha1) - def common_vm_setup(self): + def common_vm_setup(self, readwrite=False): # We also catch lack of KVM_XEN support if we fail to launch self.require_accelerator("kvm")
tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63
Quoted 7 lines
tests/avocado/kvm_xen_guest.py +4
-0@@ -56,11 +57,19 @@ def common_vm_setup(self): "367962983d0d32109998a70b45dcee4672d0b045") self.rootfs = self.get_asset("rootfs.ext4", "f1478401ea4b3fa2ea196396be44315bab2bb5e4") + if readwrite: + dest = os.path.join(self.workdir, os.path.basename(self.rootfs)) + shutil.copy(self.rootfs, dest) + self.rootfs = dest
This is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.
Quoted 60 lines
tests/avocado/kvm_xen_guest.py +12
-8- def run_and_check(self): + def run_and_check(self, readwrite=False): + if readwrite: + drive = f"file={self.rootfs},if=none,format=raw,id=drv0" + else: + drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0" self.vm.add_args('-kernel', self.kernel_path, '-append', self.kernel_params, - '-drive', f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0", + '-drive', drive, '-device', 'xen-disk,drive=drv0,vdev=xvda', '-device', 'virtio-net-pci,netdev=unet', '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')@@ -90,11 +99,11 @@ def test_kvm_xen_guest(self): :avocado: tags=kvm_xen_guest """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq.*msi /proc/interrupts') def test_kvm_xen_guest_nomsi(self):@@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self): :avocado: tags=kvm_xen_guest_nomsi """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks pci=nomsi') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq.* /proc/interrupts') def test_kvm_xen_guest_noapic_nomsi(self):@@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self): :avocado: tags=kvm_xen_guest_noapic_nomsi """ - self.common_vm_setup() + self.common_vm_setup(True) self.kernel_params = (self.KERNEL_DEFAULT + ' xen_emul_unplug=ide-disks noapic pci=nomsi') - self.run_and_check() + self.run_and_check(True) self.ssh_command('grep xen-pirq /proc/interrupts') def test_kvm_xen_guest_vapic(self):Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
Quoted 4 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa
Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.
Hm, I'm not sure *why* they require a writable image. Mostly they're just testing the interrupt routing. What's the failure mode for a read- only image?
Quoted 5 lines
Quoted 5 lines
tests/avocado/kvm_xen_guest.py +1
-0@@ -56,11 +57,19 @@ def common_vm_setup(self): "367962983d0d32109998a70b45dcee4672d0b045") self.rootfs = self.get_asset("rootfs.ext4", "f1478401ea4b3fa2ea196396be44315bab2bb5e4") + if readwrite:
Quoted 8 lines
tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63
Quoted 3 lines
tests/avocado/kvm_xen_guest.py +3
-0+ dest = os.path.join(self.workdir, os.path.basename(self.rootfs)) + shutil.copy(self.rootfs, dest) + self.rootfs = destThis is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.
Or just use -snapshot?
Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
Quoted 9 lines · David Woodhouse
Quoted 4 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa
Some of these tests actually require the root filesystem image, obtained through Avocado's asset feature and kept in a common cache location, to be writable.
Hm, I'm not sure *why* they require a writable image. Mostly they're just testing the interrupt routing. What's the failure mode for a read- only image?
There are no real failures with readonly=on although you do see init complain a bit about not being able to mount /dev and open log files. snapshot allows that to happen but doesn't change the underlying storage.
Quoted 16 lines
Quoted 14 lines
tests/avocado/kvm_xen_guest.py def common_vm_setup(self): · line 63
Quoted 9 lines
tests/avocado/kvm_xen_guest.py +4
-0@@ -56,11 +57,19 @@ def common_vm_setup(self): "367962983d0d32109998a70b45dcee4672d0b045") self.rootfs = self.get_asset("rootfs.ext4", "f1478401ea4b3fa2ea196396be44315bab2bb5e4") + if readwrite: + dest = os.path.join(self.workdir, os.path.basename(self.rootfs)) + shutil.copy(self.rootfs, dest) + self.rootfs = destThis is a very expensive way of creating a writable disk. Better to avoid adding this 'readwrite' parameter at all, and instead create a throwaway qcow2 overlay for the image for all tests. That ensures writability for everything in a cheap manner.
Or just use -snapshot?
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
Quoted 5 lines · Alex Bennée
There are no real failures with readonly=on although you do see init complain a bit about not being able to mount /dev and open log files. snapshot allows that to happen but doesn't change the underlying storage.
I could swear I experienced failures when I developed this patch. But like your patch b9371a7b90 already fixes it.
Dropping this one.
Thanks!
- Cleber.
PATCH 8/13 testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
In reply to Cleber RosaThe asset used in the mentioned test gets truncated before it's used in the test. This means that the file gets modified, and thus the asset's expected hash doesn't match anymore. This causes cache misses and re-downloads every time the test is re-run.
Let's make a copy of the asset so that the one in the cache is preserved and the cache sees a hit on re-runs.
--- tests/avocado/boot_linux_console.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
tests/avocado/boot_linux_console.py +4 -2
@@ -401,14 +401,16 @@ def test_arm_emcraft_sf2(self): 'fe371d32e50ca682391e1e70ab98c2942aeffb01/spi.bin') spi_hash = '65523a1835949b6f4553be96dec1b6a38fb05501' spi_path = self.fetch_asset(spi_url, asset_hash=spi_hash) + spi_path_rw = os.path.join(self.workdir, os.path.basename(spi_path)) + shutil.copy(spi_path, spi_path_rw) - file_truncate(spi_path, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB + file_truncate(spi_path_rw, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB self.vm.set_console() kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE self.vm.add_args('-kernel', uboot_path, '-append', kernel_command_line, - '-drive', 'file=' + spi_path + ',if=mtd,format=raw', + '-drive', 'file=' + spi_path_rw + ',if=mtd,format=raw', '-no-reboot') self.vm.launch() self.wait_for_console_pattern('Enter \'help\' for a list')
Signature · 1 line
2.45.2
Re: [PATCH 08/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
Quoted 5 lines · Cleber Rosa
The asset used in the mentioned test gets truncated before it's used in the test. This means that the file gets modified, and thus the asset's expected hash doesn't match anymore. This causes cache misses and re-downloads every time the test is re-run.
Quoted 7 lines
Let's make a copy of the asset so that the one in the cache is preserved and the cache sees a hit on re-runs.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/boot_linux_console.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 9/13 tests/avocado/boot_xen.py: fetch kernel during test setUp()
In reply to Cleber RosaThe kernel is a common blob used in all tests. By moving it to the setUp() method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.
--- tests/avocado/boot_xen.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
tests/avocado/boot_xen.py +6 -7
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest): timeout = 90 XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all' - def fetch_guest_kernel(self): + def setUp(self): + super(BootXen, self).setUp() + # Using my own built kernel - which works kernel_url = ('https://fileserver.linaro.org/' 's/JSsewXGZ6mqxPr5/download?path=%2F&files=' 'linux-5.9.9-arm64-ajb') kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83' - kernel_path = self.fetch_asset(kernel_url, - asset_hash=kernel_sha1) - - return kernel_path + self.kernel_path = self.fetch_asset(kernel_url, + asset_hash=kernel_sha1) def launch_xen(self, xen_path): """ Launch Xen with a dom0 guest kernel """ self.log.info("launch with xen_path: %s", xen_path) - kernel_path = self.fetch_guest_kernel() self.vm.set_console()
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path): '-append', self.XEN_COMMON_COMMAND_LINE, '-device', 'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0' - % (kernel_path)) + % (self.kernel_path)) self.vm.launch()
Signature · 1 line
2.45.2
Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()
Quoted 3 lines · Cleber Rosa
The kernel is a common blob used in all tests. By moving it to the
setUp()method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.
The other tests don't call fetch_asset() from their setUp method - what's different about this test that prevents the asset caching working ?
Quoted 52 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/boot_xen.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)tests/avocado/boot_xen.py +6
-7@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest): timeout = 90 XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all' - def fetch_guest_kernel(self): + def setUp(self): + super(BootXen, self).setUp() + # Using my own built kernel - which works kernel_url = ('https://fileserver.linaro.org/' 's/JSsewXGZ6mqxPr5/download?path=%2F&files=' 'linux-5.9.9-arm64-ajb') kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83' - kernel_path = self.fetch_asset(kernel_url, - asset_hash=kernel_sha1) - - return kernel_path + self.kernel_path = self.fetch_asset(kernel_url, + asset_hash=kernel_sha1) def launch_xen(self, xen_path): """ Launch Xen with a dom0 guest kernel """ self.log.info("launch with xen_path: %s", xen_path) - kernel_path = self.fetch_guest_kernel() self.vm.set_console()@@ -56,7 +55,7 @@ def launch_xen(self, xen_path): '-append', self.XEN_COMMON_COMMAND_LINE, '-device', 'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0' - % (kernel_path)) + % (self.kernel_path)) self.vm.launch()Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()
Quoted 10 lines · Daniel P. Berrangé
Quoted 3 lines · Cleber Rosa
The kernel is a common blob used in all tests. By moving it to the
setUp()method, the "fetch asset" plugin will recognize the kernel and attempt to fetch it and cache it before the tests are started.The other tests don't call
fetch_asset()from their setUp method - what's different about this test that prevents the asset caching working ?
The "wizardry" of the "fetch asset" plugin limits itself to analyzing test methods and the setUp() method. This test currently does that in the "fetch_guest_kernel()" method, which makes it blind to the "fetch asset" plugin.
If there are other tests doing that (in custom methods), then a similar change would be desirable.
Regards,
- Cleber.
PATCH 10/13 tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
In reply to Cleber Rosa--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
tests/avocado/tuxrun_baselines.py +6 -10
@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_command
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(), + 'Could not find "zstd", or it is not able to properly ' + 'decompress decompress the rootfs') def setUp(self): super().setUp() - # We need zstd for all the tuxrun tests - # See https://github.com/avocado-framework/avocado/issues/5609 - zstd = find_command('zstd', False) - if zstd is False: - self.cancel('Could not find "zstd", which is required to ' - 'decompress rootfs') - self.zstd = zstd - # Process the TuxRun specific tags, most machines work with # reasonable defaults but we sometimes need to tweak the # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None): asset_hash = isum, algorithm = "sha256") - cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4" - process.run(cmd) + archive.extract(disk_image_zst, os.path.join(self.workdir, + "rootfs.ext4")) if dt: dsum = csums.get(dt, None)
Signature · 1 line
2.45.2
Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 4 lines · Cleber Rosa
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
In reply to Cleber RosaQuoted 18 lines · Cleber Rosa
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)tests/avocado/tuxrun_baselines.py +1
-0@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_command
tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44
Quoted 4 lines
tests/avocado/tuxrun_baselines.py +1
-0@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(),
_probe_zstd_cmd() isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.
Quoted 27 lines
tests/avocado/tuxrun_baselines.py +4
-10+ 'Could not find "zstd", or it is not able to properly ' + 'decompress decompress the rootfs') def setUp(self): super().setUp() - # We need zstd for all the tuxrun tests - # See https://github.com/avocado-framework/avocado/issues/5609 - zstd = find_command('zstd', False) - if zstd is False: - self.cancel('Could not find "zstd", which is required to ' - 'decompress rootfs') - self.zstd = zstd - # Process the TuxRun specific tags, most machines work with # reasonable defaults but we sometimes need to tweak the # config. To avoid open coding everything we store all these@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None): asset_hash = isum, algorithm = "sha256") - cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4" - process.run(cmd) + archive.extract(disk_image_zst, os.path.join(self.workdir, + "rootfs.ext4")) if dt: dsum = csums.get(dt, None)
Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Cleber Rosa
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)tests/avocado/tuxrun_baselines.py +1
-0@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_command
Quoted 9 lines
tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44
Quoted 5 lines
tests/avocado/tuxrun_baselines.py +1
-0@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(),
_probe_zstd_cmd()isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.
While it's not the best practice to use private functions, I just couldn't accept rewriting that for the skip condition. I can make sure future versions (including 103.1) make it public.
Also, these patches count on the bump to 103.0 indeed.
Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 31 lines · Cleber Rosa
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 17 lines · Cleber Rosa
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/tuxrun_baselines.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)tests/avocado/tuxrun_baselines.py +1
-0@@ -17,6 +17,7 @@ from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern +from avocado.utils import archive from avocado.utils import process from avocado.utils.path import find_commandQuoted 9 lines
tests/avocado/tuxrun_baselines.py def get_tag(self, tagname, default=None): · line 44
Quoted 5 lines
tests/avocado/tuxrun_baselines.py +1
-0@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None): return default + @skipUnless(archive._probe_zstd_cmd(),
_probe_zstd_cmd()isn't public AFAICT, but more importantly this doesn't work because this method has been added in v101.0.
Quoted 5 lines
While it's not the best practice to use private functions, I just couldn't accept rewriting that for the skip condition. I can make sure future versions (including 103.1) make it public.
Also, these patches count on the bump to 103.0 indeed.
Then either mention it in the commit description to avoid wasting time to developers cherry-picking / testing this single patch, or move it after the version bump, avoiding bisectability issues.
Thanks,
Phil.
Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
Quoted 6 lines · Philippe Mathieu-Daudé
Quoted 1 line
Also, these patches count on the bump to 103.0 indeed.
Then either mention it in the commit description to avoid wasting time to developers cherry-picking / testing this single patch, or move it after the version bump, avoiding bisectability issues.
You're right. My bad.
Regards,
- Cleber.
Quoted 4 lines
Thanks,
Phil.
PATCH 11/13 tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
In reply to Cleber RosaWhen the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own copy.
--- tests/avocado/machine_aarch64_sbsaref.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tests/avocado/machine_aarch64_sbsaref.py +4 -1
@@ -7,6 +7,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later import os +import shutil from avocado import skipUnless from avocado.utils import archive
@@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu): ) img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5" - img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash) + cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash) + img_path = os.path.join(self.workdir, os.path.basename(cached_img_path)) + shutil.copy(cached_img_path, img_path) self.vm.set_console() self.vm.add_args(
Signature · 1 line
2.45.2
Re: [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
Quoted 22 lines · Cleber Rosa
When the OpenBSD based tests are run in parallel, the previously single instance of the image would become corrupt. Let's give each test its own copy.
Signed-off-by Cleber Rosa <[email protected]>--- tests/avocado/machine_aarch64_sbsaref.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)tests/avocado/machine_aarch64_sbsaref.py +1
-0@@ -7,6 +7,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later import os +import shutil from avocado import skipUnless from avocado.utils import archive
tests/avocado/machine_aarch64_sbsaref.py def boot_openbsd73(self, cpu): · line 193
Quoted 7 lines
tests/avocado/machine_aarch64_sbsaref.py +3
-1@@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu): ) img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5" - img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash) + cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash) + img_path = os.path.join(self.workdir, os.path.basename(cached_img_path)) + shutil.copy(cached_img_path, img_path)
Again I think we should be using a throwaway qcow2 overlay rather than copying the full image.
tests/avocado/machine_aarch64_sbsaref.py def boot_openbsd73(self, cpu): · line 196
Quoted 3 lines
tests/avocado/machine_aarch64_sbsaref.py +0
-0self.vm.set_console() self.vm.add_args(
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 12/13 Bump avocado to 103.0
In reply to Cleber RosaThis bumps Avocado to latest the LTS release.
An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.
Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
--- pythondeps.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pythondeps.toml +1 -1
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" } # Note that qemu.git/python/ is always implicitly installed. # Prefer an LTS version when updating the accepted versions of # avocado-framework, for example right now the limit is 92.x. -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" } pycdlib = { accepted = ">=1.11.0" }
Signature · 1 line
2.45.2
Re: [PATCH 12/13] Bump avocado to 103.0
Quoted 10 lines · Cleber Rosa
This bumps Avocado to latest the LTS release.
An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.
Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.
Rebasing to a completely re-written test harness while in freeze feels on the risky side to me, despite the known problems we have with the existing release.
Quoted 21 lines
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
Signed-off-by Cleber Rosa <[email protected]>--- pythondeps.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)pythondeps.toml +1
-1@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" } # Note that qemu.git/python/ is always implicitly installed. # Prefer an LTS version when updating the accepted versions of # avocado-framework, for example right now the limit is 92.x. -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" } pycdlib = { accepted = ">=1.11.0" }Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 12/13] Bump avocado to 103.0
In reply to Cleber RosaQuoted 10 lines · Cleber Rosa
This bumps Avocado to latest the LTS release.
An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made.
Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found.
Does that restore feature parity for macOS developers? Because this community has been left behind ignored for over 2 years and already looked at alternatives for functional testing.
Quoted 17 lines
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
Signed-off-by Cleber Rosa <[email protected]>--- pythondeps.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)pythondeps.toml +1
-1@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" } # Note that qemu.git/python/ is always implicitly installed. # Prefer an LTS version when updating the accepted versions of # avocado-framework, for example right now the limit is 92.x. -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" } pycdlib = { accepted = ">=1.11.0" }
Re: [PATCH 12/13] Bump avocado to 103.0
Quoted 4 lines · Philippe Mathieu-Daudé
Does that restore feature parity for macOS developers? Because this community has been left behind ignored for over 2 years and already looked at alternatives for functional testing.
Hi Phillipe,
As early as Avocado 102.0, macOS support is pretty complete. The exact words on the release notes[1] are:
"User of macOS will have a better experience when using Avocado. The full set of Avocado’s selftests are now run under macOS on CI. Please be advised that macOS is not currently supported at the same level of Linux-based operating systems due to the lack of contributors/maintainers with access to the needed hardware. If you are a user/developer and are willing to contribute to this, please let the Avocado team know."
When it comes to the lack of updates, that is a longer discussion indeed. When it comes to alternatives, I don't expect the QEMU project to do anything else than what it's in its best interest. As late as this can be, please take it for what it's worth. If it does any good to QEMU, please consider it.
Best,
- Cleber.
[1] - https://avocado-framework.readthedocs.io/en/latest/releases/102_0.html
Quoted 21 lines
Quoted 20 lines
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
Signed-off-by Cleber Rosa <[email protected]>---
pythondeps.toml | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)pythondeps.toml +0
-1@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" } # Note that qemu.git/python/ is always implicitly installed. # Prefer an LTS version when updating the accepted versions of # avocado-framework, for example right now the limit is 92.x. -avocado-framework = { accepted = "(>=88.1, <93.0)", installed ="88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
pycdlib = { accepted = ">=1.11.0" }
PATCH 13/13 Avocado tests: allow for parallel execution of tests
In reply to Cleber RosaThe updated Avocado version allows for the execution of tests in parallel.
While on a CI environment it may not be a good idea to increase the parallelization level in a single runner, developers may leverage that on specific CI runners or on their development environments.
This also multiplies the timeout for each test accordingly. The reason is that more concurrency can lead to less resources, and less resources can lead to some specific tests taking longer to complete and then time out. The timeout factor being used here is very conservative (being equal to the amount of parallel tasks). The worst this possibly oversized timeout value can do is making users wait a bit longer for the job to finish if a test hangs.
Overall, users can expect a much quicker turnaround on most systems with a value such as 8 on a 12 core machine.
--- docs/devel/testing.rst | 12 ++++++++++++ tests/Makefile.include | 6 +++++- 2 files changed, 17 insertions(+), 1 deletion(-)
docs/devel/testing.rst +12 -0
@@ -983,6 +983,18 @@ of Avocado or ``make check-avocado``, and can also be queried using: pyvenv/bin/avocado list tests/avocado +To run tests in parallel, the ``AVOCADO_PARALLEL`` environment +variable can be defined with a value different than ``1`` (its default +value). Example: + + .. code:: + + make check-avocado AVOCADO_PARALLEL=4 + +Please exercise care when using parallel execution with the QEMU +Avocado tests as a higher system load can cause time sensitive tests +to timeout and be interrupted. + Manual Installation ~~~~~~~~~~~~~~~~~~~
tests/Makefile.include +5 -1
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results ifndef AVOCADO_TESTS AVOCADO_TESTS=tests/avocado endif +ifndef AVOCADO_PARALLEL + AVOCADO_PARALLEL=1 +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) \ + $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \ + -p timeout_factor=$(AVOCADO_PARALLEL) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")
Signature · 1 line
2.45.2
Quoted 48 lines · Cleber Rosa
This is a *long* overdue update of the Avocado version used in QEMU. It comes a time where the role of the runner and the libraries are being discussed and questioned.
These exact commits have been staging on my side for over 30 days now, and I was exceeding what I should in terms of testing before posting. I apologize for the miscalculation.
Nevertheless, as pointed out, on the ML, these changes are needed NOW.
Some examples of runs in the CI can be seen below:
* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 432.63 s* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 148.99 sCleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugintests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access tests/avocado: use more distinct names for assetstests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado:test_arm_emcraft_sf2: handle RW requirements for assettests/avocado/boot_xen.py: fetch kernel during testsetUp()tests/avocado/tuxrun_baselines.py: use Avocado's zstd supporttests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of testsdocs/devel/testing.rst | 12 +++++++ pythondeps.toml | 2 +- tests/Makefile.include | 6 +++- tests/avocado/boot_linux_console.py | 24 ++++++++------ tests/avocado/boot_xen.py | 13 ++++---- tests/avocado/cdrom.py | 41 ++++++++++++++++++++++++ tests/avocado/intel_iommu.py | 2 ++ tests/avocado/kvm_xen_guest.py | 30 +++++++++++------ tests/avocado/machine_aarch64_sbsaref.py | 11 +++++-- tests/avocado/machine_aarch64_virt.py | 14 ++++---- tests/avocado/netdev-ethtool.py | 3 +- tests/avocado/tuxrun_baselines.py | 16 ++++----- 12 files changed, 125 insertions(+), 49 deletions(-)
Quoted 2 lines
create mode 100644
tests/avocado/cdrom.py
Quoted 6 lines · Cleber Rosa
Cleber Rosa (13):
tests/avocado: mips: fallback to HTTP given certificate expiration tests/avocado: mips: add hint for fetchasset plugintests/avocado/intel_iommu.py: increase timeout tests/avocado: add cdrom permission related tests tests/avocado: machine aarch64: standardize location and RO access
Quoted 8 lines
tests/avocado: use more distinct names for assets
tests/avocado/kvm_xen_guest.py: cope with asset RW requirements testa/avocado:test_arm_emcraft_sf2: handle RW requirements for assettests/avocado/boot_xen.py: fetch kernel during testsetUp()tests/avocado/tuxrun_baselines.py: use Avocado's zstd supporttests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image Bump avocado to 103.0
Avocado tests: allow for parallel execution of tests
Queuing patches #1, #2 and #8. Unfortunately #10 depends on Avocado 101.0 so can't be merged at this point.