Skip to contentSkip to conversation

avocado: use sha1 for fc31 imgs to avoid first time re-download

PATCH In review Daniel Henrique Barboza 5 participants 10 messages Oct 23, 2022 – Nov 15, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 avocado: use sha1 for fc31 imgs to avoid first time re-download

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

Back in QEMU, the LinuxDistro class defines the images it is going to use in the KNOWN_DISTROS attribute. All images have the 'checksum' attribute, defined using sha256, that will make avocado check if it has the right asset in the cache. So this means that we're downloading a sha1 image during check-avocado, then trying to fetch it using a sha256 hash.

This doesn't work. Avocado will match the provided checksum, a sha256 hash, with the sha1 hash from the existing CHECKSUM file for the asset, and will re-download the image again. Avocado doesn't take into account that we're using a different hash algorithm than the one used to fetch the image via get-vm-image-fedora31 (in avocado: utils/asset.py, self.find_asset_file() and fetch(self, timeout=None). The new download will take into account the chosen hash algorithm, and subsequent avocado runs will not re-download the image again because now it's comparing sha256 with sha256.

All of this, of course, assuming that the system was able to re-download the image before the avocado test timeouts. A more "fun" scenario happens if the network is not fast enough to download the image during the avocado timeout, the test will fail and the sha1 image will remain in the cache. And the test will keep trying to re-fetch it, and will keep failing with a timeout.

There's something to be said about avocado matching different sha algorithms instead of re-calculating the hash with the chosen algorithm during fetch(), especially considering that the CHECKSUM file contains the name of the algorithm used. An avocado feature request [2] was opened to see if this is a feasible change to do.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Cc Alex Bennée <[email protected]>
Cc Thomas Huth <[email protected]>
Cc John Snow <[email protected]>
Cc Cleber Rosa <[email protected]>
Cc Wainer dos Santos Moschetta <[email protected]>
---
 tests/avocado/avocado_qemu/__init__.py | 30 ++++++++++++++++++--------
 1 file changed, 21 insertions(+), 9 deletions(-)
tests/avocado/avocado_qemu/__init__.py +21 -9
index 910f3ba1ea..68c42948a3 100644
@@ -423,12 +423,24 @@ class LinuxDistro:
     Holds information of known distros.
     """
     #: A collection of known distros and their respective image checksum
+    #
+    # 'get-vm-image-fedora-31' target from Makefile.include will
+    # download the images using avocado default 'sha1' algorithm.
+    # This happens because the command line 'avocado vmimage get'
+    # from Makefile.include is not able to handle an extra --algorithm
+    # argument.
+    #
+    # To avoid a re-download of a recent image download due to a sha
+    # mismatch in the first test run, all Fedora 31 images are using
+    # 'sha1' instead of 'sha256'.
+    #
+    # FIXME: revisit this change if/when avocado learns to do CHECKSUM
+    # matches using different algorithms.
     KNOWN_DISTROS = {
         'fedora': {
             '31': {
                 'x86_64':
-                {'checksum': ('e3c1b309d9203604922d6e255c2c5d09'
-                              '8a309c2d46215d8fc026954f3c5c27a0'),
+                {'checksum': ('f62f8eabbf3687ea610c495bd59551a0025f99b7'),
                  'pxeboot_url': ('https://archives.fedoraproject.org/'
                                  'pub/archive/fedora/linux/releases/31/'
                                  'Everything/x86_64/os/images/pxeboot/'),
@@ -438,8 +450,7 @@ class LinuxDistro:
                                    'console=ttyS0,115200n8'),
                 },
                 'aarch64':
-                {'checksum': ('1e18d9c0cf734940c4b5d5ec592facae'
-                              'd2af0ad0329383d5639c997fdf16fe49'),
+                {'checksum': ('39d1fe099cdecacae894480d421be57d18b4e854'),
                 'pxeboot_url': 'https://archives.fedoraproject.org/'
                                'pub/archive/fedora/linux/releases/31/'
                                'Everything/aarch64/os/images/pxeboot/',
@@ -450,11 +461,9 @@ class LinuxDistro:
                                   ' console=ttyAMA0'),
                 },
                 'ppc64':
-                {'checksum': ('7c3528b85a3df4b2306e892199a9e1e4'
-                              '3f991c506f2cc390dc4efa2026ad2f58')},
+                {'checksum': ('9993dc28e7c49ceb52125f9513130dfe2ace026c')},
                 's390x':
-                {'checksum': ('4caaab5a434fd4d1079149a072fdc789'
-                              '1e354f834d355069ca982fdcaf5a122d')},
+                {'checksum': ('455f017b82decf32f366e06e7a7d0f6da86f96a7')},
             },
             '32': {
                 'aarch64':
@@ -595,15 +604,18 @@ def download_boot(self):
         self.log.info('Downloading/preparing boot image')
         # Fedora 31 only provides ppc64le images
         image_arch = self.arch
+        hash_algorithm = 'sha256'
         if self.distro.name == 'fedora':
             if image_arch == 'ppc64':
                 image_arch = 'ppc64le'
+            if self.distro.version == '31':
+                hash_algorithm = 'sha1'
 
         try:
             boot = vmimage.get(
                 self.distro.name, arch=image_arch, version=self.distro.version,
                 checksum=self.distro.checksum,
-                algorithm='sha256',
+                algorithm=hash_algorithm,
                 cache_dir=self.cache_dirs[0],
                 snapshot_dir=self.workdir)
         except:
Signature · 1 line
2.37.3
#1 lore raw
Quoted 10 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

Quoted 6 lines

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
Quoted 15 lines · Daniel P. Berrangé
Quoted 14 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

Quoted 8 lines
Quoted 2 lines

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Sure, I think it would be better if we can get an Avocado side fix to make the current QEMU usage viable.

For now I'll use this patch locally because this issue is 100% reproducible on my home machine :( apparently my internet is capable of 4k streaming but, for some reason, it doesn't manage to download 350Mb from the Fedora mirror in less that 180 seconds. 'make check-avocado' always gives me a timeout in 'test_pseries_tcg'.

Thanks,

Daniel

Quoted 4 lines

With regards,
Daniel

#3 lore raw
In reply to Daniel P. Berrangé
Quoted 15 lines · Daniel P. Berrangé
Quoted 14 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

Quoted 8 lines
Quoted 2 lines

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

So unless someone is really committing to spend a lot of time on updating Avocado in QEMU in the near future, I don't think that such a fix for the hash algorithm will happen any time soon, and thus I think we should consider to include this work-around for the time being.

Thomas

#4 lore raw
Quoted 25 lines · Thomas Huth
Quoted 15 lines · Daniel P. Berrangé
Quoted 14 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

Quoted 8 lines
Quoted 2 lines

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw
Quoted 36 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth
Quoted 15 lines · Daniel P. Berrangé
Quoted 14 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

Quoted 8 lines
Quoted 2 lines

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

Quoted 3 lines

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

By the way, Avocado just provided a fix for the problem this patch is trying to amend:

https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846

Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream.

Thanks,

Daniel

Quoted 3 lines

With regards,
Daniel

#6 lore raw
Quoted 42 lines · Daniel Henrique Barboza
Quoted 36 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth
Quoted 15 lines · Daniel P. Berrangé
Quoted 14 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get  --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

Quoted 8 lines
Quoted 2 lines

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

Quoted 3 lines

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

Quoted 4 lines

By the way, Avocado just provided a fix for the problem this patch is trying to amend:

https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846

Thanks Jan!

Quoted 3 lines

Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream.

See https://lore.kernel.org/qemu-devel/[email protected]/

For your case:

-- >8 --

tests/requirements.txt +1 -1
index 0ba561b6bd..e17bc3972c 100644
@@ -4,3 +4,3 @@
 # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+-e git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework
 pycdlib==1.11.0

---

Regards,

Phil.

#7 lore raw
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel Henrique Barboza
Quoted 26 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth
Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get  --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

Quoted 6 lines

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

Hi Daniel,

if the problem of migrating avocado to latest version on qemu is only in parallel run, I would suggest to disable it with nrunner.max_parallel_tasks [1]. Even that the differences between avocado legacy runner and nrunner is huge, the migration should be straight forward. So if you have more issues with migration to the nrunner, I would be happy to help you with that.

[1] https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks

- Jan

Quoted 38 lines
Quoted 10 lines
Quoted 4 lines

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

By the way, Avocado just provided a fix for the problem this patch is trying to amend:

https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846

Thanks Jan!

Quoted 3 lines

Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream.

See https://lore.kernel.org/qemu-devel/[email protected]/

For your case:

-- >8 --

tests/requirements.txt +1 -1
index 0ba561b6bd..e17bc3972c 100644
@@ -4,3 +4,3 @@
  # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+-e 

git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework
 pycdlib==1.11.0
---

Regards,

Phil.

#8 lore raw
Quoted 41 lines · Jan Richter
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel Henrique Barboza
Quoted 26 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth
Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get  --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

Quoted 6 lines

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

Quoted 5 lines

Hi Daniel,

if the problem of migrating avocado to latest version on qemu is only in parallel run, I would suggest to disable it with nrunner.max_parallel_tasks [1]. Even that the differences between avocado legacy runner and nrunner is huge, the migration should be straight forward. So if you have more issues with migration to the nrunner, I would be happy to help you with that.

[1] https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks

Thanks Jan and Phil for the infos. I didn't manage to do a successful Avocado run with the QEMU test though.

What I did, aside from the changes that Phil mentioned in tests/requirements.txt:

- created a /etc/avocado/avocado.conf to store the settings
- copied python/avocado.cfg from QEMU to avocado.conf
- added the following in avocado.conf:

[nrunner]
max_parallel_tasks = 1

This allowed me to set Avocado as it would be if running with QEMU avocado, but even then I had no success. The test get stuck indefinitely at this point:

(...)

2022-11-10 16:00:20,758 avocado.test INFO | Temporary dir: /var/tmp/avocado_tmp_znhvpbh0/avocado_job_ywyf7v30
2022-11-10 16:00:20,758 avocado.test INFO |
2022-11-10 16:00:20,758 avocado.test INFO | Job ID: 4bb3e2a12c05d84a0a06849ecef435d547a198a0
2022-11-10 16:00:20,758 avocado.test INFO |
2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| <Worker spawner="<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>" max_triaging=12 max_running=1 task_timeout=None> has been initialized
2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requested -> triaging
2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requirements OK (will proceed to check dependencies)
2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": about to be spawned with "<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>"
2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully

No error is provided. Tried to run the test for 10+ minutes and nothing happens. Hitting CTRL+C aborts the test:

$ make check-avocado AVOCADO_TESTS='tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg'
  GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
  AVOCADO Downloading avocado tests VM image for ppc64le
The image was downloaded:
Provider Version Architecture File
fedora   31      ppc64le      /home/danielhb/avocado/data/cache/by_location/d73d707673a941f5892e4bb0fa7fabaf0d41fd58/Fedora-Cloud-Base-31-1.9.ppc64le.qcow2
  AVOCADO tests/avocado
JOB ID     : 4bb3e2a12c05d84a0a06849ecef435d547a198a0
JOB LOG    : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/job.log
^C
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/results.html
JOB TIME   : 396.78 s
make: *** [/home/danielhb/kvm-project/qemu/tests/Makefile.include:142: check-avocado] Error 8

And the log registers the following:

2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully
2022-11-10 16:06:54,071 avocado.test INFO |
2022-11-10 16:06:54,123 avocado.test INFO | Test results available in /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a
2022-11-10 16:06:55,694 avocado.sysinfo DEBUG| Not logging /proc/slabinfo (lack of permissions)
2022-11-10 16:06:55,696 avocado.sysinfo DEBUG| Not logging /proc/pci (file not found)
2022-11-10 16:06:55,712 avocado.sysinfo DEBUG| Not logging /sys/kernel/debug/sched_features (file not found)

[---end of log---]

I'm probably missing more settings.

Daniel

Quoted 41 lines

- Jan

Quoted 37 lines
Quoted 10 lines
Quoted 4 lines

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

By the way, Avocado just provided a fix for the problem this patch is trying to amend:

https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846

Thanks Jan!

Quoted 3 lines

Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream.

See https://lore.kernel.org/qemu-devel/[email protected]/

For your case:

-- >8 --

tests/requirements.txt +1 -1
index 0ba561b6bd..e17bc3972c 100644
@@ -4,3 +4,3 @@
   # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+-e git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework

  pycdlib==1.11.0
---

Regards,

Phil.

#9 lore raw
Quoted 107 lines · Daniel Henrique Barboza
Quoted 41 lines · Jan Richter
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel Henrique Barboza
Quoted 26 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth
Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Daniel Henrique Barboza

'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get  --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache.

Quoted 6 lines

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing...

Quoted 5 lines

Hi Daniel,

if the problem of migrating avocado to latest version on qemu is only in parallel run, I would suggest to disable it with nrunner.max_parallel_tasks [1]. Even that the differences between avocado legacy runner and nrunner is huge, the migration should be straight forward. So if you have more issues with migration to the nrunner, I would be happy to help you with that.

[1] https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks

Thanks Jan and Phil for the infos. I didn't manage to do a successful Avocado run with the QEMU test though.

What I did, aside from the changes that Phil mentioned in tests/requirements.txt:

- created a /etc/avocado/avocado.conf to store the settings
- copied python/avocado.cfg from QEMU to avocado.conf
- added the following in avocado.conf:

[nrunner]
max_parallel_tasks  = 1

This allowed me to set Avocado as it would be if running with QEMU avocado, but even then I had no success. The test get stuck indefinitely at this point:

(...)

2022-11-10 16:00:20,758 avocado.test INFO | Temporary dir: /var/tmp/avocado_tmp_znhvpbh0/avocado_job_ywyf7v30
2022-11-10 16:00:20,758 avocado.test INFO |
2022-11-10 16:00:20,758 avocado.test INFO | Job ID: 4bb3e2a12c05d84a0a06849ecef435d547a198a0
2022-11-10 16:00:20,758 avocado.test INFO |
2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| <Worker spawner="<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>" max_triaging=12 max_running=1 task_timeout=None> has been initialized
2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requested -> triaging
2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requirements OK (will proceed to check dependencies)
2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": about to be spawned with "<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>"
2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully

No error is provided. Tried to run the test for 10+ minutes and nothing happens. Hitting CTRL+C aborts the test:

$ make check-avocado AVOCADO_TESTS='tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg'
   GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 
tests/fp/berkeley-softfloat-3 dtc
   AVOCADO Downloading avocado tests VM image for ppc64le
The image was downloaded:
Provider Version Architecture File
fedora   31      ppc64le      /home/danielhb/avocado/data/cache/by_location/d73d707673a941f5892e4bb0fa7fabaf0d41fd58/Fedora-Cloud-Base-31-1.9.ppc64le.qcow2
   AVOCADO tests/avocado
JOB ID     : 4bb3e2a12c05d84a0a06849ecef435d547a198a0
JOB LOG    : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/job.log
^C
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/results.html
JOB TIME   : 396.78 s
make: *** [/home/danielhb/kvm-project/qemu/tests/Makefile.include:142: check-avocado] Error 8

And the log registers the following:

2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully
2022-11-10 16:06:54,071 avocado.test INFO |
2022-11-10 16:06:54,123 avocado.test INFO | Test results available in /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a
2022-11-10 16:06:55,694 avocado.sysinfo DEBUG| Not logging /proc/slabinfo (lack of permissions)
2022-11-10 16:06:55,696 avocado.sysinfo DEBUG| Not logging /proc/pci (file not found)
2022-11-10 16:06:55,712 avocado.sysinfo DEBUG| Not logging /sys/kernel/debug/sched_features (file not found)
Quoted 4 lines

[---end of log---]

I'm probably missing more settings.

Hi Daniel,

I discussed this problem on Avocado meeting today and I got information that Cleber Rosa is working on a patch for qemu to update scripts to Avocado 99.0 where he will fix those things, and he will publish it soon.

- Jan

Quoted 48 lines

Daniel

Quoted 42 lines

- Jan

Quoted 38 lines
Quoted 10 lines
Quoted 4 lines

That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-(

By the way, Avocado just provided a fix for the problem this patch is trying to amend:

https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846

Thanks Jan!

Quoted 3 lines

Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream.

See https://lore.kernel.org/qemu-devel/[email protected]/

For your case:

-- >8 --

tests/requirements.txt +1 -1
index 0ba561b6bd..e17bc3972c 100644
@@ -4,3 +4,3 @@
   # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+-e 

git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework pycdlib==1.11.0
---

Regards,

Phil.