Skip to contentSkip to conversation

Acceptance tests queue

PATCH In review Philippe Mathieu-Daudé 4 participants 21 messages Apr 3, 2020 – Apr 6, 2020

lore
Share & tools
lore
#0 lore raw

cover Acceptance tests queue

I'm planning to send the first 5 patches as
This series contains acceptance test fixes, and Travis-CI improvements. I plan to send a pull request with the first 6 patches for v5.0-rc2. The last 3 patches helped debugging painfully the Travis failures, but can not be used because Avocado 78.0 is not released yet.

Patches 3 and 5 are not reviewed.

Up-to-patch 5 job:
https://travis-ci.org/github/philmd/qemu/jobs/670645531

All patches job:
https://travis-ci.org/github/philmd/qemu/jobs/670645611

Oksana Vohchana (1):
Acceptance test: Fix to EXEC migration

Philippe Mathieu-Daudé (7):
tests/acceptance/ppc_prep_40p: Use cdn.netbsd.org hostname tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com tests/acceptance/machine_sparc_leon3: Disable HelenOS test .travis.yml: Cache acceptance-test assets tests/Makefile: Add fetch-acceptance-assets rule .travis.yml: Run fetch-acceptance-assets before check-acceptance NOTFORMERGE Kludge for Avocado issue #3661

 .travis.yml                             | 4 +++-
 tests/Makefile.include                  | 7 +++++++
 tests/acceptance/machine_sparc_leon3.py | 4 ++++
 tests/acceptance/migration.py           | 6 +++---
 tests/acceptance/ppc_prep_40p.py        | 7 ++++---
 tests/requirements.txt                  | 2 +-
 6 files changed, 22 insertions(+), 8 deletions(-)
Signature · 1 line
2.21.1
#1 lore raw

PATCH 1/8 Acceptance test: Fix to EXEC migration

From: Oksana Vohchana <[email protected]>

The exec migration test isn't run a whole test scenario. This patch fixes it

Message-Id: <[email protected]>

---
 tests/acceptance/migration.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tests/acceptance/migration.py +3 -3
index a8367ca023..0365289cda 100644
@@ -70,8 +70,8 @@ def test_migration_with_unix(self):
 
     @skipUnless(find_command('nc', default=False), "'nc' command not found")
     def test_migration_with_exec(self):
-        """
-        The test works for both netcat-traditional and netcat-openbsd packages
-        """
+        """The test works for both netcat-traditional and netcat-openbsd packages."""
         free_port = self._get_free_port()
         dest_uri = 'exec:nc -l localhost %u' % free_port
+        src_uri = 'exec:nc localhost %u' % free_port
+        self.do_migrate(dest_uri, src_uri)
Signature · 1 line
2.21.1
#2 lore raw

PATCH 2/8 tests/acceptance/ppc_prep_40p: Use cdn.netbsd.org hostname

In reply to Philippe Mathieu-Daudé

Use NetBSD content delivery network to get faster downloads.

Message-Id: <[email protected]>

---
 tests/acceptance/ppc_prep_40p.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/acceptance/ppc_prep_40p.py +2 -2
index b27572f212..138064285a 100644
@@ -34,7 +34,7 @@ def test_factory_firmware_and_netbsd(self):
                     '7020-40p/P12H0456.IMG')
         bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
         bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
-        drive_url = ('https://ftp.netbsd.org/pub/NetBSD/NetBSD-archive/'
+        drive_url = ('https://cdn.netbsd.org/pub/NetBSD/NetBSD-archive/'
                      'NetBSD-4.0/prep/installation/floppy/generic_com0.fs')
         drive_hash = 'dbcfc09912e71bd5f0d82c7c1ee43082fb596ceb'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
@@ -66,7 +66,7 @@ def test_openbios_and_netbsd(self):
         :avocado: tags=arch:ppc
         :avocado: tags=machine:40p
         """
-        drive_url = ('https://ftp.netbsd.org/pub/NetBSD/iso/7.1.2/'
+        drive_url = ('https://cdn.netbsd.org/pub/NetBSD/iso/7.1.2/'
                      'NetBSD-7.1.2-prep.iso')
         drive_hash = 'ac6fa2707d888b36d6fa64de6e7fe48e'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash,
Signature · 1 line
2.21.1
#3 lore raw

PATCH 3/8 tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

In reply to Philippe Mathieu-Daudé

To avoid regular failures on Travis-CI with ftp.software.ibm.com, use a mirror.

---
 tests/acceptance/ppc_prep_40p.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/acceptance/ppc_prep_40p.py +2 -1
index 138064285a..1515561249 100644
@@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
         :avocado: tags=machine:40p
         :avocado: tags=slowness:high
         """
-        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
+        bios_url = ('http://ftpmirror.your.org/pub/misc/'
+                    'ftp.software.ibm.com/rs6000/firmware/'
                     '7020-40p/P12H0456.IMG')
         bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
         bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
Signature · 1 line
2.21.1
Reviewed-by on v1 3/8
#4 lore raw

Re: [PATCH-for-5.0 3/8] tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

Quoted 13 lines · Philippe Mathieu-Daudé

To avoid regular failures on Travis-CI with ftp.software.ibm.com, use a mirror.

---
  tests/acceptance/ppc_prep_40p.py | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
tests/acceptance/ppc_prep_40p.py +0 -0
index 138064285a..1515561249 100644

tests/acceptance/ppc_prep_40p.py def test_factory_firmware_and_netbsd(self): · line 35

Quoted 7 lines
tests/acceptance/ppc_prep_40p.py +2 -1
@@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
          :avocado: tags=machine:40p
          :avocado: tags=slowness:high
          """
-        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
+        bios_url = ('http://ftpmirror.your.org/pub/misc/'
+                    'ftp.software.ibm.com/rs6000/firmware/'
                      '7020-40p/P12H0456.IMG')

I checked the bios file can be downloaded from the mirror. I cannot judge the mirror is more reliable than ftp.boulder.ibm.com or not, though.

Code-wise, looks good. So:

Quoted 2 lines
tests/acceptance/ppc_prep_40p.py +0 -0
          bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
          bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
Tested-by Reviewed-by
#5 lore raw

Re: [PATCH-for-5.0 3/8] tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

Quoted 11 lines · Wainer dos Santos Moschetta
Quoted 8 lines · Philippe Mathieu-Daudé

To avoid regular failures on Travis-CI with ftp.software.ibm.com, use a mirror.

---
  tests/acceptance/ppc_prep_40p.py | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 16 lines

tests/acceptance/ppc_prep_40p.py def test_factory_firmware_and_netbsd(self): · line 35

Quoted 12 lines
tests/acceptance/ppc_prep_40p.py +2 -1
index 138064285a..1515561249 100644
@@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
          :avocado: tags=machine:40p
          :avocado: tags=slowness:high
          """
-        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
+        bios_url = ('http://ftpmirror.your.org/pub/misc/'
+                    'ftp.software.ibm.com/rs6000/firmware/'
                      '7020-40p/P12H0456.IMG')

I checked the bios file can be downloaded from the mirror. I cannot judge the mirror is more reliable than ftp.boulder.ibm.com or not, though.

During the tests I did to debug Avocado I constantly got a timeout on FTP. The HTTP URL seems to be more reliable. So far,

Quoted 7 lines

Code-wise, looks good. So:

Quoted 2 lines
tests/acceptance/ppc_prep_40p.py +0 -0
          bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
          bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
#6 lore raw

PATCH 4/8 tests/acceptance/machine_sparc_leon3: Disable HelenOS test

In reply to Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

This test was written/tested around beginning of 2019, but was extracted from a bigger series and posted end of June 2019 [*]. Unfortunately I did not notice commit 162abf1a8 was merged by then, which implements the AHB and APB plug and play devices.

HelenOS 0.6 is expecting the PnP registers to be not implemented by QEMU, then forces the discovered AMBA devices (see [2]).

Before 162abf1a8, the console was displaying:

HelenOS bootloader, release 0.6.0 (Elastic Horse) Built on 2014-12-21 20:17:42 for sparc32 Copyright (c) 2001-2014 HelenOS project

   0x4000bf20|0x4000bf20: kernel image (496640/128466 bytes)
   0x4002b4f2|0x4002b4f2: ns image (154195/66444 bytes)
   0x4003b87e|0x4003b87e: loader image (153182/66437 bytes)
   0x4004bc03|0x4004bc03: init image (155339/66834 bytes)
   0x4005c115|0x4005c115: locsrv image (162063/70267 bytes)
   0x4006d390|0x4006d390: rd image (152678/65889 bytes)
   0x4007d4f1|0x4007d4f1: vfs image (168480/73394 bytes)
   0x4008f3a3|0x4008f3a3: logger image (158034/68368 bytes)
   0x4009feb3|0x4009feb3: ext4fs image (234510/100301 bytes)
   0x400b8680|0x400b8680: initrd image (8388608/1668901 bytes)

ABMA devices:
<1:00c> at 0x80000100 irq 3
<1:00d> at 0x80000200
<1:011> at 0x80000300 irq 8
Memory size: 64 MB

As of this commit, it is now confused:

ABMA devices:
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
<1:3000> at 0x00000000 irq 0
...

As this test is not working as expected, simply disable it (by skipping it) for now.

[1] https://www.mail-archive.com/[email protected]/msg627094.html
[2] https://github.com/HelenOS/helenos/blob/0.6.0/boot/arch/sparc32/src/ambapp.c#L75

Message-Id: <[email protected]>

---
 tests/acceptance/machine_sparc_leon3.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/acceptance/machine_sparc_leon3.py +4 -0
index f77e210ccb..2405cd7a0d 100644
@@ -7,12 +7,16 @@
 
 from avocado_qemu import Test
 from avocado_qemu import wait_for_console_pattern
+from avocado import skip
 
 
 class Leon3Machine(Test):
 
     timeout = 60
 
+    @skip("Test currently broken")
+    # A Window Underflow exception occurs before booting the kernel,
+    # and QEMU exit calling cpu_abort(), which makes this test to fail.
     def test_leon3_helenos_uimage(self):
         """
         :avocado: tags=arch:sparc
Signature · 1 line
2.21.1
#7 lore raw

PATCH 5/8 .travis.yml: Cache acceptance-test assets

In reply to Philippe Mathieu-Daudé

Keep all acceptance-test assets in the same cache bucket.

As of v5.0.0-rc1, the cache is 2610.11MB:
https://travis-ci.org/github/philmd/qemu/jobs/670558103

---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index e0c72210b7..2fd63eceaa 100644
@@ -318,8 +318,10 @@ jobs:
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+        - du -chs $HOME/avocado/data/cache
       addons:
         apt:
           packages:
Signature · 1 line
2.21.1
Reviewed-by on v1 5/8
#8 lore raw

Re: [PATCH-for-5.0 5/8] .travis.yml: Cache acceptance-test assets

Quoted 9 lines · Philippe Mathieu-Daudé

Keep all acceptance-test assets in the same cache bucket.

As of v5.0.0-rc1, the cache is 2610.11MB:
https://travis-ci.org/github/philmd/qemu/jobs/670558103

---
  .travis.yml | 2 ++
  1 file changed, 2 insertions(+)
Quoted 16 lines
.travis.yml +2 -0
index e0c72210b7..2fd63eceaa 100644
@@ -318,8 +318,10 @@ jobs:
        env:
          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
          - TEST_CMD="make check-acceptance"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
        after_script:
          - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+        - du -chs $HOME/avocado/data/cache
        addons:
          apt:
            packages:
Acked-by on v1 5/8
#9 lore raw

Re: [PATCH-for-5.0 5/8] .travis.yml: Cache acceptance-test assets

In reply to Philippe Mathieu-Daudé
Quoted 6 lines · Philippe Mathieu-Daudé

Keep all acceptance-test assets in the same cache bucket.

As of v5.0.0-rc1, the cache is 2610.11MB:
https://travis-ci.org/github/philmd/qemu/jobs/670558103

Quoted 19 lines
---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index e0c72210b7..2fd63eceaa 100644
@@ -318,8 +318,10 @@ jobs:
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+        - du -chs $HOME/avocado/data/cache
       addons:
         apt:
           packages:
Signature · 1 line
Alex Bennée
#10 lore raw

PATCH 6/8 tests/Makefile: Add fetch-acceptance-assets rule

In reply to Philippe Mathieu-Daudé
---
 tests/Makefile.include | 7 +++++++
 1 file changed, 7 insertions(+)
tests/Makefile.include +7 -0
index 51de676298..90f457593c 100644
@@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+fetch-acceptance-assets: check-venv
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
+            tests/acceptance/*py, \
+            "AVOCADO", "tests/acceptance")
+
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
Signature · 1 line
2.21.1
#11 lore raw

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

Quoted 10 lines · Philippe Mathieu-Daudé
---
 tests/Makefile.include | 7 +++++++
 1 file changed, 7 insertions(+)
tests/Makefile.include +0 -0
index 51de676298..90f457593c 100644

tests/Makefile.include get-vm-image-fedora-31-%: check-venv · line 915

Quoted 10 lines
tests/Makefile.include +7 -0
@@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+fetch-acceptance-assets: check-venv
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
+            tests/acceptance/*py, \
+            "AVOCADO", "tests/acceptance")
+

I'm wondering if we could expand this to a rule per-test group and therefor allow parallel fetching of groups of assets?

Quoted 3 lines
tests/Makefile.include +0 -0
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
Signature · 1 line
Alex Bennée
#12 lore raw

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

Quoted 17 lines · Alex Bennée
Quoted 13 lines · Philippe Mathieu-Daudé
---
 tests/Makefile.include | 7 +++++++
 1 file changed, 7 insertions(+)
tests/Makefile.include +0 -0
index 51de676298..90f457593c 100644
@@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
Quoted 10 lines

tests/Makefile.include get-vm-image-fedora-31-%: check-venv · line 915

Quoted 7 lines
tests/Makefile.include +7 -0
+fetch-acceptance-assets: check-venv
+     $(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
+            tests/acceptance/*py, \
+            "AVOCADO", "tests/acceptance")
+

I'm wondering if we could expand this to a rule per-test group and therefor allow parallel fetching of groups of assets?

This is a valid idea! Additionally, my suggestion is to organize the tests by subdirectories under tests/acceptance/. Doing so makes it easy to handle the target rule, adding the subdirectory to the assets fetch command instead of listing specific tests for each group. Making a list of tests for each group requires extra editing to the makefile target when a new test is added. This can lead to new tests being missed in their groups by mistake.

Quoted 9 lines

tests/Makefile.include get-vm-image-fedora-31-%: check-venv · line 918

Quoted 4 lines
tests/Makefile.include +0 -0
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
      $(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \

--
Alex Bennée

#13 lore raw

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

Quoted 29 lines · Willian Rampazzo
Quoted 17 lines · Alex Bennée
Quoted 13 lines · Philippe Mathieu-Daudé
---
 tests/Makefile.include | 7 +++++++
 1 file changed, 7 insertions(+)
tests/Makefile.include +0 -0
index 51de676298..90f457593c 100644
@@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
Quoted 10 lines

tests/Makefile.include get-vm-image-fedora-31-%: check-venv · line 915

Quoted 7 lines
tests/Makefile.include +7 -0
+fetch-acceptance-assets: check-venv
+     $(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
+            tests/acceptance/*py, \
+            "AVOCADO", "tests/acceptance")
+

I'm wondering if we could expand this to a rule per-test group and therefor allow parallel fetching of groups of assets?

Quoted 7 lines

This is a valid idea! Additionally, my suggestion is to organize the tests by subdirectories under tests/acceptance/. Doing so makes it easy to handle the target rule, adding the subdirectory to the assets fetch command instead of listing specific tests for each group. Making a list of tests for each group requires extra editing to the makefile target when a new test is added. This can lead to new tests being missed in their groups by mistake.

I'd use wildcards to build up the groups so the Makefile doesn't need to be messed with again.

Quoted 10 lines
Quoted 10 lines

tests/Makefile.include get-vm-image-fedora-31-%: check-venv · line 918

Quoted 5 lines
tests/Makefile.include +0 -0
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
      $(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \

--
Alex Bennée

Signature · 1 line
Alex Bennée
#14 lore raw

PATCH 7/8 .travis.yml: Run fetch-acceptance-assets before check-acceptance

In reply to Philippe Mathieu-Daudé
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.travis.yml +1 -1
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
       dist: bionic
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets check-acceptance DEBUG=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
Signature · 1 line
2.21.1
#15 lore raw

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

Quoted 15 lines · Philippe Mathieu-Daudé
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.travis.yml +1 -1
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
       dist: bionic
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
Quoted 1 line

check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can that be done in parallel?

Also no point in -j1 here - it's implied.

Quoted 3 lines

- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

Signature · 1 line
Alex Bennée
#16 lore raw

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

Quoted 8 lines · Alex Bennée
Quoted 5 lines · Philippe Mathieu-Daudé
---
  .travis.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
Quoted 11 lines
.travis.yml +1 -1
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
        dist: bionic
        env:
          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets

check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can that be done in parallel?

I'd say calling fetch-acceptance-assets parallelized is what seems to break this Travis job (which is why I enforced -j1), but I'll refer to Willian here.

Quoted 2 lines

Also no point in -j1 here - it's implied.

You are right, I wanted to be sure this still work even if the globlal script running the tests is updated. But the correct fix is probably to use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.

Quoted 6 lines
Quoted 4 lines

- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

#17 lore raw

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 8 lines · Alex Bennée
Quoted 5 lines · Philippe Mathieu-Daudé
---
  .travis.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
.travis.yml +0 -0
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
        dist: bionic
Quoted 8 lines
Quoted 5 lines
.travis.yml +1 -1
        env:
          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets

check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can that be done in parallel?

If by parallel you mean running it with other targets, the fetch-acceptance-assets target can run at any time before the check-acceptance runs, concurrently with different targets. Now, if by parallel you mean fetch more than one asset at a time, right now it is not supported by Avocado, assets are fetched one by one, but it is an excellent idea. I have added it to my list.

Quoted 5 lines

I'd say calling fetch-acceptance-assets parallelized is what seems to break this Travis job (which is why I enforced -j1), but I'll refer to Willian here.

As I mentioned above, no problem with running it in parallel to other targets. The errors we saw when you were testing are not related to the parallel run. It should be fixed in the next release of Avocado.

Quoted 14 lines
Quoted 2 lines

Also no point in -j1 here - it's implied.

You are right, I wanted to be sure this still work even if the globlal script running the tests is updated. But the correct fix is probably to use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.

Quoted 6 lines
Quoted 4 lines

- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

#18 lore raw

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

Quoted 27 lines · Willian Rampazzo
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 8 lines · Alex Bennée
Quoted 5 lines · Philippe Mathieu-Daudé
---
  .travis.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
.travis.yml +0 -0
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
        dist: bionic
Quoted 8 lines
Quoted 5 lines
.travis.yml +1 -1
        env:
          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets

check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can that be done in parallel?

Quoted 6 lines

If by parallel you mean running it with other targets, the fetch-acceptance-assets target can run at any time before the check-acceptance runs, concurrently with different targets. Now, if by parallel you mean fetch more than one asset at a time, right now it is not supported by Avocado, assets are fetched one by one, but it is an excellent idea. I have added it to my list.

See the comment I made about the make file. We could expand the fetch-acceptance-assets target to have one dependency per python file so you could invoke avocado in parallel to fetch the assets for each test group in parallel. Of course this works best if the assets mentioned in each file are unique otherwise avocado might race with itself.

Quoted 23 lines
Quoted 4 lines

I'd say calling fetch-acceptance-assets parallelized is what seems to break this Travis job (which is why I enforced -j1), but I'll refer to Willian here.

As I mentioned above, no problem with running it in parallel to other targets. The errors we saw when you were testing are not related to the parallel run. It should be fixed in the next release of Avocado.

Quoted 14 lines
Quoted 2 lines

Also no point in -j1 here - it's implied.

You are right, I wanted to be sure this still work even if the globlal script running the tests is updated. But the correct fix is probably to use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.

Quoted 6 lines
Quoted 4 lines

- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

Signature · 1 line
Alex Bennée
#19 lore raw

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

Quoted 38 lines · Alex Bennée
Quoted 27 lines · Willian Rampazzo
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 8 lines · Alex Bennée
Quoted 5 lines · Philippe Mathieu-Daudé
---
  .travis.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
.travis.yml +0 -0
index 2fd63eceaa..c6b32da447 100644
@@ -317,7 +317,7 @@ jobs:
        dist: bionic
Quoted 8 lines
Quoted 5 lines
.travis.yml +1 -1
        env:
          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets

check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can that be done in parallel?

Quoted 6 lines

If by parallel you mean running it with other targets, the fetch-acceptance-assets target can run at any time before the check-acceptance runs, concurrently with different targets. Now, if by parallel you mean fetch more than one asset at a time, right now it is not supported by Avocado, assets are fetched one by one, but it is an excellent idea. I have added it to my list.

Quoted 5 lines

See the comment I made about the make file. We could expand the fetch-acceptance-assets target to have one dependency per python file so you could invoke avocado in parallel to fetch the assets for each test group in parallel. Of course this works best if the assets mentioned in each file are unique otherwise avocado might race with itself.

I see what you mean. About the unique asset definitions, this should not be a big problem as Avocado downloads the asset on a temporary file first and then copy it to the target file. In the worst case, multiple instances of Avocado will download the same asset at the same time, but just one will make it the target asset. The other instances will fail, but as one already succeeded, the downside is the time and bandwidth spent by the other instances.

Quoted 29 lines
Quoted 24 lines
Quoted 5 lines

I'd say calling fetch-acceptance-assets parallelized is what seems to break this Travis job (which is why I enforced -j1), but I'll refer to Willian here.

As I mentioned above, no problem with running it in parallel to other targets. The errors we saw when you were testing are not related to the parallel run. It should be fixed in the next release of Avocado.

Quoted 14 lines
Quoted 2 lines

Also no point in -j1 here - it's implied.

You are right, I wanted to be sure this still work even if the globlal script running the tests is updated. But the correct fix is probably to use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.

Quoted 6 lines
Quoted 4 lines

- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

--
Alex Bennée

#20 lore raw

PATCH 8/8 Kludge for Avocado issue #3661

In reply to Philippe Mathieu-Daudé

Use a feature from unreleased Avocado v78.0.
See https://github.com/avocado-framework/avocado/issues/3661

---
 tests/requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
index f9c84b4ba1..d625b32dbb 100644
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==76.0
+-e git+https://github.com/avocado-framework/avocado.git@f9b4dc7c58a6424eb8d0ed6781a1d76ae3a5ab06#egg=avocado-framework
 pycdlib==1.9.0
Signature · 1 line
2.21.1