Skip to contentSkip to conversation

BootLinuxSshTest: Only run the tests when explicitly requested

PATCH In review Philippe Mathieu-Daudé 2 participants 3 messages Sep 18, 2019 – Sep 19, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 BootLinuxSshTest: Only run the tests when explicitly requested

Currently the Avocado framework does not distinct the time spent downloading assets vs. the time spent running a test. With big assets (like a full VM image) the tests likely fail.

This is a limitation known by the Avocado team.
Until this issue get fixed, do not run this tests automatically.

Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED environment variable.

---
 tests/acceptance/linux_ssh_mips_malta.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tests/acceptance/linux_ssh_mips_malta.py +5 -5
index 134f10cac3..7200507a3a 100644
@@ -12,7 +12,7 @@ import logging
 import paramiko
 import time
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import Test
 from avocado.utils import process
 from avocado.utils import archive
@@ -171,7 +171,7 @@ class LinuxSSH(Test):
         self.run_common_commands()
         self.shutdown_via_ssh()
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta32eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips
@@ -186,7 +186,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('be', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta32el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mipsel
@@ -201,7 +201,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('le', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64
@@ -215,7 +215,7 @@ class LinuxSSH(Test):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
         self.check_mips_malta('be', kernel_path, 'mips64')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64el
Signature · 1 line
2.20.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 31 lines · Philippe Mathieu-Daudé

Currently the Avocado framework does not distinct the time spent downloading assets vs. the time spent running a test. With big assets (like a full VM image) the tests likely fail.

This is a limitation known by the Avocado team.
Until this issue get fixed, do not run this tests automatically.

Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED environment variable.

---
 tests/acceptance/linux_ssh_mips_malta.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tests/acceptance/linux_ssh_mips_malta.py +1 -1
index 134f10cac3..7200507a3a 100644
@@ -12,7 +12,7 @@ import logging
 import paramiko
 import time
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import Test
 from avocado.utils import process
 from avocado.utils import archive

tests/acceptance/linux_ssh_mips_malta.py class LinuxSSH(Test): · line 174

Quoted 5 lines
tests/acceptance/linux_ssh_mips_malta.py +1 -1
@@ -171,7 +171,7 @@ class LinuxSSH(Test):
         self.run_common_commands()
         self.shutdown_via_ssh()
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')

This is definitely an improvement, specially because IMO a "make check-acceptance" command should be more similar across executions, wether on Travis or not. The problem is (which is not new) is that users have to resort to the test to learn how to run those tests.

FIY, I'm working on a RFC to have a handful of "make check-acceptance-$(TYPE)" targets that hopefully can represent most users' needs. The avocado command line would still be available for more advanced users.

Quoted 33 lines
tests/acceptance/linux_ssh_mips_malta.py +3 -3
     def test_mips_malta32eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips
@@ -186,7 +186,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('be', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta32el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mipsel
@@ -201,7 +201,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('le', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64
@@ -215,7 +215,7 @@ class LinuxSSH(Test):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
         self.check_mips_malta('be', kernel_path, 'mips64')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64el
Signature · 1 line
2.20.1
#2 lore raw
Quoted 73 lines · Cleber Rosa
Quoted 31 lines · Philippe Mathieu-Daudé

Currently the Avocado framework does not distinct the time spent downloading assets vs. the time spent running a test. With big assets (like a full VM image) the tests likely fail.

This is a limitation known by the Avocado team.
Until this issue get fixed, do not run this tests automatically.

Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED environment variable.

---
 tests/acceptance/linux_ssh_mips_malta.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tests/acceptance/linux_ssh_mips_malta.py +1 -1
index 134f10cac3..7200507a3a 100644
@@ -12,7 +12,7 @@ import logging
 import paramiko
 import time
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import Test
 from avocado.utils import process
 from avocado.utils import archive

tests/acceptance/linux_ssh_mips_malta.py class LinuxSSH(Test): · line 174

Quoted 5 lines
tests/acceptance/linux_ssh_mips_malta.py +1 -1
@@ -171,7 +171,7 @@ class LinuxSSH(Test):
         self.run_common_commands()
         self.shutdown_via_ssh()
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')

This is definitely an improvement, specially because IMO a "make check-acceptance" command should be more similar across executions, wether on Travis or not. The problem is (which is not new) is that users have to resort to the test to learn how to run those tests.

FIY, I'm working on a RFC to have a handful of "make check-acceptance-$(TYPE)" targets that hopefully can represent most users' needs. The avocado command line would still be available for more advanced users.

Quoted 25 lines
tests/acceptance/linux_ssh_mips_malta.py +2 -2
     def test_mips_malta32eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips
@@ -186,7 +186,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('be', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta32el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mipsel
@@ -201,7 +201,7 @@ class LinuxSSH(Test):
 
         self.check_mips_malta('le', kernel_path, 'mips')
 
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64
@@ -215,7 +215,7 @@ class LinuxSSH(Test):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
         self.check_mips_malta('be', kernel_path, 'mips64')
 
Quoted 10 lines
Quoted 8 lines
tests/acceptance/linux_ssh_mips_malta.py +1 -1
-    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_mips_malta64el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64el
Signature · 1 line
2.20.1

BTW, also queueing this one on my python-next branch.