Skip to contentSkip to conversation

support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE

PATCH New Thomas De Schampheleire 3 participants 6 messages Feb 15, 2019 – Apr 23, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE

From: Thomas De Schampheleire <[email protected]>

Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.

---
 .../tests/download/br2-external/scp/Config.in |  0
 .../download/br2-external/scp/external.desc   |  1 +
 .../download/br2-external/scp/external.mk     |  1 +
 .../br2-external/scp/package/nohash/nohash.mk | 10 +++
 support/testing/tests/download/scp-wrapper    | 62 +++++++++++++++++++
 support/testing/tests/download/test_scp.py    | 45 ++++++++++++++
 6 files changed, 119 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/scp/Config.in
 create mode 100644 support/testing/tests/download/br2-external/scp/external.desc
 create mode 100644 support/testing/tests/download/br2-external/scp/external.mk
 create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk
 create mode 100755 support/testing/tests/download/scp-wrapper
 create mode 100644 support/testing/tests/download/test_scp.py
support/testing/tests/download/br2-external/scp/Config.in new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
support/testing/tests/download/br2-external/scp/external.desc new file +1 -0
new file mode 100644
index 0000000000..0ca0389a32
@@ -0,0 +1 @@
+name: SCP
support/testing/tests/download/br2-external/scp/external.mk new file +1 -0
new file mode 100644
index 0000000000..2636c7da24
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_SCP_PATH)/package/*/*.mk))
support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk new file +10 -0
new file mode 100644
index 0000000000..cfd49370cd
@@ -0,0 +1,10 @@
+################################################################################
+#
+# nohash
+#
+################################################################################
+
+NOHASH_VERSION = 123
+NOHASH_SITE = http://realsite.example.org/foo
+
+$(eval $(generic-package))
support/testing/tests/download/scp-wrapper new file +62 -0

62 lines not on this page · raw message

support/testing/tests/download/test_scp.py new file +45 -0

45 lines not on this page · raw message

Signature · 1 line
2.19.2
#1 lore raw

PATCH 2/2 support/testing: introduce specific error classes

From: Thomas De Schampheleire <[email protected]>

Rather than using 'SystemError' with a text string, introduce specific error classes. These can now be caught specifically, without having to check the error string, like was done in tests.download.test_scp, or without catching too much like in the case of tests.download.test_git.

---
 support/testing/infra/__init__.py           |  3 +++
 support/testing/infra/builder.py            |  4 +--
 support/testing/infra/emulator.py           |  4 +--
 support/testing/infra/exceptions.py         | 27 +++++++++++++++++++++
 support/testing/tests/download/gitremote.py |  2 +-
 support/testing/tests/download/test_git.py  |  6 ++---
 support/testing/tests/download/test_scp.py  |  8 ++----
 support/testing/tests/init/test_none.py     |  4 +--
 support/testing/tests/package/test_rust.py  |  4 +--
 9 files changed, 44 insertions(+), 18 deletions(-)
 create mode 100644 support/testing/infra/exceptions.py
support/testing/infra/__init__.py +3 -0
index e229e90852..0e4b871c8b 100644
@@ -5,6 +5,9 @@ import tempfile
 import subprocess
 from urllib2 import urlopen, HTTPError, URLError
 
+# make exceptions available to all tests with just 'import infra'
+import infra.exceptions
+
 ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 
 
support/testing/infra/builder.py +2 -2
index 018747555d..6cf6551bc5 100644
@@ -45,7 +45,7 @@ class Builder(object):
         ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile,
                               env=env)
         if ret != 0:
-            raise SystemError("Cannot olddefconfig")
+            raise infra.exceptions.ConfigurationError("Cannot olddefconfig")
 
     def build(self, make_extra_opts=[], make_extra_env={}):
         """Perform the build.
@@ -72,7 +72,7 @@ class Builder(object):
         ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile,
                               env=env)
         if ret != 0:
-            raise SystemError("Build failed")
+            raise infra.exceptions.BuildError
 
         open(self.stamp_path(), 'a').close()
 
support/testing/infra/emulator.py +2 -2
index 802e89d4b4..5e5fc6c47b 100644
@@ -84,7 +84,7 @@ class Emulator(object):
                                  timeout=60 * self.timeout_multiplier)
         if index != 0:
             self.logfile.write("==> System does not boot")
-            raise SystemError("System does not boot")
+            raise infra.exceptions.BootError
 
         self.qemu.sendline("root")
         if password:
@@ -92,7 +92,7 @@ class Emulator(object):
             self.qemu.sendline(password)
         index = self.qemu.expect(["# ", pexpect.TIMEOUT])
         if index != 0:
-            raise SystemError("Cannot login")
+            raise infra.exceptions.LoginError
         self.run("dmesg -n 1")
 
     # Run the given 'cmd' with a 'timeout' on the target
support/testing/infra/exceptions.py new file +27 -0
new file mode 100644
index 0000000000..2d32d819e3
@@ -0,0 +1,27 @@
+"""Custom exception classes for use by the test suite."""
+
+class BootError(Exception):
+    """System does not boot correctly."""
+    pass
+
+
+class LoginError(Exception):
+    """Cannot login to system."""
+    pass
+
+
+class ConfigurationError(Exception):
+    """Cannot set Buildroot configuration."""
+    pass
+
+
+class BuildError(Exception):
+    """Build failed."""
+    pass
+
+
+class TestError(Exception):
+    """Something went wrong while executing a test.
+
+    The exact cause of this problem should be specified by the caller."""
+    pass
support/testing/tests/download/gitremote.py +1 -1
index 3b35456dd1..874b1ff791 100644
@@ -38,7 +38,7 @@ class GitRemote(object):
             if ret == 0:
                 self.port = port
                 return
-        raise SystemError("Could not find a free port to run git remote")
+        raise infra.exceptions.TestError("Could not find a free port to run git remote")
 
     def stop(self):
         if self.daemon is None:
support/testing/tests/download/test_git.py +3 -3
index 40cf1afe05..1ce676cdd2 100644
@@ -52,7 +52,7 @@ class TestGitHash(GitTestBase):
     br2_external = [infra.filepath("tests/download/br2-external/git-hash")]
 
     def test_run(self):
-        with self.assertRaises(SystemError):
+        with self.assertRaises(infra.exceptions.BuildError):
             self.check_hash("bad")
         self.check_hash("good")
         self.check_hash("nohash")
@@ -62,9 +62,9 @@ class TestGitRefs(GitTestBase):
     br2_external = [infra.filepath("tests/download/br2-external/git-refs")]
 
     def test_run(self):
-        with self.assertRaises(SystemError):
+        with self.assertRaises(infra.exceptions.BuildError):
             self.check_download("git-wrong-content")
-        with self.assertRaises(SystemError):
+        with self.assertRaises(infra.exceptions.BuildError):
             self.check_download("git-wrong-sha1")
         self.check_download("git-partial-sha1-branch-head")
         self.check_download("git-partial-sha1-reachable-by-branch")
support/testing/tests/download/test_scp.py +2 -6
index 0cc9ecdd50..908b637af9 100644
@@ -37,9 +37,5 @@ class TestScpPrimarySite(infra.basetest.BRConfigTest):
             self.b.build(["{}-dirclean".format(package),
                           "{}-source".format(package)],
                          env)
-        except SystemError as e: # FIXME: introduce specific Exception classes
-            if str(e) == 'Build failed':
-                self.assertFalse('Download error, search for ERROR in the log')
-            else:
-                # an unexpected error
-                raise
+        except infra.exceptions.BuildError:
+            self.assertFalse('Download error, search for ERROR in the log')
support/testing/tests/init/test_none.py +2 -2
index 5b9b4e43f1..5c9e8ded03 100644
@@ -17,11 +17,11 @@ class TestInitSystemNone(InitSystemBase):
         index = self.emulator.qemu.expect(["/bin/sh: can't access tty; job control turned off", pexpect.TIMEOUT], timeout=60)
         if index != 0:
             self.emulator.logfile.write("==> System does not boot")
-            raise SystemError("System does not boot")
+            raise infra.exceptions.BootError
         index = self.emulator.qemu.expect(["#", pexpect.TIMEOUT], timeout=60)
         if index != 0:
             self.emulator.logfile.write("==> System does not boot")
-            raise SystemError("System does not boot")
+            raise infra.exceptions.BootError
 
         out, exit_code = self.emulator.run("sh -c 'echo $PPID'")
         self.assertEqual(exit_code, 0)
support/testing/tests/package/test_rust.py +2 -2
index 9854c3692e..0ab92fddfb 100644
@@ -37,7 +37,7 @@ class TestRustBase(infra.basetest.BRTest):
                               stderr=self.b.logfile,
                               env=env)
         if ret != 0:
-            raise SystemError("Cargo init failed")
+            raise infra.exceptions.TestError("Cargo init failed")
 
         cmd = [
             cargo, 'build', '-vv', '--target', self.target,
@@ -48,7 +48,7 @@ class TestRustBase(infra.basetest.BRTest):
                               stderr=self.b.logfile,
                               env=env)
         if ret != 0:
-            raise SystemError("Cargo build failed")
+            raise infra.exceptions.TestError("Cargo build failed")
 
         shutil.copy(prog, os.path.join(self.builddir, 'target', 'usr', 'bin'))
         self.b.build()
Signature · 1 line
2.19.2
#3 lore raw

PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE

In reply to Thomas De Schampheleire

Hello,

Sorry the long delay.
Check the method name issue, some code that can removed and also some nits below.

Quoted 5 lines · Thomas De Schampheleire

Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.

Quoted 8 lines
---
 .../tests/download/br2-external/scp/Config.in |  0
 .../download/br2-external/scp/external.desc   |  1 +
 .../download/br2-external/scp/external.mk     |  1 +
 .../br2-external/scp/package/nohash/nohash.mk | 10 +++
 support/testing/tests/download/scp-wrapper    | 62 +++++++++++++++++++
 support/testing/tests/download/test_scp.py    | 45 ++++++++++++++
 6 files changed, 119 insertions(+)

After fixing the name of the test method below, please run 'make .gitlab-ci.yml'.

Quoted 6 lines
 create mode 100644 support/testing/tests/download/br2-external/scp/Config.in
 create mode 100644 support/testing/tests/download/br2-external/scp/external.desc
 create mode 100644 support/testing/tests/download/br2-external/scp/external.mk
 create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk
 create mode 100755 support/testing/tests/download/scp-wrapper
 create mode 100644 support/testing/tests/download/test_scp.py

Please run flake8 and fix the warnings it generates.
scp-wrapper:14:1: E302 expected 2 blank lines, found 1
scp-wrapper:59:1: E305 expected 2 blank lines after class or function definition, found 1 test_scp.py:4:1: E302 expected 2 blank lines, found 1
test_scp.py:12:9: E122 continuation line missing indentation or outdented test_scp.py:40:33: E261 at least two spaces before inline comment

[snip]

Quoted 12 lines

+++ b/support/testing/tests/download/test_scp.py

support/testing/tests/download/test_scp.py new file +9 -0
@@ -0,0 +1,45 @@
+import infra
+import os
+
+class TestScpPrimarySite(infra.basetest.BRConfigTest):
+    host = 'user at server.example.org'
+    basepath = 'some/directory'
+    scp_wrapper = infra.filepath("tests/download/scp-wrapper")
+    br2_external = [infra.filepath("tests/download/br2-external/scp")]
+
 

support/testing/tests/download/test_scp.py line 19

Quoted 10 lines
support/testing/tests/download/test_scp.py new file +10 -0
+    def __init__(self, names):
+        self.config = \
+        """
+        BR2_PRIMARY_SITE="scp://%s:%s"
+        BR2_PRIMARY_SITE_ONLY=y
+        BR2_BACKUP_SITE=""
+        BR2_SCP="%s"
+        """ % (self.host, self.basepath, self.scp_wrapper)
+
+        super(TestScpPrimarySite, self).__init__(names)

This is not really needed since the values are not dynamically set, so you can just set in the class:

    config = \
        """
        BR2_PRIMARY_SITE="scp://%s:%s"
        BR2_PRIMARY_SITE_ONLY=y
        BR2_BACKUP_SITE=""
        BR2_SCP="%s"
        """ % (host, basepath, scp_wrapper)

support/testing/tests/download/test_scp.py line 24

Quoted 5 lines
support/testing/tests/download/test_scp.py new file +5 -0
+
+    def tearDown(self):
+        self.show_msg("Cleaning up")
+        if self.b and not self.keepbuilds:
+            self.b.delete()

This method already exists in the parent class so it is not needed here.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+    def test_download(self):

Please use test_run here, otherwise 'make .gitlab-ci.yml' will not pickup this test.

support/testing/tests/download/test_scp.py line 34

Quoted 8 lines
support/testing/tests/download/test_scp.py new file +8 -0
+        package = 'nohash'
+        # store downloaded tarball inside the output dir so the test infra
+        # cleans it up at the end
+        env = {
+            "BR2_DL_DIR": os.path.join(self.builddir, "dl"),
+            "SCP_WRAPPER_EXPECTED_HOST": self.host,
+            "SCP_WRAPPER_EXPECTED_BASEPATH": self.basepath,
+            "SCP_WRAPPER_EXPECTED_FILEPATH": '%s-123.tar.gz' % package

nit: .format() appear more often in the test infra then % There are another occurrences in this patch.

Regards,
Ricardo

#2 lore raw

PATCH 2/2 support/testing: introduce specific error classes

In reply to Thomas De Schampheleire

Hello,

See some nits below.

Quoted 11 lines · Thomas De Schampheleire

Rather than using 'SystemError' with a text string, introduce specific error classes. These can now be caught specifically, without having to check the error string, like was done in tests.download.test_scp, or without catching too much like in the case of tests.download.test_git.

---
 support/testing/infra/__init__.py           |  3 +++
 support/testing/infra/builder.py            |  4 +--
 support/testing/infra/emulator.py           |  4 +--
 support/testing/infra/exceptions.py         | 27 +++++++++++++++++++++

Please run flake8 and fix the warnings it generates. support/testing/infra/__init__.py:9:1: F401 'infra.exceptions' imported but unused support/testing/infra/exceptions.py:3:1: E302 expected 2 blank lines, found 1

[snip]

support/testing/infra/__init__.py import tempfile · line 9

Quoted 7 lines

+++ b/support/testing/infra/__init__.py

support/testing/infra/__init__.py +2 -0
@@ -5,6 +5,9 @@ import tempfile
 import subprocess
 from urllib2 import urlopen, HTTPError, URLError
 
+# make exceptions available to all tests with just 'import infra'
+import infra.exceptions

So here you can use:
import infra.exceptions # noqa: F401

[snip]

support/testing/tests/download/test_scp.py class TestScpPrimarySite(infra.basetest.BRConfigTest): · line 41

Quoted 13 lines
support/testing/tests/download/test_scp.py +3 -6
+++ b/support/testing/tests/download/test_scp.py
@@ -37,9 +37,5 @@ class TestScpPrimarySite(infra.basetest.BRConfigTest):
             self.b.build(["{}-dirclean".format(package),
                           "{}-source".format(package)],
                          env)
-        except SystemError as e: # FIXME: introduce specific Exception classes
-            if str(e) == 'Build failed':
-                self.assertFalse('Download error, search for ERROR in the log')
-            else:
-                # an unexpected error
-                raise
+        except infra.exceptions.BuildError:
+            self.assertFalse('Download error, search for ERROR in the log')

Any reason to not do this?

except infra.exceptions.BuildError:
raise infra.exceptions.TestError('Download error, search for ERROR in the log')

Regards,
Ricardo

#4 lore raw

PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE

In reply to Thomas De Schampheleire

Hello Thomas,

Quoted 7 lines · On Fri, 15 Feb 2019 22:35:48 +0100 Thomas De Schampheleire…

From: Thomas De Schampheleire <[email protected]>

Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.

Quoted 14 lines
---
 .../tests/download/br2-external/scp/Config.in |  0
 .../download/br2-external/scp/external.desc   |  1 +
 .../download/br2-external/scp/external.mk     |  1 +
 .../br2-external/scp/package/nohash/nohash.mk | 10 +++
 support/testing/tests/download/scp-wrapper    | 62 +++++++++++++++++++
 support/testing/tests/download/test_scp.py    | 45 ++++++++++++++
 6 files changed, 119 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/scp/Config.in
 create mode 100644 support/testing/tests/download/br2-external/scp/external.desc
 create mode 100644 support/testing/tests/download/br2-external/scp/external.mk
 create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk
 create mode 100755 support/testing/tests/download/scp-wrapper
 create mode 100644 support/testing/tests/download/test_scp.py

You got some review/comments from Ricardo, which I believe are not too difficult to address. Do you think you will have the chance to address them and send an updated version ?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE

Hi Thomas,

El s?b., 13 abr. 2019 a las 22:02, Thomas Petazzoni (<[email protected]>) escribi?:

Quoted 28 lines

Hello Thomas,

Quoted 7 lines · On Fri, 15 Feb 2019 22:35:48 +0100 Thomas De Schampheleire…

From: Thomas De Schampheleire <[email protected]>

Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.

Quoted 14 lines
---
 .../tests/download/br2-external/scp/Config.in |  0
 .../download/br2-external/scp/external.desc   |  1 +
 .../download/br2-external/scp/external.mk     |  1 +
 .../br2-external/scp/package/nohash/nohash.mk | 10 +++
 support/testing/tests/download/scp-wrapper    | 62 +++++++++++++++++++
 support/testing/tests/download/test_scp.py    | 45 ++++++++++++++
 6 files changed, 119 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/scp/Config.in
 create mode 100644 support/testing/tests/download/br2-external/scp/external.desc
 create mode 100644 support/testing/tests/download/br2-external/scp/external.mk
 create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk
 create mode 100755 support/testing/tests/download/scp-wrapper
 create mode 100644 support/testing/tests/download/test_scp.py
Quoted 3 lines

You got some review/comments from Ricardo, which I believe are not too difficult to address. Do you think you will have the chance to address them and send an updated version ?

Yes, I hope to attend these comments some time soon, but there is so much going on that I can't promise any term.

Feel free to remind me from time to time if it takes too long.

Thanks,
Thomas