support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE
PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE
From: Thomas De Schampheleire <[email protected]>
Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.
--- .../tests/download/br2-external/scp/Config.in | 0 .../download/br2-external/scp/external.desc | 1 + .../download/br2-external/scp/external.mk | 1 + .../br2-external/scp/package/nohash/nohash.mk | 10 +++ support/testing/tests/download/scp-wrapper | 62 +++++++++++++++++++ support/testing/tests/download/test_scp.py | 45 ++++++++++++++ 6 files changed, 119 insertions(+) create mode 100644 support/testing/tests/download/br2-external/scp/Config.in create mode 100644 support/testing/tests/download/br2-external/scp/external.desc create mode 100644 support/testing/tests/download/br2-external/scp/external.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk create mode 100755 support/testing/tests/download/scp-wrapper create mode 100644 support/testing/tests/download/test_scp.py
support/testing/tests/download/br2-external/scp/Config.in new file +0 -0
support/testing/tests/download/br2-external/scp/external.desc new file +1 -0
@@ -0,0 +1 @@ +name: SCP
support/testing/tests/download/br2-external/scp/external.mk new file +1 -0
@@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_SCP_PATH)/package/*/*.mk))
support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk new file +10 -0
@@ -0,0 +1,10 @@ +################################################################################ +# +# nohash +# +################################################################################ + +NOHASH_VERSION = 123 +NOHASH_SITE = http://realsite.example.org/foo + +$(eval $(generic-package))
support/testing/tests/download/scp-wrapper new file +62 -0
62 lines not on this page · raw message
support/testing/tests/download/test_scp.py new file +45 -0
45 lines not on this page · raw message
Signature · 1 line
2.19.2
PATCH 2/2 support/testing: introduce specific error classes
From: Thomas De Schampheleire <[email protected]>
Rather than using 'SystemError' with a text string, introduce specific error classes. These can now be caught specifically, without having to check the error string, like was done in tests.download.test_scp, or without catching too much like in the case of tests.download.test_git.
--- support/testing/infra/__init__.py | 3 +++ support/testing/infra/builder.py | 4 +-- support/testing/infra/emulator.py | 4 +-- support/testing/infra/exceptions.py | 27 +++++++++++++++++++++ support/testing/tests/download/gitremote.py | 2 +- support/testing/tests/download/test_git.py | 6 ++--- support/testing/tests/download/test_scp.py | 8 ++---- support/testing/tests/init/test_none.py | 4 +-- support/testing/tests/package/test_rust.py | 4 +-- 9 files changed, 44 insertions(+), 18 deletions(-) create mode 100644 support/testing/infra/exceptions.py
support/testing/infra/__init__.py +3 -0
@@ -5,6 +5,9 @@ import tempfile import subprocess from urllib2 import urlopen, HTTPError, URLError +# make exceptions available to all tests with just 'import infra' +import infra.exceptions + ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
support/testing/infra/builder.py +2 -2
@@ -45,7 +45,7 @@ class Builder(object): ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile, env=env) if ret != 0: - raise SystemError("Cannot olddefconfig") + raise infra.exceptions.ConfigurationError("Cannot olddefconfig") def build(self, make_extra_opts=[], make_extra_env={}): """Perform the build.
@@ -72,7 +72,7 @@ class Builder(object): ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile, env=env) if ret != 0: - raise SystemError("Build failed") + raise infra.exceptions.BuildError open(self.stamp_path(), 'a').close()
support/testing/infra/emulator.py +2 -2
@@ -84,7 +84,7 @@ class Emulator(object): timeout=60 * self.timeout_multiplier) if index != 0: self.logfile.write("==> System does not boot") - raise SystemError("System does not boot") + raise infra.exceptions.BootError self.qemu.sendline("root") if password:
@@ -92,7 +92,7 @@ class Emulator(object): self.qemu.sendline(password) index = self.qemu.expect(["# ", pexpect.TIMEOUT]) if index != 0: - raise SystemError("Cannot login") + raise infra.exceptions.LoginError self.run("dmesg -n 1") # Run the given 'cmd' with a 'timeout' on the target
support/testing/infra/exceptions.py new file +27 -0
@@ -0,0 +1,27 @@ +"""Custom exception classes for use by the test suite.""" + +class BootError(Exception): + """System does not boot correctly.""" + pass + + +class LoginError(Exception): + """Cannot login to system.""" + pass + + +class ConfigurationError(Exception): + """Cannot set Buildroot configuration.""" + pass + + +class BuildError(Exception): + """Build failed.""" + pass + + +class TestError(Exception): + """Something went wrong while executing a test. + + The exact cause of this problem should be specified by the caller.""" + pass
support/testing/tests/download/gitremote.py +1 -1
@@ -38,7 +38,7 @@ class GitRemote(object): if ret == 0: self.port = port return - raise SystemError("Could not find a free port to run git remote") + raise infra.exceptions.TestError("Could not find a free port to run git remote") def stop(self): if self.daemon is None:
support/testing/tests/download/test_git.py +3 -3
@@ -52,7 +52,7 @@ class TestGitHash(GitTestBase): br2_external = [infra.filepath("tests/download/br2-external/git-hash")] def test_run(self): - with self.assertRaises(SystemError): + with self.assertRaises(infra.exceptions.BuildError): self.check_hash("bad") self.check_hash("good") self.check_hash("nohash")
@@ -62,9 +62,9 @@ class TestGitRefs(GitTestBase): br2_external = [infra.filepath("tests/download/br2-external/git-refs")] def test_run(self): - with self.assertRaises(SystemError): + with self.assertRaises(infra.exceptions.BuildError): self.check_download("git-wrong-content") - with self.assertRaises(SystemError): + with self.assertRaises(infra.exceptions.BuildError): self.check_download("git-wrong-sha1") self.check_download("git-partial-sha1-branch-head") self.check_download("git-partial-sha1-reachable-by-branch")
support/testing/tests/download/test_scp.py +2 -6
@@ -37,9 +37,5 @@ class TestScpPrimarySite(infra.basetest.BRConfigTest): self.b.build(["{}-dirclean".format(package), "{}-source".format(package)], env) - except SystemError as e: # FIXME: introduce specific Exception classes - if str(e) == 'Build failed': - self.assertFalse('Download error, search for ERROR in the log') - else: - # an unexpected error - raise + except infra.exceptions.BuildError: + self.assertFalse('Download error, search for ERROR in the log')
support/testing/tests/init/test_none.py +2 -2
@@ -17,11 +17,11 @@ class TestInitSystemNone(InitSystemBase): index = self.emulator.qemu.expect(["/bin/sh: can't access tty; job control turned off", pexpect.TIMEOUT], timeout=60) if index != 0: self.emulator.logfile.write("==> System does not boot") - raise SystemError("System does not boot") + raise infra.exceptions.BootError index = self.emulator.qemu.expect(["#", pexpect.TIMEOUT], timeout=60) if index != 0: self.emulator.logfile.write("==> System does not boot") - raise SystemError("System does not boot") + raise infra.exceptions.BootError out, exit_code = self.emulator.run("sh -c 'echo $PPID'") self.assertEqual(exit_code, 0)
support/testing/tests/package/test_rust.py +2 -2
@@ -37,7 +37,7 @@ class TestRustBase(infra.basetest.BRTest): stderr=self.b.logfile, env=env) if ret != 0: - raise SystemError("Cargo init failed") + raise infra.exceptions.TestError("Cargo init failed") cmd = [ cargo, 'build', '-vv', '--target', self.target,
@@ -48,7 +48,7 @@ class TestRustBase(infra.basetest.BRTest): stderr=self.b.logfile, env=env) if ret != 0: - raise SystemError("Cargo build failed") + raise infra.exceptions.TestError("Cargo build failed") shutil.copy(prog, os.path.join(self.builddir, 'target', 'usr', 'bin')) self.b.build()
Signature · 1 line
2.19.2
PATCH 2/2 support/testing: introduce specific error classes
Hello,
See some nits below.
Quoted 11 lines · Thomas De Schampheleire
Rather than using 'SystemError' with a text string, introduce specific error classes. These can now be caught specifically, without having to check the error string, like was done in tests.download.test_scp, or without catching too much like in the case of tests.download.test_git.
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/testing/infra/__init__.py | 3 +++ support/testing/infra/builder.py | 4 +-- support/testing/infra/emulator.py | 4 +-- support/testing/infra/exceptions.py | 27 +++++++++++++++++++++
Please run flake8 and fix the warnings it generates. support/testing/infra/__init__.py:9:1: F401 'infra.exceptions' imported but unused support/testing/infra/exceptions.py:3:1: E302 expected 2 blank lines, found 1
[snip]
support/testing/infra/__init__.py import tempfile · line 9
Quoted 7 lines
+++
b/support/testing/infra/__init__.pysupport/testing/infra/__init__.py +2
-0@@ -5,6 +5,9 @@ import tempfile import subprocess from urllib2 import urlopen, HTTPError, URLError +# make exceptions available to all tests with just 'import infra' +import infra.exceptions
So here you can use:
import infra.exceptions # noqa: F401
[snip]
support/testing/tests/download/test_scp.py class TestScpPrimarySite(infra.basetest.BRConfigTest): · line 41
Quoted 13 lines
support/testing/tests/download/test_scp.py +3
-6+++ b/support/testing/tests/download/test_scp.py@@ -37,9 +37,5 @@ class TestScpPrimarySite(infra.basetest.BRConfigTest): self.b.build(["{}-dirclean".format(package), "{}-source".format(package)], env) - except SystemError as e: # FIXME: introduce specific Exception classes - if str(e) == 'Build failed': - self.assertFalse('Download error, search for ERROR in the log') - else: - # an unexpected error - raise + except infra.exceptions.BuildError: + self.assertFalse('Download error, search for ERROR in the log')
Any reason to not do this?
except infra.exceptions.BuildError:
raise infra.exceptions.TestError('Download error, search for ERROR in the log')
Regards,
Ricardo
PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE
In reply to Thomas De SchampheleireHello,
Sorry the long delay.
Check the method name issue, some code that can removed and also some nits below.
Quoted 5 lines · Thomas De Schampheleire
Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.
Signed-off-by Thomas De Schampheleire <[email protected]>
Quoted 8 lines
--- .../tests/download/br2-external/scp/Config.in | 0 .../download/br2-external/scp/external.desc | 1 + .../download/br2-external/scp/external.mk | 1 + .../br2-external/scp/package/nohash/nohash.mk | 10 +++ support/testing/tests/download/scp-wrapper | 62 +++++++++++++++++++ support/testing/tests/download/test_scp.py | 45 ++++++++++++++ 6 files changed, 119 insertions(+)
After fixing the name of the test method below, please run 'make .gitlab-ci.yml'.
Quoted 6 lines
create mode 100644 support/testing/tests/download/br2-external/scp/Config.in create mode 100644 support/testing/tests/download/br2-external/scp/external.desc create mode 100644 support/testing/tests/download/br2-external/scp/external.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk create mode 100755 support/testing/tests/download/scp-wrapper create mode 100644 support/testing/tests/download/test_scp.py
Please run flake8 and fix the warnings it generates.
scp-wrapper:14:1: E302 expected 2 blank lines, found 1
scp-wrapper:59:1: E305 expected 2 blank lines after class or function definition, found 1 test_scp.py:4:1: E302 expected 2 blank lines, found 1
test_scp.py:12:9: E122 continuation line missing indentation or outdented test_scp.py:40:33: E261 at least two spaces before inline comment
[snip]
Quoted 12 lines
+++
b/support/testing/tests/download/test_scp.pysupport/testing/tests/download/test_scp.py new file +9
-0@@ -0,0 +1,45 @@ +import infra +import os + +class TestScpPrimarySite(infra.basetest.BRConfigTest): + host = 'user at server.example.org' + basepath = 'some/directory' + scp_wrapper = infra.filepath("tests/download/scp-wrapper") + br2_external = [infra.filepath("tests/download/br2-external/scp")] +
support/testing/tests/download/test_scp.py line 19
Quoted 10 lines
support/testing/tests/download/test_scp.py new file +10
-0+ def __init__(self, names): + self.config = \ + """ + BR2_PRIMARY_SITE="scp://%s:%s" + BR2_PRIMARY_SITE_ONLY=y + BR2_BACKUP_SITE="" + BR2_SCP="%s" + """ % (self.host, self.basepath, self.scp_wrapper) + + super(TestScpPrimarySite, self).__init__(names)
This is not really needed since the values are not dynamically set, so you can just set in the class:
config = \
"""
BR2_PRIMARY_SITE="scp://%s:%s"
BR2_PRIMARY_SITE_ONLY=y
BR2_BACKUP_SITE=""
BR2_SCP="%s"
""" % (host, basepath, scp_wrapper)support/testing/tests/download/test_scp.py line 24
Quoted 5 lines
support/testing/tests/download/test_scp.py new file +5
-0+ + def tearDown(self): + self.show_msg("Cleaning up") + if self.b and not self.keepbuilds: + self.b.delete()
This method already exists in the parent class so it is not needed here.
Review
Quoted 2 lines
(quoted diff) +2
-0+ + def test_download(self):
Please use test_run here, otherwise 'make .gitlab-ci.yml' will not pickup this test.
support/testing/tests/download/test_scp.py line 34
Quoted 8 lines
support/testing/tests/download/test_scp.py new file +8
-0+ package = 'nohash' + # store downloaded tarball inside the output dir so the test infra + # cleans it up at the end + env = { + "BR2_DL_DIR": os.path.join(self.builddir, "dl"), + "SCP_WRAPPER_EXPECTED_HOST": self.host, + "SCP_WRAPPER_EXPECTED_BASEPATH": self.basepath, + "SCP_WRAPPER_EXPECTED_FILEPATH": '%s-123.tar.gz' % package
nit: .format() appear more often in the test infra then % There are another occurrences in this patch.
Regards,
Ricardo
PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE
In reply to Thomas De SchampheleireHello Thomas,
Quoted 7 lines · On Fri, 15 Feb 2019 22:35:48 +0100 Thomas De Schampheleire…
From: Thomas De Schampheleire <[email protected]>
Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.
Signed-off-by Thomas De Schampheleire <[email protected]>
Quoted 14 lines
--- .../tests/download/br2-external/scp/Config.in | 0 .../download/br2-external/scp/external.desc | 1 + .../download/br2-external/scp/external.mk | 1 + .../br2-external/scp/package/nohash/nohash.mk | 10 +++ support/testing/tests/download/scp-wrapper | 62 +++++++++++++++++++ support/testing/tests/download/test_scp.py | 45 ++++++++++++++ 6 files changed, 119 insertions(+) create mode 100644 support/testing/tests/download/br2-external/scp/Config.in create mode 100644 support/testing/tests/download/br2-external/scp/external.desc create mode 100644 support/testing/tests/download/br2-external/scp/external.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk create mode 100755 support/testing/tests/download/scp-wrapper create mode 100644 support/testing/tests/download/test_scp.py
You got some review/comments from Ricardo, which I believe are not too difficult to address. Do you think you will have the chance to address them and send an updated version ?
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 support/testing: add test to verify 'scp' download via BR2_PRIMARY_SITE
Hi Thomas,
El s?b., 13 abr. 2019 a las 22:02, Thomas Petazzoni (<[email protected]>) escribi?:
Quoted 28 lines
Hello Thomas,
Quoted 7 lines · On Fri, 15 Feb 2019 22:35:48 +0100 Thomas De Schampheleire…
From: Thomas De Schampheleire <[email protected]>
Recently it was found that the scp download infrastructure was broken. To avoid future failures, create a test that verifies that the scp command receives the expected arguments.
Signed-off-by Thomas De Schampheleire <[email protected]>Quoted 14 lines
--- .../tests/download/br2-external/scp/Config.in | 0 .../download/br2-external/scp/external.desc | 1 + .../download/br2-external/scp/external.mk | 1 + .../br2-external/scp/package/nohash/nohash.mk | 10 +++ support/testing/tests/download/scp-wrapper | 62 +++++++++++++++++++ support/testing/tests/download/test_scp.py | 45 ++++++++++++++ 6 files changed, 119 insertions(+) create mode 100644 support/testing/tests/download/br2-external/scp/Config.in create mode 100644 support/testing/tests/download/br2-external/scp/external.desc create mode 100644 support/testing/tests/download/br2-external/scp/external.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/nohash/nohash.mk create mode 100755 support/testing/tests/download/scp-wrapper create mode 100644 support/testing/tests/download/test_scp.py
Quoted 3 lines
You got some review/comments from Ricardo, which I believe are not too difficult to address. Do you think you will have the chance to address them and send an updated version ?
Yes, I hope to attend these comments some time soon, but there is so much going on that I can't promise any term.
Feel free to remind me from time to time if it takes too long.
Thanks,
Thomas