Sanity check network connectivity

RFC Applied Joshua Lock 3 participants 6 messages Jun 29, 2011 – Jul 7, 2011

Share & tools

Joshua Lock

author

2011-06-29 21:59 UTC #0 lore raw

cover Sanity check network connectivity

v3 of this change set based on extra feedback from Khem Raj.
Changes since v2:
* Remove references to Yocto, don't offer default uri's to check
* Make the error message configurable through a variable, CONNECTIVITY_CHECK_MSG, whilst providing a reasonable default.

"In response to a Yocto Bugzilla request[1] I've written a sanity test to check whether BitBake is able to fecth from http, https and git sources. The idea being that if the user is behing a proxy and this test fails we can more easily help them diagnose and fix their problem.

I've built on the existing infrastructure for less frequent sanity tests so whilst this test is reasonably heavy it will only run when TMPDIR changes (usually first run?). Further I added a variable to disable just this sanity check. People shipping offline installs to customers should just be able to set the variable in their shipped configuration and not worry about this sanity check irritating people."

The following changes since commit ff014d9634638457622f6019b163e75bafcefada:

task-base: add 3G into DISTRO_FEATURE (2011-06-29 14:46:46 +0100)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib josh/connection-test
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/connection-test

Joshua Lock (2):
sanity.bbclass: pass the data object to the less frequent test
harnesses
sanity: implement network connectivity test

 meta/classes/sanity.bbclass |   53 ++++++++++++++++++++++++++++++++++++------
 1 files changed, 45 insertions(+), 8 deletions(-)

Signature · 1 line

1.7.5.4

Joshua Lock

author

2011-06-29 21:59 UTC #1 lore raw

RFC 1/2 sanity.bbclass: pass the data object to the less frequent test harnesses

By passing the data object to the less frequently run test harnesses (check_sanity_tmpdir_change(), check_sanity_sstate_dir_change() and check_sanity_version_change()) we can run tests against BitBake data here too.

Signed-off-by Joshua Lock <[email protected]>

---
 meta/classes/sanity.bbclass |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

meta/classes/sanity.bbclass +8 -8

index d296c86..720777a 100644
@@ -21,7 +21,7 @@ def check_conf_exists(fn, data):
             return True
     return False
 
-def check_sanity_sstate_dir_change(sstate_dir):
+def check_sanity_sstate_dir_change(sstate_dir, data):
     # Sanity checks to be done when the value of SSTATE_DIR changes
 
     # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)

@@ -30,14 +30,14 @@ def check_sanity_sstate_dir_change(sstate_dir):
         testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
     return testmsg
 
-def check_sanity_tmpdir_change(tmpdir):
+def check_sanity_tmpdir_change(tmpdir, data):
     # Sanity checks to be done when the value of TMPDIR changes
 
     # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = check_create_long_filename(tmpdir, "TMPDIR")
     return testmsg
         
-def check_sanity_version_change():
+def check_sanity_version_change(data):
     # Sanity checks to be done when SANITY_VERSION changes
     return ""

@@ -266,14 +266,14 @@ def check_sanity(e):
     
     sanity_version = int(data.getVar('SANITY_VERSION', e.data, True) or 1)
     if last_sanity_version < sanity_version: 
-        messages = messages + check_sanity_version_change()
-        messages = messages + check_sanity_tmpdir_change(tmpdir)
-        messages = messages + check_sanity_sstate_dir_change(sstate_dir)
+        messages = messages + check_sanity_version_change(e.data)
+        messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
+        messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
     else: 
         if last_tmpdir != tmpdir:
-            messages = messages + check_sanity_tmpdir_change(tmpdir)
+            messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
         if last_sstate_dir != sstate_dir:
-            messages = messages + check_sanity_sstate_dir_change(sstate_dir)
+            messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
 
     if os.path.exists("conf"):
         f = file(sanityverfile, 'w')

Signature · 1 line

1.7.5.4

Joshua Lock

author

2011-06-29 21:59 UTC #2 lore raw

RFC 2/2 sanity: implement network connectivity test

In reply to Joshua Lock

Sanity test to verify files can be fetched from the network using git, http and https fetchers point users at a page to help get set up in the case of a failure.

Requires a variable CONNECTIVITY_CHECK_URIS to be set, using the same pattern as SRC_URI, of URI's to test against.
The variable CONNECTIVITY_CHECK_MSG can be set to provide a custom error message, such as a pointer to some help, when this check fails.

Addresses [YOCTO #933]

Signed-off-by Joshua Lock <[email protected]>

---
 meta/classes/sanity.bbclass |   37 +++++++++++++++++++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

meta/classes/sanity.bbclass +37 -0

index 720777a..c9d37c9 100644
@@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
 
     # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = check_create_long_filename(tmpdir, "TMPDIR")
+    # Check that we can fetch from various network transports
+    testmsg = testmsg + check_connectivity(data)
     return testmsg
         
 def check_sanity_version_change(data):

@@ -79,6 +81,41 @@ def check_create_long_filename(filepath, pathname):
             return "Failed to create a file in %s: %s" % (pathname, strerror)
     return ""
 
+def check_connectivity(d):
+    # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
+    # using the same syntax as for SRC_URI. If the variable is not set
+    # the check is skipped
+    test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or "").split()
+    retval = ""
+
+    # Only check connectivity if network enabled and the
+    # CONNECTIVITY_CHECK_URIS are set
+    network_enabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
+    check_enabled = len(test_uris)
+    if check_enabled and network_enabled:
+        data = bb.data.createCopy(d)
+        bookmark = os.getcwd()
+        dldir = bb.data.expand('${TMPDIR}/sanity', data)
+        bb.data.setVar('DL_DIR', dldir, data)
+
+        try:
+            fetcher = bb.fetch2.Fetch(test_uris, data)
+            fetcher.download()
+            fetcher.clean(test_uris)
+        except Exception:
+            # Allow the message to be configured so that users can be
+            # pointed to a support mechanism.
+            msg = bb.data.getVar('CONNECTIVITY_CHECK_MSG', d, True) or ""
+            if len(msg) == 0:
+                msg = "Failed to fetch test data from the network. Please ensure your network is configured correctly.\n"
+            retval = msg
+        finally:
+            # Make sure we tidy up the cruft
+            oe.path.remove(dldir)
+            os.chdir(bookmark)
+
+    return retval
+
 def check_sanity(e):
     from bb import note, error, data, __version__

Signature · 1 line

1.7.5.4

Richard Purdie

Applied

2011-07-07 12:48 UTC #3 lore raw

Re: [RFC 2/2] sanity: implement network connectivity test

Quoted 4 lines · Joshua Lock

Sanity test to verify files can be fetched from the network using git, http and https fetchers point users at a page to help get set up in the case of a failure.

Quoted 8 lines

Requires a variable CONNECTIVITY_CHECK_URIS to be set, using the same pattern as SRC_URI, of URI's to test against.
The variable CONNECTIVITY_CHECK_MSG can be set to provide a custom error message, such as a pointer to some help, when this check fails.
Addresses [YOCTO #933]
Signed-off-by Joshua Lock <[email protected]>

Merged to master, thanks.

Richard

Koen Kooi

2011-06-29 22:04 UTC #4 lore raw

In reply to Joshua Lock

Op 29 jun 2011, om 23:55 heeft Joshua Lock het volgende geschreven:

Quoted 3 lines

v3 of this change set based on extra feedback from Khem Raj. Changes since v2:
* Remove references to Yocto, don't offer default uri's to check

Can't you offer an oe.org url to check instead?

Joshua Lock

author

2011-06-30 00:27 UTC #5 lore raw

Quoted 7 lines · Koen Kooi

Op 29 jun 2011, om 23:55 heeft Joshua Lock het volgende geschreven:
Quoted 3 lines
v3 of this change set based on extra feedback from Khem Raj. Changes since v2:
* Remove references to Yocto, don't offer default uri's to check
Can't you offer an oe.org url to check instead?

Can we? Sure. Should we? I don't know...

If this is deemed desirable perhaps we'll need some small uri's to fetch from and test at least http, https and git fetches. Any suggestions?

Could this be a follow on patch?

Joshua

Signature · 3 lines

Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre