Skip to contentSkip to conversation

Sanity class clean ups

PATCH In review Joshua Lock 3 participants 11 messages May 24, 2012

lore
Share & tools
lore
#0 lore raw

cover Sanity class clean ups

My recent change to the sanity class to warn users when they don't have R/W permission to their SSTATE_CACHE directory has caused at least a couple of users trouble, including the Yocto project autobuilder.

I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.

The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing check_create_long_filename() call, where I make the suggestion of using SSTATE_MIRRORS if that call fails with "Permission denied" when called against SSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.

Thanks to Chris and Elizabeth for pointers as to why this was failing.

Cheers,

Joshua

The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:

tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib josh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity

Joshua Lock (7):
Revert "sanity.bbclass: check user can read and write to SSTATE_DIR" sanity.bbclass: copy the data store and finalise before running
checks
sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp) sanity.bbclass: add newline to check_create_long_filename failure
message
sanity.bbclass: add extra information when SSTATE_CACHE unusable sanity.bbclass: catch an extra exception in
check_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

 meta/classes/sanity.bbclass |  153 ++++++++++++++++++++++---------------------
 1 files changed, 78 insertions(+), 75 deletions(-)
Signature · 1 line
1.7.7.6
#1 lore raw

PATCH 1/7 Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"

This has caused problems for several users, including the Yocto Project autobuilder. Since the message was added in order to be more user friendly revert the change.

This reverts commit 0c0c4efbf92bcf0f8942f17c18525a4b4ed1798c.

---
 meta/classes/sanity.bbclass |   17 +++--------------
 1 files changed, 3 insertions(+), 14 deletions(-)
meta/classes/sanity.bbclass +3 -14
index ff3c413..05545f4 100644
@@ -96,16 +96,10 @@ def check_conf_exists(fn, data):
 def check_sanity_sstate_dir_change(sstate_dir, data):
     # Sanity checks to be done when the value of SSTATE_DIR changes
 
+    # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = ""
     if sstate_dir != "":
-        # Check that the user can read and write to SSTATE_DIR
-        sstatemsg = check_can_read_write_directory(sstate_dir) or None
-        if sstatemsg:
-            sstatemsg = sstatemsg + ". You could try using it as an SSTATE_MIRRORS instead of SSTATE_CACHE.\n"
-            testmsg = testmsg + sstatemsg
-        # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
-        testmsg = testmsg + check_create_long_filename(sstate_dir, "SSTATE_DIR")
-
+        testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
     return testmsg
 
 def check_sanity_tmpdir_change(tmpdir, data):
@@ -156,12 +150,7 @@ def check_create_long_filename(filepath, pathname):
         if errno == 36: # ENAMETOOLONG
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
-            return "Failed to create a file in %s: %s\n" % (pathname, strerror)
-    return ""
-
-def check_can_read_write_directory(directory):
-    if not os.access(directory, os.R_OK|os.W_OK):
-       return "Insufficient permissions for %s" % directory
+            return "Failed to create a file in %s: %s" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
Signature · 1 line
1.7.7.6
#2 lore raw

PATCH 2/7 sanity.bbclass: copy the data store and finalise before running checks

In reply to Joshua Lock

At the ConfigParsed event the datastore has yet to be finalised and thus appends and overrides have not been set.
To ensure the sanity check is being run against the configuration values the user has set call finalize() on a copy of the datastore and pass that for all sanity checks.

---
 meta/classes/sanity.bbclass |  129 +++++++++++++++++++++++--------------------
 1 files changed, 69 insertions(+), 60 deletions(-)
meta/classes/sanity.bbclass +69 -60
index 05545f4..5361f1f 100644
@@ -182,8 +182,8 @@ def check_connectivity(d):
 
     return retval
 
-def check_supported_distro(e):
-    tested_distros = e.data.getVar('SANITY_TESTED_DISTROS', True)
+def check_supported_distro(sanity_data):
+    tested_distros = sanity_data.getVar('SANITY_TESTED_DISTROS', True)
     if not tested_distros:
         return
 
@@ -230,26 +230,26 @@ def check_supported_distro(e):
         bb.warn('Host distribution could not be determined; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.')
 
 # Checks we should only make if MACHINE is set correctly
-def check_sanity_validmachine(e):
+def check_sanity_validmachine(sanity_data):
     from bb import data
 
     messages = ""
 
     # Check TUNE_ARCH is set
-    if data.getVar('TUNE_ARCH', e.data, True) == 'INVALID':
+    if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID':
         messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n'
 
     # Check TARGET_ARCH is set correctly
-    if data.getVar('TARGE_ARCH', e.data, False) == '${TUNE_ARCH}':
+    if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}':
         messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n'
     
     # Check TARGET_OS is set
-    if data.getVar('TARGET_OS', e.data, True) == 'INVALID':
+    if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID':
         messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n'
 
     # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS
-    pkgarchs = data.getVar('PACKAGE_ARCHS', e.data, True)
-    tunepkg = data.getVar('TUNE_PKGARCH', e.data, True)
+    pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True)
+    tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True)
     tunefound = False
     seen = {}
     dups = []
@@ -271,7 +271,7 @@ def check_sanity_validmachine(e):
     return messages
 
 
-def check_sanity(e):
+def check_sanity(sanity_data):
     from bb import note, error, data, __version__
 
     try:
@@ -281,7 +281,7 @@ def check_sanity(e):
     import commands
 
     # Check the bitbake version meets minimum requirements
-    minversion = data.getVar('BB_MIN_VERSION', e.data , True)
+    minversion = data.getVar('BB_MIN_VERSION', sanity_data , True)
     if not minversion:
         # Hack: BB_MIN_VERSION hasn't been parsed yet so return 
         # and wait for the next call
@@ -303,42 +303,42 @@ def check_sanity(e):
         messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__)
 
     # Check that the MACHINE is valid, if it is set
-    if data.getVar('MACHINE', e.data, True):
-        if not check_conf_exists("conf/machine/${MACHINE}.conf", e.data):
+    if data.getVar('MACHINE', sanity_data, True):
+        if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data):
             messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n'
         else:
-            messages = messages + check_sanity_validmachine(e)
+            messages = messages + check_sanity_validmachine(sanity_data)
     else:
         messages = messages + 'Please set a MACHINE in your local.conf or environment\n'
 
     # Check we are using a valid lacal.conf
-    current_conf  = data.getVar('CONF_VERSION', e.data, True)
-    conf_version =  data.getVar('LOCALCONF_VERSION', e.data, True)
+    current_conf  = data.getVar('CONF_VERSION', sanity_data, True)
+    conf_version =  data.getVar('LOCALCONF_VERSION', sanity_data, True)
 
     if current_conf != conf_version:
         messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n"
 
     # Check bblayers.conf is valid
-    current_lconf = data.getVar('LCONF_VERSION', e.data, True)
-    lconf_version = data.getVar('LAYER_CONF_VERSION', e.data, True)
+    current_lconf = data.getVar('LCONF_VERSION', sanity_data, True)
+    lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True)
     if current_lconf != lconf_version:
         messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n"
 
     # If we have a site.conf, check it's valid
-    if check_conf_exists("conf/site.conf", e.data):
-        current_sconf = data.getVar('SCONF_VERSION', e.data, True)
-        sconf_version = data.getVar('SITE_CONF_VERSION', e.data, True)
+    if check_conf_exists("conf/site.conf", sanity_data):
+        current_sconf = data.getVar('SCONF_VERSION', sanity_data, True)
+        sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True)
         if current_sconf != sconf_version:
             messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n"
 
-    assume_provided = data.getVar('ASSUME_PROVIDED', e.data , True).split()
+    assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split()
     # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf
     if "diffstat-native" not in assume_provided:
         messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n'
 
     # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't 
     # set, since so much relies on it being set.
-    dldir = data.getVar('DL_DIR', e.data, True)
+    dldir = data.getVar('DL_DIR', sanity_data, True)
     if not dldir:
         messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n"
     if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,33 +346,33 @@ def check_sanity(e):
     
     # Check that the DISTRO is valid, if set
     # need to take into account DISTRO renaming DISTRO
-    distro = data.getVar('DISTRO', e.data, True)
+    distro = data.getVar('DISTRO', sanity_data, True)
     if distro:
-        if not ( check_conf_exists("conf/distro/${DISTRO}.conf", e.data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", e.data) ):
-            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", e.data, True )
+        if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ):
+            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True )
 
     missing = ""
 
-    if not check_app_exists("${MAKE}", e.data):
+    if not check_app_exists("${MAKE}", sanity_data):
         missing = missing + "GNU make,"
 
-    if not check_app_exists('${BUILD_PREFIX}gcc', e.data):
-        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", e.data, True)
+    if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data):
+        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True)
 
-    if not check_app_exists('${BUILD_PREFIX}g++', e.data):
-        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", e.data, True)
+    if not check_app_exists('${BUILD_PREFIX}g++', sanity_data):
+        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True)
 
-    required_utilities = e.data.getVar('SANITY_REQUIRED_UTILITIES', True)
+    required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True)
 
     if "qemu-native" in assume_provided:
-        if not check_app_exists("qemu-arm", e.data):
+        if not check_app_exists("qemu-arm", sanity_data):
             messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH"
 
-    paths = data.getVar('PATH', e.data, True).split(":")
+    paths = data.getVar('PATH', sanity_data, True).split(":")
     if "." in paths or "" in paths:
         messages = messages + "PATH contains '.' or '', which will break the build, please remove this."
 
-    if data.getVar('TARGET_ARCH', e.data, True) == "arm":
+    if data.getVar('TARGET_ARCH', sanity_data, True) == "arm":
         # This path is no longer user-readable in modern (very recent) Linux
         try:
             if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -386,7 +386,7 @@ def check_sanity(e):
             pass
 
     for util in required_utilities.split():
-        if not check_app_exists( util, e.data ):
+        if not check_app_exists( util, sanity_data ):
             missing = missing + "%s," % util
 
     if missing != "":
@@ -397,13 +397,13 @@ def check_sanity(e):
     if pseudo_msg != "":
         messages = messages + pseudo_msg + '\n'
 
-    check_supported_distro(e)
-    toolchain_msg = check_toolchain(e.data)
+    check_supported_distro(sanity_data)
+    toolchain_msg = check_toolchain(sanity_data)
     if toolchain_msg != "":
         messages = messages + toolchain_msg + '\n'
 
     # Check if DISPLAY is set if IMAGETEST is set
-    if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':
+    if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu':
         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
 
     omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(e):
         messages = messages + "Please use a umask which allows a+rx and u+rwx\n"
     os.umask(omask)
 
-    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', e.data, True )
+    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True )
     if not oes_bb_conf:
         messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n'
 
-    nolibs = data.getVar('NO32LIBS', e.data, True)
+    nolibs = data.getVar('NO32LIBS', sanity_data, True)
     if not nolibs:
         lib32path = '/lib'
         if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(e):
         if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
             messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
 
-    tmpdir = data.getVar('TMPDIR', e.data, True)
-    sstate_dir = data.getVar('SSTATE_DIR', e.data, True)
+    tmpdir = data.getVar('TMPDIR', sanity_data, True)
+    sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True)
 
     # Check saved sanity info
     last_sanity_version = 0
@@ -442,16 +442,16 @@ def check_sanity(e):
             if line.startswith('SSTATE_DIR'):
                 last_sstate_dir = line.split()[1]
     
-    sanity_version = int(data.getVar('SANITY_VERSION', e.data, True) or 1)
+    sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1)
     if last_sanity_version < sanity_version: 
-        messages = messages + check_sanity_version_change(e.data)
-        messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
-        messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
+        messages = messages + check_sanity_version_change(sanity_data)
+        messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
+        messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data)
     else: 
         if last_tmpdir != tmpdir:
-            messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
+            messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
         if last_sstate_dir != sstate_dir:
-            messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
+            messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data)
 
     if os.path.exists("conf") and not messages:
         f = file(sanityverfile, 'w')
@@ -476,8 +476,8 @@ def check_sanity(e):
     #
     # Check the 'ABI' of TMPDIR
     #
-    current_abi = data.getVar('OELAYOUT_ABI', e.data, True)
-    abifile = data.getVar('SANITY_ABIFILE', e.data, True)
+    current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True)
+    abifile = data.getVar('SANITY_ABIFILE', sanity_data, True)
     if os.path.exists(abifile):
         f = file(abifile, "r")
         abi = f.read().strip()
@@ -486,16 +486,16 @@ def check_sanity(e):
             f.write(current_abi)
         elif abi == "2" and current_abi == "3":
             bb.note("Converting staging from layout version 2 to layout version 3")
-            os.system(e.data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots"))
-            os.system(e.data.expand("ln -s sysroots ${TMPDIR}/staging"))
-            os.system(e.data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done"))
+            os.system(sanity_data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots"))
+            os.system(sanity_data.expand("ln -s sysroots ${TMPDIR}/staging"))
+            os.system(sanity_data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done"))
             f = file(abifile, "w")
             f.write(current_abi)
         elif abi == "3" and current_abi == "4":
             bb.note("Converting staging layout from version 3 to layout version 4")
-            if os.path.exists(e.data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")):
-                os.system(e.data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}"))
-                os.system(e.data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}"))
+            if os.path.exists(sanity_data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")):
+                os.system(sanity_data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}"))
+                os.system(sanity_data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}"))
 
             f = file(abifile, "w")
             f.write(current_abi)
@@ -503,7 +503,7 @@ def check_sanity(e):
             messages = messages + "Staging layout has changed. The cross directory has been deprecated and cross packages are now built under the native sysroot.\nThis requires a rebuild.\n"
         elif abi == "5" and current_abi == "6":
             bb.note("Converting staging layout from version 5 to layout version 6")
-            os.system(e.data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}"))
+            os.system(sanity_data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}"))
             f = file(abifile, "w")
             f.write(current_abi)
         elif abi == "7" and current_abi == "8":
@@ -516,7 +516,7 @@ def check_sanity(e):
         f.write(current_abi)
     f.close()
 
-    oeroot = data.getVar('COREBASE', e.data)
+    oeroot = data.getVar('COREBASE', sanity_data)
     if oeroot.find ('+') != -1:
         messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +."
     elif oeroot.find (' ') != -1:
@@ -525,12 +525,21 @@ def check_sanity(e):
     if messages != "":
         raise_sanity_error(messages)
 
+# Create a copy of the datastore and finalise it to ensure appends and 
+# overrides are set - the datastore has yet to be finalised at ConfigParsed
+def copy_data(e):
+    sanity_data = bb.data.createCopy(e.data)
+    sanity_data.finalize()
+    return sanity_data
+
 addhandler check_sanity_eventhandler
 python check_sanity_eventhandler() {
     if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
-        check_sanity(e)
+        sanity_data = copy_data(e)
+        check_sanity(sanity_data)
     elif bb.event.getName(e) == "SanityCheck":
-        check_sanity(e)
+        sanity_data = copy_data(e)
+        check_sanity(sanity_data)
         bb.event.fire(bb.event.SanityCheckPassed(), e.data)
 
     return
Signature · 1 line
1.7.7.6
#3 lore raw

PATCH 3/7 sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp)

In reply to Joshua Lock

Replace calls to data.getVar(VARIABLE, data_object, expand) to direct calls to the getVar method the the data_object.

---
 meta/classes/sanity.bbclass |   60 +++++++++++++++++++++---------------------
 1 files changed, 30 insertions(+), 30 deletions(-)
meta/classes/sanity.bbclass +30 -30
index 5361f1f..5cf9ea1 100644
@@ -236,20 +236,20 @@ def check_sanity_validmachine(sanity_data):
     messages = ""
 
     # Check TUNE_ARCH is set
-    if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID':
+    if sanity_data.getVar('TUNE_ARCH', True) == 'INVALID':
         messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n'
 
     # Check TARGET_ARCH is set correctly
-    if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}':
+    if sanity_data.getVar('TARGE_ARCH', False) == '${TUNE_ARCH}':
         messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n'
     
     # Check TARGET_OS is set
-    if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID':
+    if sanity_data.getVar('TARGET_OS', True) == 'INVALID':
         messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n'
 
     # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS
-    pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True)
-    tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True)
+    pkgarchs = sanity_data.getVar('PACKAGE_ARCHS', True)
+    tunepkg = sanity_data.getVar('TUNE_PKGARCH', True)
     tunefound = False
     seen = {}
     dups = []
@@ -281,7 +281,7 @@ def check_sanity(sanity_data):
     import commands
 
     # Check the bitbake version meets minimum requirements
-    minversion = data.getVar('BB_MIN_VERSION', sanity_data , True)
+    minversion = sanity_data.getVar('BB_MIN_VERSION', True)
     if not minversion:
         # Hack: BB_MIN_VERSION hasn't been parsed yet so return 
         # and wait for the next call
@@ -303,7 +303,7 @@ def check_sanity(sanity_data):
         messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__)
 
     # Check that the MACHINE is valid, if it is set
-    if data.getVar('MACHINE', sanity_data, True):
+    if sanity_data.getVar('MACHINE', True):
         if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data):
             messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n'
         else:
@@ -312,33 +312,33 @@ def check_sanity(sanity_data):
         messages = messages + 'Please set a MACHINE in your local.conf or environment\n'
 
     # Check we are using a valid lacal.conf
-    current_conf  = data.getVar('CONF_VERSION', sanity_data, True)
-    conf_version =  data.getVar('LOCALCONF_VERSION', sanity_data, True)
+    current_conf  = sanity_data.getVar('CONF_VERSION', True)
+    conf_version =  sanity_data.getVar('LOCALCONF_VERSION', True)
 
     if current_conf != conf_version:
         messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n"
 
     # Check bblayers.conf is valid
-    current_lconf = data.getVar('LCONF_VERSION', sanity_data, True)
-    lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True)
+    current_lconf = sanity_data.getVar('LCONF_VERSION', True)
+    lconf_version = sanity_data.getVar('LAYER_CONF_VERSION', True)
     if current_lconf != lconf_version:
         messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n"
 
     # If we have a site.conf, check it's valid
     if check_conf_exists("conf/site.conf", sanity_data):
-        current_sconf = data.getVar('SCONF_VERSION', sanity_data, True)
-        sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True)
+        current_sconf = sanity_data.getVar('SCONF_VERSION', True)
+        sconf_version = sanity_data.getVar('SITE_CONF_VERSION', True)
         if current_sconf != sconf_version:
             messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n"
 
-    assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split()
+    assume_provided = sanity_data.getVar('ASSUME_PROVIDED', True).split()
     # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf
     if "diffstat-native" not in assume_provided:
         messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n'
 
     # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't 
     # set, since so much relies on it being set.
-    dldir = data.getVar('DL_DIR', sanity_data, True)
+    dldir = sanity_data.getVar('DL_DIR', True)
     if not dldir:
         messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n"
     if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,10 +346,10 @@ def check_sanity(sanity_data):
     
     # Check that the DISTRO is valid, if set
     # need to take into account DISTRO renaming DISTRO
-    distro = data.getVar('DISTRO', sanity_data, True)
+    distro = sanity_data.getVar('DISTRO', True)
     if distro:
         if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ):
-            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True )
+            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % sanity_data.getVar("DISTRO", True )
 
     missing = ""
 
@@ -357,10 +357,10 @@ def check_sanity(sanity_data):
         missing = missing + "GNU make,"
 
     if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data):
-        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True)
+        missing = missing + "C Compiler (%sgcc)," % sanity_data.getVar("BUILD_PREFIX", True)
 
     if not check_app_exists('${BUILD_PREFIX}g++', sanity_data):
-        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True)
+        missing = missing + "C++ Compiler (%sg++)," % sanity_data.getVar("BUILD_PREFIX", True)
 
     required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True)
 
@@ -368,11 +368,11 @@ def check_sanity(sanity_data):
         if not check_app_exists("qemu-arm", sanity_data):
             messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH"
 
-    paths = data.getVar('PATH', sanity_data, True).split(":")
+    paths = sanity_data.getVar('PATH', True).split(":")
     if "." in paths or "" in paths:
         messages = messages + "PATH contains '.' or '', which will break the build, please remove this."
 
-    if data.getVar('TARGET_ARCH', sanity_data, True) == "arm":
+    if sanity_data.getVar('TARGET_ARCH', True) == "arm":
         # This path is no longer user-readable in modern (very recent) Linux
         try:
             if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -403,7 +403,7 @@ def check_sanity(sanity_data):
         messages = messages + toolchain_msg + '\n'
 
     # Check if DISPLAY is set if IMAGETEST is set
-    if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu':
+    if not sanity_data.getVar( 'DISPLAY', True ) and sanity_data.getVar( 'IMAGETEST', True ) == 'qemu':
         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
 
     omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(sanity_data):
         messages = messages + "Please use a umask which allows a+rx and u+rwx\n"
     os.umask(omask)
 
-    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True )
+    oes_bb_conf = sanity_data.getVar( 'OES_BITBAKE_CONF', True)
     if not oes_bb_conf:
         messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n'
 
-    nolibs = data.getVar('NO32LIBS', sanity_data, True)
+    nolibs = sanity_data.getVar('NO32LIBS', True)
     if not nolibs:
         lib32path = '/lib'
         if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(sanity_data):
         if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
             messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
 
-    tmpdir = data.getVar('TMPDIR', sanity_data, True)
-    sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True)
+    tmpdir = sanity_data.getVar('TMPDIR', True)
+    sstate_dir = sanity_data.getVar('SSTATE_DIR', True)
 
     # Check saved sanity info
     last_sanity_version = 0
@@ -442,7 +442,7 @@ def check_sanity(sanity_data):
             if line.startswith('SSTATE_DIR'):
                 last_sstate_dir = line.split()[1]
     
-    sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1)
+    sanity_version = int(sanity_data.getVar('SANITY_VERSION', True) or 1)
     if last_sanity_version < sanity_version: 
         messages = messages + check_sanity_version_change(sanity_data)
         messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
@@ -476,8 +476,8 @@ def check_sanity(sanity_data):
     #
     # Check the 'ABI' of TMPDIR
     #
-    current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True)
-    abifile = data.getVar('SANITY_ABIFILE', sanity_data, True)
+    current_abi = sanity_data.getVar('OELAYOUT_ABI', True)
+    abifile = sanity_data.getVar('SANITY_ABIFILE', True)
     if os.path.exists(abifile):
         f = file(abifile, "r")
         abi = f.read().strip()
@@ -516,7 +516,7 @@ def check_sanity(sanity_data):
         f.write(current_abi)
     f.close()
 
-    oeroot = data.getVar('COREBASE', sanity_data)
+    oeroot = sanity_data.getVar('COREBASE')
     if oeroot.find ('+') != -1:
         messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +."
     elif oeroot.find (' ') != -1:
Signature · 1 line
1.7.7.6
#4 lore raw

PATCH 4/7 sanity.bbclass: add newline to check_create_long_filename failure message

In reply to Joshua Lock

Each failure in the sanity message should be reported on a new line.

---
 meta/classes/sanity.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sanity.bbclass +1 -1
index 5cf9ea1..39da14b 100644
@@ -150,7 +150,7 @@ def check_create_long_filename(filepath, pathname):
         if errno == 36: # ENAMETOOLONG
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
-            return "Failed to create a file in %s: %s" % (pathname, strerror)
+            return "Failed to create a file in %s: %s.\n" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
Signature · 1 line
1.7.7.6
#5 lore raw

PATCH 5/7 sanity.bbclass: add extra information when SSTATE_CACHE unusable

In reply to Joshua Lock

If the user does not have write permissions to SSTATE_CACHE, detected by the check_create_long_filename() test failing with a "Permission denied" value in strerror, then suggest they might want to use the location as an entry in SSTATE_MIRRORS.

---
 meta/classes/sanity.bbclass |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
meta/classes/sanity.bbclass +3 -0
index 39da14b..e86edf7 100644
@@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data):
     testmsg = ""
     if sstate_dir != "":
         testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
+        # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
+        if testmsg.split(': ')[1].strip() == "Permission denied.":
+            testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
     return testmsg
 
 def check_sanity_tmpdir_change(tmpdir, data):
Signature · 1 line
1.7.7.6
#6 lore raw

Re: [PATCH 5/7] sanity.bbclass: add extra information when SSTATE_CACHE unusable

Quoted 15 lines · Joshua Lock

If the user does not have write permissions to SSTATE_CACHE, detected by the check_create_long_filename() test failing with a "Permission denied" value in strerror, then suggest they might want to use the location as an entry in SSTATE_MIRRORS.

---
  meta/classes/sanity.bbclass |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)
meta/classes/sanity.bbclass +0 -0
index 39da14b..e86edf7 100644

meta/classes/sanity.bbclass def check_sanity_sstate_dir_change(sstate_dir, data): · line 106

Quoted 7 lines
meta/classes/sanity.bbclass +3 -0
@@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data):
      testmsg = ""
      if sstate_dir != "":
          testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
+        # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
+        if testmsg.split(': ')[1].strip() == "Permission denied.":
+            testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
      return testmsg

Whoops. This one breaks when check_create_long_filename() succeeds. I've updated the git repo with the following hunk instead of above:

+ # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
+ try:
+ err = testmsg.split(': ')[1].strip()
+ if err == "Permission denied.":
+ testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
+ except IndexError:
+ pass

Tested with both an SSTATE_DIR I can write to and one I can't.

Thanks,

Joshua

Quoted 2 lines
meta/classes/sanity.bbclass +0 -0
  def check_sanity_tmpdir_change(tmpdir, data):
Signature · 3 lines
Joshua Lock
        Yocto Project
        Intel Open Source Technology Centre
#7 lore raw

PATCH 6/7 sanity.bbclass: catch an extra exception in check_create_long_filename

In reply to Joshua Lock

The call to bb.mkdirhier() in check_create_long_filename() can fail with an OSError, explicitly catch this and report something useful to the user.

---
 meta/classes/sanity.bbclass |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
meta/classes/sanity.bbclass +2 -0
index e86edf7..4b52699 100644
@@ -154,6 +154,8 @@ def check_create_long_filename(filepath, pathname):
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
             return "Failed to create a file in %s: %s.\n" % (pathname, strerror)
+    except OSError as (errno, strerror):
+        return "Failed to create %s directory in which to run long name sanity check: %s.\n" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
Signature · 1 line
1.7.7.6
#8 lore raw

PATCH 7/7 sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

In reply to Joshua Lock

This enables a user to use bitbake -e even when the sanity checks are failing.

---
 meta/classes/sanity.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sanity.bbclass +1 -1
index 4b52699..689155c 100644
@@ -539,7 +539,7 @@ def copy_data(e):
 
 addhandler check_sanity_eventhandler
 python check_sanity_eventhandler() {
-    if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
+    if bb.event.getName(e) == "BuildStarted" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
         sanity_data = copy_data(e)
         check_sanity(sanity_data)
     elif bb.event.getName(e) == "SanityCheck":
Signature · 1 line
1.7.7.6
#9 lore raw
In reply to Joshua Lock

This series looks quite sane to me. Nicely done.

Signature · 1 line
Christopher Larson
Quoted 53 lines · Joshua Lock

My recent change to the sanity class to warn users when they don't have R/W permission to their SSTATE_CACHE directory has caused at least a couple of users trouble, including the Yocto project autobuilder.

I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.

The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing check_create_long_filename() call, where I make the suggestion of using SSTATE_MIRRORS if that call fails with "Permission denied" when called against SSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.

Thanks to Chris and Elizabeth for pointers as to why this was failing.

Cheers,

Joshua

The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:

tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib (http://git.openembedded.org/openembedded-core-contrib) josh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity

Joshua Lock (7):
Revert "sanity.bbclass: check user can read and write to SSTATE_DIR" sanity.bbclass: copy the data store and finalise before running checks
sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp) sanity.bbclass: add newline to check_create_long_filename failure message
sanity.bbclass: add extra information when SSTATE_CACHE unusable sanity.bbclass: catch an extra exception in check_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

meta/classes/sanity.bbclass | 153 ++++++++++++++++++++++---------------------

1 files changed, 78 insertions(+), 75 deletions(-)
Signature · 7 lines
1.7.7.6


_______________________________________________
Openembedded-core mailing list
[email protected] (mailto:[email protected])
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
#10 lore raw
In reply to Joshua Lock
Quoted 40 lines · Joshua Lock

My recent change to the sanity class to warn users when they don't have R/W permission to their SSTATE_CACHE directory has caused at least a couple of users trouble, including the Yocto project autobuilder.

I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.

The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing check_create_long_filename() call, where I make the suggestion of using SSTATE_MIRRORS if that call fails with "Permission denied" when called against SSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.

Thanks to Chris and Elizabeth for pointers as to why this was failing.

Cheers,

Joshua

The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:

   tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib josh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity

Joshua Lock (7):
Revert "sanity.bbclass: check user can read and write to SSTATE_DIR" sanity.bbclass: copy the data store and finalise before running
checks
sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp) sanity.bbclass: add newline to check_create_long_filename failure
message
sanity.bbclass: add extra information when SSTATE_CACHE unusable sanity.bbclass: catch an extra exception in
check_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

Quoted 3 lines
  meta/classes/sanity.bbclass |  153 ++++++++++++++++++++++---------------------
  1 files changed, 78 insertions(+), 75 deletions(-)

Merged the updated version into OE-Core

Thanks
Sau!