Sanity class clean ups
cover Sanity class clean ups
My recent change to the sanity class to warn users when they don't have R/W permission to their SSTATE_CACHE directory has caused at least a couple of users trouble, including the Yocto project autobuilder.
I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.
The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing check_create_long_filename() call, where I make the suggestion of using SSTATE_MIRRORS if that call fails with "Permission denied" when called against SSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.
Thanks to Chris and Elizabeth for pointers as to why this was failing.
Cheers,
Joshua
The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:
tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib josh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity
Joshua Lock (7):
Revert "sanity.bbclass: check user can read and write to SSTATE_DIR" sanity.bbclass: copy the data store and finalise before running
checks
sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp) sanity.bbclass: add newline to check_create_long_filename failure
message
sanity.bbclass: add extra information when SSTATE_CACHE unusable sanity.bbclass: catch an extra exception in
check_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed
meta/classes/sanity.bbclass | 153 ++++++++++++++++++++++--------------------- 1 files changed, 78 insertions(+), 75 deletions(-)
Signature · 1 line
1.7.7.6
PATCH 1/7 Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"
This has caused problems for several users, including the Yocto Project autobuilder. Since the message was added in order to be more user friendly revert the change.
This reverts commit 0c0c4efbf92bcf0f8942f17c18525a4b4ed1798c.
--- meta/classes/sanity.bbclass | 17 +++-------------- 1 files changed, 3 insertions(+), 14 deletions(-)
meta/classes/sanity.bbclass +3 -14
@@ -96,16 +96,10 @@ def check_conf_exists(fn, data): def check_sanity_sstate_dir_change(sstate_dir, data): # Sanity checks to be done when the value of SSTATE_DIR changes + # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS) testmsg = "" if sstate_dir != "": - # Check that the user can read and write to SSTATE_DIR - sstatemsg = check_can_read_write_directory(sstate_dir) or None - if sstatemsg: - sstatemsg = sstatemsg + ". You could try using it as an SSTATE_MIRRORS instead of SSTATE_CACHE.\n" - testmsg = testmsg + sstatemsg - # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS) - testmsg = testmsg + check_create_long_filename(sstate_dir, "SSTATE_DIR") - + testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR") return testmsg def check_sanity_tmpdir_change(tmpdir, data):
@@ -156,12 +150,7 @@ def check_create_long_filename(filepath, pathname): if errno == 36: # ENAMETOOLONG return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname else: - return "Failed to create a file in %s: %s\n" % (pathname, strerror) - return "" - -def check_can_read_write_directory(directory): - if not os.access(directory, os.R_OK|os.W_OK): - return "Insufficient permissions for %s" % directory + return "Failed to create a file in %s: %s" % (pathname, strerror) return "" def check_connectivity(d):
Signature · 1 line
1.7.7.6
PATCH 2/7 sanity.bbclass: copy the data store and finalise before running checks
In reply to Joshua LockAt the ConfigParsed event the datastore has yet to be finalised and thus appends and overrides have not been set.
To ensure the sanity check is being run against the configuration values the user has set call finalize() on a copy of the datastore and pass that for all sanity checks.
--- meta/classes/sanity.bbclass | 129 +++++++++++++++++++++++-------------------- 1 files changed, 69 insertions(+), 60 deletions(-)
meta/classes/sanity.bbclass +69 -60
@@ -182,8 +182,8 @@ def check_connectivity(d): return retval -def check_supported_distro(e): - tested_distros = e.data.getVar('SANITY_TESTED_DISTROS', True) +def check_supported_distro(sanity_data): + tested_distros = sanity_data.getVar('SANITY_TESTED_DISTROS', True) if not tested_distros: return
@@ -230,26 +230,26 @@ def check_supported_distro(e): bb.warn('Host distribution could not be determined; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.') # Checks we should only make if MACHINE is set correctly -def check_sanity_validmachine(e): +def check_sanity_validmachine(sanity_data): from bb import data messages = "" # Check TUNE_ARCH is set - if data.getVar('TUNE_ARCH', e.data, True) == 'INVALID': + if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID': messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n' # Check TARGET_ARCH is set correctly - if data.getVar('TARGE_ARCH', e.data, False) == '${TUNE_ARCH}': + if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}': messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n' # Check TARGET_OS is set - if data.getVar('TARGET_OS', e.data, True) == 'INVALID': + if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID': messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n' # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS - pkgarchs = data.getVar('PACKAGE_ARCHS', e.data, True) - tunepkg = data.getVar('TUNE_PKGARCH', e.data, True) + pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True) + tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True) tunefound = False seen = {} dups = []
@@ -271,7 +271,7 @@ def check_sanity_validmachine(e): return messages -def check_sanity(e): +def check_sanity(sanity_data): from bb import note, error, data, __version__ try:
@@ -281,7 +281,7 @@ def check_sanity(e): import commands # Check the bitbake version meets minimum requirements - minversion = data.getVar('BB_MIN_VERSION', e.data , True) + minversion = data.getVar('BB_MIN_VERSION', sanity_data , True) if not minversion: # Hack: BB_MIN_VERSION hasn't been parsed yet so return # and wait for the next call
@@ -303,42 +303,42 @@ def check_sanity(e): messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__) # Check that the MACHINE is valid, if it is set - if data.getVar('MACHINE', e.data, True): - if not check_conf_exists("conf/machine/${MACHINE}.conf", e.data): + if data.getVar('MACHINE', sanity_data, True): + if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data): messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n' else: - messages = messages + check_sanity_validmachine(e) + messages = messages + check_sanity_validmachine(sanity_data) else: messages = messages + 'Please set a MACHINE in your local.conf or environment\n' # Check we are using a valid lacal.conf - current_conf = data.getVar('CONF_VERSION', e.data, True) - conf_version = data.getVar('LOCALCONF_VERSION', e.data, True) + current_conf = data.getVar('CONF_VERSION', sanity_data, True) + conf_version = data.getVar('LOCALCONF_VERSION', sanity_data, True) if current_conf != conf_version: messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n" # Check bblayers.conf is valid - current_lconf = data.getVar('LCONF_VERSION', e.data, True) - lconf_version = data.getVar('LAYER_CONF_VERSION', e.data, True) + current_lconf = data.getVar('LCONF_VERSION', sanity_data, True) + lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True) if current_lconf != lconf_version: messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n" # If we have a site.conf, check it's valid - if check_conf_exists("conf/site.conf", e.data): - current_sconf = data.getVar('SCONF_VERSION', e.data, True) - sconf_version = data.getVar('SITE_CONF_VERSION', e.data, True) + if check_conf_exists("conf/site.conf", sanity_data): + current_sconf = data.getVar('SCONF_VERSION', sanity_data, True) + sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True) if current_sconf != sconf_version: messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n" - assume_provided = data.getVar('ASSUME_PROVIDED', e.data , True).split() + assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split() # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf if "diffstat-native" not in assume_provided: messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n' # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't # set, since so much relies on it being set. - dldir = data.getVar('DL_DIR', e.data, True) + dldir = data.getVar('DL_DIR', sanity_data, True) if not dldir: messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n" if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,33 +346,33 @@ def check_sanity(e): # Check that the DISTRO is valid, if set # need to take into account DISTRO renaming DISTRO - distro = data.getVar('DISTRO', e.data, True) + distro = data.getVar('DISTRO', sanity_data, True) if distro: - if not ( check_conf_exists("conf/distro/${DISTRO}.conf", e.data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", e.data) ): - messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", e.data, True ) + if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ): + messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True ) missing = "" - if not check_app_exists("${MAKE}", e.data): + if not check_app_exists("${MAKE}", sanity_data): missing = missing + "GNU make," - if not check_app_exists('${BUILD_PREFIX}gcc', e.data): - missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", e.data, True) + if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data): + missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True) - if not check_app_exists('${BUILD_PREFIX}g++', e.data): - missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", e.data, True) + if not check_app_exists('${BUILD_PREFIX}g++', sanity_data): + missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True) - required_utilities = e.data.getVar('SANITY_REQUIRED_UTILITIES', True) + required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True) if "qemu-native" in assume_provided: - if not check_app_exists("qemu-arm", e.data): + if not check_app_exists("qemu-arm", sanity_data): messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH" - paths = data.getVar('PATH', e.data, True).split(":") + paths = data.getVar('PATH', sanity_data, True).split(":") if "." in paths or "" in paths: messages = messages + "PATH contains '.' or '', which will break the build, please remove this." - if data.getVar('TARGET_ARCH', e.data, True) == "arm": + if data.getVar('TARGET_ARCH', sanity_data, True) == "arm": # This path is no longer user-readable in modern (very recent) Linux try: if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -386,7 +386,7 @@ def check_sanity(e): pass for util in required_utilities.split(): - if not check_app_exists( util, e.data ): + if not check_app_exists( util, sanity_data ): missing = missing + "%s," % util if missing != "":
@@ -397,13 +397,13 @@ def check_sanity(e): if pseudo_msg != "": messages = messages + pseudo_msg + '\n' - check_supported_distro(e) - toolchain_msg = check_toolchain(e.data) + check_supported_distro(sanity_data) + toolchain_msg = check_toolchain(sanity_data) if toolchain_msg != "": messages = messages + toolchain_msg + '\n' # Check if DISPLAY is set if IMAGETEST is set - if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu': + if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu': messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n' omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(e): messages = messages + "Please use a umask which allows a+rx and u+rwx\n" os.umask(omask) - oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', e.data, True ) + oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True ) if not oes_bb_conf: messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n' - nolibs = data.getVar('NO32LIBS', e.data, True) + nolibs = data.getVar('NO32LIBS', sanity_data, True) if not nolibs: lib32path = '/lib' if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(e): if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'): messages = messages + "You have a 32-bit libc, but no 32-bit headers. You must install the 32-bit libc headers.\n" - tmpdir = data.getVar('TMPDIR', e.data, True) - sstate_dir = data.getVar('SSTATE_DIR', e.data, True) + tmpdir = data.getVar('TMPDIR', sanity_data, True) + sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True) # Check saved sanity info last_sanity_version = 0
@@ -442,16 +442,16 @@ def check_sanity(e): if line.startswith('SSTATE_DIR'): last_sstate_dir = line.split()[1] - sanity_version = int(data.getVar('SANITY_VERSION', e.data, True) or 1) + sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1) if last_sanity_version < sanity_version: - messages = messages + check_sanity_version_change(e.data) - messages = messages + check_sanity_tmpdir_change(tmpdir, e.data) - messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data) + messages = messages + check_sanity_version_change(sanity_data) + messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data) + messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data) else: if last_tmpdir != tmpdir: - messages = messages + check_sanity_tmpdir_change(tmpdir, e.data) + messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data) if last_sstate_dir != sstate_dir: - messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data) + messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data) if os.path.exists("conf") and not messages: f = file(sanityverfile, 'w')
@@ -476,8 +476,8 @@ def check_sanity(e): # # Check the 'ABI' of TMPDIR # - current_abi = data.getVar('OELAYOUT_ABI', e.data, True) - abifile = data.getVar('SANITY_ABIFILE', e.data, True) + current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True) + abifile = data.getVar('SANITY_ABIFILE', sanity_data, True) if os.path.exists(abifile): f = file(abifile, "r") abi = f.read().strip()
@@ -486,16 +486,16 @@ def check_sanity(e): f.write(current_abi) elif abi == "2" and current_abi == "3": bb.note("Converting staging from layout version 2 to layout version 3") - os.system(e.data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots")) - os.system(e.data.expand("ln -s sysroots ${TMPDIR}/staging")) - os.system(e.data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done")) + os.system(sanity_data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots")) + os.system(sanity_data.expand("ln -s sysroots ${TMPDIR}/staging")) + os.system(sanity_data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done")) f = file(abifile, "w") f.write(current_abi) elif abi == "3" and current_abi == "4": bb.note("Converting staging layout from version 3 to layout version 4") - if os.path.exists(e.data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")): - os.system(e.data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}")) - os.system(e.data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")) + if os.path.exists(sanity_data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")): + os.system(sanity_data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}")) + os.system(sanity_data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")) f = file(abifile, "w") f.write(current_abi)
@@ -503,7 +503,7 @@ def check_sanity(e): messages = messages + "Staging layout has changed. The cross directory has been deprecated and cross packages are now built under the native sysroot.\nThis requires a rebuild.\n" elif abi == "5" and current_abi == "6": bb.note("Converting staging layout from version 5 to layout version 6") - os.system(e.data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}")) + os.system(sanity_data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}")) f = file(abifile, "w") f.write(current_abi) elif abi == "7" and current_abi == "8":
@@ -516,7 +516,7 @@ def check_sanity(e): f.write(current_abi) f.close() - oeroot = data.getVar('COREBASE', e.data) + oeroot = data.getVar('COREBASE', sanity_data) if oeroot.find ('+') != -1: messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +." elif oeroot.find (' ') != -1:
@@ -525,12 +525,21 @@ def check_sanity(e): if messages != "": raise_sanity_error(messages) +# Create a copy of the datastore and finalise it to ensure appends and +# overrides are set - the datastore has yet to be finalised at ConfigParsed +def copy_data(e): + sanity_data = bb.data.createCopy(e.data) + sanity_data.finalize() + return sanity_data + addhandler check_sanity_eventhandler python check_sanity_eventhandler() { if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1": - check_sanity(e) + sanity_data = copy_data(e) + check_sanity(sanity_data) elif bb.event.getName(e) == "SanityCheck": - check_sanity(e) + sanity_data = copy_data(e) + check_sanity(sanity_data) bb.event.fire(bb.event.SanityCheckPassed(), e.data) return
Signature · 1 line
1.7.7.6
PATCH 3/7 sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp)
In reply to Joshua LockReplace calls to data.getVar(VARIABLE, data_object, expand) to direct calls to the getVar method the the data_object.
--- meta/classes/sanity.bbclass | 60 +++++++++++++++++++++--------------------- 1 files changed, 30 insertions(+), 30 deletions(-)
meta/classes/sanity.bbclass +30 -30
@@ -236,20 +236,20 @@ def check_sanity_validmachine(sanity_data): messages = "" # Check TUNE_ARCH is set - if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID': + if sanity_data.getVar('TUNE_ARCH', True) == 'INVALID': messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n' # Check TARGET_ARCH is set correctly - if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}': + if sanity_data.getVar('TARGE_ARCH', False) == '${TUNE_ARCH}': messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n' # Check TARGET_OS is set - if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID': + if sanity_data.getVar('TARGET_OS', True) == 'INVALID': messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n' # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS - pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True) - tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True) + pkgarchs = sanity_data.getVar('PACKAGE_ARCHS', True) + tunepkg = sanity_data.getVar('TUNE_PKGARCH', True) tunefound = False seen = {} dups = []
@@ -281,7 +281,7 @@ def check_sanity(sanity_data): import commands # Check the bitbake version meets minimum requirements - minversion = data.getVar('BB_MIN_VERSION', sanity_data , True) + minversion = sanity_data.getVar('BB_MIN_VERSION', True) if not minversion: # Hack: BB_MIN_VERSION hasn't been parsed yet so return # and wait for the next call
@@ -303,7 +303,7 @@ def check_sanity(sanity_data): messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__) # Check that the MACHINE is valid, if it is set - if data.getVar('MACHINE', sanity_data, True): + if sanity_data.getVar('MACHINE', True): if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data): messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n' else:
@@ -312,33 +312,33 @@ def check_sanity(sanity_data): messages = messages + 'Please set a MACHINE in your local.conf or environment\n' # Check we are using a valid lacal.conf - current_conf = data.getVar('CONF_VERSION', sanity_data, True) - conf_version = data.getVar('LOCALCONF_VERSION', sanity_data, True) + current_conf = sanity_data.getVar('CONF_VERSION', True) + conf_version = sanity_data.getVar('LOCALCONF_VERSION', True) if current_conf != conf_version: messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n" # Check bblayers.conf is valid - current_lconf = data.getVar('LCONF_VERSION', sanity_data, True) - lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True) + current_lconf = sanity_data.getVar('LCONF_VERSION', True) + lconf_version = sanity_data.getVar('LAYER_CONF_VERSION', True) if current_lconf != lconf_version: messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n" # If we have a site.conf, check it's valid if check_conf_exists("conf/site.conf", sanity_data): - current_sconf = data.getVar('SCONF_VERSION', sanity_data, True) - sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True) + current_sconf = sanity_data.getVar('SCONF_VERSION', True) + sconf_version = sanity_data.getVar('SITE_CONF_VERSION', True) if current_sconf != sconf_version: messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n" - assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split() + assume_provided = sanity_data.getVar('ASSUME_PROVIDED', True).split() # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf if "diffstat-native" not in assume_provided: messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n' # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't # set, since so much relies on it being set. - dldir = data.getVar('DL_DIR', sanity_data, True) + dldir = sanity_data.getVar('DL_DIR', True) if not dldir: messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n" if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,10 +346,10 @@ def check_sanity(sanity_data): # Check that the DISTRO is valid, if set # need to take into account DISTRO renaming DISTRO - distro = data.getVar('DISTRO', sanity_data, True) + distro = sanity_data.getVar('DISTRO', True) if distro: if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ): - messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True ) + messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % sanity_data.getVar("DISTRO", True ) missing = ""
@@ -357,10 +357,10 @@ def check_sanity(sanity_data): missing = missing + "GNU make," if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data): - missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True) + missing = missing + "C Compiler (%sgcc)," % sanity_data.getVar("BUILD_PREFIX", True) if not check_app_exists('${BUILD_PREFIX}g++', sanity_data): - missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True) + missing = missing + "C++ Compiler (%sg++)," % sanity_data.getVar("BUILD_PREFIX", True) required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True)
@@ -368,11 +368,11 @@ def check_sanity(sanity_data): if not check_app_exists("qemu-arm", sanity_data): messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH" - paths = data.getVar('PATH', sanity_data, True).split(":") + paths = sanity_data.getVar('PATH', True).split(":") if "." in paths or "" in paths: messages = messages + "PATH contains '.' or '', which will break the build, please remove this." - if data.getVar('TARGET_ARCH', sanity_data, True) == "arm": + if sanity_data.getVar('TARGET_ARCH', True) == "arm": # This path is no longer user-readable in modern (very recent) Linux try: if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -403,7 +403,7 @@ def check_sanity(sanity_data): messages = messages + toolchain_msg + '\n' # Check if DISPLAY is set if IMAGETEST is set - if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu': + if not sanity_data.getVar( 'DISPLAY', True ) and sanity_data.getVar( 'IMAGETEST', True ) == 'qemu': messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n' omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(sanity_data): messages = messages + "Please use a umask which allows a+rx and u+rwx\n" os.umask(omask) - oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True ) + oes_bb_conf = sanity_data.getVar( 'OES_BITBAKE_CONF', True) if not oes_bb_conf: messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n' - nolibs = data.getVar('NO32LIBS', sanity_data, True) + nolibs = sanity_data.getVar('NO32LIBS', True) if not nolibs: lib32path = '/lib' if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(sanity_data): if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'): messages = messages + "You have a 32-bit libc, but no 32-bit headers. You must install the 32-bit libc headers.\n" - tmpdir = data.getVar('TMPDIR', sanity_data, True) - sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True) + tmpdir = sanity_data.getVar('TMPDIR', True) + sstate_dir = sanity_data.getVar('SSTATE_DIR', True) # Check saved sanity info last_sanity_version = 0
@@ -442,7 +442,7 @@ def check_sanity(sanity_data): if line.startswith('SSTATE_DIR'): last_sstate_dir = line.split()[1] - sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1) + sanity_version = int(sanity_data.getVar('SANITY_VERSION', True) or 1) if last_sanity_version < sanity_version: messages = messages + check_sanity_version_change(sanity_data) messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
@@ -476,8 +476,8 @@ def check_sanity(sanity_data): # # Check the 'ABI' of TMPDIR # - current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True) - abifile = data.getVar('SANITY_ABIFILE', sanity_data, True) + current_abi = sanity_data.getVar('OELAYOUT_ABI', True) + abifile = sanity_data.getVar('SANITY_ABIFILE', True) if os.path.exists(abifile): f = file(abifile, "r") abi = f.read().strip()
@@ -516,7 +516,7 @@ def check_sanity(sanity_data): f.write(current_abi) f.close() - oeroot = data.getVar('COREBASE', sanity_data) + oeroot = sanity_data.getVar('COREBASE') if oeroot.find ('+') != -1: messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +." elif oeroot.find (' ') != -1:
Signature · 1 line
1.7.7.6
PATCH 4/7 sanity.bbclass: add newline to check_create_long_filename failure message
In reply to Joshua LockEach failure in the sanity message should be reported on a new line.
--- meta/classes/sanity.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sanity.bbclass +1 -1
@@ -150,7 +150,7 @@ def check_create_long_filename(filepath, pathname): if errno == 36: # ENAMETOOLONG return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname else: - return "Failed to create a file in %s: %s" % (pathname, strerror) + return "Failed to create a file in %s: %s.\n" % (pathname, strerror) return "" def check_connectivity(d):
Signature · 1 line
1.7.7.6
PATCH 5/7 sanity.bbclass: add extra information when SSTATE_CACHE unusable
In reply to Joshua LockIf the user does not have write permissions to SSTATE_CACHE, detected by the check_create_long_filename() test failing with a "Permission denied" value in strerror, then suggest they might want to use the location as an entry in SSTATE_MIRRORS.
--- meta/classes/sanity.bbclass | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
meta/classes/sanity.bbclass +3 -0
@@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data): testmsg = "" if sstate_dir != "": testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR") + # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS + if testmsg.split(': ')[1].strip() == "Permission denied.": + testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir) return testmsg def check_sanity_tmpdir_change(tmpdir, data):
Signature · 1 line
1.7.7.6
PATCH 6/7 sanity.bbclass: catch an extra exception in check_create_long_filename
In reply to Joshua LockThe call to bb.mkdirhier() in check_create_long_filename() can fail with an OSError, explicitly catch this and report something useful to the user.
--- meta/classes/sanity.bbclass | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
meta/classes/sanity.bbclass +2 -0
@@ -154,6 +154,8 @@ def check_create_long_filename(filepath, pathname): return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname else: return "Failed to create a file in %s: %s.\n" % (pathname, strerror) + except OSError as (errno, strerror): + return "Failed to create %s directory in which to run long name sanity check: %s.\n" % (pathname, strerror) return "" def check_connectivity(d):
Signature · 1 line
1.7.7.6
PATCH 7/7 sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed
In reply to Joshua LockThis enables a user to use bitbake -e even when the sanity checks are failing.
--- meta/classes/sanity.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sanity.bbclass +1 -1
@@ -539,7 +539,7 @@ def copy_data(e): addhandler check_sanity_eventhandler python check_sanity_eventhandler() { - if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1": + if bb.event.getName(e) == "BuildStarted" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1": sanity_data = copy_data(e) check_sanity(sanity_data) elif bb.event.getName(e) == "SanityCheck":
Signature · 1 line
1.7.7.6
This series looks quite sane to me. Nicely done.
Signature · 1 line
Christopher Larson
Quoted 53 lines · Joshua Lock
My recent change to the sanity class to warn users when they don't have R/W permission to their
SSTATE_CACHEdirectory has caused at least a couple of users trouble, including the Yocto project autobuilder.I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.
The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing
check_create_long_filename()call, where I make the suggestion of usingSSTATE_MIRRORSif that call fails with "Permission denied" when called againstSSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.Thanks to Chris and Elizabeth for pointers as to why this was failing.
Cheers,
Joshua
The following changes since commit
e6333825c3482a559a0c0499e17f8f48d3042ddf:tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib(http://git.openembedded.org/openembedded-core-contrib) josh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanityJoshua Lock (7):
Revert "sanity.bbclass: check user can read and write toSSTATE_DIR" sanity.bbclass: copy the data store and finalise before running checks
sanity.bbclass:data.getVar(VAR, obj, exp)->obj.getVar(VAR, exp)sanity.bbclass: add newline tocheck_create_long_filenamefailure message
sanity.bbclass: add extra information whenSSTATE_CACHEunusable sanity.bbclass: catch an extra exception incheck_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsedmeta/classes/sanity.bbclass | 153 ++++++++++++++++++++++---------------------
1 files changed, 78 insertions(+), 75 deletions(-)Signature · 7 lines
1.7.7.6 _______________________________________________ Openembedded-core mailing list [email protected] (mailto:[email protected]) http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 5/7] sanity.bbclass: add extra information when SSTATE_CACHE unusable
In reply to Joshua LockQuoted 15 lines · Joshua Lock
If the user does not have write permissions to
SSTATE_CACHE, detected by thecheck_create_long_filename()test failing with a "Permission denied" value in strerror, then suggest they might want to use the location as an entry inSSTATE_MIRRORS.Signed-off-by Joshua Lock<[email protected]>--- meta/classes/sanity.bbclass | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)meta/classes/sanity.bbclass +0
-0
meta/classes/sanity.bbclass def check_sanity_sstate_dir_change(sstate_dir, data): · line 106
Quoted 7 lines
meta/classes/sanity.bbclass +3
-0@@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data): testmsg = "" if sstate_dir != "": testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR") + # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS + if testmsg.split(': ')[1].strip() == "Permission denied.": + testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir) return testmsg
Whoops. This one breaks when check_create_long_filename() succeeds. I've updated the git repo with the following hunk instead of above:
+ # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
+ try:
+ err = testmsg.split(': ')[1].strip()
+ if err == "Permission denied.":
+ testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
+ except IndexError:
+ pass
Tested with both an SSTATE_DIR I can write to and one I can't.
Thanks,
Joshua
Quoted 2 lines
meta/classes/sanity.bbclass +0
-0def check_sanity_tmpdir_change(tmpdir, data):
Signature · 3 lines
Joshua Lock
Yocto Project
Intel Open Source Technology CentreQuoted 40 lines · Joshua Lock
My recent change to the sanity class to warn users when they don't have R/W permission to their
SSTATE_CACHEdirectory has caused at least a couple of users trouble, including the Yocto project autobuilder.I have been unable to reproduce the issue but after discussionwith Elizabeth and Chris on #yocto I came up with the following series.
The series includes a revert my original change as it seems the simple logic has severla edge cases. I replace it later in the series with a simple piggy-back on the existing
check_create_long_filename()call, where I make the suggestion of usingSSTATE_MIRRORSif that call fails with "Permission denied" when called againstSSTATE_DIR. This check has been in use for some time and, to the best of my knowledge, doesn't trigger invalid failures.Thanks to Chris and Elizabeth for pointers as to why this was failing.
Cheers,
Joshua
The following changes since commit
e6333825c3482a559a0c0499e17f8f48d3042ddf:tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)are available in the git repository at:
git://git.openembedded.org/openembedded-core-contribjosh/sanity
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanityJoshua Lock (7):
Revert "sanity.bbclass: check user can read and write toSSTATE_DIR" sanity.bbclass: copy the data store and finalise before running
checks
sanity.bbclass:data.getVar(VAR, obj, exp)->obj.getVar(VAR, exp)sanity.bbclass: add newline tocheck_create_long_filenamefailure
message
sanity.bbclass: add extra information whenSSTATE_CACHEunusable sanity.bbclass: catch an extra exception in
check_create_long_filename
sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed
Quoted 3 lines
meta/classes/sanity.bbclass | 153 ++++++++++++++++++++++--------------------- 1 files changed, 78 insertions(+), 75 deletions(-)
Merged the updated version into OE-Core
Thanks
Sau!