Missing deps discovered in per-recipe builds

I was using simple script to build each component alone:

I was using simple script to build each component alone:

#!/bin/bash

rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*

ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u > component.list ncomponents=cat component.list | wc -l
i=1
for component in cat component.list; do
echo "Building component: ${component} ($i/$ncomponents)" bitbake -c cleansstate ${component} | tee log.${component}; bitbake ${component} | tee -a log.${component};
rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=expr $i + 1
done

Quoted 4 lines

rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*

* eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy:
ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
ERROR: Function failed: split_and_strip_files

Signed-off-by Martin Jansa <[email protected]>

---
 meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++
 1 file changed, 5 insertions(+)

meta/recipes-core/eglibc/eglibc-locale.inc +0 -0

index f4cb814..fc51022 100644

meta/recipes-core/eglibc/eglibc-locale.inc +0 -0

 # Binary locales are generated at build time if ENABLE_BINARY_LOCALE_GENERATION
 # is set. The idea is to avoid running localedef on the target (at first boot)
 # to decrease initial boot time and avoid localedef being killed by the OOM

Signature · 1 line

1.8.2.1

Quoted 9 lines · On Friday 05 July 2013 03:28:21 Martin Jansa wrote:

I was using simple script to build each component alone:

#!/bin/bash

rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*

Quoted 13 lines

ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u > component.list ncomponents=cat component.list | wc -l
i=1
for component in cat component.list; do
echo "Building component: ${component} ($i/$ncomponents)" bitbake -c cleansstate ${component} | tee log.${component}; bitbake ${component} | tee -a log.${component};
rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=expr $i + 1
done

A cleaned up form of this would be a great addition to the scripts/contrib directory if you have time...

* eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy:
ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
ERROR: Function failed: split_and_strip_files

Quoted 8 lines · Martin Jansa

* eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy:
ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
ERROR: Function failed: split_and_strip_files

Wouldn't it be better to add the necessary dep to the do_package[depends] in package.bbclass?

---
 meta/classes/package.bbclass | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

In this case yes, but what about allarch recipes, creating binutils-cross dependency in their do_package task would probably make their stamps depending on architecture.

Quoted 4 lines · Martin Jansa

In this case yes, but what about allarch recipes, creating binutils-cross dependency in their do_package task would probably make their stamps depending on architecture.

def get_binutils_dep(d):
if 'target' in d.getVar('OVERRIDES', True).split():
return d.expand('virtual/${TARGET_PREFIX}binutils')
else:
return ''

do_package[depends] += "${@get_binutils_dep(d)}"

* eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy:
ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
ERROR: Function failed: split_and_strip_files

Signed-off-by Martin Jansa <[email protected]>

---
meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++

1 file changed, 5 insertions(+)

meta/recipes-core/eglibc/eglibc-locale.inc +0 -0

index f4cb814..c3fcc6d 100644
@@ -5,6 +5,11 @@ SUMMARY = "Locale data from eglibc"

BPN = "eglibc"
LOCALEBASEPN = "${MLPREFIX}eglibc"

* cyrus-sasl is in meta-networking and isn't in subversion DEPENDS

Signed-off-by Martin Jansa <[email protected]>

---
  meta/recipes-devtools/subversion/subversion_1.6.15.bb | 1 +
  meta/recipes-devtools/subversion/subversion_1.7.10.bb | 1 +
  2 files changed, 2 insertions(+)

meta/recipes-devtools/subversion/subversion_1.6.15.bb +1 -0

index 244d7ea..2a072e8 100644
@@ -24,6 +24,7 @@ EXTRA_OECONF = " \
                  --without-berkeley-db --without-apxs --without-apache \
                  --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
                  --with-apr-util=${STAGING_BINDIR_CROSS} \
+                --without-sasl \
                  ac_cv_path_RUBY=none"
 
  inherit autotools

meta/recipes-devtools/subversion/subversion_1.7.10.bb +0 -0

index 9d189fe..ded1788 100644

* cyrus-sasl is in meta-networking and isn't in subversion DEPENDS

Signed-off-by Martin Jansa <[email protected]>

---
 meta/recipes-devtools/subversion/subversion_1.6.15.bb | 1 +
 meta/recipes-devtools/subversion/subversion_1.7.10.bb | 1 +
 2 files changed, 2 insertions(+)

meta/recipes-devtools/subversion/subversion_1.6.15.bb +1 -0

index 244d7ea..2a072e8 100644
@@ -24,6 +24,7 @@ EXTRA_OECONF = " \
                 --without-berkeley-db --without-apxs --without-apache \
                 --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
                 --with-apr-util=${STAGING_BINDIR_CROSS} \
+                --without-sasl \
                 ac_cv_path_RUBY=none"
 
 inherit autotools

meta/recipes-devtools/subversion/subversion_1.7.10.bb +1 -0

index 9d189fe..ded1788 100644
@@ -22,6 +22,7 @@ EXTRA_OECONF = " \
                 --without-berkeley-db --without-apxs \
                 --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
                 --with-apr-util=${STAGING_BINDIR_CROSS} \
+                --without-sasl \
                 ac_cv_path_RUBY=none"
 
 inherit autotools

Signature · 1 line

1.8.2.1

scripts/test-dependencies.sh line 137

Quoted 13 lines · Martin Jansa

Signed-off-by Martin Jansa <[email protected]>

---
  scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 250 insertions(+)

create mode 100755 scripts/test-dependencies.sh

scripts/test-dependencies.sh new file +7 -0

+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake $targets | tee -a ${OUTPUT1}/complete.log
+}

I've just added "-k" here to bitbake call, more improvements are expected, that's why it's only RFC, but I wanted to share it soon in case someone else also have access to some builder which is idle during weekends..

It would be nice to test and fix oe-core recipes first, but because my target is to run this in dylan I'm running this with more layers included already.

I'm sending patches for master + blacklisting broken recipes in dylan, maybe we should backport them later as missing deps and autodetected deps are relatively safe fixes.

Regards,

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 19 lines · Martin Jansa

scripts/test-dependencies.sh line 137

Quoted 13 lines · Martin Jansa

Signed-off-by Martin Jansa <[email protected]>

---
  scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 250 insertions(+)

create mode 100755 scripts/test-dependencies.sh

scripts/test-dependencies.sh new file +7 -0

+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake $targets | tee -a ${OUTPUT1}/complete.log
+}

I've just added "-k" here to bitbake call, more improvements are expected, that's why it's only RFC, but I wanted to share it soon in case someone else also have access to some builder which is idle during weekends..

Quoted 3 lines

It would be nice to test and fix oe-core recipes first, but because my target is to run this in dylan I'm running this with more layers included already.

Martin,

Such a per-recipe dependency test has been on our to do list! Once your script is in master, we can run it at least once a week.

Quoted 13 lines

I'm sending patches for master + blacklisting broken recipes in dylan, maybe we should backport them later as missing deps and autodetected deps are relatively safe fixes.

Regards,

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 2 lines

# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Signed-off-by Martin Jansa <[email protected]>

---
 scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

scripts/test-dependencies.sh new file +253 -0

new file mode 100755
index 0000000..405c14e
@@ -0,0 +1,253 @@
+#!/bin/sh
+
+# Author: Martin Jansa <[email protected]>
+#
+# Copyright (c) 2013 Martin Jansa <[email protected]>
+
+# Used to detect missing dependencies or automagically
+# enabled dependencies which aren't explicitly enabled
+# or disabled.
+
+# It does 3 builds of <target>
+# 1st to populate sstate-cache directory and sysroot
+# 2nd to rebuild each recipe with every possible
+#     dependency found in sysroot (which stays populated
+#     from 1st build
+# 3rd to rebuild each recipe only with dependencies defined
+#     in DEPENDS
+# 4th (optional) repeat build like 3rd to make sure that
+#     minimal versions of dependencies defined in DEPENDS
+#     is also enough
+
+# Global vars
+tmpdir=
+targets=
+recipes=
+buildhistory=
+buildtype=
+default_targets="world"
+default_buildhistory="buildhistory"
+default_buildtype="1 2 3 c"
+
+usage () {
+  cat << EOF
+Welcome to utility to detect missing or autoenabled dependencies.
+WARNING: this utility will completely remove your tmpdir (make sure
+         you don't have important buildhistory or persistent dir there).
+$0 <OPTION>
+
+Options:
+  -h, --help
+        Display this help and exit.
+
+  --tmpdir=<tmpdir>
+        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
+        Something like /OE/oe-core/tmp-eglibc (no / at the end).
+
+  --targets=<targets>
+        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
+        It will run "bitbake <targets>" to populate sysroots.
+        Default value is "world".
+
+  --recipes=<recipes>
+        File with list of recipes we want to rebuild with minimal and maximal sysroot.
+        Will use the environment variable RECIPES if it is not specified.
+        Default value will use all packages ever recorded in buildhistory directory.
+
+  --buildhistory=<buildhistory>
+        Path to buildhistory directory, it needs to be enabled in your config,
+        because it's used to detect different dependencies and to create list
+        of recipes to rebuild when it's not specified.
+        Will use the environment variable BUILDHISTORY if it is not specified.
+        Default value is "buildhistory"
+
+  --buildtype=<buildtype>
+        There are 4 types of build:
+          1: build to populate sstate-cache directory and sysroot
+          2: build to rebuild each recipe with every possible dep
+          3: build to rebuild each recipe with minimal dependencies
+          4: build to rebuild each recipe again with minimal dependencies
+          c: compare buildhistory directories from build 2 and 3
+        Will use the environment variable BUILDTYPE if it is not specified.
+        Default value is "1 2 3 c", order is important, type 4 is optional.
+EOF
+}
+
+# Print error information and exit.
+echo_error () {
+  echo "ERROR: $1" >&2
+  exit 1
+}
+
+while [ -n "$1" ]; do
+  case $1 in
+    --tmpdir=*)
+      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
+      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
+      shift
+        ;;
+    --targets=*)
+      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --recipes=*)
+      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildhistory=*)
+      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildtype=*)
+      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --help|-h)
+      usage
+      exit 0
+        ;;
+    *)
+      echo "Invalid arguments $*"
+      echo_error "Try '$0 -h' for more information."
+        ;;
+  esac
+done
+
+# tmpdir directory, use environment variable TMPDIR
+# if it was not specified, otherwise, error.
+[ -n "$tmpdir" ] || tmpdir=$TMPDIR
+[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
+[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
+[ -n "$targets" ] || targets=$TARGETS
+[ -n "$targets" ] || targets=$default_targets
+[ -n "$recipes" ] || recipes=$RECIPES
+[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
+[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
+[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
+[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
+[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
+[ -n "$buildtype" ] || buildtype=$BUILDTYPE
+[ -n "$buildtype" ] || buildtype=$default_buildtype
+echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
+
+OUTPUT_BASE=test-dependencies/`date "+%s"`
+
+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake -k $targets | tee -a ${OUTPUT1}/complete.log
+}
+
+build_every_recipe() {
+  if [ "${TYPE}" = "2" ] ; then
+    echo "===== 2nd build to rebuild each recipe with every possible dep ====="
+    OUTPUT_MAX=${OUTPUT_BASE}/${TYPE}_max
+    OUTPUTB=${OUTPUT_MAX}
+  else
+    echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
+    OUTPUT_MIN=${OUTPUT_BASE}/${TYPE}_min
+    OUTPUTB=${OUTPUT_MIN}
+  fi
+  
+  mkdir -p ${OUTPUTB} ${OUTPUTB}/failed ${OUTPUTB}/ok
+  echo "Logs will be stored in ${OUTPUTB} directory"
+  if [ -z "$recipes" ]; then
+    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUTB}/recipe.list
+    recipes=${OUTPUTB}/recipe.list
+  fi
+  if [ "${TYPE}" != "2" ] ; then
+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+  fi
+  i=1
+  count=`cat $recipes | wc -l`
+  for recipe in `cat $recipes`; do
+    echo "Building recipe: ${recipe} ($i/$count)"
+    bitbake -c cleansstate ${recipe} > ${OUTPUTB}/log.${recipe} 2>&1;
+    bitbake ${recipe} >> ${OUTPUTB}/log.${recipe} 2>&1;
+    grep "ERROR: Task.*failed" ${OUTPUTB}/log.${recipe} && mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/failed/${recipe} || mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/ok/${recipe}
+    if [ "${TYPE}" != "2" ] ; then
+      rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+    fi
+    i=`expr $i + 1`
+  done
+  echo "Copying buildhistory/packages to ${OUTPUTB}"
+  cp -ra $buildhistory/packages ${OUTPUTB}
+  # This will be usefull to see which library is pulling new dependency
+  echo "Copying do_package logs to ${OUTPUTB}/do_package/"
+  mkdir ${OUTPUTB}/do_package
+  find $tmpdir/work/ -name log.do_package | while read f; do
+    # pn is 3 levels back, but we don't know if there is just one log per pn (only one arch and version)
+    # dest=`echo $f | sed 's#^.*/\([^/]*\)/\([^/]*\)/\([^/]*\)/log.do_package#\1#g'`
+    dest=`echo $f | sed "s#$tmpdir/work/##g; s#/#_#g"`
+    cp $f ${OUTPUTB}/do_package/$dest
+  done
+  grep "ERROR: Task.*failed" ${OUTPUTB}/failed/*
+}
+
+compare_deps() {
+  # you can run just compare task with command like this
+  # OUTPUT_MAX=test-dependencies/1373140172/2_max \
+  # OUTPUT_MIN=test-dependencies/1373140172/3_min \
+  # OUTPUTC=test-dependencies/1373140172 \
+  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
+  echo "===== Compare dependencies recorded in \"${OUTPUT_MAX}\" and \"${OUTPUT_MIN}\" ====="
+  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
+  mkdir -p ${OUTPUTC}
+  OUTPUT_FILE=${OUTPUTC}/dependency-changes
+  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
+  echo > ${OUTPUT_FILE}
+
+  [ -d ${OUTPUT_MAX} ] || echo_error "Directory with output from build 2 \"${OUTPUT_MAX}\" does not exist"
+  [ -d ${OUTPUT_MIN} ] || echo_error "Directory with output from build 3 \"${OUTPUT_MIN}\" does not exist"
+  [ -d ${OUTPUT_MAX}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT_MAX}/packages/\" does not exist"
+  [ -d ${OUTPUT_MIN}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT_MIN}/packages/\" does not exist"
+  i=0
+  find ${OUTPUT_MAX}/packages/ -name latest | sed "s#${OUTPUT_MAX}/##g" | while read pkg; do
+    max_pkg=${OUTPUT_MAX}/${pkg}
+    min_pkg=${OUTPUT_MIN}/${pkg}
+    if [ ! -f "${min_pkg}" ] ; then
+      echo "ERROR: ${min_pkg} doesn't exist" | tee -a ${OUTPUT_FILE}
+      continue
+    fi
+    # strip version information in parenthesis
+    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    if [ "$i" = 100 ] ; then
+      echo -n "." # cheap progressbar
+      i=0
+    fi
+    if [ "${max_deps}" = "${min_deps}" ] ; then
+      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
+      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
+    else
+      missing_deps=
+      for dep in ${max_deps}; do
+        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
+      done
+      if [ -n "${missing_deps}" ] ; then
+        echo # to get rid of dots on last line
+        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
+      fi
+    fi
+    i=`expr $i + 1`
+  done
+  echo # to get rid of dots on last line
+  echo "Found differences: "
+  grep "^WARN: " ${OUTPUT_FILE}
+  echo "Found errors: "
+  grep "^ERROR: " ${OUTPUT_FILE}
+}
+
+for TYPE in $buildtype; do
+  case ${TYPE} in
+    1) build_all;;
+    2) build_every_recipe;;
+    3) build_every_recipe;;
+    4) build_every_recipe;;
+    c) compare_deps;;
+    *) echo_error "Invalid buildtype \"$TYPE\""
+  esac
+done

Signature · 1 line

1.8.2.1

Build with last version of this script is still running (after 37 hours) Building recipe: gnome-desktop (485/1848)
but I've tested it in smaller builds and I'm happy with current functionality, I'll send found issues in tomorrow or day after that when the build is complete.

Quoted 5 lines · Martin Jansa

Build with last version of this script is still running (after 37 hours) Building recipe: gnome-desktop (485/1848)
but I've tested it in smaller builds and I'm happy with current functionality, I'll send found issues in tomorrow or day after that when the build is complete.

As this has been on my todo list for a while now, come to ELC-E and I'll get you a beer. :)

Anyone volunteering to run this on oe-core master this weekend?

I added a buildserver that also exports its "sstate-cache" directory, so that other build machines can grab their stuff from it. This works fine, but I have one problem. Some packages are meant to be dependent on the system that built it. I want to enforce that each build machine creates its own package, and does not grab it from the sstate-cache of the central server.

For example,

$ cat recipes-core/meta/distro-feed-configs.bbappend
PRINC="1"
DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
DISTRO_FEED_NAME ?= "feed"
DISTRO_FEED_PREFIX = "topic"
DISTRO_FEED_URI =
"http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"

The purpose being that the host name of the machien that built the image ends up in the opkg config files. This works just fine.
Now that we have a central server, all images on all build hosts grab the package from the sstate-cache server, resulting in the feed pointing to the central server instead of the private one, which is not what I wanted to happen.

After reading the documentation, I added the following line to the recipe:

PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"

This did not have the desired effect. The package is still retrieved from the cache, and not rebuilt locally. Even if I clean and force a rebuild of the distro-feed-configs package, the package that ends up in the image is still the one from the central server.

What am I missing here?

Hi Mike,

Quoted 27 lines · On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:

I added a buildserver that also exports its "sstate-cache" directory, so that other build machines can grab their stuff from it. This works fine, but I have one problem. Some packages are meant to be dependent on the system that built it. I want to enforce that each build machine creates its own package, and does not grab it from the sstate-cache of the central server.

For example,

$ cat recipes-core/meta/distro-feed-configs.bbappend
PRINC="1"
DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
DISTRO_FEED_NAME ?= "feed"
DISTRO_FEED_PREFIX = "topic"
DISTRO_FEED_URI =
"http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"

The purpose being that the host name of the machien that built the image ends up in the opkg config files. This works just fine.
Now that we have a central server, all images on all build hosts grab the package from the sstate-cache server, resulting in the feed pointing to the central server instead of the private one, which is not what I wanted to happen.

After reading the documentation, I added the following line to the recipe:

Quoted 8 lines

PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"

This did not have the desired effect. The package is still retrieved from the cache, and not rebuilt locally. Even if I clean and force a rebuild of the distro-feed-configs package, the package that ends up in the image is still the one from the central server.

What am I missing here?

I think distro-feed-configs (from meta-oe, not OE-Core) is for package installation on the target, and has nothing to do with shared state.

I'm not sure if it's entirely appropriate, but you could take a similar approach to the one we use for preventing native sstate packages from mixing across different distributions (mostly to sidestep host glibc version dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you don't want to share and then the packages will go into a subdirectory named with that value; you could then delete these automatically from the server.

Quoted 3 lines · Martin Jansa

Signed-off-by Martin Jansa <[email protected]>

---
 scripts/test-dependencies.sh | 253

+++++++++++++++++++++++++++++++++++++++++++

Review

Quoted 5 lines

 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

(quoted diff) +2 -0

+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata

$tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null

I'll send v3 which doesn't remove $tmpdir/cache between the builds, it takes too long already without reparsing all recipes every time.

Originally I was removing it as work around for
http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
but that's not needed for master and people probably won't be changing PNBLACKLIST while this script is running.

Quoted 13 lines · Martin Jansa

Quoted 3 lines · Martin Jansa

Signed-off-by Martin Jansa <[email protected]>

---
 scripts/test-dependencies.sh | 253

+++++++++++++++++++++++++++++++++++++++++++

Review

Quoted 5 lines

 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

(quoted diff) +2 -0

+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata

$tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null

Quoted 8 lines

I'll send v3 which doesn't remove $tmpdir/cache between the builds, it takes too long already without reparsing all recipes every time.

Originally I was removing it as work around for
http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
but that's not needed for master and people probably won't be changing PNBLACKLIST while this script is running.

This is quite an interesting script. It looks quite similar to my greedy-deps-tests[1] script, though you've taken it a step further in adding builds #3 and #4.

1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts

Quoted 13 lines · Chris Larson

Quoted 10 lines · On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <[email protected] wrote:

Quoted 3 lines · Martin Jansa

Signed-off-by Martin Jansa <[email protected]>

---
 scripts/test-dependencies.sh | 253

+++++++++++++++++++++++++++++++++++++++++++

Quoted 5 lines

 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

(quoted diff) +2 -0

+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata

Quoted 8 lines

Quoted 8 lines

$tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null

I'll send v3 which doesn't remove $tmpdir/cache between the builds, it takes too long already without reparsing all recipes every time.

Originally I was removing it as work around for

http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html

Quoted 4 lines

Quoted 3 lines

but that's not needed for master and people probably won't be changing PNBLACKLIST while this script is running.

Quoted 5 lines

This is quite an interesting script. It looks quite similar to my greedy-deps-tests[1] script, though you've taken it a step further in adding builds #3 and #4.

1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts

pity I didn't know about your script before writing mine, it looks more elegant in python

Do you plan to integrate your script into oe-core? I'm fine with keeping my shell script locally until you do. Your python script would better fit in bitbake/oe environment and reloc-tests also looks useful.

Quoted 42 lines · Paul Eggleton

Hi Mike,

Quoted 27 lines · On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:

I added a buildserver that also exports its "sstate-cache" directory, so that other build machines can grab their stuff from it. This works fine, but I have one problem. Some packages are meant to be dependent on the system that built it. I want to enforce that each build machine creates its own package, and does not grab it from the sstate-cache of the central server.

For example,

$ cat recipes-core/meta/distro-feed-configs.bbappend
PRINC="1"
DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
DISTRO_FEED_NAME ?= "feed"
DISTRO_FEED_PREFIX = "topic"
DISTRO_FEED_URI =
"http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"

The purpose being that the host name of the machien that built the image ends up in the opkg config files. This works just fine.
Now that we have a central server, all images on all build hosts grab the package from the sstate-cache server, resulting in the feed pointing to the central server instead of the private one, which is not what I wanted to happen.

After reading the documentation, I added the following line to the recipe:

Quoted 8 lines

PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"

This did not have the desired effect. The package is still retrieved from the cache, and not rebuilt locally. Even if I clean and force a rebuild of the distro-feed-configs package, the package that ends up in the image is still the one from the central server.

What am I missing here?

I think distro-feed-configs (from meta-oe, not OE-Core) is for package installation on the target, and has nothing to do with shared state.

Quoted 6 lines

I'm not sure if it's entirely appropriate, but you could take a similar approach to the one we use for preventing native sstate packages from mixing across different distributions (mostly to sidestep host glibc version dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you don't want to share and then the packages will go into a subdirectory named with that value; you could then delete these automatically from the server.

I would try to add DISTRO_HOST_NAME[vardepvalue] = "${DISTRO_HOST_NAME}"

that way each builder should create own sstate archive.