fixes for multilib
cover fixes for multilib
Hi RP,
The multilib is still broken without these patches, I'd like to add oeqa test cases for them after it works.
// Robert
The following changes since commit 073d8d001033471d7fe44f52212c72a6c3541313:
dbus-test-ptest: improve reproducibility (2018-03-12 15:05:58 -0700)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/mlib
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/mlib
Robert Yang (8):
populate_sdk_ext.bbclass: fix for multilib staging.bbclass: staging_populate_sysroot_dir(): fix for multilib staging.bbclass: print searched manifest when not found package_manager.py: reverse archs correctly multilib.bbclass: extend allarch recipes toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
image
populate_sdk_ext.bbclass: remove the try...finally staging.bbclass: extend_recipe_sysroot: fix multilib manifest
meta/classes/allarch.bbclass | 3 - meta/classes/multilib.bbclass | 4 - meta/classes/multilib_global.bbclass | 4 +- meta/classes/package.bbclass | 9 --- meta/classes/populate_sdk_ext.bbclass | 142 +++++++++++++++++++-------------- meta/classes/staging.bbclass | 25 ++++++ meta/classes/toolchain-scripts.bbclass | 6 ++ meta/lib/oe/package_manager.py | 57 +++++++++++-- meta/lib/oe/sstatesig.py | 3 + 9 files changed, 168 insertions(+), 85 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/8 populate_sdk_ext.bbclass: fix for multilib
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake <image> -cpopulate_sdk_ext $ bitbake <image> -ctestsdkext [snip] Standard Output: /bin/sh: 1: i686-wrsmllib32-linux-gcc: not found [snip]
It was failed because no lib32 toolchain or lib installed.
This patch fixes:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.
--- meta/classes/populate_sdk_ext.bbclass | 72 +++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 25 deletions(-)
meta/classes/populate_sdk_ext.bbclass +47 -25
@@ -38,7 +38,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \ SDK_INHERIT_BLACKLIST ?= "buildhistory icecc" SDK_UPDATE_URL ?= "" -SDK_TARGETS ?= "${PN}" +SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}" def get_sdk_install_targets(d, images_only=False): sdk_install_targets = ''
@@ -55,9 +55,11 @@ def get_sdk_install_targets(d, images_only=False): if not images_only: if d.getVar('SDK_INCLUDE_PKGDATA') == '1': - sdk_install_targets += ' meta-world-pkgdata:do_allpackagedata' + for pn in multilib_pkg_extend(d, 'meta-world-pkgdata').split(): + sdk_install_targets += ' %s:do_allpackagedata' % pn if d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1': - sdk_install_targets += ' meta-extsdk-toolchain:do_populate_sysroot' + for pn in multilib_pkg_extend(d, 'meta-extsdk-toolchain').split(): + sdk_install_targets += ' %s:do_populate_sysroot' % pn return sdk_install_targets
@@ -601,40 +603,55 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" sdk_ext_postinst() { printf "\nExtracting buildtools...\n" cd $target_sdk_dir - env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}" - printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} > buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; } + env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`" + ./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log + if [ $? -ne 0 ]; then + echo 'ERROR: buildtools installation failed:' + cat buildtools.log + for e in $env_setup_scripts; do + echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e + done + exit 1 + fi # Delete the buildtools tar file since it won't be used again rm -f ./${SDK_BUILDTOOLS_INSTALLER} # We don't need the log either since it succeeded rm -f buildtools.log - # Make sure when the user sets up the environment, they also get - # the buildtools-tarball tools in their path. - echo ". $target_sdk_dir/buildtools/environment-setup*" >> $env_setup_script - - # Allow bitbake environment setup to be ran as part of this sdk. - echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script - # Work around runqemu not knowing how to get this information within the eSDK - echo "export DEPLOY_DIR_IMAGE=$target_sdk_dir/tmp/${@os.path.relpath(d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('TMPDIR'))}" >> $env_setup_script + for e in $env_setup_scripts; do + # Make sure when the user sets up the environment, they also get + # the buildtools-tarball tools in their path. + echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e - # A bit of another hack, but we need this in the path only for devtool - # so put it at the end of $PATH. - echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $env_setup_script + # Allow bitbake environment setup to be ran as part of this sdk. + echo "export OE_SKIP_SDK_CHECK=1" >> $e - echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script + # Work around runqemu not knowing how to get this information within the eSDK + echo "export DEPLOY_DIR_IMAGE=$target_sdk_dir/tmp/${@os.path.relpath(d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('TMPDIR'))}" >> $e - # Warn if trying to use external bitbake and the ext SDK together - echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $env_setup_script + # A bit of another hack, but we need this in the path only for devtool + # so put it at the end of $PATH. + echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e + echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $e + # Warn if trying to use external bitbake and the ext SDK together + echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $e + done if [ "$prepare_buildsystem" != "no" ]; then - printf "Preparing build system...\n" + echo "Preparing build system..." # dash which is /bin/sh on Ubuntu will not preserve the # current working directory when first ran, nor will it set $1 when # sourcing a script. That is why this has to look so ugly. LOGFILE="$target_sdk_dir/preparing_build_system.log" - sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; } - rm $target_sdk_dir/ext-sdk-prepare.py + sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" + if [ $? -ne 0 ]; then + for e in $env_setup_scripts; do + echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e + done + exit 1 + fi + rm -f $target_sdk_dir/ext-sdk-prepare.py fi echo done }
@@ -663,12 +680,17 @@ fakeroot python do_populate_sdk_ext() { def get_ext_sdk_depends(d): # Note: the deps varflag is a list not a string, so we need to specify expand=False deps = d.getVarFlag('do_image_complete', 'deps', False) - pn = d.getVar('PN') - deplist = ['%s:%s' % (pn, dep) for dep in deps] + deplist = [] + for pn in multilib_pkg_extend(d, d.getVar('BPN')).split(): + deplist += ['%s:%s' % (pn, dep) for dep in deps] tasklist = bb.build.tasksbetween('do_image_complete', 'do_build', d) tasklist.append('do_rootfs') for task in tasklist: deplist.extend((d.getVarFlag(task, 'depends') or '').split()) + + for pn in multilib_pkg_extend(d, 'meta-extsdk-toolchain').split(): + deplist += ['%s:do_populate_sysroot' % pn] + return ' '.join(deplist) python do_sdk_depends() {
@@ -682,7 +704,7 @@ python do_sdk_depends() { addtask sdk_depends do_sdk_depends[dirs] = "${WORKDIR}" -do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)} meta-extsdk-toolchain:do_populate_sysroot" +do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}" do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk', 'recrdeptask', False)}" do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}" do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
Signature · 1 line
2.7.4
PATCH 2/8 staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
In reply to Robert YangFixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-sato lib32-core-image-sato $ bitbake build-sysroots
All lib32 manifests which had been built should be installed, but only a few such as qemux86_64-lib32-base-files were installed (it was installed because MACHINE_ARCH was still qemux86-64 when multilib), but others such as x86-lib32-zlib were not installed, this was incorrect. For multilib builds, fix-up overrides to prepend :virtclass-multilib- and then again append additional multilib arches from the PACKAGE_EXTRA_ARCHS list if needed can fix the problem.
--- meta/classes/staging.bbclass | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
meta/classes/staging.bbclass +15 -0
@@ -190,8 +190,23 @@ def staging_populate_sysroot_dir(targetsysroot, nativesysroot, native, d): pkgarchs = ['${MACHINE_ARCH}'] pkgarchs = pkgarchs + list(reversed(d.getVar("PACKAGE_EXTRA_ARCHS").split())) pkgarchs.append('allarch') + + # Handle multilib archs + variants = d.getVar("MULTILIB_VARIANTS", True) or "" + for variant in variants.split(): + localdata = bb.data.createCopy(d) + overrides = localdata.getVar("OVERRIDES", False) + ":virtclass-multilib-" + variant + localdata.setVar("OVERRIDES", overrides) + bb.data.update_data(localdata) + pkgarchs_ml = localdata.getVar('PACKAGE_EXTRA_ARCHS').split() + for arch in pkgarchs_ml: + if arch not in pkgarchs: + pkgarchs.append(arch) + targetdir = targetsysroot + bb.debug(1, 'pkgarchs: %s' % pkgarchs) + bb.utils.mkdirhier(targetdir) for pkgarch in pkgarchs: for manifest in glob.glob(d.expand("${SSTATE_MANIFESTS}/manifest-%s-*.populate_sysroot" % pkgarch)):
Signature · 1 line
2.7.4
PATCH 3/8 staging.bbclass: print searched manifest when not found
In reply to Robert YangThe old warning was:
WARNING: lib32-wrlinux-image-glibc-small-1.0-r1 do_prepare_recipe_sysroot: Manifest /path/to/tmp/sstate-control/manifest-allarch-lib32-qemuwrapper-cross.populate_sysroot not found?
The message wasn't clear enough, it searched a few manifests, but only reported the last one, which confused user.
Now the warning is:
WARNING: lib32-wrlinux-image-glibc-small-1.0-r1 do_rootfs: Manifest for lib32-qemuwrapper-cross not found, searched manifests:
/path/to/tmp/sstate-control/manifest-qemux86_64-lib32-qemuwrapper-cross.populate_sysroot /path/to/tmp/sstate-control/manifest-core2-64-lib32-qemuwrapper-cross.populate_sysroot /path/to/tmp/sstate-control/manifest-x86_64-lib32-qemuwrapper-cross.populate_sysroot /path/to/tmp/sstate-control/manifest-allarch-lib32-qemuwrapper-cross.populate_sysroot
Whick makes debug easier.
--- meta/lib/oe/sstatesig.py | 3 +++ 1 file changed, 3 insertions(+)
meta/lib/oe/sstatesig.py +3 -0
@@ -394,11 +394,14 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): pkgarchs.append('allarch') pkgarchs.append('${SDK_ARCH}_${SDK_ARCH}-${SDKPKGSUFFIX}') + searched_manifest = [] for pkgarch in pkgarchs: manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname)) if os.path.exists(manifest): return manifest, d2 + searched_manifest.append(manifest) bb.warn("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) + bb.warn("Searched manifests:\n%s" % '\n'.join(searched_manifest)) return None, d2
Signature · 1 line
2.7.4
PATCH 4/8 package_manager.py: reverse archs correctly
In reply to Robert YangIt had reversed all the archs, which mixed multilib and common archs, e.g.:
"all any noarch x86_64 core2-64 qemux86_64 x86 i586 core2-32" After reversed:
"core2-32 i586 x86 qemux86_64 core2-64 x86_64 noarch any all" The core2-32 has a higher priority than core2-64 after reversed which is incorrect. Don't mix with mulitlib when reverse can fix the problem, and let multilib archs have a higher priority for multilib image.
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"IMAGE_INSTALL_append = " ca-certificates"
$ bitbake wrlinux-image-glibc-small Check rootfs.manifest, no core_32 packages is installed, this is correct, but after we build lib32-bash, it will be incorrect: $ bitbake lib32-bash $ bitbake wrlinux-image-glibc-small Check rootfs.manifest, a few lib32 packages are installed, such as lib32-bash, this is incorrect. It was because ca-certificates is allarch, and requires /bin/sh, which is provided by both bash and lib32-bash, and lib32-bash has a higher priority than bash, so it would be installed.
[YOCTO #12288]
--- meta/lib/oe/package_manager.py | 57 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 5 deletions(-)
meta/lib/oe/package_manager.py +52 -5
@@ -572,10 +572,10 @@ class RpmPM(PackageManager): self.target_rootfs = target_rootfs self.target_vendor = target_vendor self.task_name = task_name - if arch_var == None: - self.archs = self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS').replace("-","_") - else: - self.archs = self.d.getVar(arch_var).replace("-","_") + + self.archs = self._determine_archs(arch_var) + bb.note("The archs used by package manager: %s" % self.archs) + if task_name == "host": self.primary_arch = self.d.getVar('SDK_ARCH') else:
@@ -593,9 +593,56 @@ class RpmPM(PackageManager): if not os.path.exists(self.d.expand('${T}/saved')): bb.utils.mkdirhier(self.d.expand('${T}/saved')) + def _determine_archs(self, arch_var): + if arch_var: + archs = self.d.getVar(arch_var).replace("-","_").split() + else: + archs = self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS').replace("-","_").split() + # Reverse archs to ensure the -best- match is listed firstly, but need + # make sure not mixed with multilib archs, and let multilib archs have + # higher a priority for multilib image: + # For non-multilib image: (e.g., core-image-minimal) + # machine_arch:package_archs:multilib_archs:allarch_archs + # For multilib image: (e.g., lib32-core-image-minimal) + # machine_arch:multilib_archs:package_archs:allarch_archs + package_archs_var = self.d.getVar('PACKAGE_ARCHS').replace("-","_").split() + machine_arch_var = [self.d.getVar('MACHINE_ARCH').replace("-","_")] + package_archs = [] + allarch_archs = [] + ml_archs = [] + machine_arch = [] + for arch in archs: + if arch in machine_arch_var: + bb.note("Found MACHINE_ARCH: %s" % arch) + machine_arch.append(arch) + elif arch in package_archs_var: + bb.note("Found PACKAGE_ARCH: %s" % arch) + package_archs.append(arch) + elif arch in 'all any noarch'.split(): + bb.note("Found allarch: %s" % arch) + allarch_archs.append(arch) + else: + bb.note("Found multilib arch: %s" % arch) + ml_archs.append(arch) + package_archs.reverse() + if machine_arch: + archs = machine_arch + mlprefix = self.d.getVar('MLPREFIX') + if ml_archs: + ml_archs.reverse() + if mlprefix: + archs += ml_archs + package_archs + else: + archs += package_archs + ml_archs + else: + archs += package_archs + + archs += allarch_archs + return ' '.join(archs) + def _configure_dnf(self): # libsolv handles 'noarch' internally, we don't need to specify it explicitly - archs = [i for i in reversed(self.archs.split()) if i not in ["any", "all", "noarch"]] + archs = [i for i in self.archs.split() if i not in ["any", "all", "noarch"]] # This prevents accidental matching against libsolv's built-in policies if len(archs) <= 1: archs = archs + ["bogusarch"]
Signature · 1 line
2.7.4
PATCH 5/8 multilib.bbclass: extend allarch recipes
In reply to Robert YangThis can fix do_rootfs and depends chaos when multilib, for example ca-certificates is an allarch recipe:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32" IMAGE_INSTALL_append = " ca-certificates"
$ bitbake ca-certificates $ rpm -qpR tmp-glibc/deploy/rpm/noarch/ca-certificates-20170717-r0.noarch.rpm [snip] openssl [snip]
It requires openssl, but lib32-openssl doesn't proivide it:
$ rpm -qp --provides tmp-glibc/deploy/rpm/core2_32/lib32-openssl-1.0.2l-r0.core2_32.rpm lib32-openssl = 1.0.2l-r0
So openssl-1.0.2l-r0.core2_64.rpm would be installed when install ca-certificates, e.g.:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32" IMAGE_INSTALL_append = " ca-certificates"
$ bitbake lib32-core-image-minimal The openssl-1.0.2l-r0.core2_64.rpm and other 64 bit packages are installed, this is incorrect.
We have two solutions on this:
1) Let lib32-openssl provide openssl, but this would cause more problems when building since more than one providers for openssl.
Or
2) Extend allarch recipes, there would be ca-certificates-20170717-r0.noarch.rpm and lib32-ca-certificates-20170717-r0.noarch.rpm, then ca-certificates requires openssl, and lib32-ca-certificates requires lib32-openssl, the packages will be installed correctly, this is the best solution that I can find.
The problem has existed for years for deb and ipk (rpm did work since it didn't use lib32-BPN in rpm filename before changed to dnf, for example, there was no lib32-openssl.core2_32.rpm, but openssl.core2_32.rpm), this patch can fix the problem for rpm, ipk and deb.
[YOCTO #12288]
--- meta/classes/allarch.bbclass | 3 --- meta/classes/multilib.bbclass | 4 ---- meta/classes/multilib_global.bbclass | 4 +--- meta/classes/package.bbclass | 9 --------- 4 files changed, 1 insertion(+), 19 deletions(-)
meta/classes/allarch.bbclass +0 -3
@@ -42,9 +42,6 @@ python () { d.setVar("INHIBIT_PACKAGE_DEBUG_SPLIT", "1") d.setVar("INHIBIT_PACKAGE_STRIP", "1") - # These multilib values shouldn't change allarch packages so exclude them - d.appendVarFlag("emit_pkgdata", "vardepsexclude", " MULTILIB_VARIANTS") - d.appendVarFlag("write_specfile", "vardepsexclude", " MULTILIBS") elif bb.data.inherits_class('packagegroup', d) and not bb.data.inherits_class('nativesdk', d): bb.error("Please ensure recipe %s sets PACKAGE_ARCH before inherit packagegroup" % d.getVar("FILE")) }
meta/classes/multilib.bbclass +0 -4
@@ -46,10 +46,6 @@ python multilib_virtclass_handler () { if bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data): raise bb.parse.SkipRecipe("We can't extend nativesdk recipes") - if bb.data.inherits_class('allarch', e.data) and not bb.data.inherits_class('packagegroup', e.data): - raise bb.parse.SkipRecipe("Don't extend allarch recipes which are not packagegroups") - - # Expand this since this won't work correctly once we set a multilib into place e.data.setVar("ALL_MULTILIB_PACKAGE_ARCHS", e.data.getVar("ALL_MULTILIB_PACKAGE_ARCHS"))
meta/classes/multilib_global.bbclass +1 -3
@@ -165,9 +165,7 @@ python multilib_virtclass_handler_global () { return if bb.data.inherits_class('kernel', e.data) or \ - bb.data.inherits_class('module-base', e.data) or \ - (bb.data.inherits_class('allarch', e.data) and\ - not bb.data.inherits_class('packagegroup', e.data)): + bb.data.inherits_class('module-base', e.data): variants = (e.data.getVar("MULTILIB_VARIANTS") or "").split() import oe.classextend
meta/classes/package.bbclass +0 -9
@@ -475,9 +475,6 @@ def get_package_mapping (pkg, basepkg, d): key = "PKG_%s" % pkg if key in data: - # Have to avoid undoing the write_extra_pkgs(global_variants...) - if bb.data.inherits_class('allarch', d) and data[key] == basepkg: - return pkg return data[key] return pkg
@@ -1391,9 +1388,6 @@ fi if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): write_extra_pkgs(variants, pn, packages, pkgdatadir) - if (bb.data.inherits_class('allarch', d) and not bb.data.inherits_class('packagegroup', d)): - write_extra_pkgs(global_variants, pn, packages, pkgdatadir) - workdir = d.getVar('WORKDIR') for pkg in packages.split():
@@ -1479,9 +1473,6 @@ fi if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): write_extra_runtime_pkgs(variants, packages, pkgdatadir) - if bb.data.inherits_class('allarch', d) and not bb.data.inherits_class('packagegroup', d): - write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) - bb.utils.unlockfile(lf) } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
Signature · 1 line
2.7.4
PATCH 6/8 toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for image
In reply to Robert YangInstall all MULTILIB_VARIANTS to sysroot and pack them only makes sense to image recipe, so limit it to image recipe only, and it can fix:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"
$ bitbake lib32-meta-ide-support WARNING: lib32-meta-ide-support-1.0-r3 do_configure: Manifest for ncurses not found, searched manifests: /workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-qemux86_64-ncurses.populate_sysroot /workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-core2-32-ncurses.populate_sysroot /workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-i586-ncurses.populate_sysroot /workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-x86-ncurses.populate_sysroot /workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-allarch-ncurses.populate_sysroot
This is because core2-64 is not in lib32-meta-ide-support's PACKAGE_ARCHS which doesn't like image recipe (lib32-image recipe's DEFAULTTUNE is not overridded)
[YOCTO #12298]
--- meta/classes/toolchain-scripts.bbclass | 6 ++++++ 1 file changed, 6 insertions(+)
meta/classes/toolchain-scripts.bbclass +6 -0
@@ -166,6 +166,12 @@ toolchain_create_sdk_siteconfig[vardepsexclude] = "TOOLCHAIN_CONFIGSITE_SYSROOTC python __anonymous () { import oe.classextend + + # Install all MULTILIB_VARIANTS to sysroot and pack them only makes sense to + # image recipe. + if not bb.data.inherits_class('image', d): + return + deps = "" for dep in (d.getVar('TOOLCHAIN_NEED_CONFIGSITE_CACHE') or "").split(): deps += " %s:do_populate_sysroot" % dep
Signature · 1 line
2.7.4
PATCH 7/8 populate_sdk_ext.bbclass: remove the try...finally
In reply to Robert YangThe "sdkbasepath + '/conf/local.conf.bak" doesn't exist when "oe.copy_buildsystem.check_sstate_task_list()" fails, then os.replace() would raise FileNotFoundError, which overcomes the real error. Keep the error status makes debug easier, so remove the try..finally.
--- meta/classes/populate_sdk_ext.bbclass | 70 +++++++++++++++++------------------ 1 file changed, 34 insertions(+), 36 deletions(-)
meta/classes/populate_sdk_ext.bbclass +34 -36
@@ -142,43 +142,41 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath): # Create a temporary build directory that we can pass to the env setup script shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak') + with open(sdkbasepath + '/conf/local.conf', 'a') as f: + # Force the use of sstate from the build system + f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR')) + f.write('SSTATE_MIRRORS_forcevariable = "file://universal/(.*) file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n') + # Ensure TMPDIR is the default so that clean_esdk_builddir() can delete it + f.write('TMPDIR_forcevariable = "${TOPDIR}/tmp"\n') + f.write('TCLIBCAPPEND_forcevariable = ""\n') + # Drop uninative if the build isn't using it (or else NATIVELSBSTRING will + # be different and we won't be able to find our native sstate) + if not bb.data.inherits_class('uninative', d): + f.write('INHERIT_remove = "uninative"\n') + + # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake + # will not allow in its COREBASE path, so we need to rename the directory temporarily + temp_sdkbasepath = d.getVar('SDK_OUTPUT') + '/tmp-renamed-sdk' + # Delete any existing temp dir try: - with open(sdkbasepath + '/conf/local.conf', 'a') as f: - # Force the use of sstate from the build system - f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR')) - f.write('SSTATE_MIRRORS_forcevariable = "file://universal/(.*) file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n') - # Ensure TMPDIR is the default so that clean_esdk_builddir() can delete it - f.write('TMPDIR_forcevariable = "${TOPDIR}/tmp"\n') - f.write('TCLIBCAPPEND_forcevariable = ""\n') - # Drop uninative if the build isn't using it (or else NATIVELSBSTRING will - # be different and we won't be able to find our native sstate) - if not bb.data.inherits_class('uninative', d): - f.write('INHERIT_remove = "uninative"\n') - - # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake - # will not allow in its COREBASE path, so we need to rename the directory temporarily - temp_sdkbasepath = d.getVar('SDK_OUTPUT') + '/tmp-renamed-sdk' - # Delete any existing temp dir - try: - shutil.rmtree(temp_sdkbasepath) - except FileNotFoundError: - pass - os.rename(sdkbasepath, temp_sdkbasepath) - cmdprefix = '. %s .; ' % conf_initpath - logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt' - try: - oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile) - except bb.process.ExecutionError as e: - msg = 'Failed to generate filtered task list for extensible SDK:\n%s' % e.stdout.rstrip() - if 'attempted to execute unexpectedly and should have been setscened' in e.stdout: - msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n' - bb.fatal(msg) - os.rename(temp_sdkbasepath, sdkbasepath) - # Clean out residue of running bitbake, which check_sstate_task_list() - # will effectively do - clean_esdk_builddir(d, sdkbasepath) - finally: - os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf') + shutil.rmtree(temp_sdkbasepath) + except FileNotFoundError: + pass + os.rename(sdkbasepath, temp_sdkbasepath) + cmdprefix = '. %s .; ' % conf_initpath + logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt' + try: + oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile) + except bb.process.ExecutionError as e: + msg = 'Failed to generate filtered task list for extensible SDK:\n%s' % e.stdout.rstrip() + if 'attempted to execute unexpectedly and should have been setscened' in e.stdout: + msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n' + bb.fatal(msg) + os.rename(temp_sdkbasepath, sdkbasepath) + # Clean out residue of running bitbake, which check_sstate_task_list() + # will effectively do + clean_esdk_builddir(d, sdkbasepath) + os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf') python copy_buildsystem () { import re
Signature · 1 line
2.7.4
Re: [PATCH 7/8] populate_sdk_ext.bbclass: remove the try...finally
Quoted 6 lines · Robert Yang
The "sdkbasepath + '/conf/local.conf.bak" doesn't exist when "
oe.copy_buildsystem.check_sstate_task_list()" fails, thenos.replace()would
raise FileNotFoundError, which overcomes the real error. Keep the error status
makes debug easier, so remove the try..finally.
I don't think this patch is quite right. If there is a failure we *must* make sure local.conf is restored, that is important. The code should probably keep the try/finally but make the replace conditional on the file existing.
Cheers,
Richard
Re: [PATCH 7/8] populate_sdk_ext.bbclass: remove the try...finally
Quoted 8 lines · Richard Purdie
Quoted 6 lines · Robert Yang
The "sdkbasepath + '/conf/local.conf.bak" doesn't exist when "
oe.copy_buildsystem.check_sstate_task_list()" fails, thenos.replace()would
raise FileNotFoundError, which overcomes the real error. Keep the error status
makes debug easier, so remove the try..finally.
Quoted 4 lines
I don't think this patch is quite right. If there is a failure we *must* make sure local.conf is restored, that is important. The code should probably keep the try/finally but make the replace conditional on the file existing.
Thanks, updated in the repo:
Author: Robert Yang <[email protected]> Date: Wed Mar 14 10:01:51 2018 +0800
populate_sdk_ext.bbclass: only replace local.conf when local.conf.bak exists
The "sdkbasepath + '/conf/local.conf.bak'" doesn't exist when "oe.copy_buildsystem.check_sstate_task_list()" fails since the sdkbasepath had been renamed to temp_sdkbasepath, but can't rename back when oe.copy_buildsystem.check_sstate_task_list() fails, then os.replace() would raise FileNotFoundError, which overcomes the real error. Only replace local.conf when local_conf.bak exists can make the debug easier.
meta/classes/populate_sdk_ext.bbclass +4 -2
@@ -141,9 +141,11 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath): import oe.copy_buildsystem # Create a temporary build directory that we can pass to the env setup script - shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak') + local_conf = sdkbasepath + '/conf/local.conf' + local_conf_bak = local_conf + '.bak' + shutil.copyfile(local_conf, local_conf_bak) try: - with open(sdkbasepath + '/conf/local.conf', 'a') as f: + with open(local_conf, 'a') as f: # Force the use of sstate from the build system f.write('\nSSTATE_DIR_forcevariable = "%s"\n' %
d.getVar('SSTATE_DIR'))
f.write('SSTATE_MIRRORS_forcevariable = "file://universal/(.*)
file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n') @@ -178,7 +180,8 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
# will effectively do
clean_esdk_builddir(d, sdkbasepath)
finally:
- os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
+ if os.path.exists(local_conf_bak):
+ os.replace(local_conf_bak, local_conf) python copy_buildsystem () {
import re
// Robert
Quoted 5 lines
Cheers,
Richard
PATCH 8/8 staging.bbclass: extend_recipe_sysroot: fix multilib manifest
In reply to Robert YangFixed:
MACHINE = "qemux86-64"
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake <image> -cpopulate_sdk_ext [snip] Exception: subprocess.CalledProcessError: Command 'sed -e [snip]' [snip]
Subprocess output:
[snip]
sed: can't read /path/to/work/qemux86_64-wrs-linux/wrlinux-image-glibc-small/1.0-r1/recipe-sysroot/usr/lib/perl/5.24.1/ExtUtils/Liblist/Kid.pm: No such file or directory [snip]
It was failed because "/usr/lib" is in qemux86_64-wrsmllib32-linux/, not in qemux86_64-wrs-linux. The code has considered mutitlib, but seems not completed, the multilib "variant" was not in fixme, so it wasn't handled correctly, this patch fixes the problem.
--- meta/classes/staging.bbclass | 10 ++++++++++ 1 file changed, 10 insertions(+)
meta/classes/staging.bbclass +10 -0
@@ -492,11 +492,21 @@ python extend_recipe_sysroot() { if c.endswith("-native") or "-cross-" in c or "-crosssdk" in c: native = True + multilib = False + if setscenedeps[dep][2].startswith("virtual:multilib"): + variant = setscenedeps[dep][2].split(":")[2] + if variant not in fixme: + fixme[variant] = [] + multilib = True + if manifest: newmanifest = collections.OrderedDict() if native: fm = fixme['native'] targetdir = recipesysrootnative + elif multilib: + fm = fixme[variant] + targetdir = destsysroot else: fm = fixme[''] targetdir = destsysroot
Signature · 1 line
2.7.4