multilib/recipes: Use new RecipePostKeyExpansion event
PATCH v2 1/5 multilib/recipes: Use new RecipePostKeyExpansion event
There are issues with multilib due to the ordering of events where some functions see the remapped multilib dependencies and some do not. A significant problem is that the multilib class needs to make some changes before key expansion and some afterwards but by using existing event handlers, some code sees things in a partially translated state, leading to bugs.
This patch changes things to use a new event handler from bitbake which makes the ordering of the changes explcit.
The challenge in doing this is that it breaks some existing anonymous python and dyanmic assignments. In some cases these used to be translated and no longer are, meaning MLPREFIX has to be added. In some cases these are now translated and the MLPREFIX can be removed.
This change does now make it very clear when MLPREFIX is required and when it is not, its just the migration path which is harder. The patch changes the small number of cases where fixes are needed.
In particular, where a variable like RDEPENDS is conditionally extended (e.g. with an override), MLPREFIX is now required.
This patch also reverts:
base: Revert 'base.bbclass: considering multilib when setting LICENSE_EXCLUSION'
This reverts 6597130256a1609c3e05ec5891aceaf549c37985 as the changes to multilib datastore handling mean its no longer necessary.
--- meta/classes/base.bbclass | 3 +- meta/classes/fontcache.bbclass | 2 +- meta/classes/multilib.bbclass | 28 ++++++++++++--- meta/lib/oe/classextend.py | 35 +++++++++++++++++-- meta/recipes-core/glibc/glibc-package.inc | 2 +- .../packagegroups/packagegroup-base.bb | 8 ++--- meta/recipes-core/psplash/psplash_git.bb | 5 +-- meta/recipes-devtools/perl/perl_5.30.2.bb | 2 +- meta/recipes-devtools/python/python3_3.8.2.bb | 14 ++++---- .../packagegroup-core-full-cmdline.bb | 10 +++--- meta/recipes-graphics/mesa/mesa.inc | 10 +++--- .../alsa/alsa-plugins_1.2.1.bb | 6 ++-- meta/recipes-support/boost/boost.inc | 10 ++++-- 13 files changed, 95 insertions(+), 40 deletions(-)
meta/classes/base.bbclass +1 -2
@@ -570,8 +570,7 @@ python () { if unskipped_pkgs: for pkg in skipped_pkgs: bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg]))) - mlprefix = d.getVar('MLPREFIX') - d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, ' '.join(skipped_pkgs[pkg])) + d.setVar('LICENSE_EXCLUSION-' + pkg, ' '.join(skipped_pkgs[pkg])) for pkg in unskipped_pkgs: bb.debug(1, "Including the package %s" % pkg) else:
meta/classes/fontcache.bbclass +1 -1
@@ -7,7 +7,7 @@ PACKAGE_WRITE_DEPS += "qemu-native" inherit qemu FONT_PACKAGES ??= "${PN}" -FONT_EXTRA_RDEPENDS ?= "fontconfig-utils" +FONT_EXTRA_RDEPENDS ?= "${MLPREFIX}fontconfig-utils" FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig" FONTCONFIG_CACHE_PARAMS ?= "-v" # You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,
meta/classes/multilib.bbclass +23 -5
@@ -91,13 +91,12 @@ addhandler multilib_virtclass_handler multilib_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" python __anonymous () { - variant = d.getVar("BBEXTENDVARIANT") - - import oe.classextend + if bb.data.inherits_class('image', d): + variant = d.getVar("BBEXTENDVARIANT") + import oe.classextend - clsextend = oe.classextend.ClassExtender(variant, d) + clsextend = oe.classextend.ClassExtender(variant, d) - if bb.data.inherits_class('image', d): clsextend.map_depends_variable("PACKAGE_INSTALL") clsextend.map_depends_variable("LINGUAS_INSTALL") clsextend.map_depends_variable("RDEPENDS")
@@ -109,6 +108,22 @@ python __anonymous () { bb.build.deltask('do_populate_sdk', d) bb.build.deltask('do_populate_sdk_ext', d) return +} + +python multilib_virtclass_handler_postkeyexp () { + cls = d.getVar("BBEXTENDCURR") + variant = d.getVar("BBEXTENDVARIANT") + if cls != "multilib" or not variant: + return + + variant = d.getVar("BBEXTENDVARIANT") + + import oe.classextend + + clsextend = oe.classextend.ClassExtender(variant, d) + + if bb.data.inherits_class('image', d): + return clsextend.map_depends_variable("DEPENDS") clsextend.map_variable("PROVIDES")
@@ -129,6 +144,9 @@ python __anonymous () { reset_alternative_priority(d) } +addhandler multilib_virtclass_handler_postkeyexp +multilib_virtclass_handler_postkeyexp[eventmask] = "bb.event.RecipePostKeyExpansion" + def reset_alternative_priority(d): if not bb.data.inherits_class('update-alternatives', d): return
meta/lib/oe/classextend.py +32 -3
@@ -4,11 +4,21 @@ import collections +def get_packages(d): + pkgs = d.getVar("PACKAGES_NONML") + extcls = d.getVar("EXTENDERCLASS") + return extcls.rename_packages_internal(pkgs) + +def get_depends(varprefix, d): + extcls = d.getVar("EXTENDERCLASS") + return extcls.map_depends_variable(varprefix + "_NONML") + class ClassExtender(object): def __init__(self, extname, d): self.extname = extname self.d = d self.pkgs_mapping = [] + self.d.setVar("EXTENDERCLASS", self) def extend_name(self, name): if name.startswith("kernel-") or name == "virtual/kernel":
@@ -24,7 +34,7 @@ class ClassExtender(object): if not subs.startswith(self.extname): return "virtual/" + self.extname + "-" + subs return name - if name.startswith("/"): + if name.startswith("/") or (name.startswith("${") and name.endswith("}")): return name if not name.startswith(self.extname): return self.extname + "-" + name
@@ -89,8 +99,14 @@ class ClassExtender(object): for dep in deps: newdeps[self.map_depends(dep)] = deps[dep] - self.d.setVar(varname, bb.utils.join_deps(newdeps, False).replace("EXTENDPKGV", "${EXTENDPKGV}")) + if not varname.endswith("_NONML"): + #if varname == "DEPENDS": + self.d.renameVar(varname, varname + "_NONML") + self.d.setVar(varname, "${@oe.classextend.get_depends('%s', d)}" % varname) + self.d.appendVarFlag(varname, "vardeps", " " + varname + "_NONML") + ret = bb.utils.join_deps(newdeps, False).replace("EXTENDPKGV", "${EXTENDPKGV}") self.d.setVar("EXTENDPKGV", orig) + return ret def map_packagevars(self): for pkg in (self.d.getVar("PACKAGES").split() + [""]):
@@ -109,10 +125,23 @@ class ClassExtender(object): continue self.pkgs_mapping.append([pkg, self.extend_name(pkg)]) - self.d.setVar("PACKAGES", " ".join([row[1] for row in self.pkgs_mapping])) + self.d.renameVar("PACKAGES", "PACKAGES_NONML") + self.d.setVar("PACKAGES", "${@oe.classextend.get_packages(d)}") + + def rename_packages_internal(self, pkgs): + self.pkgs_mapping = [] + for pkg in (self.d.expand(pkgs) or "").split(): + if pkg.startswith(self.extname): + self.pkgs_mapping.append([pkg.split(self.extname + "-")[1], pkg]) + continue + self.pkgs_mapping.append([pkg, self.extend_name(pkg)]) + + return " ".join([row[1] for row in self.pkgs_mapping]) def rename_package_variables(self, variables): for pkg_mapping in self.pkgs_mapping: + if pkg_mapping[0].startswith("${") and pkg_mapping[0].endswith("}"): + continue for subs in variables: self.d.renameVar("%s_%s" % (subs, pkg_mapping[0]), "%s_%s" % (subs, pkg_mapping[1]))
meta/recipes-core/glibc/glibc-package.inc +1 -1
@@ -24,7 +24,7 @@ libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', INSANE_SKIP_${PN}_append_aarch64 = " libdir" FILES_${PN} = "${libc_baselibs} ${libexecdir}/*" -RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}" +RRECOMMENDS_${PN} = "${@bb.utils.contains('DISTRO_FEATURES', 'ldconfig', '${MLPREFIX}ldconfig', '', d)}" FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf" FILES_ldd = "${bindir}/ldd" FILES_libsegfault = "${base_libdir}/libSegFault*"
meta/recipes-core/packagegroups/packagegroup-base.bb +4 -4
@@ -110,16 +110,16 @@ python __anonymous () { machine_features= set(d.getVar("MACHINE_FEATURES").split()) if "bluetooth" in distro_features and not "bluetooth" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_BT", "packagegroup-base-bluetooth") + d.setVar("ADD_BT", "${MLPREFIX}packagegroup-base-bluetooth") if "wifi" in distro_features and not "wifi" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_WIFI", "packagegroup-base-wifi") + d.setVar("ADD_WIFI", "${MLPREFIX}packagegroup-base-wifi") if "3g" in distro_features and not "3g" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_3G", "packagegroup-base-3g") + d.setVar("ADD_3G", "${MLPREFIX}packagegroup-base-3g") if "nfc" in distro_features and not "nfc" in machine_features and ("usbhost" in machine_features): - d.setVar("ADD_NFC", "packagegroup-base-nfc") + d.setVar("ADD_NFC", "${MLPREFIX}packagegroup-base-nfc") } #
meta/recipes-core/psplash/psplash_git.bb +3 -2
@@ -22,6 +22,7 @@ SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default" python __anonymous() { oldpkgs = d.getVar("PACKAGES").split() splashfiles = d.getVar('SPLASH_IMAGES').split() + mlprefix = d.getVar('MLPREFIX') or '' pkgs = [] localpaths = [] for uri in splashfiles:
@@ -46,9 +47,9 @@ python __anonymous() { # Set these so that we have less work to do in do_compile and do_install_append d.setVar("SPLASH_INSTALL", " ".join(pkgs)) d.setVar("SPLASH_LOCALPATHS", " ".join(localpaths)) + for p in pkgs: + d.prependVar("PACKAGES", "%s%s " % (mlprefix, p)) - d.prependVar("PACKAGES", "%s " % (" ".join(pkgs))) - mlprefix = d.getVar('MLPREFIX') or '' pn = d.getVar('PN') or '' for p in pkgs: ep = '%s%s' % (mlprefix, p)
meta/recipes-devtools/perl/perl_5.30.2.bb +1 -1
@@ -328,7 +328,7 @@ python split_perl_packages () { python() { if d.getVar('CLASSOVERRIDE') == "class-target": - d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*(?<!native)$") + d.setVar("PACKAGES_DYNAMIC", "^${MLPREFIX}perl-module-.*(?<!native)$") elif d.getVar('CLASSOVERRIDE') == "class-native": d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*-native$") elif d.getVar('CLASSOVERRIDE') == "class-nativesdk":
meta/recipes-devtools/python/python3_3.8.2.bb +7 -7
@@ -311,8 +311,8 @@ do_create_manifest[depends] += "${PN}:do_patch" # manual dependency additions RRECOMMENDS_${PN}-core_append_class-nativesdk = " nativesdk-python3-modules" -RRECOMMENDS_${PN}-crypt_append_class-target = " openssl ca-certificates" -RRECOMMENDS_${PN}-crypt_append_class-nativesdk = " openssl ca-certificates" +RRECOMMENDS_${PN}-crypt_append_class-target = " ${MLPREFIX}openssl ${MLPREFIX}ca-certificates" +RRECOMMENDS_${PN}-crypt_append_class-nativesdk = " ${MLPREFIX}openssl ${MLPREFIX}ca-certificates" # For historical reasons PN is empty and provided by python3-modules FILES_${PN} = ""
@@ -322,7 +322,7 @@ FILES_${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAJMIN} ${bindir}/pydoc3" FILES_${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAJMIN}" # provide python-pyvenv from python3-venv -RPROVIDES_${PN}-venv += "python3-pyvenv" +RPROVIDES_${PN}-venv += "${MLPREFIX}python3-pyvenv" # package libpython3 PACKAGES =+ "libpython3 libpython3-staticdev"
@@ -333,8 +333,8 @@ INSANE_SKIP_${PN}-dev += "dev-elf" # catch all the rest (unsorted) PACKAGES += "${PN}-misc" RDEPENDS_${PN}-misc += "python3-core python3-email python3-codecs python3-pydoc python3-pickle python3-audio" -RDEPENDS_${PN}-modules_append_class-target = " python3-misc" -RDEPENDS_${PN}-modules_append_class-nativesdk = " python3-misc" +RDEPENDS_${PN}-modules_append_class-target = " ${MLPREFIX}python3-misc" +RDEPENDS_${PN}-modules_append_class-nativesdk = " ${MLPREFIX}python3-misc" FILES_${PN}-misc = "${libdir}/python${PYTHON_MAJMIN} ${libdir}/python${PYTHON_MAJMIN}/lib-dynload" # catch manpage
@@ -348,5 +348,5 @@ RDEPENDS_${PN}-ptest_append_libc-glibc = " locale-base-tr-tr.iso-8859-9" RDEPENDS_${PN}-tkinter += "${@bb.utils.contains('PACKAGECONFIG', 'tk', 'tk tk-lib', '', d)}" RDEPENDS_${PN}-dev = "" -RDEPENDS_${PN}-tests_append_class-target = " bash" -RDEPENDS_${PN}-tests_append_class-nativesdk = " bash" +RDEPENDS_${PN}-tests_append_class-target = " ${MLPREFIX}bash" +RDEPENDS_${PN}-tests_append_class-nativesdk = " ${MLPREFIX}bash"
meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb +6 -4
@@ -32,21 +32,23 @@ python __anonymous () { namemap["packagegroup-core-full-cmdline-sys-services"] = "packagegroup-core-sys-services" packages = d.getVar("PACKAGES").split() + mlprefix = d.getVar("MLPREFIX") for pkg in packages: + pkg2 = pkg[len(mlprefix):] if pkg.endswith('-dev'): - mapped = namemap.get(pkg[:-4], None) + mapped = namemap.get(pkg2[:-4], None) if mapped: mapped += '-dev' elif pkg.endswith('-dbg'): - mapped = namemap.get(pkg[:-4], None) + mapped = namemap.get(pkg2[:-4], None) if mapped: mapped += '-dbg' else: - mapped = namemap.get(pkg, None) + mapped = namemap.get(pkg2, None) if mapped: oldtaskname = mapped.replace("packagegroup-core", "task-core") - mapstr = " %s %s" % (mapped, oldtaskname) + mapstr = " %s%s %s%s" % (mlprefix, mapped, mlprefix, oldtaskname) d.appendVar("RPROVIDES_%s" % pkg, mapstr) d.appendVar("RREPLACES_%s" % pkg, mapstr) d.appendVar("RCONFLICTS_%s" % pkg, mapstr)
meta/recipes-graphics/mesa/mesa.inc +6 -4
@@ -212,18 +212,20 @@ python __anonymous() { ("gles", "libgles3",)): if not p[0] in pkgconfig: continue - fullp = p[1] + "-mesa" - pkgs = " ".join(p[1:]) + mlprefix = d.getVar("MLPREFIX") + fullp = mlprefix + p[1] + "-mesa" + mlprefix = d.getVar("MLPREFIX") + pkgs = " ".join(mlprefix + x for x in p[1:]) d.setVar("DEBIAN_NOAUTONAME_" + fullp, "1") d.appendVar("RREPLACES_" + fullp, pkgs) d.appendVar("RPROVIDES_" + fullp, pkgs) d.appendVar("RCONFLICTS_" + fullp, pkgs) - d.appendVar("RRECOMMENDS_" + fullp, " mesa-megadriver") + d.appendVar("RRECOMMENDS_" + fullp, " ${MLPREFIX}mesa-megadriver") # For -dev, the first element is both the Debian and original name fullp += "-dev" - pkgs = p[1] + "-dev" + pkgs = mlprefix + p[1] + "-dev" d.setVar("DEBIAN_NOAUTONAME_" + fullp, "1") d.appendVar("RREPLACES_" + fullp, pkgs) d.appendVar("RPROVIDES_" + fullp, pkgs)
meta/recipes-multimedia/alsa/alsa-plugins_1.2.1.bb +3 -3
@@ -167,7 +167,7 @@ FILES_${PN}-pulseaudio-conf += "\ " RDEPENDS_${PN}-pulseaudio-conf += "\ - libasound-module-conf-pulse \ - libasound-module-ctl-pulse \ - libasound-module-pcm-pulse \ + ${MLPREFIX}libasound-module-conf-pulse \ + ${MLPREFIX}libasound-module-ctl-pulse \ + ${MLPREFIX}libasound-module-pcm-pulse \ "
meta/recipes-support/boost/boost.inc +7 -3
@@ -62,12 +62,16 @@ PACKAGES = "${PN}-dbg ${BOOST_PACKAGES}" python __anonymous () { packages = [] extras = [] + mlprefix = d.getVar("MLPREFIX") for lib in d.getVar('BOOST_LIBS').split(): extras.append("--with-%s" % lib) - pkg = "boost-%s" % lib.replace("_", "-") - packages.append(pkg) + pkg = "boost-%s" % (lib.replace("_", "-")) + packages.append(mlprefix + pkg) if not d.getVar("FILES_%s" % pkg): - d.setVar("FILES_%s" % pkg, "${libdir}/libboost_%s*.so.*" % lib) + d.setVar("FILES_%s%s" % (mlprefix, pkg), "${libdir}/libboost_%s*.so.*" % lib) + else: + d.setVar("FILES_%s%s" % (mlprefix, pkg), d.getVar("FILES_%s" % pkg)) + d.setVar("BOOST_PACKAGES", " ".join(packages)) d.setVar("BJAM_EXTRA", " ".join(extras)) }
Signature · 1 line
2.25.1
PATCH 2/5 ltp: Add missing dependencies on coreutils, bc, e2fsprogs and gdb
When the tests are run we see messages like:
/opt/ltp/testcases/bin/run_cpuctl_stress_test.sh: line 242: nice: command not found /opt/ltp/testcases/bin/run_cpuctl_test_fj.sh: line 66: tac: command not found vma05 1 TCONF: 'gdb' not foundmemcg_failcnt 1 TCONF: 'bc' not found
Owner=nobody; perms=-rw-------; sudo: lsattr: command not found
so add missing dependencies to avoid these.
--- meta/recipes-extended/ltp/ltp_20200515.bb | 4 ++++ 1 file changed, 4 insertions(+)
meta/recipes-extended/ltp/ltp_20200515.bb +4 -0
@@ -77,13 +77,17 @@ do_install(){ RDEPENDS_${PN} = "\ attr \ bash \ + bc \ + coreutils \ cpio \ cronie \ curl \ + e2fsprogs \ e2fsprogs-mke2fs \ expect \ file \ gawk \ + gdb \ gzip \ iproute2 \ ldd \
Signature · 1 line
2.25.1
PATCH 3/5 resulttool/report: Remove leftover debugging
In reply to Richard PurdieI've long since wondered why there was some odd output in result reports, remove the leftover debug which was causing it.
--- scripts/lib/resulttool/report.py | 1 - 1 file changed, 1 deletion(-)
scripts/lib/resulttool/report.py +0 -1
@@ -96,7 +96,6 @@ class ResultsTextReport(object): if 'ltpresult.sections' in result and suite not in result['ltpresult.sections']: try: _, suite, suite1, test = k.split(".", 3) - print("split2: %s %s %s" % (suite, suite1, test)) if suite + "." + suite1 in result['ltpresult.sections']: suite = suite + "." + suite1 except ValueError:
Signature · 1 line
2.25.1
PATCH 4/5 resulttool/log: Add ability to dump ltp logs as well as ptest
In reply to Richard PurdieCurrently only ptest logs are accessible with the log command, this adds support so the ltp logs can be extracted too.
--- scripts/lib/resulttool/log.py | 21 ++++++++++++++------- scripts/lib/resulttool/resultutils.py | 22 ++++++++++++++-------- 2 files changed, 28 insertions(+), 15 deletions(-)
scripts/lib/resulttool/log.py +14 -7
@@ -34,13 +34,17 @@ def log(args, logger): return 1 for _, run_name, _, r in resultutils.test_run_results(results): - if args.dump_ptest and 'ptestresult.sections' in r: - for name, ptest in r['ptestresult.sections'].items(): - logdata = resultutils.ptestresult_get_log(r, name) + if args.dump_ptest: + for sectname in ['ptestresult.sections', 'ltpposixresult.sections', 'ltpresult.sections']: + if sectname in r: + for name, ptest in r[sectname].items(): + logdata = resultutils.generic_get_log(sectname, r, name) if logdata is not None: dest_dir = args.dump_ptest if args.prepend_run: dest_dir = os.path.join(dest_dir, run_name) + if not sectname.startswith("ptest"): + dest_dir = os.path.join(dest_dir, sectname.split(".")[0]) os.makedirs(dest_dir, exist_ok=True) dest = os.path.join(dest_dir, '%s.log' % name)
@@ -49,10 +53,13 @@ def log(args, logger): f.write(logdata) if args.raw_ptest: - rawlog = resultutils.ptestresult_get_rawlogs(r) - if rawlog is not None: - print(rawlog) - else: + found = False + for sectname in ['ptestresult.rawlogs', 'ltpposixresult.rawlogs', 'ltpresult.rawlogs']: + rawlog = resultutils.generic_get_rawlogs(sectname, r) + if rawlog is not None: + print(rawlog) + found = True + if not found: print('Raw ptest logs not found') return 1
scripts/lib/resulttool/resultutils.py +14 -8
@@ -130,23 +130,29 @@ def decode_log(logdata): return data.decode("utf-8", errors='ignore') return None -def ptestresult_get_log(results, section): - if 'ptestresult.sections' not in results: +def generic_get_log(sectionname, results, section): + if sectionname not in results: return None - if section not in results['ptestresult.sections']: + if section not in results[sectionname]: return None - ptest = results['ptestresult.sections'][section] + ptest = results[sectionname][section] if 'log' not in ptest: return None return decode_log(ptest['log']) -def ptestresult_get_rawlogs(results): - if 'ptestresult.rawlogs' not in results: +def ptestresult_get_log(results, section): + return generic_get_log('ptestresuls.sections', results, section) + +def generic_get_rawlogs(sectname, results): + if sectname not in results: return None - if 'log' not in results['ptestresult.rawlogs']: + if 'log' not in results[sectname]: return None - return decode_log(results['ptestresult.rawlogs']['log']) + return decode_log(results[sectname]['log']) + +def ptestresult_get_rawlogs(results): + return generic_get_rawlogs('ptestresult.rawlogs', results) def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, ptestlogs=False): for res in results:
Signature · 1 line
2.25.1
PATCH 5/5 ltp: Exclude the memcg_stress tests due to timeout problems
In reply to Richard PurdieThis test runs for 900s, we often see tests killed after 300s without output which makes the test results unreliable and inconsistent. The easiest solution for now is to skip this long running test, patching it out wth sed.
--- meta/recipes-extended/ltp/ltp_20200515.bb | 4 ++++ 1 file changed, 4 insertions(+)
meta/recipes-extended/ltp/ltp_20200515.bb +4 -0
@@ -72,6 +72,10 @@ do_install(){ -e 's@[^ ]*-fdebug-prefix-map=[^ "]*@@g' \ -e 's@[^ ]*-fmacro-prefix-map=[^ "]*@@g' \ -e 's@[^ ]*--sysroot=[^ "]*@@g' + + # The controllers memcg_stree test seems to cause us hangs and takes 900s + # (maybe we expect more regular output?), anyhow, skip it + sed -e '/^memcg_stress/d' -i ${D}${prefix}/runtest/controllers } RDEPENDS_${PN} = "\
Signature · 1 line
2.25.1
The one question I did have is there are a couple of places where you add the ML prefix for nativesdk classes. If we don't have them, it shouldn't hurt anything, but do we actually support multilib nativesdk packages?
Quoted 424 lines · Richard Purdie
There are issues with multilib due to the ordering of events where some functions see the remapped multilib dependencies and some do not. A significant problem is that the multilib class needs to make some changes before key expansion and some afterwards but by using existing event handlers, some code sees things in a partially translated state, leading to bugs.
This patch changes things to use a new event handler from bitbake which makes the ordering of the changes explcit.
The challenge in doing this is that it breaks some existing anonymous python and dyanmic assignments. In some cases these used to be translated and no longer are, meaning MLPREFIX has to be added. In some cases these are now translated and the MLPREFIX can be removed.
This change does now make it very clear when MLPREFIX is required and when it is not, its just the migration path which is harder. The patch changes the small number of cases where fixes are needed.
In particular, where a variable like RDEPENDS is conditionally extended (e.g. with an override), MLPREFIX is now required.
This patch also reverts:
base: Revert 'base.bbclass: considering multilib when settingLICENSE_EXCLUSION'This reverts
6597130256a1609c3e05ec5891aceaf549c37985as the changes to multilib datastore handling mean its no longer necessary.Signed-off-by Richard Purdie <[email protected]>--- meta/classes/base.bbclass | 3 +- meta/classes/fontcache.bbclass | 2 +- meta/classes/multilib.bbclass | 28 ++++++++++++--- meta/lib/oe/classextend.py | 35 +++++++++++++++++-- meta/recipes-core/glibc/glibc-package.inc | 2 +- .../packagegroups/packagegroup-base.bb | 8 ++--- meta/recipes-core/psplash/psplash_git.bb | 5 +-- meta/recipes-devtools/perl/perl_5.30.2.bb | 2 +- meta/recipes-devtools/python/python3_3.8.2.bb | 14 ++++---- .../packagegroup-core-full-cmdline.bb | 10 +++--- meta/recipes-graphics/mesa/mesa.inc | 10 +++--- .../alsa/alsa-plugins_1.2.1.bb | 6 ++-- meta/recipes-support/boost/boost.inc | 10 ++++-- 13 files changed, 95 insertions(+), 40 deletions(-)meta/classes/base.bbclass +1
-2@@ -570,8 +570,7 @@ python () { if unskipped_pkgs: for pkg in skipped_pkgs: bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg]))) - mlprefix = d.getVar('MLPREFIX') - d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, ' '.join(skipped_pkgs[pkg])) + d.setVar('LICENSE_EXCLUSION-' + pkg, ' '.join(skipped_pkgs[pkg])) for pkg in unskipped_pkgs: bb.debug(1, "Including the package %s" % pkg) else:meta/classes/fontcache.bbclass +1
-1@@ -7,7 +7,7 @@ PACKAGE_WRITE_DEPS += "qemu-native" inherit qemu FONT_PACKAGES ??= "${PN}" -FONT_EXTRA_RDEPENDS ?= "fontconfig-utils" +FONT_EXTRA_RDEPENDS ?= "${MLPREFIX}fontconfig-utils" FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig" FONTCONFIG_CACHE_PARAMS ?= "-v" # You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,meta/classes/multilib.bbclass +23
-5@@ -91,13 +91,12 @@ addhandler multilib_virtclass_handler multilib_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" python __anonymous () { - variant = d.getVar("BBEXTENDVARIANT") - - import oe.classextend + if bb.data.inherits_class('image', d): + variant = d.getVar("BBEXTENDVARIANT") + import oe.classextend - clsextend = oe.classextend.ClassExtender(variant, d) + clsextend = oe.classextend.ClassExtender(variant, d) - if bb.data.inherits_class('image', d): clsextend.map_depends_variable("PACKAGE_INSTALL") clsextend.map_depends_variable("LINGUAS_INSTALL") clsextend.map_depends_variable("RDEPENDS")@@ -109,6 +108,22 @@ python __anonymous () { bb.build.deltask('do_populate_sdk', d) bb.build.deltask('do_populate_sdk_ext', d) return +} + +python multilib_virtclass_handler_postkeyexp () { + cls = d.getVar("BBEXTENDCURR") + variant = d.getVar("BBEXTENDVARIANT") + if cls != "multilib" or not variant: + return + + variant = d.getVar("BBEXTENDVARIANT") + + import oe.classextend + + clsextend = oe.classextend.ClassExtender(variant, d) + + if bb.data.inherits_class('image', d): + return clsextend.map_depends_variable("DEPENDS") clsextend.map_variable("PROVIDES")@@ -129,6 +144,9 @@ python __anonymous () { reset_alternative_priority(d) } +addhandler multilib_virtclass_handler_postkeyexp +multilib_virtclass_handler_postkeyexp[eventmask] = "bb.event.RecipePostKeyExpansion" + def reset_alternative_priority(d): if not bb.data.inherits_class('update-alternatives', d): returnmeta/lib/oe/classextend.py +32
-3@@ -4,11 +4,21 @@ import collections +def get_packages(d): + pkgs = d.getVar("PACKAGES_NONML") + extcls = d.getVar("EXTENDERCLASS") + return extcls.rename_packages_internal(pkgs) + +def get_depends(varprefix, d): + extcls = d.getVar("EXTENDERCLASS") + return extcls.map_depends_variable(varprefix + "_NONML") + class ClassExtender(object): def __init__(self, extname, d): self.extname = extname self.d = d self.pkgs_mapping = [] + self.d.setVar("EXTENDERCLASS", self) def extend_name(self, name): if name.startswith("kernel-") or name == "virtual/kernel":@@ -24,7 +34,7 @@ class ClassExtender(object): if not subs.startswith(self.extname): return "virtual/" + self.extname + "-" + subs return name - if name.startswith("/"): + if name.startswith("/") or (name.startswith("${") and name.endswith("}")): return name if not name.startswith(self.extname): return self.extname + "-" + name@@ -89,8 +99,14 @@ class ClassExtender(object): for dep in deps: newdeps[self.map_depends(dep)] = deps[dep] - self.d.setVar(varname, bb.utils.join_deps(newdeps, False).replace("EXTENDPKGV", "${EXTENDPKGV}")) + if not varname.endswith("_NONML"): + #if varname == "DEPENDS": + self.d.renameVar(varname, varname + "_NONML") + self.d.setVar(varname, "${@oe.classextend.get_depends('%s', d)}" % varname) + self.d.appendVarFlag(varname, "vardeps", " " + varname + "_NONML") + ret = bb.utils.join_deps(newdeps, False).replace("EXTENDPKGV", "${EXTENDPKGV}") self.d.setVar("EXTENDPKGV", orig) + return ret def map_packagevars(self): for pkg in (self.d.getVar("PACKAGES").split() + [""]):@@ -109,10 +125,23 @@ class ClassExtender(object): continue self.pkgs_mapping.append([pkg, self.extend_name(pkg)]) - self.d.setVar("PACKAGES", " ".join([row[1] for row in self.pkgs_mapping])) + self.d.renameVar("PACKAGES", "PACKAGES_NONML") + self.d.setVar("PACKAGES", "${@oe.classextend.get_packages(d)}") + + def rename_packages_internal(self, pkgs): + self.pkgs_mapping = [] + for pkg in (self.d.expand(pkgs) or "").split(): + if pkg.startswith(self.extname): + self.pkgs_mapping.append([pkg.split(self.extname + "-")[1], pkg]) + continue + self.pkgs_mapping.append([pkg, self.extend_name(pkg)]) + + return " ".join([row[1] for row in self.pkgs_mapping]) def rename_package_variables(self, variables): for pkg_mapping in self.pkgs_mapping: + if pkg_mapping[0].startswith("${") and pkg_mapping[0].endswith("}"): + continue for subs in variables: self.d.renameVar("%s_%s" % (subs, pkg_mapping[0]), "%s_%s" % (subs, pkg_mapping[1]))meta/recipes-core/glibc/glibc-package.inc +1
-1@@ -24,7 +24,7 @@ libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', INSANE_SKIP_${PN}_append_aarch64 = " libdir" FILES_${PN} = "${libc_baselibs} ${libexecdir}/*" -RRECOMMENDS_${PN} = "${@bb.utils.filter('DISTRO_FEATURES', 'ldconfig', d)}" +RRECOMMENDS_${PN} = "${@bb.utils.contains('DISTRO_FEATURES', 'ldconfig', '${MLPREFIX}ldconfig', '', d)}" FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf" FILES_ldd = "${bindir}/ldd" FILES_libsegfault = "${base_libdir}/libSegFault*"meta/recipes-core/packagegroups/packagegroup-base.bb +4
-4@@ -110,16 +110,16 @@ python __anonymous () { machine_features= set(d.getVar("MACHINE_FEATURES").split()) if "bluetooth" in distro_features and not "bluetooth" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_BT", "packagegroup-base-bluetooth") + d.setVar("ADD_BT", "${MLPREFIX}packagegroup-base-bluetooth") if "wifi" in distro_features and not "wifi" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_WIFI", "packagegroup-base-wifi") + d.setVar("ADD_WIFI", "${MLPREFIX}packagegroup-base-wifi") if "3g" in distro_features and not "3g" in machine_features and ("pcmcia" in machine_features or "pci" in machine_features or "usbhost" in machine_features): - d.setVar("ADD_3G", "packagegroup-base-3g") + d.setVar("ADD_3G", "${MLPREFIX}packagegroup-base-3g") if "nfc" in distro_features and not "nfc" in machine_features and ("usbhost" in machine_features): - d.setVar("ADD_NFC", "packagegroup-base-nfc") + d.setVar("ADD_NFC", "${MLPREFIX}packagegroup-base-nfc") } #meta/recipes-core/psplash/psplash_git.bb +3
-2@@ -22,6 +22,7 @@ SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default" python __anonymous() { oldpkgs = d.getVar("PACKAGES").split() splashfiles = d.getVar('SPLASH_IMAGES').split() + mlprefix = d.getVar('MLPREFIX') or '' pkgs = [] localpaths = [] for uri in splashfiles:@@ -46,9 +47,9 @@ python __anonymous() { # Set these so that we have less work to do in do_compile and do_install_append d.setVar("SPLASH_INSTALL", " ".join(pkgs)) d.setVar("SPLASH_LOCALPATHS", " ".join(localpaths)) + for p in pkgs: + d.prependVar("PACKAGES", "%s%s " % (mlprefix, p)) - d.prependVar("PACKAGES", "%s " % (" ".join(pkgs))) - mlprefix = d.getVar('MLPREFIX') or '' pn = d.getVar('PN') or '' for p in pkgs: ep = '%s%s' % (mlprefix, p)meta/recipes-devtools/perl/perl_5.30.2.bb +1
-1@@ -328,7 +328,7 @@ python split_perl_packages () { python() { if d.getVar('CLASSOVERRIDE') == "class-target": - d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*(?<!native)$") + d.setVar("PACKAGES_DYNAMIC", "^${MLPREFIX}perl-module-.*(?<!native)$") elif d.getVar('CLASSOVERRIDE') == "class-native": d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*-native$") elif d.getVar('CLASSOVERRIDE') == "class-nativesdk":meta/recipes-devtools/python/python3_3.8.2.bb +7
-7@@ -311,8 +311,8 @@ do_create_manifest[depends] += "${PN}:do_patch" # manual dependency additions RRECOMMENDS_${PN}-core_append_class-nativesdk = " nativesdk-python3-modules" -RRECOMMENDS_${PN}-crypt_append_class-target = " openssl ca-certificates" -RRECOMMENDS_${PN}-crypt_append_class-nativesdk = " openssl ca-certificates" +RRECOMMENDS_${PN}-crypt_append_class-target = " ${MLPREFIX}openssl ${MLPREFIX}ca-certificates" +RRECOMMENDS_${PN}-crypt_append_class-nativesdk = " ${MLPREFIX}openssl ${MLPREFIX}ca-certificates" # For historical reasons PN is empty and provided by python3-modules FILES_${PN} = ""@@ -322,7 +322,7 @@ FILES_${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAJMIN} ${bindir}/pydoc3" FILES_${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAJMIN}" # provide python-pyvenv from python3-venv -RPROVIDES_${PN}-venv += "python3-pyvenv" +RPROVIDES_${PN}-venv += "${MLPREFIX}python3-pyvenv" # package libpython3 PACKAGES =+ "libpython3 libpython3-staticdev"@@ -333,8 +333,8 @@ INSANE_SKIP_${PN}-dev += "dev-elf" # catch all the rest (unsorted) PACKAGES += "${PN}-misc" RDEPENDS_${PN}-misc += "python3-core python3-email python3-codecs python3-pydoc python3-pickle python3-audio" -RDEPENDS_${PN}-modules_append_class-target = " python3-misc" -RDEPENDS_${PN}-modules_append_class-nativesdk = " python3-misc" +RDEPENDS_${PN}-modules_append_class-target = " ${MLPREFIX}python3-misc" +RDEPENDS_${PN}-modules_append_class-nativesdk = " ${MLPREFIX}python3-misc" FILES_${PN}-misc = "${libdir}/python${PYTHON_MAJMIN} ${libdir}/python${PYTHON_MAJMIN}/lib-dynload" # catch manpage@@ -348,5 +348,5 @@ RDEPENDS_${PN}-ptest_append_libc-glibc = " locale-base-tr-tr.iso-8859-9" RDEPENDS_${PN}-tkinter += "${@bb.utils.contains('PACKAGECONFIG', 'tk', 'tk tk-lib', '', d)}" RDEPENDS_${PN}-dev = "" -RDEPENDS_${PN}-tests_append_class-target = " bash" -RDEPENDS_${PN}-tests_append_class-nativesdk = " bash" +RDEPENDS_${PN}-tests_append_class-target = " ${MLPREFIX}bash" +RDEPENDS_${PN}-tests_append_class-nativesdk = " ${MLPREFIX}bash"meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb +6
-4@@ -32,21 +32,23 @@ python __anonymous () { namemap["packagegroup-core-full-cmdline-sys-services"] = "packagegroup-core-sys-services" packages = d.getVar("PACKAGES").split() + mlprefix = d.getVar("MLPREFIX") for pkg in packages: + pkg2 = pkg[len(mlprefix):] if pkg.endswith('-dev'): - mapped = namemap.get(pkg[:-4], None) + mapped = namemap.get(pkg2[:-4], None) if mapped: mapped += '-dev' elif pkg.endswith('-dbg'): - mapped = namemap.get(pkg[:-4], None) + mapped = namemap.get(pkg2[:-4], None) if mapped: mapped += '-dbg' else: - mapped = namemap.get(pkg, None) + mapped = namemap.get(pkg2, None) if mapped: oldtaskname = mapped.replace("packagegroup-core", "task-core") - mapstr = " %s %s" % (mapped, oldtaskname) + mapstr = " %s%s %s%s" % (mlprefix, mapped, mlprefix, oldtaskname) d.appendVar("RPROVIDES_%s" % pkg, mapstr) d.appendVar("RREPLACES_%s" % pkg, mapstr) d.appendVar("RCONFLICTS_%s" % pkg, mapstr)meta/recipes-graphics/mesa/mesa.inc +6
-4@@ -212,18 +212,20 @@ python __anonymous() { ("gles", "libgles3",)): if not p[0] in pkgconfig: continue - fullp = p[1] + "-mesa" - pkgs = " ".join(p[1:]) + mlprefix = d.getVar("MLPREFIX") + fullp = mlprefix + p[1] + "-mesa" + mlprefix = d.getVar("MLPREFIX") + pkgs = " ".join(mlprefix + x for x in p[1:]) d.setVar("DEBIAN_NOAUTONAME_" + fullp, "1") d.appendVar("RREPLACES_" + fullp, pkgs) d.appendVar("RPROVIDES_" + fullp, pkgs) d.appendVar("RCONFLICTS_" + fullp, pkgs) - d.appendVar("RRECOMMENDS_" + fullp, " mesa-megadriver") + d.appendVar("RRECOMMENDS_" + fullp, " ${MLPREFIX}mesa-megadriver") # For -dev, the first element is both the Debian and original name fullp += "-dev" - pkgs = p[1] + "-dev" + pkgs = mlprefix + p[1] + "-dev" d.setVar("DEBIAN_NOAUTONAME_" + fullp, "1") d.appendVar("RREPLACES_" + fullp, pkgs) d.appendVar("RPROVIDES_" + fullp, pkgs)meta/recipes-multimedia/alsa/alsa-plugins_1.2.1.bb +3
-3@@ -167,7 +167,7 @@ FILES_${PN}-pulseaudio-conf += "\ " RDEPENDS_${PN}-pulseaudio-conf += "\ - libasound-module-conf-pulse \ - libasound-module-ctl-pulse \ - libasound-module-pcm-pulse \ + ${MLPREFIX}libasound-module-conf-pulse \ + ${MLPREFIX}libasound-module-ctl-pulse \ + ${MLPREFIX}libasound-module-pcm-pulse \ "meta/recipes-support/boost/boost.inc +7
-3@@ -62,12 +62,16 @@ PACKAGES = "${PN}-dbg ${BOOST_PACKAGES}" python __anonymous () { packages = [] extras = [] + mlprefix = d.getVar("MLPREFIX") for lib in d.getVar('BOOST_LIBS').split(): extras.append("--with-%s" % lib) - pkg = "boost-%s" % lib.replace("_", "-") - packages.append(pkg) + pkg = "boost-%s" % (lib.replace("_", "-")) + packages.append(mlprefix + pkg) if not d.getVar("FILES_%s" % pkg): - d.setVar("FILES_%s" % pkg, "${libdir}/libboost_%s*.so.*" % lib) + d.setVar("FILES_%s%s" % (mlprefix, pkg), "${libdir}/libboost_%s*.so.*" % lib) + else: + d.setVar("FILES_%s%s" % (mlprefix, pkg), d.getVar("FILES_%s" % pkg)) + d.setVar("BOOST_PACKAGES", " ".join(packages)) d.setVar("BJAM_EXTRA", " ".join(extras)) }
Signature · 2 lines
Jeremy A. Puhlman [email protected]
Quoted 5 lines · Jeremy Puhlman
The one question I did have is there are a couple of places where you add the ML prefix for
nativesdk classes. If we don't have them, it shouldn't hurt anything, but do we actually support
multilib nativesdk packages?
nativesdk actually uses the same mechanism to extend the package names as multilib does so for nativesdk, MLPREFIX="nativesdk-".
So no, we don't support mulitlib nativesdk but nativesdk is a kind of multilib in some ways as it also sets MLPREFIX.
Cheers,
Richard
Hi Richard,
Quoted 18 lines · Richard Purdie
There are issues with multilib due to the ordering of events where some functions see the remapped multilib dependencies and some do not. A significant problem is that the multilib class needs to make some changes before key expansion and some afterwards but by using existing event handlers, some code sees things in a partially translated state, leading to bugs.
This patch changes things to use a new event handler from bitbake which makes the ordering of the changes explcit.
The challenge in doing this is that it breaks some existing anonymous python and dyanmic assignments. In some cases these used to be translated and no longer are, meaning MLPREFIX has to be added. In some cases these are now translated and the MLPREFIX can be removed.
This change does now make it very clear when MLPREFIX is required and when it is not, its just the migration path which is harder. The patch changes the small number of cases where fixes are needed.
Quoted 9 lines
In particular, where a variable like RDEPENDS is conditionally extended (e.g. with an override), MLPREFIX is now required.
This patch also reverts:
base: Revert 'base.bbclass: considering multilib when settingLICENSE_EXCLUSION'This reverts
6597130256a1609c3e05ec5891aceaf549c37985as the changes to multilib datastore handling mean its no longer necessary.
Is it possible to link the bugzilla entry in the commit log?
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13865
Otherwise, the main question is: do builds actually show warnings or fail if MLPREFIX is forgotten in some places? I'm thinking about other layers needing similar changes.
In either case, maybe a proper "note" in the migration steps for the impacted release would be a good addition I think?
Some nitpicking/cosmetic review:
[...]
meta/lib/oe/classextend.py class ClassExtender(object): · line 103
Quoted 3 lines
meta/lib/oe/classextend.py +2
-1- self.d.setVar(varname, bb.utils.join_deps(newdeps, False).replace("EXTENDPKGV", "${EXTENDPKGV}")) + if not varname.endswith("_NONML"): + #if varname == "DEPENDS":
Is it an actual comment or is it a leftover of some debugging session?
[...]
meta/recipes-graphics/mesa/mesa.inc python __anonymous() { · line 217
Quoted 13 lines
meta/recipes-graphics/mesa/mesa.inc +3
-2@@ -212,18 +212,20 @@ python __anonymous() { ("gles", "libgles3",)): if not p[0] in pkgconfig: continue - fullp = p[1] + "-mesa" - pkgs = " ".join(p[1:]) + mlprefix = d.getVar("MLPREFIX") + fullp = mlprefix + p[1] + "-mesa" + mlprefix = d.getVar("MLPREFIX")
Duplicate mlprefix = d.getVar("MLPREFIX"). Any reason for that?
[...]
Thanks a bunch for the patches, I know it was painful. Quentin