Install rdependent of allarch package with same multilib variant of image
cover Install rdependent of allarch package with same multilib variant of image
From: Kai Kang <[email protected]>
Hi Richard and Mark,
The series rfc patch is trying to install 64bits packages which is required by allarch package to 32 bits image. Work for rpm and need little more work and test for dpkg and opkg.
Take ca-certificates as example. It requires openssl. So it intalls openssl for both core-image-sato and lib32-core-image-sato. But lib32-openssl is expected to be installed to lib32-core-image-sato.
So extend rdpendent of allarch package with all multilib variants and 'noarch-' as well.
Extend
ca-certificates -> openssl
to
ca-certificates -> openssl, lib32-openssl, noarch-openssl
Rdepends on openssl and lib32-openssl make both of them exist in oe-core-repo when do rootfs. And then make both openssl and lib32-openssl provide noarch-openssl. And only keep noarch-openssl in RDEPENDS when write package for ca-certificates. When do image rootfs for lib32-core-image-sato, remove openssl if installed already and install lib32-openssl to provide noarch-openssl for allarch pacakge ca-certificates.
And extend rprovides of allarch package with prefix 'noarch-' too that it may be required by other allarch packages. Put all dependent packages of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.
Any comment for this method is welcomed. Thanks a lot.
--Kai
Kai Kang (2):
make allarch packages don't dependent specific multilib package
WIP: install rdependent packages of allarch packages according to
image rootfs
meta/classes/allarch.bbclass | 3 -- meta/classes/multilib.bbclass | 3 +- meta/classes/multilib_global.bbclass | 31 ++++++++++++++++- meta/classes/nativesdk.bbclass | 8 +++++ meta/classes/package_deb.bbclass | 7 +++- meta/classes/package_ipk.bbclass | 7 +++- meta/classes/package_rpm.bbclass | 8 ++++- meta/conf/multilib.conf | 16 +++++++++ meta/lib/oe/package_manager.py | 51 ++++++++++++++++++++++++++-- meta/lib/oe/rootfs.py | 28 +++++++++++++++ 10 files changed, 152 insertions(+), 10 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH 1/2 make allarch packages don't dependent specific multilib package
From: Kai Kang <[email protected]>
The dependencies of allarch packages doesn't not extend when enable multilib. So take ca-certificates as an example, it always requires openssl no matter whether multilib is enabled. For multilib image such as lib32-core-image-sato, it installs 64 bits openssl rather than lib32-openssl which is expected.
So extend rdependencies of allarch package with prefix 'noarch-' and all of multilib variants. For
ca-certificates -> openssl
exends to
ca-certificates -> openssl, lib32-openssl, noarch-openssl
Rdepends on openssl and lib32-openssl make both of them exist in oe-core-repo when do rootfs. And then make both openssl and lib32-openssl provide noarch-openssl. And only keep noarch-openssl in RDEPENDS when write package for ca-certificates. So when do image rootfs for lib32-core-image-sato, remove openssl if installed already and install lib32-openssl for allarch pacakge ca-certificates.
And extend rprovides of allarch package with prefix 'noarch-' too that it may be required by other allarch packages. Put all dependent packages of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.
--- meta/classes/allarch.bbclass | 3 --- meta/classes/multilib.bbclass | 3 ++- meta/classes/multilib_global.bbclass | 31 +++++++++++++++++++++++++++- meta/classes/nativesdk.bbclass | 8 +++++++ meta/classes/package_deb.bbclass | 7 ++++++- meta/classes/package_ipk.bbclass | 7 ++++++- meta/classes/package_rpm.bbclass | 8 ++++++- meta/conf/multilib.conf | 16 ++++++++++++++ 8 files changed, 75 insertions(+), 8 deletions(-)
meta/classes/allarch.bbclass +0 -3
@@ -42,9 +42,6 @@ python () { d.setVar("INHIBIT_PACKAGE_DEBUG_SPLIT", "1") d.setVar("INHIBIT_PACKAGE_STRIP", "1") - # These multilib values shouldn't change allarch packages so exclude them - d.appendVarFlag("emit_pkgdata", "vardepsexclude", " MULTILIB_VARIANTS") - d.appendVarFlag("write_specfile", "vardepsexclude", " MULTILIBS") d.appendVarFlag("do_package", "vardepsexclude", " package_do_shlibs") elif bb.data.inherits_class('packagegroup', d) and not bb.data.inherits_class('nativesdk', d): bb.error("Please ensure recipe %s sets PACKAGE_ARCH before inherit packagegroup" % d.getVar("FILE"))
meta/classes/multilib.bbclass +2 -1
@@ -134,7 +134,8 @@ python do_package_qa_multilib() { i = i[len('virtual/'):] if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)) and \ (not 'cross-canadian' in i) and (not i.startswith("nativesdk-")) and \ - (not i.startswith("rtld")) and (not i.startswith('kernel-vmlinux')): + (not i.startswith("rtld")) and (not i.startswith('kernel-vmlinux')) and \ + (not i.startswith("noarch")): candidates.append(i) if len(candidates) > 0: msg = "%s package %s - suspicious values '%s' in %s" \
meta/classes/multilib_global.bbclass +30 -1
@@ -161,6 +161,16 @@ multilib_virtclass_handler_vendor[eventmask] = "bb.event.ConfigParsed" python multilib_virtclass_handler_global () { variant = e.data.getVar("BBEXTENDVARIANT") + + # make packages from ALLARCH_RDEP_PKGS provide noarch-pkg + if not (bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data)): + for pkg in (e.data.getVar("PACKAGES") or "").split(): + bpkg = pkg.lstrip(variant + "-" if variant else "") + if bpkg in (e.data.getVar("ALLARCH_RDEP_PKGS") or ""): + rprovs = d.getVar("RPROVIDES_%s" % pkg) or "" + rprovs = rprovs + " noarch-" + bpkg + d.setVar("RPROVIDES_%s" % pkg, rprovs.strip()) + if variant: return
@@ -194,9 +204,28 @@ python multilib_virtclass_handler_global () { for clsextend in clsextends: rprovs = rprovs + " " + clsextend.map_variable("RPROVIDES_%s" % pkg, setvar=False) rprovs = rprovs + " " + clsextend.extname + "-" + pkg + + if bb.data.inherits_class('allarch', e.data): + # make allarch pkg rprovides noarch-itself + if not pkg.endswith(("-dbg", "-dev", "-staticdev")): + rprovs = rprovs + " noarch-" + pkg + e.data.setVar("RPROVIDES_%s" % pkg, rprovs) + + # Process RDEPENDS for noarch recipe + if bb.data.inherits_class('allarch', e.data) and \ + not (bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data)): + noarch_clsext = oe.classextend.ClassExtender('noarch', e.data) + for pkg in (e.data.getVar("PACKAGES") or "").split(): + if pkg.endswith(("-dbg", "-dev", "-staticdev")): + continue + + rdeps = e.data.getVar("RDEPENDS_%s" % pkg) or "" + rdeps = rdeps + " " + noarch_clsext.map_variable("RDEPENDS_%s" % pkg, setvar=False) + for clsextend in clsextends: + rdeps = rdeps + " " + clsextend.map_variable("RDEPENDS_%s" % pkg, setvar=False) + e.data.setVar("RDEPENDS_%s" % pkg, rdeps) } addhandler multilib_virtclass_handler_global multilib_virtclass_handler_global[eventmask] = "bb.event.RecipeParsed" -
meta/classes/nativesdk.bbclass +8 -0
@@ -95,6 +95,14 @@ python () { clsextend.map_packagevars() clsextend.map_variable("PROVIDES") clsextend.map_regexp_variable("PACKAGES_DYNAMIC") + + # make variant-pkg from ALLARCH_RDEP_PKGS provides noarch-pkg + for pkg in (d.getVar("PACKAGES") or "").split(): + bpkg = pkg.replace("nativesdk-", "") + if bpkg in (d.getVar("ALLARCH_RDEP_PKGS") or "").split(): + rprovs = d.getVar("RPROVIDES_%s" % pkg) or "" + rprovs = rprovs + " noarch-" + pkg + d.setVar("RPROVIDES_%s" % pkg, rprovs) } addhandler nativesdk_virtclass_handler
meta/classes/package_deb.bbclass +6 -1
@@ -245,7 +245,12 @@ def deb_write_pkg(pkg, d): elif (v or "").startswith("> "): var[dep][i] = var[dep][i].replace("> ", ">> ") - rdepends = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "") + rdepends = localdata.getVar("RDEPENDS") or "" + if bb.data.inherits_class('allarch', localdata) \ + and not bb.data.inherits_class('packagegroup', localdata) \ + and not pkg.endswith(("-dbg", "-dev", "-staticdev")): + rdepends = " ".join(list(filter(lambda dep: dep.startswith('noarch-'), rdepends.split()))) + rdepends = bb.utils.explode_dep_versions2(rdepends) debian_cmp_remap(rdepends) for dep in list(rdepends.keys()): if dep == pkg:
meta/classes/package_ipk.bbclass +6 -1
@@ -225,7 +225,12 @@ def ipk_write_pkg(pkg, d): elif (v or "").startswith("> "): var[dep][i] = var[dep][i].replace("> ", ">> ") - rdepends = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "") + rdepends = localdata.getVar("RDEPENDS") or "" + if bb.data.inherits_class('allarch', localdata) \ + and not bb.data.inherits_class('packagegroup', localdata) \ + and not pkg.endswith(("-dbg", "-dev", "-staticdev")): + rdepends = " ".join(list(filter(lambda dep: dep.startswith('noarch-'), rdepends))) + rdepends = bb.utils.explode_dep_versions2(rdepends) debian_cmp_remap(rdepends) rrecommends = bb.utils.explode_dep_versions2(localdata.getVar("RRECOMMENDS") or "") debian_cmp_remap(rrecommends)
meta/classes/package_rpm.bbclass +7 -1
@@ -333,6 +333,13 @@ python write_specfile () { localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg) + if bb.data.inherits_class('allarch', localdata) \ + and not bb.data.inherits_class('packagegroup', localdata) \ + and not pkg.endswith(("-dbg", "-dev", "-staticdev")): + rdeps = (localdata.getVar('RDEPENDS') or "").split() + rdeps = list(filter(lambda dep: dep.startswith('noarch-'), rdeps)) + localdata.setVar('RDEPENDS', ' '.join(rdeps)) + conffiles = get_conffiles(pkg, d) dirfiles = localdata.getVar('DIRFILES') if dirfiles is not None:
@@ -372,7 +379,6 @@ python write_specfile () { splitrprerm = localdata.getVar('pkg_prerm') splitrpostrm = localdata.getVar('pkg_postrm') - if not perfiledeps: # Add in summary of per file dependencies splitrdepends = splitrdepends + " " + get_perfile('RDEPENDS', pkg, d)
meta/conf/multilib.conf +16 -0
@@ -26,3 +26,19 @@ OPKG_ARGS_append = " --force-maintainer --force-overwrite" PKG_CONFIG_PATH .= ":${WORKDIR}/recipe-sysroot/${datadir}/pkgconfig" PKG_CONFIG_PATH[vardepsexclude] = "datadir WORKDIR" PKG_CONFIG_PATH[vardepvalueexclude] = ":${WORKDIR}/recipe-sysroot/${datadir}/pkgconfig" + +# resolve runtime dependencies of allarch packages when enable multilib +ALLARCH_RDEP_PKGS ?= "\ + bash \ + busybox-mdev \ + font-util \ + fontconfig-utils \ + kbd \ + openssl \ + perl \ + ppp \ + udev \ + udev-extraconf \ + weston \ + ${VIRTUAL-RUNTIME_base-utils} \ +"
Signature · 1 line
2.17.1
RFC PATCH 2/2 WIP: install rdependent packages of allarch packages according to image rootfs
In reply to [email protected]From: Kai Kang <[email protected]>
Add interface is_leaf to package managers. If any multilib variant of package from 'ALLARCH_RDEP_PKGS' is installed to image rootfs, then install the packages with 'BBEXTENDVARIANT' of current image and remove the already installed package if not required by others and not add to image intentionally.
--- meta/lib/oe/package_manager.py | 51 ++++++++++++++++++++++++++++++++-- meta/lib/oe/rootfs.py | 28 +++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +49 -2
@@ -249,7 +249,7 @@ class DpkgIndexer(Indexer): release.write("Label: %s\n" % arch) cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive - + index_cmds.append(cmd) deb_dirs_found = True
@@ -448,6 +448,13 @@ class PackageManager(object, metaclass=ABCMeta): def list_installed(self): pass + @abstractmethod + def is_leaf(self, pkg): + """ + Whether pkg is required by other packages + """ + pass + @abstractmethod def extract(self, pkg): """
@@ -587,7 +594,7 @@ def create_packages_dir(d, rpm_repo_dir, deploydir, taskname, filterbydependenci pn = d.getVar("PN") seendirs = set() multilibs = {} - + rpm_subrepo_dir = oe.path.join(rpm_repo_dir, "rpm") bb.utils.remove(rpm_subrepo_dir, recurse=True)
@@ -895,6 +902,20 @@ class RpmPM(PackageManager): return packages + def is_leaf(self, pkg): + if not pkg: + return False + + cmd = bb.utils.which(os.getenv('PATH'), "rpm") + args = ["-q", "--whatrequires", "--root=%s" % self.target_rootfs] + + bb.note("Running %s" % ' '.join([cmd] + args + [pkg])) + # return value 0 stands for other(s) require(s) pkg + if subprocess.call([cmd] + args + [pkg], shell=True): + return True + else: + return False + def update(self): self._invoke_dnf(["makecache", "--refresh"])
@@ -1327,6 +1348,12 @@ class OpkgPM(OpkgDpkgPM): def list_installed(self): return OpkgPkgsList(self.d, self.target_rootfs, self.config_file).list_pkgs() + def is_leaf(self, pkg): + if not pkg: + return False + + + def handle_bad_recommendations(self): bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS") or "" if bad_recommendations.strip() == "":
@@ -1753,6 +1780,26 @@ class DpkgPM(OpkgDpkgPM): def list_installed(self): return DpkgPkgsList(self.d, self.target_rootfs).list_pkgs() + def is_leaf(self, pkg): + bb.warn("=====================in dpkg is_leaf()\n") + if not pkg: + return False + + cmd = "%s rdepends %s" % (self.apt_cache_cmd, pkg) + + try: + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8") + except subprocess.CalledProcessError as e: + bb.fatal("Unable to list rdepends packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8"))) + + bb.warn(output) + bb.warn(len(output.split())) + if len(output.split()) > 2: + return False + else: + return True + def package_info(self, pkg): """ Returns a dictionary with the package info.
meta/lib/oe/rootfs.py +28 -0
@@ -170,6 +170,7 @@ class Rootfs(object, metaclass=ABCMeta): def create(self): bb.note("###### Generate rootfs #######") + bb.warn(" in create ###### Generate rootfs #######") pre_process_cmds = self.d.getVar("ROOTFS_PREPROCESS_COMMAND") post_process_cmds = self.d.getVar("ROOTFS_POSTPROCESS_COMMAND") rootfs_post_install_cmds = self.d.getVar('ROOTFS_POSTINSTALL_COMMAND')
@@ -213,6 +214,9 @@ class Rootfs(object, metaclass=ABCMeta): self._uninstall_unneeded() + bb.warn("========= in rootfs() before self._fix_allarch_ml_rdeps\n") + self._fix_allarch_ml_rdeps() + if self.progress_reporter: self.progress_reporter.next_stage()
@@ -279,6 +283,30 @@ class Rootfs(object, metaclass=ABCMeta): # Remove the package manager data files self.pm.remove_packaging_data() + def _fix_allarch_ml_rdeps(self): + variant = self.d.getVar("BBEXTENDVARIANT") or "" + ml_variants = self.d.getVar("MULTILIB_VARIANTS") or "" + + packages = self.pm.list_installed() + + pkg_to_remove = list() + bb.warn("======= in _fix_allarch_ml_rdeps \n") + for var in ml_variants.split(): + for pkg in (self.d.getVar("ALLARCH_RDEP_PKGS") or "").split(): + ml_pkg = pkg if var == variant else var + "-" + pkg + if ml_pkg in packages: + cur_pkg = variant + "-" + pkg if variant else pkg + if not cur_pkg in packages: + self.pm.install([cur_pkg]) + if self.pm.is_leaf(ml_pkg) \ + and ml_pkg not in (self.d.getVar("IMAGE_INSTALL") or "").split(): + pkg_to_remove.append(ml_pkg) + + bb.note("Following duplicated packages required by all arch packages \ + will be removed: %s" % " ".join(pkg_to_remove)) + self.pm.update() + self.pm.remove(pkg_to_remove) + def _run_ldconfig(self): if self.d.getVar('LDCONFIGDEPEND'): bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
Signature · 1 line
2.17.1
Hi Kai,
Quoted 26 lines · On Thu, 2018-07-26 at 00:41 +0800, [email protected] wrote:
The series rfc patch is trying to install 64bits packages which is required by allarch package to 32 bits image. Work for rpm and need little more work and test for dpkg and opkg.
Take ca-certificates as example. It requires openssl. So it intalls openssl for both core-image-sato and lib32-core-image-sato. But lib32-openssl is expected to be installed to lib32-core-image-sato.
So extend rdpendent of allarch package with all multilib variants and 'noarch-' as well.
Extend
ca-certificates -> opensslto
ca-certificates -> openssl, lib32-openssl, noarch-openssl
Rdepends on openssl and lib32-openssl make both of them exist in oe-core-repo when do rootfs. And then make both openssl and lib32-openssl provide noarch-openssl. And only keep noarch-openssl in RDEPENDS when write package for ca-certificates. When do image rootfs for lib32-core-image-sato, remove openssl if installed already and install lib32-openssl to provide noarch-openssl for allarch pacakge ca-certificates.
Quoted 8 lines
And extend rprovides of allarch package with prefix 'noarch-' too that
it may be required by other allarch packages. Put all dependent packages
of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.Any comment for this method is welcomed. Thanks a lot.
I think this is kind of the right direction but its not ready in this form. Taking a step back and thinking about this problem, its actually similar to the allarch recipes signature problem.
Normally if A DEPENDS on B, the taskhash of A includes the taskhash of B. In the allarch case this is problematic so you see ca-certificates-
Quoted 1 line
openssl listed in
SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.
I think (but am not 100% sure) that everything you list in ALLARCH_RDEP_PKGS is already covered somewhere in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.
I'd therefore propose using SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS instead of adding a ALLARCH_RDEP_PKGS which duplicates information. I can't see any case where you'd want an entry in one but not the other.
Rather than changing the RPROVIDES/RDEPENDS in package_{deb|ipk|rpm}, you should do this once in package.bbclass.
There is a bigger problem though since the allarch recipe signature filtering doesn't change what bitbake will actually build, only how its represented in sstate and the circumstances under which it would rebuild it. If ca-certificates DEPENDS on openssl, it will build openssl whether you're targeting core-image-sato or lib32-core-image- sato. This potentially means you can end up in a situation where it wouldn't build lib32-openssl if nothing else in lib32-core-image-sato depended upon it more directly.
There is no easy way to solve this problem since somehow, if you're building lib32-<image> it needs to build lib32-openssl and if you're building <image> it needs to build openssl. "pushing" that information back up the dependency chain is very hard, we have no such mechanism.
I'm afraid I don't have a solution for this piece, its a very hard problem to solve. Your current patches happen to work as there are other more direct dependencies which tend to ensure the right thing happens but we can't rely on that and it probably isn't the case for all the dependencies.
Finally, the package manager shouldn't really need changes for this approach to work. If things RPROVIDE and RDEPEND on the noarch versions, the feed priorities should mean the right things happen. If that doesn't happen automatically, the feed prorities are not working correctly and that needs fixing, not the 2/2 patch you're proposing. I suspect this issue needs fixing regardless of anything else.
Cheers,
Richard
Quoted 37 lines · On 2018年07月26日 06:47, [email protected] wrote:
Hi Kai,
Quoted 34 lines · On Thu, 2018-07-26 at 00:41 +0800, [email protected] wrote:
The series rfc patch is trying to install 64bits packages which is required by allarch package to 32 bits image. Work for rpm and need little more work and test for dpkg and opkg.
Take ca-certificates as example. It requires openssl. So it intalls openssl for both core-image-sato and lib32-core-image-sato. But lib32-openssl is expected to be installed to lib32-core-image-sato.
So extend rdpendent of allarch package with all multilib variants and 'noarch-' as well.
Extend
ca-certificates -> opensslto
ca-certificates -> openssl, lib32-openssl, noarch-openssl
Rdepends on openssl and lib32-openssl make both of them exist in oe-core-repo when do rootfs. And then make both openssl and lib32-openssl provide noarch-openssl. And only keep noarch-openssl in RDEPENDS when write package for ca-certificates. When do image rootfs for lib32-core-image-sato, remove openssl if installed already and install lib32-openssl to provide noarch-openssl for allarch pacakge ca-certificates.
And extend rprovides of allarch package with prefix 'noarch-' too that
it may be required by other allarch packages. Put all dependent packages
of all allarch recipes in oe-core to a new ariable 'ALLARCH_RDEP_PKGS'.Any comment for this method is welcomed. Thanks a lot.
Hi Richard,
Thanks for detailed reply.
Quoted 6 lines
I think this is kind of the right direction but its not ready in this form. Taking a step back and thinking about this problem, its actually similar to the allarch recipes signature problem.
Normally if A DEPENDS on B, the taskhash of A includes the taskhash of B. In the allarch case this is problematic so you see ca-certificates-
Quoted 8 lines
Quoted 1 line
openssl listed in
SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.I think (but am not 100% sure) that everything you list in
ALLARCH_RDEP_PKGSis already covered somewhere inSIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.I'd therefore propose using
SIGGEN_EXCLUDE_SAFE_RECIPE_DEPSinstead of adding aALLARCH_RDEP_PKGSwhich duplicates information. I can't see any case where you'd want an entry in one but not the other.
Yes, it seems ALLARCH_RDEP_PKGS and I'll try to use SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS instead.
Quoted 3 lines
Rather than changing the RPROVIDES/RDEPENDS in package_{deb|ipk|rpm}, you should do this once in package.bbclass.
ok.
Quoted 9 lines
There is a bigger problem though since the allarch recipe signature filtering doesn't change what bitbake will actually build, only how its represented in sstate and the circumstances under which it would rebuild it. If ca-certificates DEPENDS on openssl, it will build openssl whether you're targeting core-image-sato or lib32-core-image- sato. This potentially means you can end up in a situation where it wouldn't build lib32-openssl if nothing else in lib32-core-image-sato depended upon it more directly.
That's why extend rdepends of allarch package with all multilib variant. In bitbake inner environment, make ca-certificates depends on openssl, lib32-openssl and even libn32-openssl for qemumips64. It may cost a little more build time(I didn't test yet), but the dependency relationship is fixed and make sure that all multilib version of openssl exist in 'oe-core-repo' which is used for do_rootfs.
When build ca-certificates, both openssl and lib32-openssl will be built. And when write package of ca-certificates, remove openssl and lib32-openssl from RDEPENDS and keep noarch-openssl. And make both openssl and lib32-openssl provides noarch-openssl. Then some tweak according image multilib type could be done at then end of do_rootfs.
Quoted 5 lines
There is no easy way to solve this problem since somehow, if you're building lib32-<image> it needs to build lib32-openssl and if you're building <image> it needs to build openssl. "pushing" that information back up the dependency chain is very hard, we have no such mechanism.
I once want to set 'PREFERRED_PROVIDER_noarch-openssl' in image.bbclass, and finally find that the setting only exists in d of <image>, and can't pass to outer context of 'defaultpkgname'
Quoted 7 lines
I'm afraid I don't have a solution for this piece, its a very hard problem to solve. Your current patches happen to work as there are other more direct dependencies which tend to ensure the right thing happens but we can't rely on that and it probably isn't the case for all the dependencies.
Quoted 6 lines
Finally, the package manager shouldn't really need changes for this approach to work. If things RPROVIDE and RDEPEND on the noarch versions, the feed priorities should mean the right things happen. If that doesn't happen automatically, the feed prorities are not working correctly and that needs fixing, not the 2/2 patch you're proposing. I suspect this issue needs fixing regardless of anything else.
Make feed repo with different priority once was an option. But I selected this way first. I'll work on it.
Regards,
Kai
Quoted 5 lines
Cheers,
Richard
Signature · 2 lines
Regards, Neil | Kai Kang
Quoted 21 lines · Kang Kai
Quoted 11 lines · On 2018年07月26日 06:47, [email protected] wrote:
There is a bigger problem though since the allarch recipe signature filtering doesn't change what bitbake will actually build, only how its
represented in sstate and the circumstances under which it would rebuild it. If ca-certificates DEPENDS on openssl, it will build openssl whether you're targeting core-image-sato or lib32-core- image-
sato. This potentially means you can end up in a situation where it wouldn't build lib32-openssl if nothing else in lib32-core-image- sato
depended upon it more directly.That's why extend rdepends of allarch package with all multilib variant.
In bitbake inner environment, make ca-certificates depends on openssl, lib32-openssl and even libn32-openssl for qemumips64. It may cost a little more build time(I didn't test yet), but the dependency relationship is fixed and make sure that all multilib version of openssl exist in 'oe-core-repo' which is used fordo_rootfs.
Quoted 6 lines
When build ca-certificates, both openssl and lib32-openssl will be built. And when write package of ca-certificates, remove openssl and lib32-openssl from RDEPENDS and keep noarch-openssl.
And make both openssl and lib32-openssl provides noarch-openssl. Then some tweak according image multilib type could be done at then end ofdo_rootfs.
Requiring that all variants of openssl build in order to build ca- certificates will have huge build time impact as it will need to build both multilib toolchains.
At that point we'd probably be better either building allarch recipes twice (which I don't like), or, banning non-allarch dependencies from allarch recipes. If a recipe needs a non-allarch dependency we'd make it non-allarch. We may just have to accept the latter...
Cheers,
Richard
FWIW: the later is what I did in morty based build to get multilib working with allarch.
It was actually combination of both, in some cases the dependency was already pulled by something else and it was relatively safe to drop from allarch recipe (e.g. I did that for ca-certificates), in some cases I've just dropped allarch.
I also don't like building things twice, we have allarch web apps which have runtime dependency on web engine (which includes chromium build) and I really don't want to build chromium twice just to satisfy allarch dependencies, when we really want to have just one chromium in the rootfs in the end. So it's not only the multilib toochain and small things like openssl, but it can get huge. Building small allarch web app multiple times (per TUNE_PKGARCH) on the other hand is relatively small price to pay.
Quoted 48 lines · On Thu, Jul 26, 2018 at 11:07 AM <[email protected]> wrote:
Quoted 21 lines · Kang Kai
Quoted 11 lines · On 2018年07月26日 06:47, [email protected] wrote:
There is a bigger problem though since the allarch recipe signature filtering doesn't change what bitbake will actually build, only how its
represented in sstate and the circumstances under which it would rebuild it. If ca-certificates DEPENDS on openssl, it will build openssl whether you're targeting core-image-sato or lib32-core- image-
sato. This potentially means you can end up in a situation where it wouldn't build lib32-openssl if nothing else in lib32-core-image- sato
depended upon it more directly.That's why extend rdepends of allarch package with all multilib variant.
In bitbake inner environment, make ca-certificates depends on openssl, lib32-openssl and even libn32-openssl for qemumips64. It may cost a little more build time(I didn't test yet), but the dependency relationship is fixed and make sure that all multilib version of openssl exist in 'oe-core-repo' which is used fordo_rootfs.Quoted 6 lines
When build ca-certificates, both openssl and lib32-openssl will be built. And when write package of ca-certificates, remove openssl and lib32-openssl from RDEPENDS and keep noarch-openssl.
And make both openssl and lib32-openssl provides noarch-openssl. Then some tweak according image multilib type could be done at then end ofdo_rootfs.Requiring that all variants of openssl build in order to build ca- certificates will have huge build time impact as it will need to build both multilib toolchains.
At that point we'd probably be better either building allarch recipes twice (which I don't like), or, banning non-allarch dependencies from allarch recipes. If a recipe needs a non-allarch dependency we'd make it non-allarch. We may just have to accept the latter...
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
2018-07-26 11:07 GMT+02:00 <[email protected]>:
Quoted 8 lines
Requiring that all variants of openssl build in order to build ca- certificates will have huge build time impact as it will need to build both multilib toolchains.
At that point we'd probably be better either building allarch recipes twice (which I don't like), or, banning non-allarch dependencies from allarch recipes. If a recipe needs a non-allarch dependency we'd make it non-allarch. We may just have to accept the latter...
Another issue is that whatever solution ends up being merged it needs to be understandable by a human. I remember I ripped out a large chunk multilib-related renaming of packages (probably dealing with the same issue) when transitioning from rpm5 to rpm4, and the reason was, essentially, that I was not able to understand what the code is doing and why.
Alex
Quoted 29 lines · On 7/26/18 4:07 AM, [email protected] wrote:
Quoted 21 lines · Kang Kai
Quoted 11 lines · On 2018年07月26日 06:47, [email protected] wrote:
There is a bigger problem though since the allarch recipe signature filtering doesn't change what bitbake will actually build, only how its
represented in sstate and the circumstances under which it would rebuild it. If ca-certificates DEPENDS on openssl, it will build openssl whether you're targeting core-image-sato or lib32-core- image-
sato. This potentially means you can end up in a situation where it wouldn't build lib32-openssl if nothing else in lib32-core-image- sato
depended upon it more directly.That's why extend rdepends of allarch package with all multilib variant.
In bitbake inner environment, make ca-certificates depends on openssl, lib32-openssl and even libn32-openssl for qemumips64. It may cost a little more build time(I didn't test yet), but the dependency relationship is fixed and make sure that all multilib version of openssl exist in 'oe-core-repo' which is used fordo_rootfs.Quoted 6 lines
When build ca-certificates, both openssl and lib32-openssl will be built. And when write package of ca-certificates, remove openssl and lib32-openssl from RDEPENDS and keep noarch-openssl.
And make both openssl and lib32-openssl provides noarch-openssl. Then some tweak according image multilib type could be done at then end ofdo_rootfs.
Quoted 8 lines
Requiring that all variants of openssl build in order to build ca- certificates will have huge build time impact as it will need to build both multilib toolchains.
At that point we'd probably be better either building allarch recipes twice (which I don't like), or, banning non-allarch dependencies from allarch recipes. If a recipe needs a non-allarch dependency we'd make it non-allarch. We may just have to accept the latter...
allarch requiring (runtime) dependencies on non-allarch might just have to be banned. I hadn't considered that -- but it would resolve most of the problems that are being worked on.
If we can produce the allarch package -once- w/o rdepends on anything then they should still be fine.. since the dependency would be:
arch1 |
| -> allarch
arch2 |Avoiding conflicts and preserving the approach we want.
--Mark
Quoted 4 lines
Cheers,
Richard