fix populate_sdk for package_deb
cover fix populate_sdk for package_deb
This series fixes bugs in lib/oe/package_manager which are triggered when using populate_sdk in a distro using PACKAGE_CLASSES = "package_ipk".
The main issue is that apt install will try to remove already installed packages which conflict against those requested in the current install step. Is is triggered here by the coreutils conflict in target-sdk-provides-dummy, which provides perl (amongst others). As apt and dpkg depend on perl, this causes apt to try to remove itself, failing at the essential package check. As this happens during the complementary package installation, which is allowed to fail with a warning, this leads to an SDK which is missing the -dev/-dbg packages.
This issue also affects zeus, so I'd be grateful if they could be applied there as well, to avoid having to maintain a modified OE-core locally. I've tested them on master and zeus.
Jan Luebbe (5):
lib/oe/package_manager: make sure to not remove packages in apt
install
lib/oe/package_manager: fix handling of last package lib/oe/package_manager: collect provided package names when using debs lib/oe/package_manager: avoid installing provided packages via apt lib/oe/package_manager: don't try to rm /var/lib/opkg
meta/lib/oe/package_manager.py | 37 +++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 14 deletions(-)
Signature · 1 line
2.26.0.rc2
PATCH 1/5 lib/oe/package_manager: make sure to not remove packages in apt install
apt install can decide to remove already installed packages if there are conflicts. Avoid this by explicitly specifying --no-remove. This will then cause a "E: Packages need to be removed but remove is disabled." message.
--- meta/lib/oe/package_manager.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/package_manager.py +1 -1
@@ -1624,7 +1624,7 @@ class DpkgPM(OpkgDpkgPM): os.environ['APT_CONFIG'] = self.apt_conf_file - cmd = "%s %s install --force-yes --allow-unauthenticated %s" % \ + cmd = "%s %s install --force-yes --allow-unauthenticated --no-remove %s" % \ (self.apt_get_cmd, self.apt_args, ' '.join(pkgs)) try:
Signature · 1 line
2.26.0.rc2
PATCH 2/5 lib/oe/package_manager: fix handling of last package
In reply to Jan LuebbeIn commit 7d214b34e11dc57316ed5c1c7747c4601286f6d2, only the code in the for loop was modified to store the pkgarch value. The code used if there was no empty line at the end was not modified.
Instead of fixing the duplicated code, remove it and just make sure that a final empty line is processed.
--- meta/lib/oe/package_manager.py | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/lib/oe/package_manager.py +1 -7
@@ -41,7 +41,7 @@ def opkg_query(cmd_output): filename = "" dep = [] pkgarch = "" - for line in cmd_output.splitlines(): + for line in cmd_output.splitlines()+['']: line = line.rstrip() if ':' in line: if line.startswith("Package: "):
@@ -80,12 +80,6 @@ def opkg_query(cmd_output): dep = [] pkgarch = "" - if pkg: - if not filename: - filename = "%s_%s_%s.ipk" % (pkg, ver, arch) - output[pkg] = {"arch":arch, "ver":ver, - "filename":filename, "deps": dep } - return output def failed_postinsts_abort(pkgs, log_path):
Signature · 1 line
2.26.0.rc2
PATCH 3/5 lib/oe/package_manager: collect provided package names when using debs
In reply to Jan LuebbeThis is needed for a later change to avoid installing packages which are already provided by an installed package.
--- meta/lib/oe/package_manager.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +8 -2
@@ -40,6 +40,7 @@ def opkg_query(cmd_output): ver = "" filename = "" dep = [] + prov = [] pkgarch = "" for line in cmd_output.splitlines()+['']: line = line.rstrip()
@@ -64,6 +65,10 @@ def opkg_query(cmd_output): dep.append("%s [REC]" % recommend) elif line.startswith("PackageArch: "): pkgarch = line.split(": ")[1] + elif line.startswith("Provides: "): + provides = verregex.sub('', line.split(": ")[1]) + for provide in provides.split(", "): + prov.append(provide) # When there is a blank line save the package information elif not line:
@@ -72,12 +77,13 @@ def opkg_query(cmd_output): filename = "%s_%s_%s.ipk" % (pkg, ver, arch) if pkg: output[pkg] = {"arch":arch, "ver":ver, - "filename":filename, "deps": dep, "pkgarch":pkgarch } + "filename":filename, "deps": dep, "pkgarch":pkgarch, "provs": prov} pkg = "" arch = "" ver = "" filename = "" dep = [] + prov = [] pkgarch = "" return output
@@ -355,7 +361,7 @@ class DpkgPkgsList(PkgsList): "--admindir=%s/var/lib/dpkg" % self.rootfs_dir, "-W"] - cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\n\n") + cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\nProvides: ${Provides}\n\n") try: cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip().decode("utf-8")
Signature · 1 line
2.26.0.rc2
PATCH 4/5 lib/oe/package_manager: avoid installing provided packages via apt
In reply to Jan LuebbeIf there already is a package providing (and conflicting against) packages what should be installed, apt will try remove the conflicting package (target-sdk-provides-dummy) and any that depend on it (like apt and dpkg). This usually fails because of the protection of essential packages. In that case, no -dev/-dbg packages are installed to the SDK.
Avoid this problem by checking which packages are already provided and removing them from the list to be installed. Also sort the list to make it easier to read when debugging.
--- meta/lib/oe/package_manager.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +12 -2
@@ -581,6 +581,11 @@ class PackageManager(object, metaclass=ABCMeta): # oe-pkgdata-util reads it from a file with tempfile.NamedTemporaryFile(mode="w+", prefix="installed-pkgs") as installed_pkgs: pkgs = self.list_installed() + + provided_pkgs = set() + for pkg in pkgs.values(): + provided_pkgs |= set(pkg.get('provs', [])) + output = oe.utils.format_pkg_list(pkgs, "arch") installed_pkgs.write(output) installed_pkgs.flush()
@@ -592,10 +597,15 @@ class PackageManager(object, metaclass=ABCMeta): if exclude: cmd.extend(['--exclude=' + '|'.join(exclude.split())]) try: - bb.note("Installing complementary packages ...") bb.note('Running %s' % cmd) complementary_pkgs = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8") - self.install(complementary_pkgs.split(), attempt_only=True) + complementary_pkgs = set(complementary_pkgs.split()) + skip_pkgs = sorted(complementary_pkgs & provided_pkgs) + install_pkgs = sorted(complementary_pkgs - provided_pkgs) + bb.note("Installing complementary packages ... %s (skipped already provided packages %s)" % ( + ' '.join(install_pkgs), + ' '.join(skip_pkgs))) + self.install(install_pkgs, attempt_only=True) except subprocess.CalledProcessError as e: bb.fatal("Could not compute complementary packages list. Command " "'%s' returned %d:\n%s" %
Signature · 1 line
2.26.0.rc2
PATCH 5/5 lib/oe/package_manager: don't try to rm /var/lib/opkg
In reply to Jan LuebbeAs opkglibdir starts with a /, os.path.join will ignore self.target_rootfs, leading to an attempt to remove /var/lib/opkg.
This only fails if it exists on the host, explaining why this remained undiscovered for long.
--- meta/lib/oe/package_manager.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oe/package_manager.py +1 -2
@@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM): open(os.path.join(target_dpkg_dir, "available"), "w+").close() def remove_packaging_data(self): - bb.utils.remove(os.path.join(self.target_rootfs, - self.d.getVar('opkglibdir')), True) + bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True) bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True) def fix_broken_dependencies(self):
Signature · 1 line
2.26.0.rc2
Re: [OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm /var/lib/opkg
Quoted 26 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Jan Luebbe
Sent: den 1 april 2020 16:58
To: [email protected]
Cc: Jan Luebbe <[email protected]>
Subject: [OE-core][PATCH 5/5]lib/oe/package_manager: don't try to rm/var/lib/opkgAs opkglibdir starts with a /, os.path.join will ignore self.target_rootfs, leading to an attempt to remove
/var/lib/opkg.This only fails if it exists on the host, explaining why this remained undiscovered for long.
Signed-off-by Jan Luebbe <[email protected]>--- meta/lib/oe/package_manager.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)meta/lib/oe/package_manager.py +0
-0
meta/lib/oe/package_manager.py class DpkgPM(OpkgDpkgPM): · line 1799
Quoted 6 lines
meta/lib/oe/package_manager.py +1
-2@@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM): open(os.path.join(target_dpkg_dir, "available"), "w+").close() def remove_packaging_data(self): - bb.utils.remove(os.path.join(self.target_rootfs, - self.d.getVar('opkglibdir')), True) + bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True)
Or you could change it to use oe.path.join() instead:
bb.utils.remove(oe.path.join(self.target_rootfs,
self.d.getVar('opkglibdir')), True)Quoted 5 lines
meta/lib/oe/package_manager.py +0
-1bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True) def fix_broken_dependencies(self): --2.26.0.rc2
//Peter