Skip to contentSkip to conversation

fix populate_sdk for package_deb

PATCH In review Jan Luebbe 2 participants 7 messages Apr 1, 2020 – Apr 3, 2020

lore
Share & tools
lore

Jan Luebbe

author
#0 lore raw

cover fix populate_sdk for package_deb

This series fixes bugs in lib/oe/package_manager which are triggered when using populate_sdk in a distro using PACKAGE_CLASSES = "package_ipk".

The main issue is that apt install will try to remove already installed packages which conflict against those requested in the current install step. Is is triggered here by the coreutils conflict in target-sdk-provides-dummy, which provides perl (amongst others). As apt and dpkg depend on perl, this causes apt to try to remove itself, failing at the essential package check. As this happens during the complementary package installation, which is allowed to fail with a warning, this leads to an SDK which is missing the -dev/-dbg packages.

This issue also affects zeus, so I'd be grateful if they could be applied there as well, to avoid having to maintain a modified OE-core locally. I've tested them on master and zeus.

Jan Luebbe (5):
lib/oe/package_manager: make sure to not remove packages in apt
install
lib/oe/package_manager: fix handling of last package lib/oe/package_manager: collect provided package names when using debs lib/oe/package_manager: avoid installing provided packages via apt lib/oe/package_manager: don't try to rm /var/lib/opkg

 meta/lib/oe/package_manager.py | 37 +++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 14 deletions(-)
Signature · 1 line
2.26.0.rc2
author
#1 lore raw

PATCH 1/5 lib/oe/package_manager: make sure to not remove packages in apt install

apt install can decide to remove already installed packages if there are conflicts. Avoid this by explicitly specifying --no-remove. This will then cause a "E: Packages need to be removed but remove is disabled." message.

---
 meta/lib/oe/package_manager.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/package_manager.py +1 -1
index 2ea8046332af..dfe4197ce1fb 100644
@@ -1624,7 +1624,7 @@ class DpkgPM(OpkgDpkgPM):
 
         os.environ['APT_CONFIG'] = self.apt_conf_file
 
-        cmd = "%s %s install --force-yes --allow-unauthenticated %s" % \
+        cmd = "%s %s install --force-yes --allow-unauthenticated --no-remove %s" % \
               (self.apt_get_cmd, self.apt_args, ' '.join(pkgs))
 
         try:
Signature · 1 line
2.26.0.rc2
author
#2 lore raw

PATCH 2/5 lib/oe/package_manager: fix handling of last package

In reply to Jan Luebbe

In commit 7d214b34e11dc57316ed5c1c7747c4601286f6d2, only the code in the for loop was modified to store the pkgarch value. The code used if there was no empty line at the end was not modified.

Instead of fixing the duplicated code, remove it and just make sure that a final empty line is processed.

---
 meta/lib/oe/package_manager.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/lib/oe/package_manager.py +1 -7
index dfe4197ce1fb..072372892732 100644
@@ -41,7 +41,7 @@ def opkg_query(cmd_output):
     filename = ""
     dep = []
     pkgarch = ""
-    for line in cmd_output.splitlines():
+    for line in cmd_output.splitlines()+['']:
         line = line.rstrip()
         if ':' in line:
             if line.startswith("Package: "):
@@ -80,12 +80,6 @@ def opkg_query(cmd_output):
             dep = []
             pkgarch = ""
 
-    if pkg:
-        if not filename:
-            filename = "%s_%s_%s.ipk" % (pkg, ver, arch)
-        output[pkg] = {"arch":arch, "ver":ver,
-                "filename":filename, "deps": dep }
-
     return output
 
 def failed_postinsts_abort(pkgs, log_path):
Signature · 1 line
2.26.0.rc2
author
#3 lore raw

PATCH 3/5 lib/oe/package_manager: collect provided package names when using debs

In reply to Jan Luebbe

This is needed for a later change to avoid installing packages which are already provided by an installed package.

---
 meta/lib/oe/package_manager.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +8 -2
index 072372892732..e862915b1b75 100644
@@ -40,6 +40,7 @@ def opkg_query(cmd_output):
     ver = ""
     filename = ""
     dep = []
+    prov = []
     pkgarch = ""
     for line in cmd_output.splitlines()+['']:
         line = line.rstrip()
@@ -64,6 +65,10 @@ def opkg_query(cmd_output):
                     dep.append("%s [REC]" % recommend)
             elif line.startswith("PackageArch: "):
                 pkgarch = line.split(": ")[1]
+            elif line.startswith("Provides: "):
+                provides = verregex.sub('', line.split(": ")[1])
+                for provide in provides.split(", "):
+                    prov.append(provide)
 
         # When there is a blank line save the package information
         elif not line:
@@ -72,12 +77,13 @@ def opkg_query(cmd_output):
                 filename = "%s_%s_%s.ipk" % (pkg, ver, arch)
             if pkg:
                 output[pkg] = {"arch":arch, "ver":ver,
-                        "filename":filename, "deps": dep, "pkgarch":pkgarch }
+                        "filename":filename, "deps": dep, "pkgarch":pkgarch, "provs": prov}
             pkg = ""
             arch = ""
             ver = ""
             filename = ""
             dep = []
+            prov = []
             pkgarch = ""
 
     return output
@@ -355,7 +361,7 @@ class DpkgPkgsList(PkgsList):
                "--admindir=%s/var/lib/dpkg" % self.rootfs_dir,
                "-W"]
 
-        cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\n\n")
+        cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\nProvides: ${Provides}\n\n")
 
         try:
             cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip().decode("utf-8")
Signature · 1 line
2.26.0.rc2
author
#4 lore raw

PATCH 4/5 lib/oe/package_manager: avoid installing provided packages via apt

In reply to Jan Luebbe

If there already is a package providing (and conflicting against) packages what should be installed, apt will try remove the conflicting package (target-sdk-provides-dummy) and any that depend on it (like apt and dpkg). This usually fails because of the protection of essential packages. In that case, no -dev/-dbg packages are installed to the SDK.

Avoid this problem by checking which packages are already provided and removing them from the list to be installed. Also sort the list to make it easier to read when debugging.

---
 meta/lib/oe/package_manager.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +12 -2
index e862915b1b75..2a5d7ef539e6 100644
@@ -581,6 +581,11 @@ class PackageManager(object, metaclass=ABCMeta):
         # oe-pkgdata-util reads it from a file
         with tempfile.NamedTemporaryFile(mode="w+", prefix="installed-pkgs") as installed_pkgs:
             pkgs = self.list_installed()
+
+            provided_pkgs = set()
+            for pkg in pkgs.values():
+                provided_pkgs |= set(pkg.get('provs', []))
+
             output = oe.utils.format_pkg_list(pkgs, "arch")
             installed_pkgs.write(output)
             installed_pkgs.flush()
@@ -592,10 +597,15 @@ class PackageManager(object, metaclass=ABCMeta):
             if exclude:
                 cmd.extend(['--exclude=' + '|'.join(exclude.split())])
             try:
-                bb.note("Installing complementary packages ...")
                 bb.note('Running %s' % cmd)
                 complementary_pkgs = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8")
-                self.install(complementary_pkgs.split(), attempt_only=True)
+                complementary_pkgs = set(complementary_pkgs.split())
+                skip_pkgs = sorted(complementary_pkgs & provided_pkgs)
+                install_pkgs = sorted(complementary_pkgs - provided_pkgs)
+                bb.note("Installing complementary packages ... %s (skipped already provided packages %s)" % (
+                    ' '.join(install_pkgs),
+                    ' '.join(skip_pkgs)))
+                self.install(install_pkgs, attempt_only=True)
             except subprocess.CalledProcessError as e:
                 bb.fatal("Could not compute complementary packages list. Command "
                          "'%s' returned %d:\n%s" %
Signature · 1 line
2.26.0.rc2
author
#5 lore raw

PATCH 5/5 lib/oe/package_manager: don't try to rm /var/lib/opkg

In reply to Jan Luebbe

As opkglibdir starts with a /, os.path.join will ignore self.target_rootfs, leading to an attempt to remove /var/lib/opkg.

This only fails if it exists on the host, explaining why this remained undiscovered for long.

---
 meta/lib/oe/package_manager.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oe/package_manager.py +1 -2
index 2a5d7ef539e6..b0660411eaf5 100644
@@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM):
             open(os.path.join(target_dpkg_dir, "available"), "w+").close()
 
     def remove_packaging_data(self):
-        bb.utils.remove(os.path.join(self.target_rootfs,
-                                     self.d.getVar('opkglibdir')), True)
+        bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True)
         bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True)
 
     def fix_broken_dependencies(self):
Signature · 1 line
2.26.0.rc2
#6 lore raw

Re: [OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm /var/lib/opkg

Quoted 26 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Jan Luebbe
Sent: den 1 april 2020 16:58
To: [email protected]
Cc: Jan Luebbe <[email protected]>
Subject: [OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm /var/lib/opkg

As opkglibdir starts with a /, os.path.join will ignore self.target_rootfs, leading to an attempt to remove /var/lib/opkg.

This only fails if it exists on the host, explaining why this remained undiscovered for long.

---
 meta/lib/oe/package_manager.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oe/package_manager.py +0 -0
index 2a5d7ef539e6..b0660411eaf5 100644

meta/lib/oe/package_manager.py class DpkgPM(OpkgDpkgPM): · line 1799

Quoted 6 lines
meta/lib/oe/package_manager.py +1 -2
@@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM):
             open(os.path.join(target_dpkg_dir, "available"), "w+").close()
 
     def remove_packaging_data(self):
-        bb.utils.remove(os.path.join(self.target_rootfs,
-                                     self.d.getVar('opkglibdir')), True)
+        bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True)

Or you could change it to use oe.path.join() instead:

        bb.utils.remove(oe.path.join(self.target_rootfs,
                                     self.d.getVar('opkglibdir')), True)
Quoted 5 lines
meta/lib/oe/package_manager.py +0 -1
         bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True)
 
     def fix_broken_dependencies(self):
--

2.26.0.rc2

//Peter