Skip to contentSkip to conversation

python-smartpm-native: prefer same arch when install

PATCH In review Robert Yang 3 participants 5 messages Oct 27, 2015 – Oct 28, 2015

lore
Share & tools
lore
#0 lore raw

cover python-smartpm-native: prefer same arch when install

The following changes since commit f787b688f2884ce3fa888b4041030538c7d2bf55:

oeqa/utils/decorators: fix missing keyword arguments on decorators (2015-10-27 07:22:22 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/smart
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/smart

Robert Yang (1):
python-smartpm-native: prefer same arch when install

 .../python-smartpm/smart-prefer-same-arch.patch    |  119 ++++++++++++++++++++
 meta/recipes-devtools/python/python-smartpm_git.bb |    2 +
 2 files changed, 121 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/1 python-smartpm-native: prefer same arch when install

We had made smart install multilib RDEPENDS correctly from package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix the issue from python-smartpm-native, and make it work well.

The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use pkg_A's arch when possible.

This patch fixed:
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
RPM_PREFER_ELF_ARCH = "1"
IMAGE_INSTALL_append = " lib32-connman-gnome"
MACHINE = "qemux86-64"

$ bitbake core-image-sato

Only 64bit loaders like libgdk-pixbuf-2.0-loader-jpeg and libgdk-pixbuf-2.0-loader-png were installed before this patch since they are in RRECOMMANDS, now both 32bit and 64bit are installed.

This patch is for native only to minimize the impact for this release, the main problem is the "if attempt:", otherwise, more packages like lib32-libc6-dev will be installed when populate_sdk, but only libc6-dev were installed before. We need reconstruct the way that we install the complentary packages in future release, the "if attempt:" make they work together atm.

[YOCTO #8570]

---
 .../python-smartpm/smart-prefer-same-arch.patch    |  119 ++++++++++++++++++++
 meta/recipes-devtools/python/python-smartpm_git.bb |    2 +
 2 files changed, 121 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch
meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch new file +119 -0
new file mode 100644
index 0000000..b1305b5
@@ -0,0 +1,119 @@
+From 1749592a76425449c1f3119fb14db3f5e14226ed Mon Sep 17 00:00:00 2001
+From: Robert Yang <[email protected]>
+Date: Tue, 27 Oct 2015 06:20:45 -0700
+Subject: [PATCH] smart/transaction.py: prefer same arch when install
+
+We had made smart install multilib RDEPENDS correctly from
+package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
+the issue from python-smartpm-native, and make it work well.
+
+The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
+pkg_A's arch when possible.
+
+This patch fixed:
+require conf/multilib.conf
+MULTILIBS = "multilib:lib32"
+DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
+RPM_PREFER_ELF_ARCH = "1"
+IMAGE_INSTALL_append = " lib32-connman-gnome"
+MACHINE = "qemux86-64"
+
+$ bitbake core-image-sato
+
+Only 64bit loaders like libgdk-pixbuf-2.0-loader-jpeg and
+libgdk-pixbuf-2.0-loader-png were installed before this patch since they
+are in RRECOMMANDS, now both 32bit and 64bit are installed.
+
+This patch is for native only to minimize the impact for this release,
+the main problem is the "if attempt:", otherwise, more packages like
+lib32-libc6-dev will be installed when populate_sdk, but only libc6-dev
+were installed before. We need reconstruct the way that we install the
+complentary packages in future release, the "if attempt:" make they work
+together atm.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Robert Yang <[email protected]>
+---
+ smart/transaction.py |   28 +++++++++++++++++++++++++---
+ 1 file changed, 25 insertions(+), 3 deletions(-)
+
+diff --git a/smart/transaction.py b/smart/transaction.py
+index f022aae..63a837c 100644
+--- a/smart/transaction.py
++++ b/smart/transaction.py
+@@ -22,6 +22,7 @@
+ from smart.const import INSTALL, REMOVE, UPGRADE, FIX, REINSTALL, KEEP, LOCKED_EXCLUDE, LOCKED_INSTALL, LOCKED_CONFLICT, LOCKED_CONFLICT_BY, LOCKED_NO_COEXIST, LOCKED_SYSCONF, LOCKED_REMOVE
+ from smart.cache import PreRequires, Package
+ from smart import *
++from smart.backends.rpm.rpmver import splitarch
+ 
+ def lock_reason(pkg, lockvalue):
+     try:
+@@ -618,6 +619,7 @@ class Transaction(object):
+                     self._remove(namepkg, changeset, locked, pending, depth)
+ 
+         # Install packages required by this one.
++        pkgarch = splitarch(pkg.version)[1]
+         for req in pkg.requires + pkg.recommends:
+ 
+             reqrequired = req in pkg.requires
+@@ -626,6 +628,8 @@ class Transaction(object):
+             prvpkgs = {}
+             lockedpkgs = {}
+             found = False
++            found_same_arch = False
++            same_arch_in_provides = False
+             for prv in req.providedby:
+                 for prvpkg in prv.packages:
+                     if not reqrequired:
+@@ -633,9 +637,19 @@ class Transaction(object):
+                             continue
+                         elif pkgconf.testFlag("ignore-recommends", prvpkg):
+                             continue
++                    prvpkgarch = splitarch(prvpkg.version)[1]
++                    if pkgarch == prvpkgarch:
++                        # Found a same arch in provides
++                        same_arch_in_provides = True
+                     if isinst(prvpkg):
+                         found = True
+-                        break
++                        if attempt:
++                            break
++                        if pkgarch == prvpkgarch:
++                            # Found the best match
++                            found_same_arch = True
++                            break
++                        continue
+                     if prvpkg not in locked:
+                         prvpkgs[prvpkg] = True
+                     else:
+@@ -643,7 +657,7 @@ class Transaction(object):
+                 else:
+                     continue
+                 break
+-            if found:
++            if found_same_arch or (found and attempt) or (found and not same_arch_in_provides):
+                 # Someone is already providing it. Good.
+                 continue
+ 
+@@ -672,7 +686,15 @@ class Transaction(object):
+ 
+                     # It's only a recommend, skip
+                     continue
+-
++            if len(prvpkgs) > 1 and not attempt:
++                prvarchs = set()
++                for prv in prvpkgs:
++                    prvarchs.add(splitarch(prv.version)[1])
++                if pkgarch in prvarchs:
++                    # Prefer the one which has the same arch
++                    for prv in prvpkgs.copy():
++                        if splitarch(prv.version)[1] != pkgarch:
++                                del prvpkgs[prv]
+             if len(prvpkgs) == 1:
+                 # Don't check locked here. prvpkgs was
+                 # already filtered above.
+-- 
+1.7.9.5
+
meta/recipes-devtools/python/python-smartpm_git.bb +2 -0
index d6c378b..1ebf590 100644
@@ -26,6 +26,8 @@ SRC_URI = "\
           file://smart-cache.py-getPackages-matches-name-version.patch \
          "
 
+SRC_URI_append_class-native = "file://smart-prefer-same-arch.patch"
+
 SRCREV = "407a7eca766431257dcd1da15175cc36a1bb22d0"
 PV = "1.5+git${SRCPV}"
 
Signature · 1 line
1.7.9.5
#2 lore raw
Quoted 7 lines · Robert Yang

We had made smart install multilib RDEPENDS correctly from package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix the issue from python-smartpm-native, and make it work well.

The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use pkg_A's arch when possible.

I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:

Changes to images/intel_corei7_64/glibc/core-image-sato (installed-package-names.txt):
lib32-ncurses-terminfo-base was added lib32-libjpeg9 was added lib32-glibc-gconv-iso8859-1 was added lib32-shared-mime-info was added lib32-libxml2 was added lib32-wpa-supplicant-cli was added lib32-libgdk-pixbuf-2.0-loader-png was added lib32-wpa-supplicant-passphrase was added lib32-pango-module-basic-fc was added lib32-bash was added lib32-libgdk-pixbuf-2.0-loader-jpeg was added lib32-libgdk-pixbuf-2.0-loader-gif was added lib32-libgdk-pixbuf-2.0-loader-xpm was added lib32-glibc-gconv was added

Some of those are good (gdk-pixbuf and pango), some bad (bash, wpa-supplicant-cli), some probably bugs in packages that should be allarch (termino, shared-mime-info).

I'm undecided what we do about this. Fixing the dependency chains is good, but we're definitely pulling in more than expected.

Ross

#3 lore raw
Quoted 31 lines · Ross

On 27 October 2015 at 14:05, Robert Yang <[email protected] <mailto:[email protected]>> wrote:

We had made smart install multilib RDEPENDS correctly from package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix the issue from python-smartpm-native, and make it work well.

The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use pkg_A's arch when possible.

I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:

Changes to images/intel_corei7_64/glibc/core-image-sato (installed-package-names.txt):
lib32-ncurses-terminfo-base was added lib32-libjpeg9 was added lib32-glibc-gconv-iso8859-1 was added lib32-shared-mime-info was added lib32-libxml2 was added lib32-wpa-supplicant-cli was added lib32-libgdk-pixbuf-2.0-loader-png was added lib32-wpa-supplicant-passphrase was added lib32-pango-module-basic-fc was added lib32-bash was added lib32-libgdk-pixbuf-2.0-loader-jpeg was added lib32-libgdk-pixbuf-2.0-loader-gif was added lib32-libgdk-pixbuf-2.0-loader-xpm was added lib32-glibc-gconv was added

Quoted 3 lines

Some of those are good (gdk-pixbuf and pango), some bad (bash, wpa-supplicant-cli), some probably bugs in packages that should be allarch (termino, shared-mime-info).

bash, wpa-supplicant-cli is a good example of why this is an issue. RPM luckily says, well we've got two versions of bash, lets resolve this by following the ELF resolution rule. So while two packages may exist in the manifest, you won't get any conflicts and space is likely not duplicated.

But you can't automatically say "give me any bash, but give me arch specific versions of everything else". The default in RPM is if I ask for 'bash', give me any version of bash.. if I ask for 'bash.i586' I was -the- i586 version.

This is why the fix for this is either adjust the behavior (as Robert has done) to suck in more dependencies and consider the ELF resolver to be adequate -- or to add some kind of a hint to the recipe mechanisms that select dependencies must be of the same arch type. One we handle in the resolver, and one we could handle when we generate the packages themselves.

I'm not sure there is an obvious solution, I prefer the later with a variant of the former myself. (Variant being if all things are equal we weight dependencies to match the arch.. but if something is already in with that provide we use it..) Then the hint would cause the produced packages to be able to say they need explicit versions of the plugins.

Quoted 2 lines

I'm undecided what we do about this. Fixing the dependency chains is good, but we're definitely pulling in more than expected.

I agree, I have a feeling what I wrote about is the right answer... but I don't have any code to prove it.. and I'm not sure that we should be making that big of a change right now.

Robert's patch SHOULD resolve the problem at the expense of a slightly larger resolution manifest, which may be acceptable.

--Mark

Quoted 3 lines

Ross

#4 lore raw
In reply to Burton, Ross
Quoted 31 lines · Ross

On 27 October 2015 at 14:05, Robert Yang <[email protected] <mailto:[email protected]>> wrote:

We had made smart install multilib RDEPENDS correctly from package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix the issue from python-smartpm-native, and make it work well.

The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use pkg_A's arch when possible.

I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:

Changes to images/intel_corei7_64/glibc/core-image-sato (installed-package-names.txt):
lib32-ncurses-terminfo-base was added lib32-libjpeg9 was added lib32-glibc-gconv-iso8859-1 was added lib32-shared-mime-info was added lib32-libxml2 was added lib32-wpa-supplicant-cli was added lib32-libgdk-pixbuf-2.0-loader-png was added lib32-wpa-supplicant-passphrase was added lib32-pango-module-basic-fc was added lib32-bash was added lib32-libgdk-pixbuf-2.0-loader-jpeg was added lib32-libgdk-pixbuf-2.0-loader-gif was added lib32-libgdk-pixbuf-2.0-loader-xpm was added lib32-glibc-gconv was added

Quoted 3 lines

Some of those are good (gdk-pixbuf and pango), some bad (bash, wpa-supplicant-cli), some probably bugs in packages that should be allarch (termino, shared-mime-info).

I think that lacking something so that it doesn't work is worse than have more things but works ?

// Robert

Quoted 5 lines

I'm undecided what we do about this. Fixing the dependency chains is good, but we're definitely pulling in more than expected.

Ross