package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

While a pkg name (such as python3) not existed in variable PACKAGES, but provided by another pkg (such as python-core), in this situation, rpm based do_rootfs could not search it in the feeds.

The fix is to invoke 'smart query --provides' to do the search if the feeds search failed. We don't drop the feeds search, because the smart query search is too slow, and the feeds search is fast and could work well with most pkgs.

[YOCTO #6918]

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

meta/lib/oe/package_manager.py +16 -0

index ffb83b2..a0ecf3a 100644
@@ -671,6 +671,22 @@ class RpmPM(PackageManager):
 
         return ""
 
+    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
+        output = self._invoke_smart('query  --provides %s' % pkg) or ''
+        for provider in output.split('\n'):
+            # Filter out the line which is empty or start with space
+            if provider.strip() == '' or provider[0] == ' ':
+                continue
+
+            for arch in feed_archs:
+                arch = '@' + arch.replace('-', '_')
+                if arch in provider:
+                    # First found is best match
+                    return provider
+
+        return ''
+
+
     '''
     Translate the OE multilib format names to the RPM/Smart format names
     It searched the RPM/Smart format names in probable multilib feeds

first, @@ -691,7 +707,8 @@ class RpmPM(PackageManager):

                 # if the pkg in this multilib feed
                 if subst != pkg:
                     feed_archs = self.ml_prefix_list[mlib]
-                    new_pkg = self._search_pkg_name_in_feeds(subst,
feed_archs) +                    new_pkg =
self._search_pkg_name_in_feeds(subst, feed_archs) or \ +                   
    self._search_pkg_provider_in_smart(subst, feed_archs) if not new_pkg:
                         # Failed to translate, package not found!
                         err_msg = '%s not found in the %s feeds (%s).\n' %
\ @@ -709,7 +726,8 @@ class RpmPM(PackageManager):
             if pkg == new_pkg:

                 # Search new_pkg in default archs
                 default_archs = self.ml_prefix_list['default']
-                new_pkg = self._search_pkg_name_in_feeds(pkg,
default_archs) +                new_pkg =
self._search_pkg_name_in_feeds(pkg, default_archs) or \ +                  
 self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:
                     err_msg = '%s not found in the base feeds (%s).\n' % \
                               (pkg, ' '.join(default_archs))

Quoted 5 lines

default_archs) + new_pkg =

Quoted 4 lines

self._search_pkg_name_in_feeds(pkg, default_archs) or \ +
self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:

                     err_msg = '%s not found in the base feeds (%s).\n' % \
                               (pkg, ' '.join(default_archs))

It took a long time to reply to this, my apologies. This patch looks OK but Richard pointed out that it may have an impact on the speed of do_rootfs because of the extra calls out to smart. Do you have any performance impact figures?

Thanks,
Paul

Quoted 9 lines · Paul Eggleton

Quoted 5 lines

default_archs) + new_pkg =

Quoted 4 lines

self._search_pkg_name_in_feeds(pkg, default_archs) or \ +
self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:

                     err_msg = '%s not found in the base feeds (%s).\n' % \
                               (pkg, ' '.join(default_archs))

It took a long time to reply to this, my apologies. This patch looks OK but Richard pointed out that it may have an impact on the speed of do_rootfs because of the extra calls out to smart. Do you have any performance impact figures?

Hi Paul,

The extra 'smart' calling takes 3-5 seconds for one search, but it will not do that for most packages, it goes into self._search_pkg_name_in_feeds first, if not found and then in to self._search_pkg_provider_in_smart, only if we have such as python3 in the build session, it will take extra 3-5 seconds.

Changed in V2:
- Remove debug commented out line;
- Tweak the way of function invoking;

vim local.conf
...
IMAGE_INSTALL_append " python3"
...

bitbake core-image-minimal

//Hongxu

The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:

   nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs

Hongxu Jia (1):
package_manager.py: fix rpm based do_rootfs failed while
IMAGE_INSTALL_append = " python3"

  meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)

Changed in V2:
- Remove debug commented out line;
- Tweak the way of function invoking;

vim local.conf
...
IMAGE_INSTALL_append " python3"
...

bitbake core-image-minimal

//Hongxu

The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:

   nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs

Hongxu Jia (1):
package_manager.py: fix rpm based do_rootfs failed while
IMAGE_INSTALL_append = " python3"

  meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)

Changed in V2:
- Remove debug commented out line;
- Tweak the way of function invoking;

vim local.conf
...
IMAGE_INSTALL_append " python3"
...

bitbake core-image-minimal

//Hongxu

The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:

   nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs

Hongxu Jia (1):
package_manager.py: fix rpm based do_rootfs failed while
IMAGE_INSTALL_append = " python3"

  meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)