shlibs providers improvements

PATCH Superseded Martin Jansa 3 participants 14 messages Jan 18, 2014 – Jan 19, 2014

lore

Share & tools

lore

Martin Jansa

author

2014-01-18 14:02 UTC #0 lore raw

cover shlibs providers improvements

Fixes for [YOCTO #4628]

Richard had few more ideas to improve it even more, but this is good start.

Most important changes:
* PRIVATE_LIBS aren't duplicitly added to RDEPENDS (when recipe provides own libfoo it shouldn't get runtime dependency on global libfoo shlib provider)
* warning is shown for undeterministic providers (multiple packages recorded to provide the same library) - recipe maintainers can decide if such library should be considered private or make sure that only one package is built (most common case are gst plugins where 0.10 and 1.* provide the same gst modules, but as long as nobody tries to linke directly to some gst plugin we shouldn't list them as shlib providers).
* list files which require given shlib, often I was surprised to see that something got runtime dependency on libfoo, showing which binary is linked to it makes it easier to debug when you have only log.do_package and package directory is already gone (e.g. analyzing test-dependencies.sh script outputs)

The following changes since commit 8163854adf87ac42a8f08ee25685d0ce1efb4724:

oe-selftest: separated the SStateBase and SStateTests in different modules (2014-01-16 12:18:44 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib jansa/shlib-providers
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/shlib-providers

Martin Jansa (5):
package.bbclass: move reading shlibs providers to separate function package.bbclass: show warning when package is trying to provide
already provided shlib
package.bbclass: add SHLIBSSEARCHDIRS to define where to search for
shlib providers
package.bbclass: Don't search for prividers of PRIVATE_LIBS package.bbclass: Show which files require given dependency in debug
output

 meta/classes/package.bbclass | 102 +++++++++++++++++++++++++++++++------------
 1 file changed, 75 insertions(+), 27 deletions(-)

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-18 14:02 UTC #1 lore raw

PATCH 1/5 package.bbclass: move reading shlibs providers to separate function

* prepare for reading shlibs providers only from dependency tree of current recipe

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 43 +++++++++++++++++++++++--------------------
 1 file changed, 23 insertions(+), 20 deletions(-)

meta/classes/package.bbclass +23 ~~-20~~

index 768047c..7dcec5e 100644
@@ -1352,6 +1352,28 @@ python package_do_shlibs() {
     # Take shared lock since we're only reading, not writing
     lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
 
+    def read_shlib_providers():
+        list_re = re.compile('^(.*)\.list$')
+        # Go from least to most specific since the last one found wins
+        for dir in reversed(shlibs_dirs):
+            if not os.path.exists(dir):
+                continue
+            for file in os.listdir(dir):
+                m = list_re.match(file)
+                if m:
+                    dep_pkg = m.group(1)
+                    fd = open(os.path.join(dir, file))
+                    lines = fd.readlines()
+                    fd.close()
+                    ver_file = os.path.join(dir, dep_pkg + '.ver')
+                    lib_ver = None
+                    if os.path.exists(ver_file):
+                        fd = open(ver_file)
+                        lib_ver = fd.readline().rstrip()
+                        fd.close()
+                    for l in lines:
+                        shlib_provider[l.rstrip()] = (dep_pkg, lib_ver)
+
     def linux_so(file):
         needs_ldconfig = False
         cmd = d.getVar('OBJDUMP', True) + " -p " + pipes.quote(file) + " 2>/dev/null"

@@ -1496,26 +1518,7 @@ python package_do_shlibs() {
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
 
-    list_re = re.compile('^(.*)\.list$')
-    # Go from least to most specific since the last one found wins
-    for dir in reversed(shlibs_dirs):
-        if not os.path.exists(dir):
-            continue
-        for file in os.listdir(dir):
-            m = list_re.match(file)
-            if m:
-                dep_pkg = m.group(1)
-                fd = open(os.path.join(dir, file))
-                lines = fd.readlines()
-                fd.close()
-                ver_file = os.path.join(dir, dep_pkg + '.ver')
-                lib_ver = None
-                if os.path.exists(ver_file):
-                    fd = open(ver_file)
-                    lib_ver = fd.readline().rstrip()
-                    fd.close()
-                for l in lines:
-                    shlib_provider[l.rstrip()] = (dep_pkg, lib_ver)
+    read_shlib_providers()
 
     bb.utils.unlockfile(lf)

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-18 14:02 UTC #2 lore raw

PATCH 2/5 package.bbclass: show warning when package is trying to provide already provided shlib

In reply to Martin Jansa

* move read_shlib_providers before registering package as provider and don't change provider if it already exists, show warning instead

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +10 -2

index 7dcec5e..aa8156d 100644
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):

@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False

@@ -1504,6 +1507,12 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s is already registered as shlib provider for %s, ignoring %s-%s trying to register the same' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                        continue
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()

@@ -1517,8 +1526,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-18 14:02 UTC #9 lore raw

PATCH 3/5 package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

In reply to Martin Jansa

* when package contains some files matching "^.*\.so", but in directory not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so) don't register it as libfoo provider, because it's possible that there is different package providing libfoo.so in ${libdir} and that would be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can define own SHLIBSSEARCHDIRS value

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

meta/classes/package.bbclass +23 -3

index aa8156d..fe25a6b 100644
@@ -1322,6 +1322,9 @@ python package_do_filedeps() {
 SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
 SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
 
+# default search path when searching for shlibs provided by package
+SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
+
 python package_do_shlibs() {
     import re, pipes

@@ -1332,6 +1335,20 @@ python package_do_shlibs() {
 
     lib_re = re.compile("^.*\.so")
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
+    
+    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
+        if shlibs_search_dirs_re_txt:
+            shlibs_search_dirs_re_txt += "|"
+        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)

@@ -1390,9 +1407,12 @@ python package_do_shlibs() {
             if m:
                 this_soname = m.group(1)
                 if not this_soname in sonames:
-                    # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or -1 == private_libs.find(this_soname):
-                        sonames.append(this_soname)
+                    if shlibs_search_dirs_re.match(file):
+                        # if library is private (only used by package) then do not build shlib for it
+                        if not private_libs or -1 == private_libs.find(this_soname):
+                            sonames.append(this_soname)
+                    else:
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-18 14:02 UTC #12 lore raw

PATCH 4/5 package.bbclass: Don't search for prividers of PRIVATE_LIBS

In reply to Martin Jansa

* split PRIVATE_LIBS and don't use find(), so that libfoo cannot be found in PRIVATE_LIBS = "libfoobar"

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +11 -2

index fe25a6b..7820955 100644
@@ -1409,7 +1409,7 @@ python package_do_shlibs() {
                 if not this_soname in sonames:
                     if shlibs_search_dirs_re.match(file):
                         # if library is private (only used by package) then do not build shlib for it
-                        if not private_libs or -1 == private_libs.find(this_soname):
+                        if not private_libs or this_soname not in private_libs:
                             sonames.append(this_soname)
                     else:
                         bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))

@@ -1495,7 +1495,8 @@ python package_do_shlibs() {
     read_shlib_providers()
 
     for pkg in packages.split():
-        private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
+        private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True) or ""
+        private_libs = private_libs.split()
         needs_ldconfig = False
         bb.debug(2, "calculating shlib provides for %s" % pkg)

@@ -1566,6 +1567,14 @@ python package_do_shlibs() {
 
         deps = list()
         for n in needed[pkg]:
+            # if n is in private libraries, don't try to search provider for it
+            # this could cause problem in case some abc.bb provides private
+            # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1
+            # but skipping it is still better alternative than providing own
+            # version and then adding runtime dependency for the same system library
+            if private_libs and n in private_libs:
+                bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n))
+                continue
             if n in shlib_provider.keys():
                 (dep_pkg, ver_needed) = shlib_provider[n]

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-18 14:02 UTC #13 lore raw

PATCH 5/5 package.bbclass: Show which files require given dependency in debug output

In reply to Martin Jansa

* when log.do_package shows some unexpected dependency, people usually need to grep package directory to find which binary was creating that dependency, show it directly in the debug output

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +10 -2

index 7820955..854a120 100644
@@ -1403,6 +1403,9 @@ python package_do_shlibs() {
             if m:
                 if m.group(1) not in needed[pkg]:
                     needed[pkg].append(m.group(1))
+                if m.group(1) not in needed_from:
+                    needed_from[m.group(1)] = []
+                needed_from[m.group(1)].append(file)
             m = re.match("\s+SONAME\s+([^\s]*)", l)
             if m:
                 this_soname = m.group(1)

@@ -1478,6 +1481,10 @@ python package_do_shlibs() {
                                 needed[pkg] = []
                             if name and name not in needed[pkg]:
                                 needed[pkg].append(name)
+                            if name not in needed_from:
+                                needed_from[name] = []
+                            if lafile and lafile not in needed_from[name]:
+                                needed_from[name].append(lafile)
                                 #bb.note("Adding %s for %s" % (name, pkg))
 
     if d.getVar('PACKAGE_SNAP_LIB_SYMLINKS', True) == "1":

@@ -1491,6 +1498,7 @@ python package_do_shlibs() {
         use_ldconfig = False
 
     needed = {}
+    needed_from = {}
     shlib_provider = {}
     read_shlib_providers()

@@ -1578,7 +1586,7 @@ python package_do_shlibs() {
             if n in shlib_provider.keys():
                 (dep_pkg, ver_needed) = shlib_provider[n]
 
-                bb.debug(2, '%s: Dependency %s requires package %s' % (pkg, n, dep_pkg))
+                bb.debug(2, '%s: Dependency %s requires package %s (used by files: %s)' % (pkg, n, dep_pkg, needed_from[n]))
 
                 if dep_pkg == pkg:
                     continue

@@ -1590,7 +1598,7 @@ python package_do_shlibs() {
                 if not dep in deps:
                     deps.append(dep)
             else:
-                bb.note("Couldn't find shared library provider for %s" % n)
+                bb.note("Couldn't find shared library provider for %s, used by files: %s" % (n, needed_from[n]))
 
         deps_file = os.path.join(pkgdest, pkg + ".shlibdeps")
         if os.path.exists(deps_file):

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-19 13:13 UTC #3 lore raw

Re: [PATCH 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

In reply to Martin Jansa

Quoted 32 lines · Martin Jansa

* move read_shlib_providers before registering package as provider and don't change provider if it already exists, show warning instead

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +3 -0

index 7dcec5e..aa8156d 100644
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):

@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False

meta/classes/package.bbclass python package_do_shlibs() { · line 1514

Quoted 8 lines

meta/classes/package.bbclass +5 -0

@@ -1504,6 +1507,12 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s is already registered as shlib provider for %s, ignoring %s-%s trying to register the same' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                        continue

I'll drop continue and update warning message, changing registered provider for last built is still undeterministic but slightly better e.g. when changing packaging and the shlib is moved between packages (I've run into this in efl, when moving to split packages and they were still getting runtime dependency on old efl package).

Quoted 17 lines

meta/classes/package.bbclass +2 -2

+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()

@@ -1517,8 +1526,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)

Signature · 1 line

1.8.5.2

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Martin Jansa

author

2014-01-19 13:28 UTC #4 lore raw

PATCH v2 2/5 package.bbclass: show warning when package is trying to provide already provided shlib

In reply to Martin Jansa

* move read_shlib_providers before registering package as provider and don't change provider if it already exists, show warning instead

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +9 -2

index 7dcec5e..9b511a6 100644
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):

@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False

@@ -1504,6 +1507,11 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()

@@ -1517,8 +1525,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)

Signature · 1 line

1.8.5.2

Phil Blundell

2014-01-19 14:42 UTC #5 lore raw

Re: [PATCHv2 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

Quoted 2 lines · Martin Jansa

* move read_shlib_providers before registering package as provider and don't change provider if it already exists, show warning instead

This new patch doesn't seem to match the commit message any more.

Quoted 5 lines

meta/classes/package.bbclass +5 -0

+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))

Martin Jansa

author

2014-01-19 15:23 UTC #6 lore raw

PATCH v2 3/5 package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

meta/classes/package.bbclass +23 -3

index 9b511a6..a9e32f2 100644
@@ -1322,6 +1322,9 @@ python package_do_filedeps() {
 SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
 SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
 
+# default search path when searching for shlibs provided by package
+SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
+
 python package_do_shlibs() {
     import re, pipes

@@ -1332,6 +1335,20 @@ python package_do_shlibs() {
 
     lib_re = re.compile("^.*\.so")
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
+    
+    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
+        if shlibs_search_dirs_re_txt:
+            shlibs_search_dirs_re_txt += "|"
+        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)

@@ -1390,9 +1407,12 @@ python package_do_shlibs() {
             if m:
                 this_soname = m.group(1)
                 if not this_soname in sonames:
-                    # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or -1 == private_libs.find(this_soname):
-                        sonames.append(this_soname)
+                    if shlibs_search_dirs_re.match(file):
+                        # if library is private (only used by package) then do not build shlib for it
+                        if not private_libs or -1 == private_libs.find(this_soname):
+                            sonames.append(this_soname)
+                    else:
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-19 15:24 UTC #8 lore raw

PATCH v3 2/5 package.bbclass: show warning when package is providing already provided shlib

In reply to Phil Blundell

* move read_shlib_providers before registering package as provider and show warning when different package tries to provide something already provided.

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

meta/classes/package.bbclass +9 -2

index 7dcec5e..9b511a6 100644
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):

@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False

@@ -1504,6 +1507,11 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()

@@ -1517,8 +1525,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)

Signature · 1 line

1.8.5.2

Martin Jansa

author

2014-01-19 15:25 UTC #7 lore raw

Re: [PATCHv2 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

In reply to Martin Jansa

Quoted 7 lines · Martin Jansa

* when package contains some files matching "^.*\.so", but in directory not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so) don't register it as libfoo provider, because it's possible that there is different package providing libfoo.so in ${libdir} and that would be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can define own SHLIBSSEARCHDIRS value

This one is the same as v1, sent by accident when sending v3 of 2/5.

Quoted 61 lines

[YOCTO #4628]

Signed-off-by Martin Jansa <[email protected]>

---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

meta/classes/package.bbclass +23 -3

index 9b511a6..a9e32f2 100644
@@ -1322,6 +1322,9 @@ python package_do_filedeps() {
 SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
 SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
 
+# default search path when searching for shlibs provided by package
+SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
+
 python package_do_shlibs() {
     import re, pipes

@@ -1332,6 +1335,20 @@ python package_do_shlibs() {
 
     lib_re = re.compile("^.*\.so")
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
+    
+    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
+        if shlibs_search_dirs_re_txt:
+            shlibs_search_dirs_re_txt += "|"
+        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)

@@ -1390,9 +1407,12 @@ python package_do_shlibs() {
             if m:
                 this_soname = m.group(1)
                 if not this_soname in sonames:
-                    # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or -1 == private_libs.find(this_soname):
-                        sonames.append(this_soname)
+                    if shlibs_search_dirs_re.match(file):
+                        # if library is private (only used by package) then do not build shlib for it
+                        if not private_libs or -1 == private_libs.find(this_soname):
+                            sonames.append(this_soname)
+                    else:
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):

Signature · 1 line

1.8.5.2

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Richard Purdie

2014-01-19 17:14 UTC #10 lore raw

Re: [PATCH 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

In reply to Martin Jansa

Quoted 8 lines · Martin Jansa

* when package contains some files matching "^.*\.so", but in directory not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so) don't register it as libfoo provider, because it's possible that there is different package providing libfoo.so in ${libdir} and that would be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can define own SHLIBSSEARCHDIRS value

Quoted 6 lines

[YOCTO #4628]
Signed-off-by Martin Jansa <[email protected]>
---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

Doesn't this mean we lose automatic dependencies for libraries not in the default system paths after this change?

We should really be respecting the libraries RPATH in this code and searching the places it will look...

Cheers,

Richard

Martin Jansa

author

2014-01-19 18:55 UTC #11 lore raw

Re: [PATCH 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

Quoted 11 lines · Richard Purdie

Quoted 10 lines · Martin Jansa
* when package contains some files matching "^.*\.so", but in directory not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so) don't register it as libfoo provider, because it's possible that there is different package providing libfoo.so in ${libdir} and that would be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can define own SHLIBSSEARCHDIRS value
[YOCTO #4628]

Quoted 7 lines

Quoted 4 lines
Signed-off-by Martin Jansa <[email protected]>
---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
Doesn't this mean we lose automatic dependencies for libraries not in the default system paths after this change?

True, recipes with such libraries would need to append to SHLIBSSEARCHDIRS

Quoted 2 lines

We should really be respecting the libraries RPATH in this code and searching the places it will look...

I agree we should respect RPATH, but sometimes it is more complicated, because your own binaries can have RPATH pointing to /opt/package/bundled-lib/libfoo.so, but you don't want any other recipe to find your libfoo.so, because there is preferred "system" /usr/lib/libfoo.so

So in ideal case we should respect combination of default search path, ld.so.conf, RPATH and PRIVATE_LIBS :/.

I don't mind if you drop this one from patchset, it's the most controversial one and can be work-arounded by carefully maintained PRIVATE_LIBS list.

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]