Skip to contentSkip to conversation

package.bbclass: allow shell-style wildcards in PRIVATE_LIBS

PATCH In review Alexander Kanavin 2 participants 8 messages Sep 3, 2019 – Sep 4, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package.bbclass: allow shell-style wildcards in PRIVATE_LIBS

PRIVATE_LIBS is used to exclude 'private' libraries from getting added to automatic runtime dependency resolution. This variable currently has to list all libraries by name, which becomes a maintenance issue if the list of such libraries frequently changes, or is very large.

This change allows using shell-style wildcards in the variable, similar to how FILES lists what gets packaged.

---
 meta/classes/package.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/classes/package.bbclass +4 -2
index 114d6559f5e..aa8451ffe8b 100644
@@ -1646,7 +1646,8 @@ python package_do_shlibs() {
                 prov = (this_soname, ldir, pkgver)
                 if not prov in sonames:
                     # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or this_soname not in private_libs:
+                    import fnmatch
+                    if not private_libs or len([i for i in private_libs if fnmatch.fnmatch(this_soname, i)]) == 0:
                         sonames.add(prov)
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
@@ -1829,7 +1830,8 @@ python package_do_shlibs() {
             # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1
             # but skipping it is still better alternative than providing own
             # version and then adding runtime dependency for the same system library
-            if private_libs and n[0] in private_libs:
+            import fnmatch
+            if private_libs and len([i for i in private_libs if fnmatch.fnmatch(n[0], i)]) > 0:
                 bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n[0]))
                 continue
             if n[0] in shlib_provider.keys():
Signature · 1 line
2.17.1
#1 lore raw

Hi Alexander,

Quoted 18 lines · Alexander Kanavin

PRIVATE_LIBS is used to exclude 'private' libraries from getting added to automatic runtime dependency resolution. This variable currently has to list all libraries by name, which becomes a maintenance issue if the list of such libraries frequently changes, or is very large.

This change allows using shell-style wildcards in the variable, similar to how FILES lists what gets packaged.

---
 meta/classes/package.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/classes/package.bbclass +0 -0
index 114d6559f5e..aa8451ffe8b 100644

meta/classes/package.bbclass python package_do_shlibs() { · line 1650

Quoted 6 lines
meta/classes/package.bbclass +2 -1
@@ -1646,7 +1646,8 @@ python package_do_shlibs() {
                 prov = (this_soname, ldir, pkgver)
                 if not prov in sonames:
                     # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or this_soname not in private_libs:
+                    import fnmatch
+                    if not private_libs or len([i for i in private_libs if fnmatch.fnmatch(this_soname, i)]) == 0:

We only need to know if this_soname is matching one of the patterns, we don't need to check for each and every pattern.

Something like:

for pattern in private_libs:

	if fnmatch.fnmatch(this_soname, pattern)]) == 0:
		sonames.add(prov)
		break

would be possible?

AFAICT, private_libs is either the content of PRIVATE_LIBS_PN or PRIVATE_LIBS or "" then passed to .split(). This should be safe for the forloop without a check.

meta/classes/package.bbclass python package_do_shlibs() { · line 1834

Quoted 10 lines
meta/classes/package.bbclass +2 -1
                         sonames.add(prov)
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
@@ -1829,7 +1830,8 @@ python package_do_shlibs() {
             # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1
             # but skipping it is still better alternative than providing own
             # version and then adding runtime dependency for the same system library
-            if private_libs and n[0] in private_libs:
+            import fnmatch
+            if private_libs and len([i for i in private_libs if fnmatch.fnmatch(n[0], i)]) > 0:

Same here?

Just a suggestion :)

BR,
Quentin

Signature · 3 lines
StreamUnlimited Engineering GmbH
High Tech Campus Vienna, Gutheil-Schoder-Gasse 10, 1100 Vienna, Austria
[email protected], www.streamunlimited.com
#2 lore raw
Quoted 11 lines · Quentin Schulz
Quoted 1 line
                     # if library is private (only used by package) then

do not build shlib for it

Quoted 1 line

- if not private_libs or this_soname not in

private_libs:

Review

Quoted 2 lines
(quoted diff) +2 -0
+                    import fnmatch
+                    if not private_libs or len([i for i in private_libs

if fnmatch.fnmatch(this_soname, i)]) == 0:

We only need to know if this_soname is matching one of the patterns, we don't need to check for each and every pattern.

Quoted 9 lines

Something like:

for pattern in private_libs:

        if fnmatch.fnmatch(this_soname, pattern)]) == 0:
                sonames.add(prov)
                break

would be possible?

Yes, although there should only be very few patterns, and most of the time just one: *.so*. Not worth it I'd say. There are other spots in do_package which truly slow down things and could use a bit of optimization.

Alex

#3 lore raw
In reply to Quentin Schulz
Quoted 15 lines · Quentin Schulz

meta/classes/package.bbclass python package_do_shlibs() { · line 1648

Quoted 5 lines

+++ b/meta/classes/package.bbclass

meta/classes/package.bbclass +0 -0
@@ -1646,7 +1646,8 @@ python package_do_shlibs() {
                 prov = (this_soname, ldir, pkgver)
                 if not prov in sonames:
                     # if library is private (only used by package) then

do not build shlib for it

Review python package_do_shlibs() { · line 1649

Quoted 1 line
(quoted diff) +0 -1
-                    if not private_libs or this_soname not in

private_libs:

Review python package_do_shlibs() { · line 1650

Quoted 2 lines
(quoted diff) +2 -0
+                    import fnmatch
+                    if not private_libs or len([i for i in private_libs

if fnmatch.fnmatch(this_soname, i)]) == 0:

We only need to know if this_soname is matching one of the patterns, we don't need to check for each and every pattern.

Quoted 9 lines

Something like:

for pattern in private_libs:

        if fnmatch.fnmatch(this_soname, pattern)]) == 0:
                sonames.add(prov)
                break

would be possible?

Actually, this suggested code snippet looks altogether wrong to me. We do need to check the file against every pattern, and only if it matches none of them, it gets added to sonames. Note the original code says 'this_soname not in private_libs', which also means going over every item in the list.

Alex

#4 lore raw

Hi Alex,

Quoted 25 lines · Alexander Kanavin
Quoted 22 lines · Quentin Schulz

meta/classes/package.bbclass python package_do_shlibs() { · line 1648

Quoted 5 lines

+++ b/meta/classes/package.bbclass

meta/classes/package.bbclass +0 -0
@@ -1646,7 +1646,8 @@ python package_do_shlibs() {
                 prov = (this_soname, ldir, pkgver)
                 if not prov in sonames:
                     # if library is private (only used by package) then

do not build shlib for it

Review python package_do_shlibs() { · line 1649

Quoted 1 line
(quoted diff) +0 -1
-                    if not private_libs or this_soname not in

private_libs:

Review python package_do_shlibs() { · line 1650

Quoted 2 lines
(quoted diff) +2 -0
+                    import fnmatch
+                    if not private_libs or len([i for i in private_libs

if fnmatch.fnmatch(this_soname, i)]) == 0:

We only need to know if this_soname is matching one of the patterns, we don't need to check for each and every pattern.

Something like:

for pattern in private_libs:

        if fnmatch.fnmatch(this_soname, pattern)]) == 0:
                sonames.add(prov)
                break
Quoted 4 lines
Quoted 2 lines

would be possible?

Actually, this suggested code snippet looks altogether wrong to me. We do

Indeed.

Quoted 2 lines

need to check the file against every pattern, and only if it matches none of them, it gets added to sonames. Note the original code says 'this_soname

So we just need to invert the logic actually.

matched = False
for pattern in private_libs:

	if fnmatch.fnmatch(this_soname, pattern)]) == 0:
		matched = True
		break

if not matched:
sonames.add(prov)

Quoted 2 lines

not in private_libs', which also means going over every item in the list.

I'd hope Python devs did optimization on this one and stopped at the first match in the list, but I'm not one of them so...

BR,
Quentin

Signature · 3 lines
StreamUnlimited Engineering GmbH
High Tech Campus Vienna, Gutheil-Schoder-Gasse 10, 1100 Vienna, Austria
[email protected], www.streamunlimited.com
#5 lore raw
Quoted 11 lines · Quentin Schulz

So we just need to invert the logic actually.

matched = False
for pattern in private_libs:
if fnmatch.fnmatch(this_soname, pattern)]) == 0:
matched = True
break

if not matched:
sonames.add(prov)

I think you are looking for the "for...else" feature in Python :) Still, the original code is not adding any lines at all while yours adds eight so I'd just keep the patch as it is.

Alex

#6 lore raw

Hi Alex,

Quoted 11 lines · Alexander Kanavin
Quoted 8 lines · Quentin Schulz

So we just need to invert the logic actually.

matched = False
for pattern in private_libs:
if fnmatch.fnmatch(this_soname, pattern)]) == 0:
matched = True
break

Quoted 5 lines
Quoted 3 lines

if not matched:
sonames.add(prov)

I think you are looking for the "for...else" feature in Python :) Still,

I'm discovering this one, thanks!

Quoted 3 lines

the original code is not adding any lines at all while yours adds eight so I'd just keep the patch as it is.

Added lines vs added execution time.

Anyway, glad to have this feature one day in Yocto, I was about to use it in some project and was upset for having to name each and every lib in the package.

Thanks!
Quentin

Signature · 3 lines
StreamUnlimited Engineering GmbH
High Tech Campus Vienna, Gutheil-Schoder-Gasse 10, 1100 Vienna, Austria
[email protected], www.streamunlimited.com
#7 lore raw
Quoted 7 lines · Quentin Schulz
Quoted 1 line

the original code is not adding any lines at all while yours adds eight

so

Quoted 2 lines

I'd just keep the patch as it is.

Added lines vs added execution time.

I am fairly certain do_package spends the bulk of its time elsewhere, so you are really optimizing the wrong spot here :)

Alex