Skip to contentSkip to conversation

package: Generate srclist and filelics json files

PATCH In review Meh Mbeh Ida Delphine 2 participants 8 messages Mar 16, 2021 – Mar 28, 2021 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/6 package: Generate srclist and filelics json files

From: Richard Purdie <[email protected]>

Change do_package to:

a) Save the debug object -> source file mapping information (in TEMPDBGSRCMAPPING)

b) Save a .srclist file continaing that information (in json format)

c) Perform a grep of all the sources in the build of this recipe looking for SPDX license headers

d) Save out a file listing the SPDX headers found for each file

---
 meta/classes/package.bbclass | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
meta/classes/package.bbclass +31 -0
index 247bdc7bbf..42bd38f948 100644
@@ -1232,6 +1232,9 @@ python split_and_strip_files () {
                 for file in staticlibs:
                     results.append( (file,source_info(file, d)) )
 
+        # Stash this for emit_pkgdata
+        d.setVar('TEMPDBGSRCMAPPING', results)
+
         sources = set()
         for r in results:
             sources.update(r[1])
@@ -1544,6 +1547,7 @@ PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS
 python emit_pkgdata() {
     from glob import glob
     import json
+    import subprocess
 
     def process_postinst_on_target(pkg, mlprefix):
         pkgval = d.getVar('PKG_%s' % pkg)
@@ -1695,6 +1699,33 @@ fi
         and not bb.data.inherits_class('packagegroup', d):
         write_extra_runtime_pkgs(global_variants, packages, pkgdatadir)
 
+    sourceresult = d.getVar('TEMPDBGSRCMAPPING', False)
+    sources = {}
+    if sourceresult:
+        for r in sourceresult:
+            sources[r[0]] = r[1]
+        with open(data_file + ".srclist", 'w') as f:
+            f.write(json.dumps(sources, sort_keys=True))
+
+        filelics = {}
+        for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
+            p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
+            out, err = p.communicate()
+            if p.returncode == 0:
+                for l in out.decode("utf-8").split("\n"):
+                    l = l.strip()
+                    if not l:
+                        continue
+                    l = l.split(":")
+                    if len(l) < 3:
+                        bb.warn(str(l))
+                        continue
+                    fn = "/" + l[0]
+                    lic = l[2]
+                    filelics[fn] = lic
+        with open(data_file + ".filelics", 'w') as f:
+            f.write(json.dumps(filelics, sort_keys=True))
+
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 2/6 package: Add license computation experiment

From: Richard Purdie <[email protected]>

This code creates a list of SPDX headers found for the sources that make up a given set of binaries that make up an individual package.

This is then compared with the license field of the gven package containing those binaries.

---
 meta/classes/package.bbclass | 39 +++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +38 -1
index 42bd38f948..ab96f141ae 100644
@@ -1633,6 +1633,8 @@ fi
 
     workdir = d.getVar('WORKDIR')
 
+    filemap = {}
+
     for pkg in packages.split():
         pkgval = d.getVar('PKG_%s' % pkg)
         if pkgval is None:
@@ -1645,6 +1647,9 @@ fi
         seen = set()
         for f in pkgfiles[pkg]:
             relpth = os.path.relpath(f, pkgdestpkg)
+            if not pkg in filemap:
+                filemap[pkg] = []
+            filemap[pkg].append(os.sep + relpth)
             fstat = os.lstat(f)
             files[os.sep + relpth] = fstat.st_size
             if fstat.st_ino not in seen:
@@ -1721,11 +1726,43 @@ fi
                         bb.warn(str(l))
                         continue
                     fn = "/" + l[0]
-                    lic = l[2]
+                    lic = l[2].strip()
+                    if lic.endswith("*/"):
+                        lic = lic[:-2]
+                    lic = lic.strip()
                     filelics[fn] = lic
         with open(data_file + ".filelics", 'w') as f:
             f.write(json.dumps(filelics, sort_keys=True))
 
+        computedlics = {}
+        computedpkglics = {}
+        for r in sourceresult:
+            for subf in r[1]:
+                if subf in filelics:
+                    if r[0] not in computedlics:
+                        computedlics[r[0]] = set()
+                    computedlics[r[0]].add(filelics[subf])
+        #if computedlics:
+        #    bb.warn(str(computedlics))
+        dvar = d.getVar('PKGD')
+        for f in computedlics:
+            shortf = f.replace(dvar, "")
+            found = False
+            for pkg in filemap:
+                if shortf in filemap[pkg]:
+                    found = True
+                    if pkg not in computedpkglics:
+                        computedpkglics[pkg] = set()
+                    computedpkglics[pkg].update(computedlics[f])
+            if not found:
+                bb.warn("%s not in %s" % (f, str(filemap)))
+        #if computedpkglics:
+        #    bb.warn(str(computedpkglics))
+        for pkg in computedpkglics:
+            lic = d.getVar('LICENSE_%s' % (pkg))
+            if not lic:
+                lic = d.getVar('LICENSE')
+            bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic))
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 3/6 oe.license: Add functions to split and canonicalise license strings.

In reply to Meh Mbeh Ida Delphine

These functions ensures recipe LICENSE strings are split into their individual licenses and then canonicalised to easily enable their matching with licenses of sources during the packaging process.

---
 meta/lib/oe/license.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
meta/lib/oe/license.py +19 -0
index c1274a61de..dc1bc6682c 100644
@@ -243,3 +243,22 @@ def list_licenses(licensestr):
     except SyntaxError as exc:
         raise LicenseSyntaxError(licensestr, exc)
     return visitor.licenses
+
+def canonical_license(license, d):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0), for the license named 'X+', return canonical form of
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
+    or the passed license if there is no canonical form.
+    """
+    lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
+    return lic or license
+
+def split_spdx_lic(licensestr,d):
+    """
+    Split the license strings and returns a set of the
+    canonicalised licenses.
+    """
+    split_lic = list_licenses(licensestr)
+    spdx_lic = set([canonical_license(l, d) for l in split_lic])
+    return spdx_lic
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 4/6 package.bbclass: Displays warnings for just licences in recipes.

In reply to Meh Mbeh Ida Delphine

After the comparism between computedpkglics and recipe lics, the warnings are displayed if and only if there are licenses in the recipes not found in the sources.

---
 meta/classes/package.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +6 -1
index ab96f141ae..efdc169275 100644
@@ -1548,6 +1548,7 @@ python emit_pkgdata() {
     from glob import glob
     import json
     import subprocess
+    import oe.license
 
     def process_postinst_on_target(pkg, mlprefix):
         pkgval = d.getVar('PKG_%s' % pkg)
@@ -1762,7 +1763,11 @@ fi
             lic = d.getVar('LICENSE_%s' % (pkg))
             if not lic:
                 lic = d.getVar('LICENSE')
-            bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic))
+            spdx_lic = oe.license.split_spdx_lic(lic, d)
+
+            # Displays warnings for licenses found in the recipes and not sources
+            if spdx_lic - computedpkglics[pkg]:
+                bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic))
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
Signature · 1 line
2.25.1
#4 lore raw

PATCH v2 5/6 package.bbclass: Add handle_qa_error check

In reply to Meh Mbeh Ida Delphine

package_qa_handle_error() from insane.bbclass handles display of warnings by allowing the user decide whether they want the issues treated as warnings, errors or hidden completely. This is done by setting a variable "LICENSE_SOURCE_SPDX" in local.conf to either "0" or "1".
Add SPDXLICENSEMAP variable to emit_pkgdata[vardeps]

---
 meta/classes/package.bbclass | 127 ++++++++++++++++++-----------------
 1 file changed, 65 insertions(+), 62 deletions(-)
meta/classes/package.bbclass +65 -62
index efdc169275..8cd7f3ad21 100644
@@ -1549,6 +1549,7 @@ python emit_pkgdata() {
     import json
     import subprocess
     import oe.license
+    import bb.utils
 
     def process_postinst_on_target(pkg, mlprefix):
         pkgval = d.getVar('PKG_%s' % pkg)
@@ -1706,70 +1707,72 @@ fi
         write_extra_runtime_pkgs(global_variants, packages, pkgdatadir)
 
     sourceresult = d.getVar('TEMPDBGSRCMAPPING', False)
-    sources = {}
-    if sourceresult:
-        for r in sourceresult:
-            sources[r[0]] = r[1]
-        with open(data_file + ".srclist", 'w') as f:
-            f.write(json.dumps(sources, sort_keys=True))
-
-        filelics = {}
-        for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
-            p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
-            out, err = p.communicate()
-            if p.returncode == 0:
-                for l in out.decode("utf-8").split("\n"):
-                    l = l.strip()
-                    if not l:
-                        continue
-                    l = l.split(":")
-                    if len(l) < 3:
-                        bb.warn(str(l))
-                        continue
-                    fn = "/" + l[0]
-                    lic = l[2].strip()
-                    if lic.endswith("*/"):
-                        lic = lic[:-2]
-                    lic = lic.strip()
-                    filelics[fn] = lic
-        with open(data_file + ".filelics", 'w') as f:
-            f.write(json.dumps(filelics, sort_keys=True))
-
-        computedlics = {}
-        computedpkglics = {}
-        for r in sourceresult:
-            for subf in r[1]:
-                if subf in filelics:
-                    if r[0] not in computedlics:
-                        computedlics[r[0]] = set()
-                    computedlics[r[0]].add(filelics[subf])
-        #if computedlics:
-        #    bb.warn(str(computedlics))
-        dvar = d.getVar('PKGD')
-        for f in computedlics:
-            shortf = f.replace(dvar, "")
-            found = False
-            for pkg in filemap:
-                if shortf in filemap[pkg]:
-                    found = True
-                    if pkg not in computedpkglics:
-                        computedpkglics[pkg] = set()
-                    computedpkglics[pkg].update(computedlics[f])
-            if not found:
-                bb.warn("%s not in %s" % (f, str(filemap)))
-        #if computedpkglics:
-        #    bb.warn(str(computedpkglics))
-        for pkg in computedpkglics:
-            lic = d.getVar('LICENSE_%s' % (pkg))
-            if not lic:
-                lic = d.getVar('LICENSE')
-            spdx_lic = oe.license.split_spdx_lic(lic, d)
-
-            # Displays warnings for licenses found in the recipes and not sources
-            if spdx_lic - computedpkglics[pkg]:
-                bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic))
+    if bb.utils.to_boolean(d.getVar("LICENSE_SOURCE_SPDX")):
+        sources = {}
+        if sourceresult:
+            for r in sourceresult:
+                sources[r[0]] = r[1]
+            with open(data_file + ".srclist", 'w') as f:
+                f.write(json.dumps(sources, sort_keys=True))
+
+            filelics = {}
+            for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
+                p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
+                out, err = p.communicate()
+                if p.returncode == 0:
+                    for l in out.decode("utf-8").split("\n"):
+                        l = l.strip()
+                        if not l:
+                            continue
+                        l = l.split(":")
+                        if len(l) < 3:
+                            bb.warn(str(l))
+                            continue
+                        fn = "/" + l[0]
+                        lic = l[2].strip()
+                        if lic.endswith("*/"):
+                            lic = lic[:-2]
+                        lic = lic.strip()
+                        filelics[fn] = lic
+            with open(data_file + ".filelics", 'w') as f:
+                f.write(json.dumps(filelics, sort_keys=True))
+
+            computedlics = {}
+            computedpkglics = {}
+            for r in sourceresult:
+                for subf in r[1]:
+                    if subf in filelics:
+                        if r[0] not in computedlics:
+                            computedlics[r[0]] = set()
+                        computedlics[r[0]].add(filelics[subf])
+            #if computedlics:
+            #    bb.warn(str(computedlics))
+            dvar = d.getVar('PKGD')
+            for f in computedlics:
+                shortf = f.replace(dvar, "")
+                found = False
+                for pkg in filemap:
+                    if shortf in filemap[pkg]:
+                        found = True
+                        if pkg not in computedpkglics:
+                            computedpkglics[pkg] = set()
+                        computedpkglics[pkg].update(computedlics[f])
+                if not found:
+                    bb.warn("%s not in %s" % (f, str(filemap)))
+            #if computedpkglics:
+            #    bb.warn(str(computedpkglics))
+            for pkg in computedpkglics:
+                lic = d.getVar('LICENSE_%s' % (pkg))
+                if not lic:
+                    lic = d.getVar('LICENSE')
+                spdx_lic = oe.license.split_spdx_lic(lic, d)
+
+                # Displays warnings for licenses found in the recipes and not sources
+                if spdx_lic - computedpkglics[pkg]:
+                    package_qa_handle_error("LICENSE_SOURCE_SPDX", ("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic)), d)
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
+emit_pkgdata[vardeps] += "SPDXLICENSEMAP"
 
 ldconfig_postinst_fragment() {
 if [ x"$D" = "x" ]; then
Signature · 1 line
2.25.1
#5 lore raw

PATCH v2 6/6 Moved logic to get filelics from package.bbclass to license.py

In reply to Meh Mbeh Ida Delphine

The logic to obtain filelics is moved from package.bbclass to license.py as a standalone function.
Also, the check in package.bbclass after obtaining filelics ensures licenses with * WITH Linux-syscall-note" are ignored, as long as they're only in header files.

---
 meta/classes/package.bbclass | 35 ++++++++++++-----------------------
 meta/lib/oe/license.py       | 29 +++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+), 23 deletions(-)
meta/classes/package.bbclass +12 -23
index 8cd7f3ad21..a2144ec08e 100644
@@ -1715,28 +1715,19 @@ fi
             with open(data_file + ".srclist", 'w') as f:
                 f.write(json.dumps(sources, sort_keys=True))
 
+            flics = oe.license.get_filelics([d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')])
             filelics = {}
-            for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
-                p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
-                out, err = p.communicate()
-                if p.returncode == 0:
-                    for l in out.decode("utf-8").split("\n"):
-                        l = l.strip()
-                        if not l:
-                            continue
-                        l = l.split(":")
-                        if len(l) < 3:
-                            bb.warn(str(l))
-                            continue
-                        fn = "/" + l[0]
-                        lic = l[2].strip()
-                        if lic.endswith("*/"):
-                            lic = lic[:-2]
-                        lic = lic.strip()
-                        filelics[fn] = lic
+            header_list = [".h", ".hh", ".hpp"]
+            for k, v in flics.items():
+                # This check ignores "*WITH Linux-syscall-note" licenses in header files since the headers aren't part of the original sources and come from dependencies the software links to.
+                #C/C++ headers because most of the ones being match are from the Linux kernel.
+                if k.endswith(tuple(header_list)) and v.endswith("WITH Linux-syscall-note"):
+                    continue
+                else:
+                    filelics[k] = v
             with open(data_file + ".filelics", 'w') as f:
                 f.write(json.dumps(filelics, sort_keys=True))
-
+  
             computedlics = {}
             computedpkglics = {}
             for r in sourceresult:
@@ -1745,8 +1736,7 @@ fi
                         if r[0] not in computedlics:
                             computedlics[r[0]] = set()
                         computedlics[r[0]].add(filelics[subf])
-            #if computedlics:
-            #    bb.warn(str(computedlics))
+            
             dvar = d.getVar('PKGD')
             for f in computedlics:
                 shortf = f.replace(dvar, "")
@@ -1759,8 +1749,7 @@ fi
                         computedpkglics[pkg].update(computedlics[f])
                 if not found:
                     bb.warn("%s not in %s" % (f, str(filemap)))
-            #if computedpkglics:
-            #    bb.warn(str(computedpkglics))
+            
             for pkg in computedpkglics:
                 lic = d.getVar('LICENSE_%s' % (pkg))
                 if not lic:
meta/lib/oe/license.py +29 -0
index dc1bc6682c..45f90ff151 100644
@@ -6,6 +6,8 @@
 import ast
 import re
 from fnmatch import fnmatchcase as fnmatch
+import subprocess
+import bb
 
 def license_ok(license, dont_want_licenses):
     """ Return False if License exist in dont_want_licenses else True """
@@ -262,3 +264,30 @@ def split_spdx_lic(licensestr,d):
     split_lic = list_licenses(licensestr)
     spdx_lic = set([canonical_license(l, d) for l in split_lic])
     return spdx_lic
+
+def get_filelics(dirs):
+    """"
+    This function returns a dictionary of file paths (keys) and their corresponding SPDX header identifiers (values).
+    """
+    filelics = {}
+    for dirent in dirs:
+        p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R','-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
+        out, err = p.communicate()
+        if p.returncode == 0:
+            for l in out.decode("utf-8").split("\n"):
+                l = l.strip()
+                if not l:
+                    continue
+                l = l.split(":")
+                if len(l) < 3:
+                    bb.warn(str(l))
+                    continue
+                fn = "/" + l[0]
+                lic = l[2].strip()
+                if lic.endswith("*/"):
+                    lic = lic[:-2]
+                lic = lic.strip()
+                filelics[fn] = lic
+    return filelics
+    
+
Signature · 1 line
2.25.1
#8 lore raw

Re: [OE-core] [poky-contrib][PATCH v2 3/6] oe.license: Add functions to split and canonicalise license strings.

In reply to a message not in the archive

Hello,
A gentle reminder to please review this patch.

Cheers,
Ida.

Quoted 43 lines · Meh Mbeh Ida Delphine via lists.openembedded.org,

These functions ensures recipe LICENSE strings are split into their individual licenses and then canonicalised to easily enable their matching with licenses of sources during the packaging process.

---
 meta/lib/oe/license.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
meta/lib/oe/license.py +19 -1
index c1274a61de..dc1bc6682c 100644
@@ -243,3 +243,22 @@ def list_licenses(licensestr):
     except SyntaxError as exc:
         raise LicenseSyntaxError(licensestr, exc)
     return visitor.licenses
+
+def canonical_license(license, d):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0), for the license named 'X+', return canonical form of
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
+    or the passed license if there is no canonical form.
+    """
+    lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
+    return lic or license
+
+def split_spdx_lic(licensestr,d):
+    """
+    Split the license strings and returns a set of the
+    canonicalised licenses.
+    """
+    split_lic = list_licenses(licensestr)
+    spdx_lic = set([canonical_license(l, d) for l in split_lic])
+    return spdx_lic
--

2.25.1

#6 lore raw

Re: [OE-core] [poky-contrib][PATCH v2 6/6] Moved logic to get filelics from package.bbclass to license.py

In reply to Meh Mbeh Ida Delphine

Hi Ida,

Quoted 10 lines · Meh Mbeh Ida Delphine

The logic to obtain filelics is moved from package.bbclass to license.py as a standalone function.
Also, the check in package.bbclass after obtaining filelics ensures licenses with * WITH Linux-syscall-note" are ignored, as long as they're only in header files.

---
 meta/classes/package.bbclass | 35 ++++++++++++-----------------------
 meta/lib/oe/license.py | 29 +++++++++++++++++++++++++++++

 2 files changed, 41 insertions(+), 23 deletions(-)

I have spent quite a bit of time thinking about this patchset and I'm not sure we're yet into a position where it can merge. The code has definitely moved on from where things started but I'm not sure it is far enough along to be useful to others yet in this form.

For example, we should only ever generate warnings if there is a clear path the user needs to take or a clear benefit for fixing it. The warnings from this code, whilst reduced from where they were, are still going to be large in number and not focused enough for general use. I realise I added the warnings in the original code but the idea was that was only a temporary thing to help debug it with and illustrate what it was doing, it was never intended to merge like that. Ultimately a warning should only be shown when we know there is something serious wrong.

If this data was being written to a log for someone to review, that would perhaps be better but even then, I'm not sure the quality of the output data is yet high enough for that to be useful for people review.

Some of the function abstraction does look useful but it is entwined in with the rest of the patches so can't be merged without them I do also have some concerns about the function names, I think they'd need to be improved before it could be merged too.

What we do have here is a set of building blocks that the next steps in license development can take place from so its good to have the progress.

Cheers,

Richard