package: Generate srclist and filelics json files
PATCH v2 1/6 package: Generate srclist and filelics json files
From: Richard Purdie <[email protected]>
Change do_package to:
a) Save the debug object -> source file mapping information (in TEMPDBGSRCMAPPING)
b) Save a .srclist file continaing that information (in json format)
c) Perform a grep of all the sources in the build of this recipe looking for SPDX license headers
d) Save out a file listing the SPDX headers found for each file
--- meta/classes/package.bbclass | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
meta/classes/package.bbclass +31 -0
@@ -1232,6 +1232,9 @@ python split_and_strip_files () { for file in staticlibs: results.append( (file,source_info(file, d)) ) + # Stash this for emit_pkgdata + d.setVar('TEMPDBGSRCMAPPING', results) + sources = set() for r in results: sources.update(r[1])
@@ -1544,6 +1547,7 @@ PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS python emit_pkgdata() { from glob import glob import json + import subprocess def process_postinst_on_target(pkg, mlprefix): pkgval = d.getVar('PKG_%s' % pkg)
@@ -1695,6 +1699,33 @@ fi and not bb.data.inherits_class('packagegroup', d): write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) + sourceresult = d.getVar('TEMPDBGSRCMAPPING', False) + sources = {} + if sourceresult: + for r in sourceresult: + sources[r[0]] = r[1] + with open(data_file + ".srclist", 'w') as f: + f.write(json.dumps(sources, sort_keys=True)) + + filelics = {} + for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]: + p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent) + out, err = p.communicate() + if p.returncode == 0: + for l in out.decode("utf-8").split("\n"): + l = l.strip() + if not l: + continue + l = l.split(":") + if len(l) < 3: + bb.warn(str(l)) + continue + fn = "/" + l[0] + lic = l[2] + filelics[fn] = lic + with open(data_file + ".filelics", 'w') as f: + f.write(json.dumps(filelics, sort_keys=True)) + } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
Signature · 1 line
2.25.1
PATCH v2 2/6 package: Add license computation experiment
From: Richard Purdie <[email protected]>
This code creates a list of SPDX headers found for the sources that make up a given set of binaries that make up an individual package.
This is then compared with the license field of the gven package containing those binaries.
--- meta/classes/package.bbclass | 39 +++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +38 -1
@@ -1633,6 +1633,8 @@ fi workdir = d.getVar('WORKDIR') + filemap = {} + for pkg in packages.split(): pkgval = d.getVar('PKG_%s' % pkg) if pkgval is None:
@@ -1645,6 +1647,9 @@ fi seen = set() for f in pkgfiles[pkg]: relpth = os.path.relpath(f, pkgdestpkg) + if not pkg in filemap: + filemap[pkg] = [] + filemap[pkg].append(os.sep + relpth) fstat = os.lstat(f) files[os.sep + relpth] = fstat.st_size if fstat.st_ino not in seen:
@@ -1721,11 +1726,43 @@ fi bb.warn(str(l)) continue fn = "/" + l[0] - lic = l[2] + lic = l[2].strip() + if lic.endswith("*/"): + lic = lic[:-2] + lic = lic.strip() filelics[fn] = lic with open(data_file + ".filelics", 'w') as f: f.write(json.dumps(filelics, sort_keys=True)) + computedlics = {} + computedpkglics = {} + for r in sourceresult: + for subf in r[1]: + if subf in filelics: + if r[0] not in computedlics: + computedlics[r[0]] = set() + computedlics[r[0]].add(filelics[subf]) + #if computedlics: + # bb.warn(str(computedlics)) + dvar = d.getVar('PKGD') + for f in computedlics: + shortf = f.replace(dvar, "") + found = False + for pkg in filemap: + if shortf in filemap[pkg]: + found = True + if pkg not in computedpkglics: + computedpkglics[pkg] = set() + computedpkglics[pkg].update(computedlics[f]) + if not found: + bb.warn("%s not in %s" % (f, str(filemap))) + #if computedpkglics: + # bb.warn(str(computedpkglics)) + for pkg in computedpkglics: + lic = d.getVar('LICENSE_%s' % (pkg)) + if not lic: + lic = d.getVar('LICENSE') + bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic)) } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
Signature · 1 line
2.25.1
PATCH v2 3/6 oe.license: Add functions to split and canonicalise license strings.
In reply to Meh Mbeh Ida DelphineThese functions ensures recipe LICENSE strings are split into their individual licenses and then canonicalised to easily enable their matching with licenses of sources during the packaging process.
--- meta/lib/oe/license.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
meta/lib/oe/license.py +19 -0
@@ -243,3 +243,22 @@ def list_licenses(licensestr): except SyntaxError as exc: raise LicenseSyntaxError(licensestr, exc) return visitor.licenses + +def canonical_license(license, d): + """ + Return the canonical (SPDX) form of the license if available (so GPLv3 + becomes GPL-3.0), for the license named 'X+', return canonical form of + 'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+), + or the passed license if there is no canonical form. + """ + lic = d.getVarFlag('SPDXLICENSEMAP', license) or "" + return lic or license + +def split_spdx_lic(licensestr,d): + """ + Split the license strings and returns a set of the + canonicalised licenses. + """ + split_lic = list_licenses(licensestr) + spdx_lic = set([canonical_license(l, d) for l in split_lic]) + return spdx_lic
Signature · 1 line
2.25.1
PATCH v2 4/6 package.bbclass: Displays warnings for just licences in recipes.
In reply to Meh Mbeh Ida DelphineAfter the comparism between computedpkglics and recipe lics, the warnings are displayed if and only if there are licenses in the recipes not found in the sources.
--- meta/classes/package.bbclass | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +6 -1
@@ -1548,6 +1548,7 @@ python emit_pkgdata() { from glob import glob import json import subprocess + import oe.license def process_postinst_on_target(pkg, mlprefix): pkgval = d.getVar('PKG_%s' % pkg)
@@ -1762,7 +1763,11 @@ fi lic = d.getVar('LICENSE_%s' % (pkg)) if not lic: lic = d.getVar('LICENSE') - bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic)) + spdx_lic = oe.license.split_spdx_lic(lic, d) + + # Displays warnings for licenses found in the recipes and not sources + if spdx_lic - computedpkglics[pkg]: + bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic)) } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
Signature · 1 line
2.25.1
PATCH v2 5/6 package.bbclass: Add handle_qa_error check
In reply to Meh Mbeh Ida Delphinepackage_qa_handle_error() from insane.bbclass handles display of warnings by allowing the user decide whether they want the issues treated as warnings, errors or hidden completely. This is done by setting a variable "LICENSE_SOURCE_SPDX" in local.conf to either "0" or "1".
Add SPDXLICENSEMAP variable to emit_pkgdata[vardeps]
--- meta/classes/package.bbclass | 127 ++++++++++++++++++----------------- 1 file changed, 65 insertions(+), 62 deletions(-)
meta/classes/package.bbclass +65 -62
@@ -1549,6 +1549,7 @@ python emit_pkgdata() { import json import subprocess import oe.license + import bb.utils def process_postinst_on_target(pkg, mlprefix): pkgval = d.getVar('PKG_%s' % pkg)
@@ -1706,70 +1707,72 @@ fi write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) sourceresult = d.getVar('TEMPDBGSRCMAPPING', False) - sources = {} - if sourceresult: - for r in sourceresult: - sources[r[0]] = r[1] - with open(data_file + ".srclist", 'w') as f: - f.write(json.dumps(sources, sort_keys=True)) - - filelics = {} - for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]: - p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent) - out, err = p.communicate() - if p.returncode == 0: - for l in out.decode("utf-8").split("\n"): - l = l.strip() - if not l: - continue - l = l.split(":") - if len(l) < 3: - bb.warn(str(l)) - continue - fn = "/" + l[0] - lic = l[2].strip() - if lic.endswith("*/"): - lic = lic[:-2] - lic = lic.strip() - filelics[fn] = lic - with open(data_file + ".filelics", 'w') as f: - f.write(json.dumps(filelics, sort_keys=True)) - - computedlics = {} - computedpkglics = {} - for r in sourceresult: - for subf in r[1]: - if subf in filelics: - if r[0] not in computedlics: - computedlics[r[0]] = set() - computedlics[r[0]].add(filelics[subf]) - #if computedlics: - # bb.warn(str(computedlics)) - dvar = d.getVar('PKGD') - for f in computedlics: - shortf = f.replace(dvar, "") - found = False - for pkg in filemap: - if shortf in filemap[pkg]: - found = True - if pkg not in computedpkglics: - computedpkglics[pkg] = set() - computedpkglics[pkg].update(computedlics[f]) - if not found: - bb.warn("%s not in %s" % (f, str(filemap))) - #if computedpkglics: - # bb.warn(str(computedpkglics)) - for pkg in computedpkglics: - lic = d.getVar('LICENSE_%s' % (pkg)) - if not lic: - lic = d.getVar('LICENSE') - spdx_lic = oe.license.split_spdx_lic(lic, d) - - # Displays warnings for licenses found in the recipes and not sources - if spdx_lic - computedpkglics[pkg]: - bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic)) + if bb.utils.to_boolean(d.getVar("LICENSE_SOURCE_SPDX")): + sources = {} + if sourceresult: + for r in sourceresult: + sources[r[0]] = r[1] + with open(data_file + ".srclist", 'w') as f: + f.write(json.dumps(sources, sort_keys=True)) + + filelics = {} + for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]: + p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent) + out, err = p.communicate() + if p.returncode == 0: + for l in out.decode("utf-8").split("\n"): + l = l.strip() + if not l: + continue + l = l.split(":") + if len(l) < 3: + bb.warn(str(l)) + continue + fn = "/" + l[0] + lic = l[2].strip() + if lic.endswith("*/"): + lic = lic[:-2] + lic = lic.strip() + filelics[fn] = lic + with open(data_file + ".filelics", 'w') as f: + f.write(json.dumps(filelics, sort_keys=True)) + + computedlics = {} + computedpkglics = {} + for r in sourceresult: + for subf in r[1]: + if subf in filelics: + if r[0] not in computedlics: + computedlics[r[0]] = set() + computedlics[r[0]].add(filelics[subf]) + #if computedlics: + # bb.warn(str(computedlics)) + dvar = d.getVar('PKGD') + for f in computedlics: + shortf = f.replace(dvar, "") + found = False + for pkg in filemap: + if shortf in filemap[pkg]: + found = True + if pkg not in computedpkglics: + computedpkglics[pkg] = set() + computedpkglics[pkg].update(computedlics[f]) + if not found: + bb.warn("%s not in %s" % (f, str(filemap))) + #if computedpkglics: + # bb.warn(str(computedpkglics)) + for pkg in computedpkglics: + lic = d.getVar('LICENSE_%s' % (pkg)) + if not lic: + lic = d.getVar('LICENSE') + spdx_lic = oe.license.split_spdx_lic(lic, d) + + # Displays warnings for licenses found in the recipes and not sources + if spdx_lic - computedpkglics[pkg]: + package_qa_handle_error("LICENSE_SOURCE_SPDX", ("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], spdx_lic)), d) } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides" +emit_pkgdata[vardeps] += "SPDXLICENSEMAP" ldconfig_postinst_fragment() { if [ x"$D" = "x" ]; then
Signature · 1 line
2.25.1
PATCH v2 6/6 Moved logic to get filelics from package.bbclass to license.py
In reply to Meh Mbeh Ida DelphineThe logic to obtain filelics is moved from package.bbclass to license.py as a standalone function.
Also, the check in package.bbclass after obtaining filelics ensures licenses with * WITH Linux-syscall-note" are ignored, as long as they're only in header files.
--- meta/classes/package.bbclass | 35 ++++++++++++----------------------- meta/lib/oe/license.py | 29 +++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 23 deletions(-)
meta/classes/package.bbclass +12 -23
@@ -1715,28 +1715,19 @@ fi with open(data_file + ".srclist", 'w') as f: f.write(json.dumps(sources, sort_keys=True)) + flics = oe.license.get_filelics([d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]) filelics = {} - for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]: - p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent) - out, err = p.communicate() - if p.returncode == 0: - for l in out.decode("utf-8").split("\n"): - l = l.strip() - if not l: - continue - l = l.split(":") - if len(l) < 3: - bb.warn(str(l)) - continue - fn = "/" + l[0] - lic = l[2].strip() - if lic.endswith("*/"): - lic = lic[:-2] - lic = lic.strip() - filelics[fn] = lic + header_list = [".h", ".hh", ".hpp"] + for k, v in flics.items(): + # This check ignores "*WITH Linux-syscall-note" licenses in header files since the headers aren't part of the original sources and come from dependencies the software links to. + #C/C++ headers because most of the ones being match are from the Linux kernel. + if k.endswith(tuple(header_list)) and v.endswith("WITH Linux-syscall-note"): + continue + else: + filelics[k] = v with open(data_file + ".filelics", 'w') as f: f.write(json.dumps(filelics, sort_keys=True)) - + computedlics = {} computedpkglics = {} for r in sourceresult:
@@ -1745,8 +1736,7 @@ fi if r[0] not in computedlics: computedlics[r[0]] = set() computedlics[r[0]].add(filelics[subf]) - #if computedlics: - # bb.warn(str(computedlics)) + dvar = d.getVar('PKGD') for f in computedlics: shortf = f.replace(dvar, "")
@@ -1759,8 +1749,7 @@ fi computedpkglics[pkg].update(computedlics[f]) if not found: bb.warn("%s not in %s" % (f, str(filemap))) - #if computedpkglics: - # bb.warn(str(computedpkglics)) + for pkg in computedpkglics: lic = d.getVar('LICENSE_%s' % (pkg)) if not lic:
meta/lib/oe/license.py +29 -0
@@ -6,6 +6,8 @@ import ast import re from fnmatch import fnmatchcase as fnmatch +import subprocess +import bb def license_ok(license, dont_want_licenses): """ Return False if License exist in dont_want_licenses else True """
@@ -262,3 +264,30 @@ def split_spdx_lic(licensestr,d): split_lic = list_licenses(licensestr) spdx_lic = set([canonical_license(l, d) for l in split_lic]) return spdx_lic + +def get_filelics(dirs): + """" + This function returns a dictionary of file paths (keys) and their corresponding SPDX header identifiers (values). + """ + filelics = {} + for dirent in dirs: + p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R','-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent) + out, err = p.communicate() + if p.returncode == 0: + for l in out.decode("utf-8").split("\n"): + l = l.strip() + if not l: + continue + l = l.split(":") + if len(l) < 3: + bb.warn(str(l)) + continue + fn = "/" + l[0] + lic = l[2].strip() + if lic.endswith("*/"): + lic = lic[:-2] + lic = lic.strip() + filelics[fn] = lic + return filelics + +
Signature · 1 line
2.25.1
Re: [OE-core] [poky-contrib][PATCH v2 3/6] oe.license: Add functions to split and canonicalise license strings.
In reply to a message not in the archiveHello,
A gentle reminder to please review this patch.
Cheers,
Ida.
Quoted 43 lines · Meh Mbeh Ida Delphine via lists.openembedded.org,
These functions ensures recipe LICENSE strings are split into their individual licenses and then canonicalised to easily enable their matching with licenses of sources during the packaging process.
Signed-off-by Ida Delphine <[email protected]>--- meta/lib/oe/license.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)meta/lib/oe/license.py +19
-1@@ -243,3 +243,22 @@ def list_licenses(licensestr): except SyntaxError as exc: raise LicenseSyntaxError(licensestr, exc) return visitor.licenses + +def canonical_license(license, d): + """ + Return the canonical (SPDX) form of the license if available (so GPLv3 + becomes GPL-3.0), for the license named 'X+', return canonical form of + 'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+), + or the passed license if there is no canonical form. + """ + lic = d.getVarFlag('SPDXLICENSEMAP', license) or "" + return lic or license + +def split_spdx_lic(licensestr,d): + """ + Split the license strings and returns a set of the + canonicalised licenses. + """ + split_lic = list_licenses(licensestr) + spdx_lic = set([canonical_license(l, d) for l in split_lic]) + return spdx_lic --2.25.1
Re: [OE-core] [poky-contrib][PATCH v2 6/6] Moved logic to get filelics from package.bbclass to license.py
In reply to Meh Mbeh Ida DelphineHi Ida,
Quoted 10 lines · Meh Mbeh Ida Delphine
The logic to obtain filelics is moved from package.bbclass to
license.pyas a standalone function.
Also, the check in package.bbclass after obtaining filelics ensures licenses with * WITH Linux-syscall-note" are ignored, as long as they're only in header files.Signed-off-by Ida Delphine <[email protected]>---
meta/classes/package.bbclass | 35 ++++++++++++-----------------------
meta/lib/oe/license.py | 29 +++++++++++++++++++++++++++++2 files changed, 41 insertions(+), 23 deletions(-)
I have spent quite a bit of time thinking about this patchset and I'm not sure we're yet into a position where it can merge. The code has definitely moved on from where things started but I'm not sure it is far enough along to be useful to others yet in this form.
For example, we should only ever generate warnings if there is a clear path the user needs to take or a clear benefit for fixing it. The warnings from this code, whilst reduced from where they were, are still going to be large in number and not focused enough for general use. I realise I added the warnings in the original code but the idea was that was only a temporary thing to help debug it with and illustrate what it was doing, it was never intended to merge like that. Ultimately a warning should only be shown when we know there is something serious wrong.
If this data was being written to a log for someone to review, that would perhaps be better but even then, I'm not sure the quality of the output data is yet high enough for that to be useful for people review.
Some of the function abstraction does look useful but it is entwined in with the rest of the patches so can't be merged without them I do also have some concerns about the function names, I think they'd need to be improved before it could be merged too.
What we do have here is a set of building blocks that the next steps in license development can take place from so its good to have the progress.
Cheers,
Richard