license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses
PATCH 1/7 license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses
Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.
This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.
--- meta/classes/license.bbclass | 27 ++++++++++++++++--- meta/conf/documentation.conf | 1 + .../oeqa/selftest/cases/incompatible_lic.py | 6 ++--- 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +23 -4
@@ -252,7 +252,7 @@ def canonical_license(d, license): """ Return the canonical (SPDX) form of the license if available (so GPLv3 becomes GPL-3.0), for the license named 'X+', return canonical form of - 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), + 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), or the passed license if there is no canonical form. """ lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license): lic += '+' return lic or license +def available_licenses(d): + """ + Return the available licenses by searching the directories specified by + COMMON_LICENSE_DIR and LICENSE_PATH. + """ + lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' + + (d.getVar('LICENSE_PATH') or '')).split() + + licenses = [] + for lic_dir in lic_dirs: + licenses += os.listdir(lic_dir) + + licenses = sorted(licenses) + return licenses + +# Only determine the list of all available licenses once. This assumes that any +# additions to LICENSE_PATH have been done before this file is parsed. +AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}" + def expand_wildcard_licenses(d, wildcard_licenses): """ - Return actual spdx format license names if wildcard used. We expand - wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values. + Return actual spdx format license names if wildcards are used. We expand + wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES. """ import fnmatch licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses): spdxflags = fnmatch.filter(spdxmapkeys, wld_lic) licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags] - spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or '').split() + spdx_lics = d.getVar('AVAILABLE_LICENSES').split() for wld_lic in wildcard_licenses: licenses += fnmatch.filter(spdx_lics, wld_lic)
meta/conf/documentation.conf +1 -0
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib p AUTHOR[doc] = "Email address used to contact the original author(s) in order to send patches and forward bugs." AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the syslinux bootloader." AUTOREV[doc] = "When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository." +AVAILABLE_LICENSES[doc] = "List of licenses found in the directories specified by COMMON_LICENSE_DIR and LICENSE_PATH." #B
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase): if error_msg not in result.output: raise AssertionError(result.output) - # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES) + # Verify that a package with an SPDX license (from AVAILABLE_LICENSES) # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX license def test_incompatible_spdx_license(self): self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0') - # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES) + # Verify that a package with an SPDX license (from AVAILABLE_LICENSES) # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in # SPDXLICENSEMAP) of this SPDX license def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase): self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3') # Verify that a package with a non-SPDX license (neither in - # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when + # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when # INCOMPATIBLE_LICENSE contains this license def test_incompatible_nonspdx_license(self): self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
Signature · 1 line
2.21.0
PATCH 2/7 licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variable
The SRC_DISTRIBUTE_LICENSES variable and its static list of licenses has been replaced by AVAILABLE_LICENSES, which automatically contains all available licenses.
--- meta/conf/licenses.conf | 43 ----------------------------------------- 1 file changed, 43 deletions(-)
meta/conf/licenses.conf +0 -43
@@ -1,44 +1,3 @@ -# These aren't actually used anywhere that I can tell -# They may be in the future (or are used by someone else -# For completion sake, I've updated them -SRC_DISTRIBUTE_LICENSES += "AAL Adobe AFL-1.2 AFL-2.0 AFL-2.1 AFL-3.0" -SRC_DISTRIBUTE_LICENSES += "AGPL-3.0 ANTLR-PD Apache-1.0 Apache-1.1 Apache-2.0" -SRC_DISTRIBUTE_LICENSES += "APL-1.0 APSL-1.0 APSL-1.1 APSL-1.2 APSL-2.0" -SRC_DISTRIBUTE_LICENSES += "Artistic-1.0 Artistic-2.0 BitstreamVera BSD" -SRC_DISTRIBUTE_LICENSES += "BSD-2-Clause BSD-3-Clause BSD-4-Clause BSL-1.0" -SRC_DISTRIBUTE_LICENSES += "CATOSL-1.1 CC0-1.0 CC-BY-1.0 CC-BY-2.0 CC-BY-2.5" -SRC_DISTRIBUTE_LICENSES += "CC-BY-3.0 CC-BY-NC-1.0 CC-BY-NC-2.0 CC-BY-NC-2.5" -SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-3.0 CC-BY-NC-ND-1.0 CC-BY-NC-ND-2.0" -SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-ND-2.5 CC-BY-NC-ND-3.0 CC-BY-NC-SA-1.0" -SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-SA-2.0 CC-BY-NC-SA-2.5 CC-BY-NC-SA-3.0" -SRC_DISTRIBUTE_LICENSES += "CC-BY-ND-1.0 CC-BY-ND-2.0 CC-BY-ND-2.5 CC-BY-ND-3.0" -SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0 CC-BY-SA-2.5 CC-BY-SA-3.0 CC-BY-SA-4.0" -SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B CECILL-C" -SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL" -SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0 EFL-2.0" -SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1" -SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair Frameworx-1.0" -SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0" -SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception" -SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-classpath-exception" -SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-font-exception" -SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-GCC-exception" -SRC_DISTRIBUTE_LICENSES += "GPL-2-with-bison-exception GPL-3.0" -SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-autoconf-exception" -SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-GCC-exception" -SRC_DISTRIBUTE_LICENSES += "gSOAP-1 gSOAP-1.3b HPND IPA IPL-1.0 ISC LGPL-2.0" -SRC_DISTRIBUTE_LICENSES += "LGPL-2.1 LGPL-3.0 Libpng LPL-1.02 LPPL-1.0 LPPL-1.1" -SRC_DISTRIBUTE_LICENSES += "LPPL-1.2 LPPL-1.3c MirOS MIT Motosoto MPL-1.0" -SRC_DISTRIBUTE_LICENSES += "MPL-1.1 MS-PL MS-RL Multics NASA-1.3 Nauman NCSA" -SRC_DISTRIBUTE_LICENSES += "NGPL Nokia NPOSL-3.0 NTP OASIS OCLC-2.0 ODbL-1.0" -SRC_DISTRIBUTE_LICENSES += "OFL-1.1 OGTSL OLDAP-2.8 OpenSSL OSL-1.0 OSL-2.0" -SRC_DISTRIBUTE_LICENSES += "OSL-3.0 PD PHP-3.0 PostgreSQL Proprietary" -SRC_DISTRIBUTE_LICENSES += "Python-2.0 QPL-1.0 RHeCos-1 RHeCos-1.1 RPL-1.5" -SRC_DISTRIBUTE_LICENSES += "RPSL-1.0 RSCPL Ruby SAX-PD SGI-1 Simple-2.0 Sleepycat" -SRC_DISTRIBUTE_LICENSES += "SPL-1.0 SugarCRM-1 SugarCRM-1.1.3 UCB VSL-1.0 W3C" -SRC_DISTRIBUTE_LICENSES += "Watcom-1.0 WXwindows XFree86-1.0 XFree86-1.1 Xnet XSL YPL-1.1" -SRC_DISTRIBUTE_LICENSES += "Zimbra-1.3 Zlib ZPL-1.1 ZPL-2.0 ZPL-2.1" - # Standards are great! Everyone has their own. In an effort to standardize licensing # names, common-licenses will use the SPDX standard license names. In order to not # break the non-standardized license names that we find in LICENSE, we'll set
@@ -188,5 +147,3 @@ FOSS_BASE_URL = "http://localhost/repo/?mod=spdx_license_once" FOSS_SERVER = "${FOSS_BASE_URL}&fullSPDXFlag=${FOSS_FULL_SPDX}&noCopyright=${FOSS_NO_COPYRIGHT}&recursiveUnpack=${FOSS_RECURSIVE_UNPACK}" FOSS_WGET_FLAGS = "-qO - --no-check-certificate --timeout=0" - -
Signature · 1 line
2.21.0
PATCH 3/7 license.bbclass: Make incompatible_pkg_license return incompatible lics
In reply to Peter KjellerstedtThis makes it possible to report the incompatible licenses.
--- meta/classes/license.bbclass | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
meta/classes/license.bbclass +9 -3
@@ -310,15 +310,21 @@ def incompatible_pkg_license(d, dont_want_licenses, license): # Handles an "or" or two license sets provided by # flattened_licenses(), pick one that works if possible. def choose_lic_set(a, b): - return a if all(oe.license.license_ok(canonical_license(d, lic), + return a if all(oe.license.license_ok(canonical_license(d, lic), dont_want_licenses) for lic in a) else b try: licenses = oe.license.flattened_licenses(license, choose_lic_set) except oe.license.LicenseError as exc: bb.fatal('%s: %s' % (d.getVar('P'), exc)) - return any(not oe.license.license_ok(canonical_license(d, l), \ - dont_want_licenses) for l in licenses) + + incompatible_lic = [] + for l in licenses: + license = canonical_license(d, l) + if not oe.license.license_ok(license, dont_want_licenses): + incompatible_lic.append(license) + + return sorted(incompatible_lic) def incompatible_license(d, dont_want_licenses, package=None): """
Signature · 1 line
2.21.0
PATCH 4/7 base.bbclass: Simplify the check for whitelisted licenses
In reply to Peter KjellerstedtAfter a number of rewrites, the code checking if a package has been whitelisted for an incompatible license was calculating the whitelisted packages twice (as 'whitelist' and as 'incompatwl').
--- meta/classes/base.bbclass | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
meta/classes/base.bbclass +4 -9
@@ -531,23 +531,21 @@ python () { bad_licenses = expand_wildcard_licenses(d, bad_licenses) whitelist = [] - incompatwl = [] for lic in bad_licenses: spdx_license = return_spdx(d, lic) whitelist.extend((d.getVar("WHITELIST_" + lic) or "").split()) if spdx_license: whitelist.extend((d.getVar("WHITELIST_" + spdx_license) or "").split()) + + if pn in whitelist: ''' We need to track what we are whitelisting and why. If pn is incompatible we need to be able to note that the image that is created may infact contain incompatible licenses despite INCOMPATIBLE_LICENSE being set. ''' - incompatwl.extend((d.getVar("WHITELIST_" + lic) or "").split()) - if spdx_license: - incompatwl.extend((d.getVar("WHITELIST_" + spdx_license) or "").split()) - - if not pn in whitelist: + bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn) + else: pkgs = d.getVar('PACKAGES').split() skipped_pkgs = [] unskipped_pkgs = []
@@ -567,9 +565,6 @@ python () { elif all_skipped or incompatible_license(d, bad_licenses): bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license)) raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license) - elif pn in whitelist: - if pn in incompatwl: - bb.note("INCLUDING " + pn + " as buildable despite INCOMPATIBLE_LICENSE because it has been whitelisted") # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a # superset of all per-package licenses. We do not do advanced (pattern)
Signature · 1 line
2.21.0
PATCH 5/7 base.bbclass: Report only the licenses that are incompatible for a package
In reply to Peter KjellerstedtInstead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.
--- meta/classes/base.bbclass | 26 ++++++++++++------- .../oeqa/selftest/cases/incompatible_lic.py | 6 ++--- 2 files changed, 19 insertions(+), 13 deletions(-)
meta/classes/base.bbclass +16 -10
@@ -547,24 +547,30 @@ python () { bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn) else: pkgs = d.getVar('PACKAGES').split() - skipped_pkgs = [] + skipped_pkgs = {} unskipped_pkgs = [] for pkg in pkgs: - if incompatible_license(d, bad_licenses, pkg): - skipped_pkgs.append(pkg) + incompatible_lic = incompatible_license(d, bad_licenses, pkg) + if incompatible_lic: + skipped_pkgs[pkg] = incompatible_lic else: unskipped_pkgs.append(pkg) - all_skipped = skipped_pkgs and not unskipped_pkgs if unskipped_pkgs: for pkg in skipped_pkgs: - bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + license) + bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg]))) mlprefix = d.getVar('MLPREFIX') - d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, 1) + d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, ' '.join(skipped_pkgs[pkg])) for pkg in unskipped_pkgs: - bb.debug(1, "INCLUDING the package " + pkg) - elif all_skipped or incompatible_license(d, bad_licenses): - bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license)) - raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license) + bb.debug(1, "Including the package %s" % pkg) + else: + incompatible_lic = incompatible_license(d, bad_licenses) + for pkg in skipped_pkgs: + incompatible_lic += skipped_pkgs[pkg] + incompatible_lic = sorted(list(set(incompatible_lic))) + + if incompatible_lic: + bb.debug(1, "Skipping recipe %s because of incompatible license(s): %s" % (pn, ' '.join(incompatible_lic))) + raise bb.parse.SkipRecipe("it has incompatible license(s): %s" % ' '.join(incompatible_lic)) # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a # superset of all per-package licenses. We do not do advanced (pattern)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake class IncompatibleLicenseTests(OESelftestTestCase): def lic_test(self, pn, pn_lic, lic): - error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has an incompatible license: %s' % (pn, pn, pn_lic) + error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has incompatible license(s): %s' % (pn, pn, pn_lic) self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))
@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase): # license cannot be built when INCOMPATIBLE_LICENSE contains this SPDX # license def test_incompatible_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0') # Verify that a package with an alias (from SPDXLICENSEMAP) to an SPDX # license cannot be built when INCOMPATIBLE_LICENSE contains this alias def test_incompatible_alias_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3') # Verify that a package with a non-SPDX license (neither in # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
Signature · 1 line
2.21.0
Re: [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a package
Hi Peter,
Quoted 6 lines · Peter Kjellerstedt
Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.
Signed-off-by Peter Kjellerstedt <[email protected]>---
[...]
Quoted 16 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1
-1@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake class IncompatibleLicenseTests(OESelftestTestCase): def lic_test(self, pn, pn_lic, lic): - error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has an incompatible license: %s' % (pn, pn, pn_lic) + error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has incompatible license(s): %s' % (pn, pn, pn_lic) self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase): # license cannot be built when INCOMPATIBLE_LICENSE contains this SPDX # license
meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 36
Quoted 10 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +2
-2def test_incompatible_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0') # Verify that a package with an alias (from SPDXLICENSEMAP) to an SPDX # license cannot be built when INCOMPATIBLE_LICENSE contains this alias def test_incompatible_alias_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')
We currently have 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
They test that the alias system (SPXLICENSEMAP) still works, which is rather important since we never enforced the use of licenses from SRC_DISTRIBUTE_LICENSES in LICENSE in recipes and allowed the use of aliases (hence SPXLICENSEMAP).
If I'm not mistaken this change induces the following 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
which duplicates two tests.
The questions are: which error warranted this code change?
Are SPDXLICENSEMAP aliases still working? Can it still be checked?
If the original tests are indeed not needed anymore, we need to explicit why (and we can actually just remove them).
I'm still advocating for new tests with wildcards :)
Thanks,
Quentin
Re: [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a package
Quoted 63 lines
-----Original Message-----
From: Quentin Schulz <[email protected]>
Sent: den 9 december 2019 11:20
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a packageHi Peter,
Quoted 6 lines · Peter Kjellerstedt
Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.
Signed-off-by Peter Kjellerstedt <[email protected]>---
[...]
Review
Quoted 1 line
(quoted diff) +0
-0
b/meta/lib/oeqa/selftest/cases/incompatible_lic.pymeta/lib/oeqa/selftest/cases/incompatible_lic.py from oeqa.utils.commands import bitbake · line 7
Quoted 8 lines
index
ad878571b5..9d1e801117100644meta/lib/oeqa/selftest/cases/incompatible_lic.py +0
-1@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake class IncompatibleLicenseTests(OESelftestTestCase): def lic_test(self, pn, pn_lic, lic): - error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: ithas an incompatible license: %s' % (pn, pn,
pn_lic)meta/lib/oeqa/selftest/cases/incompatible_lic.py from oeqa.utils.commands import bitbake · line 7
Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1
-0+ error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: ithas incompatible license(s): %s' % (pn, pn,
pn_lic)meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 27
Quoted 5 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0
-0self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase): # license cannot be built when INCOMPATIBLE_LICENSE contains thisSPDX
meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase):
Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1
-1# license def test_incompatible_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0')
meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 31
Quoted 2 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0
-0# Verify that a package with an alias (from SPDXLICENSEMAP) to anSPDX
meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 32
Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0
-0# license cannot be built when INCOMPATIBLE_LICENSE contains thisalias
meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 36
Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1
-1def test_incompatible_alias_spdx_license_alias(self): - self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3') + self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')We currently have 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 inINCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 inINCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPL-3.0 inINCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPLv3 inINCOMPATIBLE_LICENSEThey test that the alias system (SPXLICENSEMAP) still works, which is rather important since we never enforced the use of licenses from
SRC_DISTRIBUTE_LICENSESin LICENSE in recipes and allowed the use of aliases (hence SPXLICENSEMAP).
Quoted 7 lines
If I'm not mistaken this change induces the following 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 inINCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 inINCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 inINCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 inINCOMPATIBLE_LICENSEwhich duplicates two tests.
Actually not. Please note that different recipes are used for the first two tests (incompatible-license.bb) and the last two tests (incompatible-license-alias.bb).
Quoted 1 line
The questions are: which error warranted this code change?
Before, the LICENSE string was reported as is, whereas after my changes, the canonical SPDX names are reported.
Quoted 1 line
Are SPDXLICENSEMAP aliases still working? Can it still be checked?
It is still working and checked.
Quoted 4 lines
If the original tests are indeed not needed anymore, we need to explicit why (and we can actually just remove them).
I'm still advocating for new tests with wildcards :)
Ah, now I think I get your comment from the previous mail. Yes, I should probably add some tests that validate the wildcard support.
Quoted 2 lines
Thanks,
Quentin
//Peter
PATCH 6/7 package.bbclass: Report only the licenses that are incompatible
In reply to Peter KjellerstedtWhen excluding a package from being packaged due to incompatible licenses, report the license(s) that are actually incompatible.
--- meta/classes/package.bbclass | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
meta/classes/package.bbclass +3 -2
@@ -1263,8 +1263,9 @@ python populate_packages () { # Handle LICENSE_EXCLUSION package_list = [] for pkg in packages: - if d.getVar('LICENSE_EXCLUSION-' + pkg): - msg = "%s has an incompatible license. Excluding from packaging." % pkg + licenses = d.getVar('LICENSE_EXCLUSION-' + pkg) + if licenses: + msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, ' '.join(licenses)) package_qa_handle_error("incompatible-license", msg, d) else: package_list.append(pkg)
Signature · 1 line
2.21.0
PATCH 7/7 license_image.bbclass: Report only the licenses that are incompatible
In reply to Peter KjellerstedtInstead of reporting ${LICENSE} when a package cannot be installed into an image because it is using an incompatible license, report the license(s) that are actually incompatible.
--- meta/classes/license_image.bbclass | 5 +++-- meta/lib/oeqa/selftest/cases/incompatible_lic.py | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
meta/classes/license_image.bbclass +3 -2
@@ -51,8 +51,9 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): for pkg in sorted(pkg_dic): if bad_licenses and pkg not in whitelist: try: - if incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"]): - bb.fatal("Package %s has an incompatible license %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"])) + licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"]) + if licenses: + bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(licenses))) (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \ oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"], bad_licenses, canonical_license, d)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +2 -2
@@ -49,7 +49,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0" def test_bash_default(self): self.write_config(self.default_config()) - error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ and cannot be installed into the image." + error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+" result = bitbake('core-image-minimal', ignore_status=True) if error_msg not in result.output:
@@ -57,7 +57,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0" def test_bash_and_license(self): self.write_config(self.default_config() + '\nLICENSE_append_pn-bash = " & SomeLicense"') - error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ & SomeLicense and cannot be installed into the image." + error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+" result = bitbake('core-image-minimal', ignore_status=True) if error_msg not in result.output:
Signature · 1 line
2.21.0
Hi Peter,
Quoted 9 lines · Peter Kjellerstedt
Previously, there was
SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used byexpand_wildcard_licenses()to expand any wildcards used in, e.g.,INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.
Quoted 6 lines
This introduces a new (documented) variable,
AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead ofSRC_DISTRIBUTE_LICENSESwhen expanding wildcards.
Great addition, thanks!
Quoted 12 lines
Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/license.bbclass | 27 ++++++++++++++++--- meta/conf/documentation.conf | 1 + .../oeqa/selftest/cases/incompatible_lic.py | 6 ++--- 3 files changed, 27 insertions(+), 7 deletions(-)meta/classes/license.bbclass +0
-0
meta/classes/license.bbclass def canonical_license(d, license): · line 255
Quoted 5 lines
meta/classes/license.bbclass +1
-1@@ -252,7 +252,7 @@ def canonical_license(d, license): """ Return the canonical (SPDX) form of the license if available (so GPLv3 becomes GPL-3.0), for the license named 'X+', return canonical form of - 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), + 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
Might not be fully awake yet but looks like it's a spurious change here. Though, there's actually a typo in availabel that could be fixed :)
[...]
Quoted 8 lines
meta/classes/license.bbclass +1
-1# Verify that a package with a non-SPDX license (neither in - # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when + # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when # INCOMPATIBLE_LICENSE contains this licensedef
test_incompatible_nonspdx_license(self):
self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
I think it could be great to have a new test in that selftest file to check that INCOMPATIBLE_LICENSE works with wildcards for licenses not in SRC_DISTRIBUTE_LICENSES.
Thanks,
Quentin
Quoted 36 lines
-----Original Message-----
From: Quentin Schulz <[email protected]>
Sent: den 9 december 2019 10:41
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 1/7] license.bbclass: IntroduceAVAILABLE_LICENSESthat lists all licensesHi Peter,
Quoted 9 lines · Peter Kjellerstedt
Previously, there was
SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used byexpand_wildcard_licenses()to expand any wildcards used in, e.g.,INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.Quoted 6 lines
This introduces a new (documented) variable,
AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead ofSRC_DISTRIBUTE_LICENSESwhen expanding wildcards.Great addition, thanks!
Quoted 7 lines
Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/license.bbclass | 27 ++++++++++++++++--- meta/conf/documentation.conf | 1 + .../oeqa/selftest/cases/incompatible_lic.py | 6 ++--- 3 files changed, 27 insertions(+), 7 deletions(-)
Quoted 12 lines
meta/classes/license.bbclass def canonical_license(d, license): · line 255
Quoted 10 lines
meta/classes/license.bbclass +1
-1@@ -252,7 +252,7 @@ def canonical_license(d, license): """ Return the canonical (SPDX) form of the license if available (so GPLv3 becomes GPL-3.0), for the license named 'X+', return canonical form of - 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), + 'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),Might not be fully awake yet but looks like it's a spurious change here.
There was a trailing space on that line that I removed.
Quoted 1 line
Though, there's actually a typo in availabel that could be fixed :)
Now that you mention it, there actually is. Might as well fix that too now that it has been discovered.
Quoted 10 lines
[...]
meta/classes/license.bbclass
Quoted 8 lines
# Verify that a package with a non-SPDX license (neither in
meta/classes/license.bbclass +1
-2- # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when + # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when # INCOMPATIBLE_LICENSE contains this license def test_incompatible_nonspdx_license(self): self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense') --
Quoted 3 lines
I think it could be great to have a new test in that selftest file to check that
INCOMPATIBLE_LICENSEworks with wildcards for licenses not inSRC_DISTRIBUTE_LICENSES.
Well, I remove SRC_DISTRIBUTE_LICENSES in the next commit so I do not see what that would add?
Quoted 2 lines
Thanks,
Quentin
//Peter