Skip to contentSkip to conversation

license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

PATCH Superseded Peter Kjellerstedt 2 participants 11 messages Dec 7, 2019 – Dec 11, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/7 license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

---
 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +23 -4
index 648a4d7892..20af6d34b6 100644
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), 
+    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
     or the passed license if there is no canonical form.
     """
     lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license):
             lic += '+'
     return lic or license
 
+def available_licenses(d):
+    """
+    Return the available licenses by searching the directories specified by
+    COMMON_LICENSE_DIR and LICENSE_PATH.
+    """
+    lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' +
+                (d.getVar('LICENSE_PATH') or '')).split()
+
+    licenses = []
+    for lic_dir in lic_dirs:
+        licenses += os.listdir(lic_dir)
+
+    licenses = sorted(licenses)
+    return licenses
+
+# Only determine the list of all available licenses once. This assumes that any
+# additions to LICENSE_PATH have been done before this file is parsed.
+AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}"
+
 def expand_wildcard_licenses(d, wildcard_licenses):
     """
-    Return actual spdx format license names if wildcard used. We expand
-    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
+    Return actual spdx format license names if wildcards are used. We expand
+    wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES.
     """
     import fnmatch
     licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
-    spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or '').split()
+    spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:
         licenses += fnmatch.filter(spdx_lics, wld_lic)
 
meta/conf/documentation.conf +1 -0
index fca36f3cf6..0b21d1f63e 100644
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib p
 AUTHOR[doc] = "Email address used to contact the original author(s) in order to send patches and forward bugs."
 AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the syslinux bootloader."
 AUTOREV[doc] = "When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository."
+AVAILABLE_LICENSES[doc] = "List of licenses found in the directories specified by COMMON_LICENSE_DIR and LICENSE_PATH."
 
 #B
 
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
index 904b5b4094..ad878571b5 100644
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         if error_msg not in result.output:
             raise AssertionError(result.output)
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX license
     def test_incompatible_spdx_license(self):
         self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0')
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in
     # SPDXLICENSEMAP) of this SPDX license
     def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
Signature · 1 line
2.21.0
#1 lore raw

PATCH 2/7 licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variable

The SRC_DISTRIBUTE_LICENSES variable and its static list of licenses has been replaced by AVAILABLE_LICENSES, which automatically contains all available licenses.

---
 meta/conf/licenses.conf | 43 -----------------------------------------
 1 file changed, 43 deletions(-)
meta/conf/licenses.conf +0 -43
index 7b01c57632..1c586a3bc6 100644
@@ -1,44 +1,3 @@
-# These aren't actually used anywhere that I can tell
-# They may be in the future (or are used by someone else
-# For completion sake, I've updated them
-SRC_DISTRIBUTE_LICENSES += "AAL Adobe AFL-1.2 AFL-2.0 AFL-2.1 AFL-3.0"
-SRC_DISTRIBUTE_LICENSES += "AGPL-3.0 ANTLR-PD Apache-1.0 Apache-1.1 Apache-2.0"
-SRC_DISTRIBUTE_LICENSES += "APL-1.0 APSL-1.0 APSL-1.1 APSL-1.2 APSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Artistic-1.0 Artistic-2.0 BitstreamVera BSD"
-SRC_DISTRIBUTE_LICENSES += "BSD-2-Clause BSD-3-Clause BSD-4-Clause BSL-1.0"
-SRC_DISTRIBUTE_LICENSES += "CATOSL-1.1 CC0-1.0 CC-BY-1.0 CC-BY-2.0 CC-BY-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-3.0 CC-BY-NC-1.0 CC-BY-NC-2.0 CC-BY-NC-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-3.0 CC-BY-NC-ND-1.0 CC-BY-NC-ND-2.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-ND-2.5 CC-BY-NC-ND-3.0 CC-BY-NC-SA-1.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-SA-2.0 CC-BY-NC-SA-2.5 CC-BY-NC-SA-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-ND-1.0 CC-BY-ND-2.0 CC-BY-ND-2.5 CC-BY-ND-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0 CC-BY-SA-2.5 CC-BY-SA-3.0 CC-BY-SA-4.0"
-SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B CECILL-C"
-SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL"
-SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0 EFL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair Frameworx-1.0"
-SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-classpath-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-font-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2-with-bison-exception GPL-3.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "gSOAP-1 gSOAP-1.3b HPND IPA IPL-1.0 ISC LGPL-2.0"
-SRC_DISTRIBUTE_LICENSES += "LGPL-2.1 LGPL-3.0 Libpng LPL-1.02 LPPL-1.0 LPPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "LPPL-1.2 LPPL-1.3c MirOS MIT Motosoto MPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "MPL-1.1 MS-PL MS-RL Multics NASA-1.3 Nauman NCSA"
-SRC_DISTRIBUTE_LICENSES += "NGPL Nokia NPOSL-3.0 NTP OASIS OCLC-2.0 ODbL-1.0"
-SRC_DISTRIBUTE_LICENSES += "OFL-1.1 OGTSL OLDAP-2.8 OpenSSL OSL-1.0 OSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "OSL-3.0 PD PHP-3.0 PostgreSQL Proprietary"
-SRC_DISTRIBUTE_LICENSES += "Python-2.0 QPL-1.0 RHeCos-1 RHeCos-1.1 RPL-1.5"
-SRC_DISTRIBUTE_LICENSES += "RPSL-1.0 RSCPL Ruby SAX-PD SGI-1 Simple-2.0 Sleepycat"
-SRC_DISTRIBUTE_LICENSES += "SPL-1.0 SugarCRM-1 SugarCRM-1.1.3 UCB VSL-1.0 W3C"
-SRC_DISTRIBUTE_LICENSES += "Watcom-1.0 WXwindows XFree86-1.0 XFree86-1.1 Xnet XSL YPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "Zimbra-1.3 Zlib ZPL-1.1 ZPL-2.0 ZPL-2.1"
-
 # Standards are great! Everyone has their own. In an effort to standardize licensing
 # names, common-licenses will use the SPDX standard license names. In order to not
 # break the non-standardized license names that we find in LICENSE, we'll set
@@ -188,5 +147,3 @@ FOSS_BASE_URL = "http://localhost/repo/?mod=spdx_license_once"
 FOSS_SERVER = "${FOSS_BASE_URL}&fullSPDXFlag=${FOSS_FULL_SPDX}&noCopyright=${FOSS_NO_COPYRIGHT}&recursiveUnpack=${FOSS_RECURSIVE_UNPACK}"
 
 FOSS_WGET_FLAGS = "-qO - --no-check-certificate --timeout=0"
-
-
Signature · 1 line
2.21.0
#2 lore raw

PATCH 3/7 license.bbclass: Make incompatible_pkg_license return incompatible lics

In reply to Peter Kjellerstedt

This makes it possible to report the incompatible licenses.

---
 meta/classes/license.bbclass | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
meta/classes/license.bbclass +9 -3
index 20af6d34b6..8094883a87 100644
@@ -310,15 +310,21 @@ def incompatible_pkg_license(d, dont_want_licenses, license):
     # Handles an "or" or two license sets provided by
     # flattened_licenses(), pick one that works if possible.
     def choose_lic_set(a, b):
-        return a if all(oe.license.license_ok(canonical_license(d, lic), 
+        return a if all(oe.license.license_ok(canonical_license(d, lic),
                             dont_want_licenses) for lic in a) else b
 
     try:
         licenses = oe.license.flattened_licenses(license, choose_lic_set)
     except oe.license.LicenseError as exc:
         bb.fatal('%s: %s' % (d.getVar('P'), exc))
-    return any(not oe.license.license_ok(canonical_license(d, l), \
-               dont_want_licenses) for l in licenses)
+
+    incompatible_lic = []
+    for l in licenses:
+        license = canonical_license(d, l)
+        if not oe.license.license_ok(license, dont_want_licenses):
+            incompatible_lic.append(license)
+
+    return sorted(incompatible_lic)
 
 def incompatible_license(d, dont_want_licenses, package=None):
     """
Signature · 1 line
2.21.0
#3 lore raw

PATCH 4/7 base.bbclass: Simplify the check for whitelisted licenses

In reply to Peter Kjellerstedt

After a number of rewrites, the code checking if a package has been whitelisted for an incompatible license was calculating the whitelisted packages twice (as 'whitelist' and as 'incompatwl').

---
 meta/classes/base.bbclass | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
meta/classes/base.bbclass +4 -9
index 3306b316bb..5176168cf8 100644
@@ -531,23 +531,21 @@ python () {
             bad_licenses = expand_wildcard_licenses(d, bad_licenses)
 
             whitelist = []
-            incompatwl = []
             for lic in bad_licenses:
                 spdx_license = return_spdx(d, lic)
                 whitelist.extend((d.getVar("WHITELIST_" + lic) or "").split())
                 if spdx_license:
                     whitelist.extend((d.getVar("WHITELIST_" + spdx_license) or "").split())
+
+            if pn in whitelist:
                 '''
                 We need to track what we are whitelisting and why. If pn is
                 incompatible we need to be able to note that the image that
                 is created may infact contain incompatible licenses despite
                 INCOMPATIBLE_LICENSE being set.
                 '''
-                incompatwl.extend((d.getVar("WHITELIST_" + lic) or "").split())
-                if spdx_license:
-                    incompatwl.extend((d.getVar("WHITELIST_" + spdx_license) or "").split())
-
-            if not pn in whitelist:
+                bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn)
+            else:
                 pkgs = d.getVar('PACKAGES').split()
                 skipped_pkgs = []
                 unskipped_pkgs = []
@@ -567,9 +565,6 @@ python () {
                 elif all_skipped or incompatible_license(d, bad_licenses):
                     bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license))
                     raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license)
-            elif pn in whitelist:
-                if pn in incompatwl:
-                    bb.note("INCLUDING " + pn + " as buildable despite INCOMPATIBLE_LICENSE because it has been whitelisted")
 
         # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a
         # superset of all per-package licenses. We do not do advanced (pattern)
Signature · 1 line
2.21.0
#4 lore raw

PATCH 5/7 base.bbclass: Report only the licenses that are incompatible for a package

In reply to Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.

---
 meta/classes/base.bbclass                     | 26 ++++++++++++-------
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 2 files changed, 19 insertions(+), 13 deletions(-)
meta/classes/base.bbclass +16 -10
index 5176168cf8..87d8ba2cf8 100644
@@ -547,24 +547,30 @@ python () {
                 bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn)
             else:
                 pkgs = d.getVar('PACKAGES').split()
-                skipped_pkgs = []
+                skipped_pkgs = {}
                 unskipped_pkgs = []
                 for pkg in pkgs:
-                    if incompatible_license(d, bad_licenses, pkg):
-                        skipped_pkgs.append(pkg)
+                    incompatible_lic = incompatible_license(d, bad_licenses, pkg)
+                    if incompatible_lic:
+                        skipped_pkgs[pkg] = incompatible_lic
                     else:
                         unskipped_pkgs.append(pkg)
-                all_skipped = skipped_pkgs and not unskipped_pkgs
                 if unskipped_pkgs:
                     for pkg in skipped_pkgs:
-                        bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + license)
+                        bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
                         mlprefix = d.getVar('MLPREFIX')
-                        d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, 1)
+                        d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, ' '.join(skipped_pkgs[pkg]))
                     for pkg in unskipped_pkgs:
-                        bb.debug(1, "INCLUDING the package " + pkg)
-                elif all_skipped or incompatible_license(d, bad_licenses):
-                    bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license))
-                    raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license)
+                        bb.debug(1, "Including the package %s" % pkg)
+                else:
+                    incompatible_lic = incompatible_license(d, bad_licenses)
+                    for pkg in skipped_pkgs:
+                        incompatible_lic += skipped_pkgs[pkg]
+                    incompatible_lic = sorted(list(set(incompatible_lic)))
+
+                    if incompatible_lic:
+                        bb.debug(1, "Skipping recipe %s because of incompatible license(s): %s" % (pn, ' '.join(incompatible_lic)))
+                        raise bb.parse.SkipRecipe("it has incompatible license(s): %s" % ' '.join(incompatible_lic))
 
         # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a
         # superset of all per-package licenses. We do not do advanced (pattern)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
index ad878571b5..9d1e801117 100644
@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake
 class IncompatibleLicenseTests(OESelftestTestCase):
 
     def lic_test(self, pn, pn_lic, lic):
-        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has an incompatible license: %s' % (pn, pn, pn_lic)
+        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has incompatible license(s): %s' % (pn, pn, pn_lic)
 
         self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))
 
@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
     # license cannot be built when INCOMPATIBLE_LICENSE contains this SPDX
     # license
     def test_incompatible_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0')
 
     # Verify that a package with an alias (from SPDXLICENSEMAP) to an SPDX
     # license cannot be built when INCOMPATIBLE_LICENSE contains this alias
     def test_incompatible_alias_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
     # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
Signature · 1 line
2.21.0
#5 lore raw

Re: [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a package

Hi Peter,

Quoted 6 lines · Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.

---

[...]

Quoted 16 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -1
index ad878571b5..9d1e801117 100644
@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake
 class IncompatibleLicenseTests(OESelftestTestCase):
 
     def lic_test(self, pn, pn_lic, lic):
-        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has an incompatible license: %s' % (pn, pn, pn_lic)
+        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has incompatible license(s): %s' % (pn, pn, pn_lic)
 
         self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))
 
@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
     # license cannot be built when INCOMPATIBLE_LICENSE contains this SPDX
     # license

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 36

Quoted 10 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +2 -2
     def test_incompatible_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0')
 
     # Verify that a package with an alias (from SPDXLICENSEMAP) to an SPDX
     # license cannot be built when INCOMPATIBLE_LICENSE contains this alias
     def test_incompatible_alias_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')
 

We currently have 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE

They test that the alias system (SPXLICENSEMAP) still works, which is rather important since we never enforced the use of licenses from SRC_DISTRIBUTE_LICENSES in LICENSE in recipes and allowed the use of aliases (hence SPXLICENSEMAP).

If I'm not mistaken this change induces the following 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE

which duplicates two tests.

The questions are: which error warranted this code change?

Are SPDXLICENSEMAP aliases still working? Can it still be checked?

If the original tests are indeed not needed anymore, we need to explicit why (and we can actually just remove them).

I'm still advocating for new tests with wildcards :)

Thanks,
Quentin

#6 lore raw

Re: [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a package

Quoted 63 lines

-----Original Message-----
From: Quentin Schulz <[email protected]>
Sent: den 9 december 2019 11:20
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 5/7] base.bbclass: Report only the licenses that are incompatible for a package

Hi Peter,

Quoted 6 lines · Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.

---

[...]

Review

Quoted 1 line
(quoted diff) +0 -0

b/meta/lib/oeqa/selftest/cases/incompatible_lic.py

meta/lib/oeqa/selftest/cases/incompatible_lic.py from oeqa.utils.commands import bitbake · line 7

Quoted 8 lines

index ad878571b5..9d1e801117 100644

meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -1
@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake
 class IncompatibleLicenseTests(OESelftestTestCase):
 
     def lic_test(self, pn, pn_lic, lic):
-        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it

has an incompatible license: %s' % (pn, pn, pn_lic)

meta/lib/oeqa/selftest/cases/incompatible_lic.py from oeqa.utils.commands import bitbake · line 7

Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -0
+        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it

has incompatible license(s): %s' % (pn, pn, pn_lic)

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 27

Quoted 5 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -0
         self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))
 
@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
     # license cannot be built when INCOMPATIBLE_LICENSE contains this

SPDX

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase):

Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -1
     # license
     def test_incompatible_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-

3.0')

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 31

Quoted 2 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -0
     # Verify that a package with an alias (from SPDXLICENSEMAP) to an

SPDX

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 32

Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -0
     # license cannot be built when INCOMPATIBLE_LICENSE contains this

alias

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 36

Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -1
     def test_incompatible_alias_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')
 

We currently have 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPLv3 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE

They test that the alias system (SPXLICENSEMAP) still works, which is rather important since we never enforced the use of licenses from SRC_DISTRIBUTE_LICENSES in LICENSE in recipes and allowed the use of aliases (hence SPXLICENSEMAP).

Quoted 7 lines

If I'm not mistaken this change induces the following 4 tests:
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPL-3.0 in INCOMPATIBLE_LICENSE
- GPL-3.0 in LICENSE field in recipe and GPLv3 in INCOMPATIBLE_LICENSE

which duplicates two tests.

Actually not. Please note that different recipes are used for the first two tests (incompatible-license.bb) and the last two tests (incompatible-license-alias.bb).

Quoted 1 line

The questions are: which error warranted this code change?

Before, the LICENSE string was reported as is, whereas after my changes, the canonical SPDX names are reported.

Quoted 1 line

Are SPDXLICENSEMAP aliases still working? Can it still be checked?

It is still working and checked.

Quoted 4 lines

If the original tests are indeed not needed anymore, we need to explicit why (and we can actually just remove them).

I'm still advocating for new tests with wildcards :)

Ah, now I think I get your comment from the previous mail. Yes, I should probably add some tests that validate the wildcard support.

Quoted 2 lines

Thanks,
Quentin

//Peter

#7 lore raw

PATCH 6/7 package.bbclass: Report only the licenses that are incompatible

In reply to Peter Kjellerstedt

When excluding a package from being packaged due to incompatible licenses, report the license(s) that are actually incompatible.

---
 meta/classes/package.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
meta/classes/package.bbclass +3 -2
index e0d6ff6701..6f5939e12f 100644
@@ -1263,8 +1263,9 @@ python populate_packages () {
     # Handle LICENSE_EXCLUSION
     package_list = []
     for pkg in packages:
-        if d.getVar('LICENSE_EXCLUSION-' + pkg):
-            msg = "%s has an incompatible license. Excluding from packaging." % pkg
+        licenses = d.getVar('LICENSE_EXCLUSION-' + pkg)
+        if licenses:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, ' '.join(licenses))
             package_qa_handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
Signature · 1 line
2.21.0
#8 lore raw

PATCH 7/7 license_image.bbclass: Report only the licenses that are incompatible

In reply to Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package cannot be installed into an image because it is using an incompatible license, report the license(s) that are actually incompatible.

---
 meta/classes/license_image.bbclass               | 5 +++--
 meta/lib/oeqa/selftest/cases/incompatible_lic.py | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
meta/classes/license_image.bbclass +3 -2
index b5399b6d96..a8c72da3cb 100644
@@ -51,8 +51,9 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
         for pkg in sorted(pkg_dic):
             if bad_licenses and pkg not in whitelist:
                 try:
-                    if incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"]):
-                        bb.fatal("Package %s has an incompatible license %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"]))
+                    licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                    if licenses:
+                        bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(licenses)))
                     (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                         oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
                         bad_licenses, canonical_license, d)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +2 -2
index 9d1e801117..458a940618 100644
@@ -49,7 +49,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
 
     def test_bash_default(self):
         self.write_config(self.default_config())
-        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ and cannot be installed into the image."
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+"
 
         result = bitbake('core-image-minimal', ignore_status=True)
         if error_msg not in result.output:
@@ -57,7 +57,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
 
     def test_bash_and_license(self):
         self.write_config(self.default_config() + '\nLICENSE_append_pn-bash = " & SomeLicense"')
-        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ & SomeLicense and cannot be installed into the image."
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+"
 
         result = bitbake('core-image-minimal', ignore_status=True)
         if error_msg not in result.output:
Signature · 1 line
2.21.0
#9 lore raw
In reply to Peter Kjellerstedt

Hi Peter,

Quoted 9 lines · Peter Kjellerstedt

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

Quoted 6 lines

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

Great addition, thanks!

Quoted 12 lines
---
 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +0 -0
index 648a4d7892..20af6d34b6 100644

meta/classes/license.bbclass def canonical_license(d, license): · line 255

Quoted 5 lines
meta/classes/license.bbclass +1 -1
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), 
+    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),

Might not be fully awake yet but looks like it's a spurious change here. Though, there's actually a typo in availabel that could be fixed :)

[...]

Quoted 8 lines
meta/classes/license.bbclass +1 -1
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license

def test_incompatible_nonspdx_license(self):
self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')

I think it could be great to have a new test in that selftest file to check that INCOMPATIBLE_LICENSE works with wildcards for licenses not in SRC_DISTRIBUTE_LICENSES.

Thanks,
Quentin

#10 lore raw
Quoted 36 lines

-----Original Message-----
From: Quentin Schulz <[email protected]>
Sent: den 9 december 2019 10:41
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 1/7] license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

Hi Peter,

Quoted 9 lines · Peter Kjellerstedt

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

Quoted 6 lines

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

Great addition, thanks!

Quoted 7 lines
---
 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
Quoted 12 lines

meta/classes/license.bbclass def canonical_license(d, license): · line 255

Quoted 10 lines
meta/classes/license.bbclass +1 -1
index 648a4d7892..20af6d34b6 100644
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
+    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),

Might not be fully awake yet but looks like it's a spurious change here.

There was a trailing space on that line that I removed.

Quoted 1 line

Though, there's actually a typo in availabel that could be fixed :)

Now that you mention it, there actually is. Might as well fix that too now that it has been discovered.

Quoted 10 lines

[...]

meta/classes/license.bbclass

Quoted 8 lines

# Verify that a package with a non-SPDX license (neither in

meta/classes/license.bbclass +1 -2
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
--
Quoted 3 lines

I think it could be great to have a new test in that selftest file to check that INCOMPATIBLE_LICENSE works with wildcards for licenses not in SRC_DISTRIBUTE_LICENSES.

Well, I remove SRC_DISTRIBUTE_LICENSES in the next commit so I do not see what that would add?

Quoted 2 lines

Thanks,
Quentin

//Peter