Skip to contentSkip to conversation

license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

PATCH In review Peter Kjellerstedt 4 participants 11 messages Dec 11, 2019 – Aug 22, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/8 license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

---

PATCHv2: Correct spelling of 'available'.

 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +23 -4
index 648a4d7892..c388740003 100644
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), 
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
     or the passed license if there is no canonical form.
     """
     lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license):
             lic += '+'
     return lic or license
 
+def available_licenses(d):
+    """
+    Return the available licenses by searching the directories specified by
+    COMMON_LICENSE_DIR and LICENSE_PATH.
+    """
+    lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' +
+                (d.getVar('LICENSE_PATH') or '')).split()
+
+    licenses = []
+    for lic_dir in lic_dirs:
+        licenses += os.listdir(lic_dir)
+
+    licenses = sorted(licenses)
+    return licenses
+
+# Only determine the list of all available licenses once. This assumes that any
+# additions to LICENSE_PATH have been done before this file is parsed.
+AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}"
+
 def expand_wildcard_licenses(d, wildcard_licenses):
     """
-    Return actual spdx format license names if wildcard used. We expand
-    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
+    Return actual spdx format license names if wildcards are used. We expand
+    wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES.
     """
     import fnmatch
     licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
-    spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or '').split()
+    spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:
         licenses += fnmatch.filter(spdx_lics, wld_lic)
 
meta/conf/documentation.conf +1 -0
index fca36f3cf6..0b21d1f63e 100644
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib p
 AUTHOR[doc] = "Email address used to contact the original author(s) in order to send patches and forward bugs."
 AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the syslinux bootloader."
 AUTOREV[doc] = "When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository."
+AVAILABLE_LICENSES[doc] = "List of licenses found in the directories specified by COMMON_LICENSE_DIR and LICENSE_PATH."
 
 #B
 
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
index 904b5b4094..ad878571b5 100644
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         if error_msg not in result.output:
             raise AssertionError(result.output)
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX license
     def test_incompatible_spdx_license(self):
         self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0')
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in
     # SPDXLICENSEMAP) of this SPDX license
     def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
Signature · 1 line
2.21.0
#1 lore raw

PATCH v3 2/8 licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variable

The SRC_DISTRIBUTE_LICENSES variable and its static list of licenses has been replaced by AVAILABLE_LICENSES, which automatically contains all available licenses.

---
 meta/conf/licenses.conf | 43 -----------------------------------------
 1 file changed, 43 deletions(-)
meta/conf/licenses.conf +0 -43
index 7b01c57632..1c586a3bc6 100644
@@ -1,44 +1,3 @@
-# These aren't actually used anywhere that I can tell
-# They may be in the future (or are used by someone else
-# For completion sake, I've updated them
-SRC_DISTRIBUTE_LICENSES += "AAL Adobe AFL-1.2 AFL-2.0 AFL-2.1 AFL-3.0"
-SRC_DISTRIBUTE_LICENSES += "AGPL-3.0 ANTLR-PD Apache-1.0 Apache-1.1 Apache-2.0"
-SRC_DISTRIBUTE_LICENSES += "APL-1.0 APSL-1.0 APSL-1.1 APSL-1.2 APSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Artistic-1.0 Artistic-2.0 BitstreamVera BSD"
-SRC_DISTRIBUTE_LICENSES += "BSD-2-Clause BSD-3-Clause BSD-4-Clause BSL-1.0"
-SRC_DISTRIBUTE_LICENSES += "CATOSL-1.1 CC0-1.0 CC-BY-1.0 CC-BY-2.0 CC-BY-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-3.0 CC-BY-NC-1.0 CC-BY-NC-2.0 CC-BY-NC-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-3.0 CC-BY-NC-ND-1.0 CC-BY-NC-ND-2.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-ND-2.5 CC-BY-NC-ND-3.0 CC-BY-NC-SA-1.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-SA-2.0 CC-BY-NC-SA-2.5 CC-BY-NC-SA-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-ND-1.0 CC-BY-ND-2.0 CC-BY-ND-2.5 CC-BY-ND-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0 CC-BY-SA-2.5 CC-BY-SA-3.0 CC-BY-SA-4.0"
-SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B CECILL-C"
-SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL"
-SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0 EFL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair Frameworx-1.0"
-SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-classpath-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-font-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2-with-bison-exception GPL-3.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "gSOAP-1 gSOAP-1.3b HPND IPA IPL-1.0 ISC LGPL-2.0"
-SRC_DISTRIBUTE_LICENSES += "LGPL-2.1 LGPL-3.0 Libpng LPL-1.02 LPPL-1.0 LPPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "LPPL-1.2 LPPL-1.3c MirOS MIT Motosoto MPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "MPL-1.1 MS-PL MS-RL Multics NASA-1.3 Nauman NCSA"
-SRC_DISTRIBUTE_LICENSES += "NGPL Nokia NPOSL-3.0 NTP OASIS OCLC-2.0 ODbL-1.0"
-SRC_DISTRIBUTE_LICENSES += "OFL-1.1 OGTSL OLDAP-2.8 OpenSSL OSL-1.0 OSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "OSL-3.0 PD PHP-3.0 PostgreSQL Proprietary"
-SRC_DISTRIBUTE_LICENSES += "Python-2.0 QPL-1.0 RHeCos-1 RHeCos-1.1 RPL-1.5"
-SRC_DISTRIBUTE_LICENSES += "RPSL-1.0 RSCPL Ruby SAX-PD SGI-1 Simple-2.0 Sleepycat"
-SRC_DISTRIBUTE_LICENSES += "SPL-1.0 SugarCRM-1 SugarCRM-1.1.3 UCB VSL-1.0 W3C"
-SRC_DISTRIBUTE_LICENSES += "Watcom-1.0 WXwindows XFree86-1.0 XFree86-1.1 Xnet XSL YPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "Zimbra-1.3 Zlib ZPL-1.1 ZPL-2.0 ZPL-2.1"
-
 # Standards are great! Everyone has their own. In an effort to standardize licensing
 # names, common-licenses will use the SPDX standard license names. In order to not
 # break the non-standardized license names that we find in LICENSE, we'll set
@@ -188,5 +147,3 @@ FOSS_BASE_URL = "http://localhost/repo/?mod=spdx_license_once"
 FOSS_SERVER = "${FOSS_BASE_URL}&fullSPDXFlag=${FOSS_FULL_SPDX}&noCopyright=${FOSS_NO_COPYRIGHT}&recursiveUnpack=${FOSS_RECURSIVE_UNPACK}"
 
 FOSS_WGET_FLAGS = "-qO - --no-check-certificate --timeout=0"
-
-
Signature · 1 line
2.21.0
#2 lore raw

PATCH v3 3/8 license.bbclass: Make incompatible_pkg_license return incompatible lics

In reply to Peter Kjellerstedt

This makes it possible to report the incompatible licenses.

---
 meta/classes/license.bbclass | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
meta/classes/license.bbclass +9 -3
index c388740003..f90176d6c0 100644
@@ -310,15 +310,21 @@ def incompatible_pkg_license(d, dont_want_licenses, license):
     # Handles an "or" or two license sets provided by
     # flattened_licenses(), pick one that works if possible.
     def choose_lic_set(a, b):
-        return a if all(oe.license.license_ok(canonical_license(d, lic), 
+        return a if all(oe.license.license_ok(canonical_license(d, lic),
                             dont_want_licenses) for lic in a) else b
 
     try:
         licenses = oe.license.flattened_licenses(license, choose_lic_set)
     except oe.license.LicenseError as exc:
         bb.fatal('%s: %s' % (d.getVar('P'), exc))
-    return any(not oe.license.license_ok(canonical_license(d, l), \
-               dont_want_licenses) for l in licenses)
+
+    incompatible_lic = []
+    for l in licenses:
+        license = canonical_license(d, l)
+        if not oe.license.license_ok(license, dont_want_licenses):
+            incompatible_lic.append(license)
+
+    return sorted(incompatible_lic)
 
 def incompatible_license(d, dont_want_licenses, package=None):
     """
Signature · 1 line
2.21.0
#3 lore raw

PATCH v3 4/8 base.bbclass: Simplify the check for whitelisted licenses

In reply to Peter Kjellerstedt

After a number of rewrites, the code checking if a package has been whitelisted for an incompatible license was calculating the whitelisted packages twice (as 'whitelist' and as 'incompatwl').

---
 meta/classes/base.bbclass | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
meta/classes/base.bbclass +4 -9
index 31457f9f12..a3170d219c 100644
@@ -526,23 +526,21 @@ python () {
             bad_licenses = expand_wildcard_licenses(d, bad_licenses)
 
             whitelist = []
-            incompatwl = []
             for lic in bad_licenses:
                 spdx_license = return_spdx(d, lic)
                 whitelist.extend((d.getVar("WHITELIST_" + lic) or "").split())
                 if spdx_license:
                     whitelist.extend((d.getVar("WHITELIST_" + spdx_license) or "").split())
+
+            if pn in whitelist:
                 '''
                 We need to track what we are whitelisting and why. If pn is
                 incompatible we need to be able to note that the image that
                 is created may infact contain incompatible licenses despite
                 INCOMPATIBLE_LICENSE being set.
                 '''
-                incompatwl.extend((d.getVar("WHITELIST_" + lic) or "").split())
-                if spdx_license:
-                    incompatwl.extend((d.getVar("WHITELIST_" + spdx_license) or "").split())
-
-            if not pn in whitelist:
+                bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn)
+            else:
                 pkgs = d.getVar('PACKAGES').split()
                 skipped_pkgs = []
                 unskipped_pkgs = []
@@ -562,9 +560,6 @@ python () {
                 elif all_skipped or incompatible_license(d, bad_licenses):
                     bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license))
                     raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license)
-            elif pn in whitelist:
-                if pn in incompatwl:
-                    bb.note("INCLUDING " + pn + " as buildable despite INCOMPATIBLE_LICENSE because it has been whitelisted")
 
         # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a
         # superset of all per-package licenses. We do not do advanced (pattern)
Signature · 1 line
2.21.0
#4 lore raw

PATCH v3 5/8 base.bbclass: Report only the licenses that are incompatible for a package

In reply to Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package is identified as using an incompatible license, report the license(s) that are actually incompatible.

---
 meta/classes/base.bbclass                     | 26 ++++++++++++-------
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 2 files changed, 19 insertions(+), 13 deletions(-)
meta/classes/base.bbclass +16 -10
index a3170d219c..145daea3ec 100644
@@ -542,24 +542,30 @@ python () {
                 bb.note("Including %s as buildable despite it having an incompatible license because it has been whitelisted" % pn)
             else:
                 pkgs = d.getVar('PACKAGES').split()
-                skipped_pkgs = []
+                skipped_pkgs = {}
                 unskipped_pkgs = []
                 for pkg in pkgs:
-                    if incompatible_license(d, bad_licenses, pkg):
-                        skipped_pkgs.append(pkg)
+                    incompatible_lic = incompatible_license(d, bad_licenses, pkg)
+                    if incompatible_lic:
+                        skipped_pkgs[pkg] = incompatible_lic
                     else:
                         unskipped_pkgs.append(pkg)
-                all_skipped = skipped_pkgs and not unskipped_pkgs
                 if unskipped_pkgs:
                     for pkg in skipped_pkgs:
-                        bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + license)
+                        bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
                         mlprefix = d.getVar('MLPREFIX')
-                        d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, 1)
+                        d.setVar('LICENSE_EXCLUSION-' + mlprefix + pkg, ' '.join(skipped_pkgs[pkg]))
                     for pkg in unskipped_pkgs:
-                        bb.debug(1, "INCLUDING the package " + pkg)
-                elif all_skipped or incompatible_license(d, bad_licenses):
-                    bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, license))
-                    raise bb.parse.SkipRecipe("it has an incompatible license: %s" % license)
+                        bb.debug(1, "Including the package %s" % pkg)
+                else:
+                    incompatible_lic = incompatible_license(d, bad_licenses)
+                    for pkg in skipped_pkgs:
+                        incompatible_lic += skipped_pkgs[pkg]
+                    incompatible_lic = sorted(list(set(incompatible_lic)))
+
+                    if incompatible_lic:
+                        bb.debug(1, "Skipping recipe %s because of incompatible license(s): %s" % (pn, ' '.join(incompatible_lic)))
+                        raise bb.parse.SkipRecipe("it has incompatible license(s): %s" % ' '.join(incompatible_lic))
 
         # Try to verify per-package (LICENSE_<pkg>) values. LICENSE should be a
         # superset of all per-package licenses. We do not do advanced (pattern)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
index ad878571b5..9d1e801117 100644
@@ -4,7 +4,7 @@ from oeqa.utils.commands import bitbake
 class IncompatibleLicenseTests(OESelftestTestCase):
 
     def lic_test(self, pn, pn_lic, lic):
-        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has an incompatible license: %s' % (pn, pn, pn_lic)
+        error_msg = 'ERROR: Nothing PROVIDES \'%s\'\n%s was skipped: it has incompatible license(s): %s' % (pn, pn, pn_lic)
 
         self.write_config("INCOMPATIBLE_LICENSE += \"%s\"" % (lic))
 
@@ -27,12 +27,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
     # license cannot be built when INCOMPATIBLE_LICENSE contains this SPDX
     # license
     def test_incompatible_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPL-3.0')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPL-3.0')
 
     # Verify that a package with an alias (from SPDXLICENSEMAP) to an SPDX
     # license cannot be built when INCOMPATIBLE_LICENSE contains this alias
     def test_incompatible_alias_spdx_license_alias(self):
-        self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
+        self.lic_test('incompatible-license-alias', 'GPL-3.0', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
     # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
Signature · 1 line
2.21.0
#5 lore raw

PATCH v3 6/8 package.bbclass: Report only the licenses that are incompatible

In reply to Peter Kjellerstedt

When excluding a package from being packaged due to incompatible licenses, report the license(s) that are actually incompatible.

---

PATCHv2: Show the names of the incompatible licenses correctly.

 meta/classes/package.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
meta/classes/package.bbclass +3 -2
index e0d6ff6701..ef3de35961 100644
@@ -1263,8 +1263,9 @@ python populate_packages () {
     # Handle LICENSE_EXCLUSION
     package_list = []
     for pkg in packages:
-        if d.getVar('LICENSE_EXCLUSION-' + pkg):
-            msg = "%s has an incompatible license. Excluding from packaging." % pkg
+        licenses = d.getVar('LICENSE_EXCLUSION-' + pkg)
+        if licenses:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
             package_qa_handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
Signature · 1 line
2.21.0
#6 lore raw

PATCH v3 7/8 license_image.bbclass: Report only the licenses that are incompatible

In reply to Peter Kjellerstedt

Instead of reporting ${LICENSE} when a package cannot be installed into an image because it is using an incompatible license, report the license(s) that are actually incompatible.

---
 meta/classes/license_image.bbclass               | 5 +++--
 meta/lib/oeqa/selftest/cases/incompatible_lic.py | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
meta/classes/license_image.bbclass +3 -2
index b5399b6d96..a8c72da3cb 100644
@@ -51,8 +51,9 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
         for pkg in sorted(pkg_dic):
             if bad_licenses and pkg not in whitelist:
                 try:
-                    if incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"]):
-                        bb.fatal("Package %s has an incompatible license %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"]))
+                    licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                    if licenses:
+                        bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(licenses)))
                     (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                         oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
                         bad_licenses, canonical_license, d)
meta/lib/oeqa/selftest/cases/incompatible_lic.py +2 -2
index 9d1e801117..458a940618 100644
@@ -49,7 +49,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
 
     def test_bash_default(self):
         self.write_config(self.default_config())
-        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ and cannot be installed into the image."
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+"
 
         result = bitbake('core-image-minimal', ignore_status=True)
         if error_msg not in result.output:
@@ -57,7 +57,7 @@ INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
 
     def test_bash_and_license(self):
         self.write_config(self.default_config() + '\nLICENSE_append_pn-bash = " & SomeLicense"')
-        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ & SomeLicense and cannot be installed into the image."
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0+"
 
         result = bitbake('core-image-minimal', ignore_status=True)
         if error_msg not in result.output:
Signature · 1 line
2.21.0
#7 lore raw

PATCH v3 8/8 incompatible_lic.py: Add tests for incompatible licenses with wildcards

In reply to Peter Kjellerstedt

---

PATCHv2: New in this patch set.
PATCHv3: Added the missing incompatible-licenses.bb recipe file.

 .../license/incompatible-licenses.bb          |  3 ++
 .../oeqa/selftest/cases/incompatible_lic.py   | 42 +++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100644 meta-selftest/recipes-test/license/incompatible-licenses.bb
meta-selftest/recipes-test/license/incompatible-licenses.bb new file +3 -0
new file mode 100644
index 0000000000..9c1545efba
@@ -0,0 +1,3 @@
+SUMMARY = "Recipe with multiple SPDX licenses"
+DESCRIPTION = "Is licensed with multiple SPDX licenses to be used for testing"
+LICENSE = "GPL-2.0 & GPL-3.0 & LGPL-3.0"
meta/lib/oeqa/selftest/cases/incompatible_lic.py +42 -0

54 lines not on this page · raw message

Signature · 1 line
2.21.0
#8 lore raw
In reply to Peter Kjellerstedt

This actually fails yocto-check-layer script for Yocto compliance checking, when a layer provides own licenses via LICENSE_PATH (in layer.conf - LICENSE_PATH += "${LAYERDIR}/licenses"):

Variable AVAILABLE_LICENSES value changed:
...

Is there a new way to provide additional licenses from a layer or should the yocto-check-layer script be modified to accomodate (i.e. ignore) this change?

Signature · 1 line
Denys
Quoted 127 lines · Peter Kjellerstedt

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

---

PATCHv2: Correct spelling of 'available'.

 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +23 -4
index 648a4d7892..c388740003 100644
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), 
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
     or the passed license if there is no canonical form.
     """
     lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license):
             lic += '+'
     return lic or license
 
+def available_licenses(d):
+    """
+    Return the available licenses by searching the directories specified by
+    COMMON_LICENSE_DIR and LICENSE_PATH.
+    """
+    lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' +
+                (d.getVar('LICENSE_PATH') or '')).split()
+
+    licenses = []
+    for lic_dir in lic_dirs:
+        licenses += os.listdir(lic_dir)
+
+    licenses = sorted(licenses)
+    return licenses
+
+# Only determine the list of all available licenses once. This assumes that any
+# additions to LICENSE_PATH have been done before this file is parsed.
+AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}"
+
 def expand_wildcard_licenses(d, wildcard_licenses):
     """
-    Return actual spdx format license names if wildcard used. We expand
-    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
+    Return actual spdx format license names if wildcards are used. We expand
+    wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES.
     """
     import fnmatch
     licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
-    spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or '').split()
+    spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:
         licenses += fnmatch.filter(spdx_lics, wld_lic)
 
meta/conf/documentation.conf +1 -0
index fca36f3cf6..0b21d1f63e 100644
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib p
 AUTHOR[doc] = "Email address used to contact the original author(s) in order to send patches and forward bugs."
 AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the syslinux bootloader."
 AUTOREV[doc] = "When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository."
+AVAILABLE_LICENSES[doc] = "List of licenses found in the directories specified by COMMON_LICENSE_DIR and LICENSE_PATH."
 
 #B
 
meta/lib/oeqa/selftest/cases/incompatible_lic.py +3 -3
index 904b5b4094..ad878571b5 100644
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         if error_msg not in result.output:
             raise AssertionError(result.output)
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX license
     def test_incompatible_spdx_license(self):
         self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0')
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in
     # SPDXLICENSEMAP) of this SPDX license
     def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
Signature · 7 lines
2.21.0

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
#9 lore raw

This actually also results in checksums changing for recipes that inherit license and call incompatible_license_contains() when adding/removing layers that add to the license path, such as lvm2.

Quoted 166 lines · Denys Dmytriyenko

This actually fails yocto-check-layer script for Yocto compliance checking, when a layer provides own licenses via LICENSE_PATH (in layer.conf - LICENSE_PATH += "${LAYERDIR}/licenses"):

Variable AVAILABLE_LICENSES value changed:
...

Is there a new way to provide additional licenses from a layer or should the yocto-check-layer script be modified to accomodate (i.e. ignore) this change?

--
Denys

Quoted 30 lines · Peter Kjellerstedt

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible.

This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards.

---

PATCHv2: Correct spelling of 'available'.

 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)
meta/classes/license.bbclass +0 -0
index 648a4d7892..c388740003 100644

meta/classes/license.bbclass def canonical_license(d, license): · line 253

Quoted 2 lines
meta/classes/license.bbclass +0 -0
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so

GPLv3

meta/classes/license.bbclass def canonical_license(d, license): · line 254

Quoted 1 line
meta/classes/license.bbclass +0 -0
     becomes GPL-3.0), for the license named 'X+', return canonical form

of

meta/classes/license.bbclass def canonical_license(d, license): · line 267

Quoted 12 lines
meta/classes/license.bbclass +4 -1
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
     or the passed license if there is no canonical form.
     """
     lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license):
             lic += '+'
     return lic or license
 
+def available_licenses(d):
+    """
+    Return the available licenses by searching the directories

specified by

meta/classes/license.bbclass def canonical_license(d, license):

Quoted 13 lines
meta/classes/license.bbclass +13 -0
+    COMMON_LICENSE_DIR and LICENSE_PATH.
+    """
+    lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' +
+                (d.getVar('LICENSE_PATH') or '')).split()
+
+    licenses = []
+    for lic_dir in lic_dirs:
+        licenses += os.listdir(lic_dir)
+
+    licenses = sorted(licenses)
+    return licenses
+
+# Only determine the list of all available licenses once. This assumes

that any

meta/classes/license.bbclass def canonical_license(d, license):

Quoted 7 lines
meta/classes/license.bbclass +3 -2
+# additions to LICENSE_PATH have been done before this file is parsed.
+AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}"
+
 def expand_wildcard_licenses(d, wildcard_licenses):
     """
-    Return actual spdx format license names if wildcard used. We expand
-    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES

values.

meta/classes/license.bbclass def canonical_license(d, license): · line 286

Quoted 1 line
meta/classes/license.bbclass +1 -0
+    Return actual spdx format license names if wildcards are used. We

expand

meta/classes/license.bbclass def expand_wildcard_licenses(d, wildcard_licenses): · line 294

Quoted 7 lines
meta/classes/license.bbclass +1 -0
+    wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES.
     """
     import fnmatch
     licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in

spdxflags]

meta/classes/license.bbclass def expand_wildcard_licenses(d, wildcard_licenses): · line 276

Quoted 2 lines
meta/classes/license.bbclass +0 -1
-    spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or

'').split()

meta/conf/documentation.conf ASSUME_SHLIBS[doc] = "List of shlib:package[_version]

Quoted 9 lines
meta/classes/license.bbclass +1 -0
+    spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:
         licenses += fnmatch.filter(spdx_lics, wld_lic)
 
meta/conf/documentation.conf +0 -0
index fca36f3cf6..0b21d1f63e 100644
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version]

mappings. Useful for lib p

meta/conf/documentation.conf ASSUME_SHLIBS[doc] = "List of shlib:package[_version] · line 69

Quoted 1 line
meta/conf/documentation.conf +0 -0
 AUTHOR[doc] = "Email address used to contact the original author(s) in

order to send patches and forward bugs."

meta/conf/documentation.conf ASSUME_SHLIBS[doc] = "List of shlib:package[_version] · line 70

Quoted 1 line
meta/conf/documentation.conf +0 -0
 AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the

syslinux bootloader."

meta/conf/documentation.conf ASSUME_SHLIBS[doc] = "List of shlib:package[_version] · line 71

Quoted 1 line
meta/conf/documentation.conf +0 -0
 AUTOREV[doc] = "When SRCREV is set to the value of this variable, it

specifies to use the latest source revision in the repository."

meta/conf/documentation.conf ASSUME_SHLIBS[doc] = "List of shlib:package[_version] · line 72

Quoted 1 line
meta/conf/documentation.conf +1 -0
+AVAILABLE_LICENSES[doc] = "List of licenses found in the directories

specified by COMMON_LICENSE_DIR and LICENSE_PATH."

Review

Quoted 4 lines
meta/conf/documentation.conf +0 -0
 #B
 
(quoted diff) +0 -0

b/meta/lib/oeqa/selftest/cases/incompatible_lic.py

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 15

Quoted 8 lines

index 904b5b4094..ad878571b5 100644

meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -1
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         if error_msg not in result.output:
             raise AssertionError(result.output)
 
-    # Verify that a package with an SPDX license (from

SRC_DISTRIBUTE_LICENSES)

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 15

Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -0
+    # Verify that a package with an SPDX license (from

AVAILABLE_LICENSES)

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 16

Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -0
     # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX

license

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase):

Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -1
     def test_incompatible_spdx_license(self):
         self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0')
 
-    # Verify that a package with an SPDX license (from

SRC_DISTRIBUTE_LICENSES)

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 20

Quoted 1 line
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -0
+    # Verify that a package with an SPDX license (from

AVAILABLE_LICENSES)

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase): · line 38

Quoted 8 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +0 -1
     # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in
     # SPDXLICENSEMAP) of this SPDX license
     def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built

when

meta/lib/oeqa/selftest/cases/incompatible_lic.py class IncompatibleLicenseTests(OESelftestTestCase):

Quoted 4 lines
meta/lib/oeqa/selftest/cases/incompatible_lic.py +1 -0
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense',

'FooLicense')

Quoted 8 lines

--
2.21.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 4 lines
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
#10 lore raw
Quoted 4 lines · Christopher Larson

This actually also results in checksums changing for recipes that inherit license and call incompatible_license_contains() when adding/removing layers that add to the license path, such as lvm2.

Quoted 12 lines
Quoted 10 lines · Denys Dmytriyenko

This actually fails yocto-check-layer script for Yocto compliance checking, when a layer provides own licenses via LICENSE_PATH (in layer.conf - LICENSE_PATH += "${LAYERDIR}/licenses"):

Variable AVAILABLE_LICENSES value changed:
...

Is there a new way to provide additional licenses from a layer or should the yocto-check-layer script be modified to accomodate (i.e. ignore) this change?

I think we either need to vardepsexclude this variable from its usage sites or we need to figure out a different way of handling these license issues.

I worry about generically whitelisting it as there are cases where adding a new license should cause rebuilds.

We'll therefore have to look at the usage sites and decide if we want a rebuild for each usage.

Cheers,

Richard