Skip to contentSkip to conversation

Incompatible license handling fixes

PATCH Superseded Joshua Watt 2 participants 20 messages Oct 18, 2024 – Oct 24, 2024 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover Incompatible license handling fixes

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-* packages at packaging time like other packages. This meant that if there were *-locale-* packages with an incompatible license, they would still be generated, and then the final image would likely fail when the packages were added to it.

These patches are more less ordered from least controversial to most controversial :)

I suspect the first two are probably fine optimizations, the third uses the new library code to make sure INCOMPATIBLE_LICENCE correctly applies to *-locale-* packages. This same call would likely also need to be called anywhere else dynamic package generation happens, unless there is some smart way to detect that.

Finally, the last patch makes it possible to set the LICENSE field of the generated *-locale-* packages, by making them match LICENSE:${PN}-locale, if set. This provides a means for recipe writes to affect the license of the split locale packages in the event that it doesn't match the recipe LICENSE. This makes logical sense to me, but please let me know if I'm missing some important assumption

Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check for incompatible licenses in locale packages
lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 203 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |   7 +
 5 files changed, 224 insertions(+), 207 deletions(-)
Signature · 1 line
2.46.2
#1 lore raw

PATCH 1/4 classes-global/license: Move functions to library code

Moves several of the functions in license.bbclass to be library code

---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)
meta/classes-global/base.bbclass +5 -5
index b6940bbb6ff..88b932fc3f0 100644
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
meta/classes-global/license.bbclass +0 -165

171 lines not on this page · raw message

meta/classes-recipe/license_image.bbclass +7 -7
index 0e953856a63..d2c5ab902ce 100644
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
meta/lib/oe/license.py +163 -0

166 lines not on this page · raw message

Signature · 1 line
2.46.2
#2 lore raw

PATCH 2/4 lib: license: Move package license skip to library

In reply to Joshua Watt

Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations

---
 meta/classes-global/base.bbclass | 36 ++++------------------------
 meta/lib/oe/license.py           | 40 ++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 32 deletions(-)
meta/classes-global/base.bbclass +4 -32
index 88b932fc3f0..b81e61fdb72 100644
@@ -573,42 +573,14 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/lib/oe/license.py +40 -0
index 7739697c401..75b1e70ba0e 100644
@@ -422,3 +422,43 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+            d.setVar('_exclude_incompatible-' + pkg, ' '.join(incompatible_lic))
+
+    return skipped_pkgs
Signature · 1 line
2.46.2
#3 lore raw

PATCH 3/4 lib: package: Check for incompatible licenses in locale packages

In reply to Joshua Watt

Checks if incompatible licenses are present in the dynamically generated locale packages

---
 meta/lib/oe/package.py | 3 +++
 1 file changed, 3 insertions(+)
meta/lib/oe/package.py +3 -0
index c213a9a3ca6..37fbfe355cb 100644
@@ -685,9 +685,11 @@ def split_locales(d):
     description = d.getVar('DESCRIPTION') or ""
     locale_section = d.getVar('LOCALE_SECTION')
     mlprefix = d.getVar('MLPREFIX') or ""
+    locale_pkgs = []
     for l in sorted(locales):
         ln = legitimize_package_name(l)
         pkg = pn + '-locale-' + ln
+        locale_pkgs.append(pkg)
         packages.insert(locale_index, pkg)
         locale_index += 1
         files = []
@@ -702,6 +704,7 @@ def split_locales(d):
             d.setVar('SECTION:' + pkg, locale_section)
 
     d.setVar('PACKAGES', ' '.join(packages))
+    oe.license.skip_incompatible_package_licenses(d, locale_pkgs)
 
     # Disabled by RP 18/06/07
     # Wildcards aren't supported in debian
Signature · 1 line
2.46.2
#4 lore raw

PATCH 4/4 lib: package: Copy locale license

In reply to Joshua Watt

When creating split locales, copy the license from LICENSE:${PN}-locale if set, otherwise leave it unspecified (which will result in falling back to LICENSE)

---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)
meta/lib/oe/package.py +4 -0
index 37fbfe355cb..be8bffbd37b 100644
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -700,6 +702,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
Signature · 1 line
2.46.2
#5 lore raw

v2 cover Incompatible Licenses in Dynamic Packages

In reply to Joshua Watt

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-* packages at packaging time like other packages. This meant that if there were *-locale-* packages with an incompatible license, they would still be generated, and then the final image would likely fail when the packages were added to it. This is actually not just a problem with *-locale-* packages, but in fact any packages generated dynamically.

To fix this for all cases, the code to detect incompatible licenses in packages is moved to a library function, and instead of using the '_exclude-incompatible-' variable to transfer this knowledge from base.bbclass to do_package, the library function is called a second time in do_package. While slightly less efficient, this means that do_package is checking the actual final list of packages for license exclusions, make it accurate even when dynamic packages are added.

Finally, the last patch makes it possible to set the LICENSE field of the generated *-locale-* packages, by making them match LICENSE:${PN}-locale, if set. This provides a means for recipe writes to affect the license of the split locale packages in the event that it doesn't match the recipe LICENSE. This makes logical sense to me, but please let me know if I'm missing some important assumption

Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check incompatible licenses at packaging time
lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 202 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |  10 +-
 5 files changed, 223 insertions(+), 210 deletions(-)
Signature · 1 line
2.46.2
#6 lore raw

PATCH v2 1/4 classes-global/license: Move functions to library code

Moves several of the functions in license.bbclass to be library code

New function dependencies were manually verified using bitbake-dumpsigs to ensure that bitbake identified the same dependencies even though they are now in library code (although the new function names mean that the task hashes still change)

---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)
meta/classes-global/base.bbclass +5 -5
index b6940bbb6ff..88b932fc3f0 100644
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
meta/classes-global/license.bbclass +0 -165

171 lines not on this page · raw message

meta/classes-recipe/license_image.bbclass +7 -7
index 0e953856a63..d2c5ab902ce 100644
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
meta/lib/oe/license.py +163 -0

166 lines not on this page · raw message

Signature · 1 line
2.46.2
#7 lore raw

PATCH v2 2/4 lib: license: Move package license skip to library

In reply to Joshua Watt

Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations

---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)
meta/classes-global/base.bbclass +4 -31
index 88b932fc3f0..5b8663f454d 100644
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
meta/lib/oe/license.py +39 -0
index 7739697c401..866a876d7f0 100644
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
Signature · 1 line
2.46.2
#8 lore raw

RE: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

Quoted 79 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joshua Watt
Sent: den 23 oktober 2024 23:15
To: [email protected]
Cc: Joshua Watt <[email protected]>
Subject: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations

---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)
meta/classes-global/base.bbclass +4 -31
index 88b932fc3f0..5b8663f454d 100644
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
meta/lib/oe/license.py +9 -0
index 7739697c401..866a876d7f0 100644
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True

meta/lib/oe/license.py def check_license_format(d): · line 443

Quoted 10 lines
meta/lib/oe/license.py +10 -0
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}

For efficiency, it would be better to return early, i.e.:

    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
    if not bad_licenses:
        return {}

    pn = d.getVar("PN")
    if if pn.startswith("nativesdk-") or pn.startswith("gcc-source-"):
        return {}
    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
        if pn.endswith(d.expand(t)):
            return {}
Quoted 21 lines
meta/lib/oe/license.py +20 -1
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
--
Quoted 1 line

2.46.2

//Peter

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

RE: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

Quoted 23 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Peter Kjellerstedt
Sent: den 24 oktober 2024 01:39
To: Joshua Watt <[email protected]>; [email protected]
Subject: Re: [OE-core][PATCH v2 2/4] lib: license: Move package license
skip to library

Quoted 16 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joshua Watt
Sent: den 23 oktober 2024 23:15
To: [email protected]
Cc: Joshua Watt <[email protected]>
Subject: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations

---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)
Quoted 7 lines

meta/classes-global/base.bbclass python () { · line 574

Quoted 7 lines
meta/classes-global/base.bbclass +0 -0
index 88b932fc3f0..5b8663f454d 100644
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()

Missed this in the first review pass: you can move bad_licenses down to where it is being used.

Quoted 98 lines
Quoted 56 lines
meta/classes-global/base.bbclass +4 -31
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
meta/lib/oe/license.py +9 -0
index 7739697c401..866a876d7f0 100644
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True

meta/lib/oe/license.py def check_license_format(d): · line 443

Quoted 10 lines
meta/lib/oe/license.py +10 -0
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}

For efficiency, it would be better to return early, i.e.:

    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
    if not bad_licenses:
        return {}

    pn = d.getVar("PN")
    if if pn.startswith("nativesdk-") or pn.startswith("gcc-source-"):
        return {}
    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
        if pn.endswith(d.expand(t)):
            return {}
Quoted 16 lines
meta/lib/oe/license.py +16 -0
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
Quoted 8 lines
Quoted 6 lines
meta/lib/oe/license.py +4 -1
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
--

2.46.2

//Peter

//Peter

#11 lore raw

PATCH v2 3/4 lib: package: Check incompatible licenses at packaging time

In reply to Joshua Watt

Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since its it was added as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution.

---
 meta/classes-global/base.bbclass | 1 -
 meta/lib/oe/package.py           | 6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)
meta/classes-global/base.bbclass +0 -1
index 5b8663f454d..b81e61fdb72 100644
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/lib/oe/package.py +3 -3
index c213a9a3ca6..480408e41e3 100644
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
Signature · 1 line
2.46.2
#12 lore raw

RE: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

Quoted 7 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joshua Watt
Sent: den 23 oktober 2024 23:15
To: [email protected]
Cc: Joshua Watt <[email protected]>
Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

Quoted 7 lines

Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since its it was added

Typo: its it -> it

Quoted 42 lines

as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution.

---
 meta/classes-global/base.bbclass | 1 -
 meta/lib/oe/package.py           | 6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)
meta/classes-global/base.bbclass +0 -1
index 5b8663f454d..b81e61fdb72 100644
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/lib/oe/package.py +3 -4
index c213a9a3ca6..480408e41e3 100644
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
--
Quoted 1 line

2.46.2

There is still a reference to _exclude_incompatible in package.bbclass. Not sure what to do about it (it has to do with variable dependencies to trigger do_package to re-run for recipes that are affected by changes to INCOMPATIBLE_LICENSE).

//Peter

#13 lore raw

RE: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

In reply to Joshua Watt
Quoted 42 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joshua Watt
Sent: den 23 oktober 2024 23:15
To: [email protected]
Cc: Joshua Watt <[email protected]>
Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since its it was added as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution.

---
 meta/classes-global/base.bbclass | 1 -
 meta/lib/oe/package.py           | 6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)
meta/classes-global/base.bbclass +0 -1
index 5b8663f454d..b81e61fdb72 100644
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/lib/oe/package.py +0 -0
index c213a9a3ca6..480408e41e3 100644

meta/lib/oe/package.py def populate_packages(d): · line 1450

Quoted 4 lines
meta/lib/oe/package.py +1 -0
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):

Umm, how well did you test this? ;)
The Python interpreter is not too happy about the trailing colon on the line above...

Quoted 11 lines
meta/lib/oe/package.py +2 -4
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
--

2.46.2

//Peter

#14 lore raw

Re: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

Quoted 44 lines · Peter Kjellerstedt
Quoted 44 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joshua Watt
Sent: den 23 oktober 2024 23:15
To: [email protected]
Cc: Joshua Watt <[email protected]>
Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since its it was added as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution.

---
 meta/classes-global/base.bbclass | 1 -
 meta/lib/oe/package.py           | 6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)
meta/classes-global/base.bbclass +0 -1
index 5b8663f454d..b81e61fdb72 100644
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/lib/oe/package.py +0 -0
index c213a9a3ca6..480408e41e3 100644
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
Quoted 6 lines

meta/lib/oe/package.py def populate_packages(d): · line 1450

Quoted 3 lines
meta/lib/oe/package.py +1 -0
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):

Umm, how well did you test this? ;)
The Python interpreter is not too happy about the trailing colon on the line above...

Yep, because I found that, fixed it, and forgot to squash the fix in before I submitted

I'll send a V3

Quoted 15 lines
Quoted 11 lines
meta/lib/oe/package.py +2 -4
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
--
Quoted 1 line

2.46.2

//Peter

#15 lore raw

PATCH v2 4/4 lib: package: Copy locale license

In reply to Joshua Watt

When creating split locales, copy the license from LICENSE:${PN}-locale if set, otherwise leave it unspecified (which will result in falling back to LICENSE)

---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)
meta/lib/oe/package.py +4 -0
index 480408e41e3..236a0edbcb2 100644
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
Signature · 1 line
2.46.2
#16 lore raw

v3 cover Incompatible Licenses in Dynamic Packages

In reply to Joshua Watt

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-* packages at packaging time like other packages. This meant that if there were *-locale-* packages with an incompatible license, they would still be generated, and then the final image would likely fail when the packages were added to it. This is actually not just a problem with *-locale-* packages, but in fact any packages generated dynamically.

To fix this for all cases, the code to detect incompatible licenses in packages is moved to a library function, and instead of using the '_exclude-incompatible-' variable to transfer this knowledge from base.bbclass to do_package, the library function is called a second time in do_package. While slightly less efficient, this means that do_package is checking the actual final list of packages for license exclusions, make it accurate even when dynamic packages are added.

Finally, the last patch makes it possible to set the LICENSE field of the generated *-locale-* packages, by making them match LICENSE:${PN}-locale, if set. This provides a means for recipe writes to affect the license of the split locale packages in the event that it doesn't match the recipe LICENSE. This makes logical sense to me, but please let me know if I'm missing some important assumption

v3: Fixed several bugs that were accidentally omitted from the previous patch series, and feedback.

Joshua Watt (4):
classes-global/license: Move functions to library code
lib: license: Move package license skip to library
lib: package: Check incompatible licenses at packaging time
lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-global/package.bbclass       |   4 -
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 202 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |  10 +-
 6 files changed, 223 insertions(+), 214 deletions(-)
Signature · 1 line
2.46.2
#17 lore raw

PATCH v3 1/4 classes-global/license: Move functions to library code

Moves several of the functions in license.bbclass to be library code

New function dependencies were manually verified using bitbake-dumpsigs to ensure that bitbake identified the same dependencies even though they are now in library code (although the new function names mean that the task hashes still change)

---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)
meta/classes-global/base.bbclass +5 -5
index b6940bbb6ff..88b932fc3f0 100644
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
meta/classes-global/license.bbclass +0 -165

171 lines not on this page · raw message

meta/classes-recipe/license_image.bbclass +7 -7
index 0e953856a63..d2c5ab902ce 100644
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
meta/lib/oe/license.py +163 -0

166 lines not on this page · raw message

Signature · 1 line
2.46.2
#18 lore raw

PATCH v3 2/4 lib: license: Move package license skip to library

In reply to Joshua Watt

Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations

---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)
meta/classes-global/base.bbclass +4 -31
index 88b932fc3f0..5b8663f454d 100644
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
meta/lib/oe/license.py +39 -0
index 7739697c401..32c77fa204d 100644
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
Signature · 1 line
2.46.2
#19 lore raw

PATCH v3 3/4 lib: package: Check incompatible licenses at packaging time

In reply to Joshua Watt

Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since it was added as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution.

---
 meta/classes-global/base.bbclass    | 1 -
 meta/classes-global/package.bbclass | 4 ----
 meta/lib/oe/package.py              | 6 +++---
 3 files changed, 3 insertions(+), 8 deletions(-)
meta/classes-global/base.bbclass +0 -1
index 5b8663f454d..b81e61fdb72 100644
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
meta/classes-global/package.bbclass +0 -4
index 6cd8c0140f2..9be1d6a5b17 100644
@@ -447,10 +447,6 @@ def gen_packagevar(d, pkgvars="PACKAGEVARS"):
     for p in pkgs:
         for v in vars:
             ret.append(v + ":" + p)
-
-        # Ensure that changes to INCOMPATIBLE_LICENSE re-run do_package for
-        # affected recipes.
-        ret.append('_exclude_incompatible-%s' % p)
     return " ".join(ret)
 
 
meta/lib/oe/package.py +3 -3
index c213a9a3ca6..104b880b9e1 100644
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, packages)
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
Signature · 1 line
2.46.2
#20 lore raw

PATCH v3 4/4 lib: package: Copy locale license

In reply to Joshua Watt

When creating split locales, copy the license from LICENSE:${PN}-locale if set, otherwise leave it unspecified (which will result in falling back to LICENSE)

---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)
meta/lib/oe/package.py +4 -0
index 104b880b9e1..16359232ecd 100644
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
Signature · 1 line
2.46.2