Skip to contentSkip to conversation

pull-spdx-license.py: add the SPDX licenses exceptions

PATCH In review Jan Vermaete 3 participants 4 messages Jun 27 – Jun 29

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pull-spdx-license.py: add the SPDX licenses exceptions

Fetch and extract the SPDX exception licenses as the normal licenses are handled.

[YOCTO #16294]

---
 scripts/pull-spdx-licenses.py | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
scripts/pull-spdx-licenses.py +12 -10
index 597a62133f..de1f5b7bad 100755
@@ -55,20 +55,22 @@ def main():
             data = json.load(response)
             version = data["tag_name"]
 
-    print(f"Pulling SPDX license list version {version}")
-    req = urllib.request.Request(
-        f"https://raw.githubusercontent.com/spdx/license-list-data/{version}/json/licenses.json"
-    )
-    with urllib.request.urlopen(req) as response:
-        spdx_licenses = json.load(response)
+    spdx_licenses = {}
+    for jsonfile in ["licenses.json", "exceptions.json"]:
+        req = urllib.request.Request(
+            f"https://raw.githubusercontent.com/spdx/license-list-data/{version}/json/{jsonfile}"
+        )
+        with urllib.request.urlopen(req) as response:
+            spdx_licenses.update(json.load(response))
+            print(len(spdx_licenses))
 
     with (TOP_DIR / "meta" / "files" / "spdx-licenses.json").open("w") as f:
         json.dump(spdx_licenses, f, sort_keys=True, indent=2)
 
-    total_count = len(spdx_licenses["licenses"])
+    total_count = len(spdx_licenses["licenses"] + spdx_licenses["exceptions"])
     updated = 0
-    for idx, lic in enumerate(spdx_licenses["licenses"]):
-        lic_id = lic["licenseId"]
+    for idx, lic in enumerate(spdx_licenses["licenses"] + spdx_licenses["exceptions"]):
+        lic_id = lic.get("licenseId") or lic.get("licenseExceptionId")
 
         print(f"[{idx + 1} of {total_count}] ", end="")
 
@@ -88,7 +90,7 @@ def main():
             continue
 
         with dest_license_file.open("w") as f:
-            f.write(lic_data["licenseText"])
+            f.write(lic_data.get("licenseText") or lic_data.get("licenseExceptionText"))
         updated += 1
         print("done")
 
Signature · 1 line
2.47.3
#1 lore raw
Quoted 3 lines · Jan Vermaete via lists.openembedded.org

Fetch and extract the SPDX exception licenses as the normal licenses are handled.

Thanks, but we can't actually use the license exceptions this way. The problem is that exceptions are only valid after an spdx WITH statement (e.g. "GPL-3.0 AND OpenSSL-exception" is not valid), which our license expressions do not support. Also, it's better to keep the licenses and exceptions in separate files so that it's easier to distinguish them when parsing.

Regardless, I have been working on converting all our license expressions to valid SPDX expressions, but it got sidetracked by some SBoM work. Part of that was somewhat similar patch to fetch the exceptions. If you would like to take a look, the branch is contrib/jpew/spdx-licenses. I hope to get back to working on that as soon as the SBoM stuff is sorted out.

Quoted 76 lines

[YOCTO #16294]

---
 scripts/pull-spdx-licenses.py | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
scripts/pull-spdx-licenses.py +0 -3
index 597a62133f..de1f5b7bad 100755
@@ -55,20 +55,22 @@ def main():
             data = json.load(response)
             version = data["tag_name"]
 
-    print(f"Pulling SPDX license list version {version}")
-    req = urllib.request.Request(
-        f"

https://raw.githubusercontent.com/spdx/license-list-data/{version}/json/licenses.json
"

scripts/pull-spdx-licenses.py +4 -3
-    )
-    with urllib.request.urlopen(req) as response:
-        spdx_licenses = json.load(response)
+    spdx_licenses = {}
+    for jsonfile in ["licenses.json", "exceptions.json"]:
+        req = urllib.request.Request(
+            f"

https://raw.githubusercontent.com/spdx/license-list-data/{version}/json/{jsonfile}
"

scripts/pull-spdx-licenses.py +4 -0
+        )
+        with urllib.request.urlopen(req) as response:
+            spdx_licenses.update(json.load(response))
+            print(len(spdx_licenses))
 
     with (TOP_DIR / "meta" / "files" / "spdx-licenses.json").open("w") as

f:
json.dump(spdx_licenses, f, sort_keys=True, indent=2)

scripts/pull-spdx-licenses.py +1 -1
-    total_count = len(spdx_licenses["licenses"])
+    total_count = len(spdx_licenses["licenses"] +

spdx_licenses["exceptions"])
updated = 0

scripts/pull-spdx-licenses.py +1 -2
-    for idx, lic in enumerate(spdx_licenses["licenses"]):
-        lic_id = lic["licenseId"]
+    for idx, lic in enumerate(spdx_licenses["licenses"] +

spdx_licenses["exceptions"]):

scripts/pull-spdx-licenses.py +2 -2
+        lic_id = lic.get("licenseId") or lic.get("licenseExceptionId")
 
         print(f"[{idx + 1} of {total_count}] ", end="")
@@ -88,7 +90,7 @@ def main():
             continue
 
         with dest_license_file.open("w") as f:
-            f.write(lic_data["licenseText"])
+            f.write(lic_data.get("licenseText") or
 lic_data.get("licenseExceptionText"))
         updated += 1
         print("done")
 
--

2.47.3

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#239697):
https://lists.openembedded.org/g/openembedded-core/message/239697
Mute This Topic: https://lists.openembedded.org/mt/120005594/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
Quoted 3 lines · Joshua Watt via lists.openembedded.org

Thanks, but we can't actually use the license exceptions this way. The problem is that exceptions are only valid after an spdx WITH statement (e.g. "GPL-3.0 AND OpenSSL-exception" is not valid), which our license expressions do not support. Also, it's better to keep the licenses and exceptions in separate files so that it's easier to distinguish them when parsing.

Regardless, I have been working on converting all our license expressions to valid SPDX expressions, but it got sidetracked by some SBoM work. Part of that was somewhat similar patch to fetch the exceptions. If you would like to take a look, the branch is contrib/jpew/spdx-licenses. I hope to get back to working on that as soon as the SBoM stuff is sorted out.

A related issue is that lack of support for SPDX exceptions holds up correctly specifying fmt's license terms (which is 'MIT WITH fmt-exception' if I understood you right):

https://spdx.org/licenses/fmt-exception.html
https://github.com/fmtlib/fmt/blob/main/doc/LICENSE-exception

The current recipe says simply 'MIT'.

Alex

#3 lore raw
Quoted 15 lines · Alexander Kanavin
Quoted 1 line · Joshua Watt via lists.openembedded.org

Thanks, but we can't actually use the license exceptions this way. The

problem is that exceptions are only valid after an spdx WITH statement (e.g. "GPL-3.0 AND OpenSSL-exception" is not valid), which our license expressions do not support. Also, it's better to keep the licenses and exceptions in separate files so that it's easier to distinguish them when parsing.

Quoted 2 lines

Regardless, I have been working on converting all our license

expressions to valid SPDX expressions, but it got sidetracked by some SBoM work. Part of that was somewhat similar patch to fetch the exceptions. If you would like to take a look, the branch is contrib/jpew/spdx-licenses. I hope to get back to working on that as soon as the SBoM stuff is sorted out.

Quoted 9 lines

A related issue is that lack of support for SPDX exceptions holds up correctly specifying fmt's license terms (which is 'MIT WITH fmt-exception' if I understood you right):

https://spdx.org/licenses/fmt-exception.html
https://github.com/fmtlib/fmt/blob/main/doc/LICENSE-exception

The current recipe says simply 'MIT'.

Correct. In the short term, you can make a new "OE" license that is "MIT-with-fmt-exception" to capture this (we have examples of this already, see Apache-2.0-with-LLVM-exception), and then once my changes are in you can change to the correct SPDX expression "MIT WITH fmt-exception"

Quoted 3 lines

Alex