Skip to contentSkip to conversation

scripts/spdxcheck.py: Lets strictly read license files in utf-8

PATCH Superseded Nishanth Menon 3 participants 3 messages Jul 3, 2021 – Jul 7, 2021 Licenses and spdx stuff

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts/spdxcheck.py: Lets strictly read license files in utf-8

Commit bc41a7f36469 ("LICENSES: Add the CC-BY-4.0 license") unfortunately introduced LICENSES/dual/CC-BY-4.0 in UTF-8 Unicode text While python will barf at it with:

FAIL: 'ascii' codec can't decode byte 0xe2 in position 2109: ordinal not in range(128) Traceback (most recent call last):
File "scripts/spdxcheck.py", line 244, in <module>
spdx = read_spdxdata(repo)
File "scripts/spdxcheck.py", line 47, in read_spdxdata
for l in open(el.path).readlines():
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2109: ordinal not in range(128)

While it is indeed debatable if 'Licensor.' used in the license file needs unicode quotes, instead, let us force spdxcheck to read utf-8 instead.

---
 scripts/spdxcheck.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/spdxcheck.py +1 -1
index 3e784cf9f401..ebd06ae642c9 100755
@@ -44,7 +44,7 @@ def read_spdxdata(repo):
                 continue
 
             exception = None
-            for l in open(el.path).readlines():
+            for l in open(el.path, encoding="utf-8").readlines():
                 if l.startswith('Valid-License-Identifier:'):
                     lid = l.split(':')[1].strip().upper()
                     if lid in spdx.licenses:
Signature · 1 line
2.32.0
Reviewed-by on v1 1/1
#1 lore raw

Nishanth,

Quoted 2 lines · Nishanth Menon

Commit bc41a7f36469 ("LICENSES: Add the CC-BY-4.0 license") unfortunately introduced LICENSES/dual/CC-BY-4.0 in UTF-8 Unicode text

Sigh. Why are people adding such things w/o running this script in the first place.

Quoted 12 lines

While python will barf at it with:

FAIL: 'ascii' codec can't decode byte 0xe2 in position 2109: ordinal not in range(128) Traceback (most recent call last):
File "scripts/spdxcheck.py", line 244, in <module>
spdx = read_spdxdata(repo)
File "scripts/spdxcheck.py", line 47, in read_spdxdata
for l in open(el.path).readlines():
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2109: ordinal not in range(128)

Quoted 3 lines

While it is indeed debatable if 'Licensor.' used in the license file needs unicode quotes, instead, let us force spdxcheck to read utf-8 instead.

s/let us//

Ditto for the $subject. See Documentation/process/ for further enlightment.

Quoted 2 lines

With that fixed:

#2 lore raw
Quoted 7 lines · Thomas Gleixner

Nishanth,

Quoted 2 lines · Nishanth Menon

Commit bc41a7f36469 ("LICENSES: Add the CC-BY-4.0 license") unfortunately introduced LICENSES/dual/CC-BY-4.0 in UTF-8 Unicode text

Sigh. Why are people adding such things w/o running this script in the first place.

I have a guess on that front ... there is nothing in our documentation that says anybody should run it, and the script itself gives no indication of what it does, when it should be run, or how to run it. That might just reduce uptake a little bit...:)

I increasingly believe that anything we add to scripts/ should start with a "usage" header describing why it exists and how to make it do its thing. That would be a welcome addition to spdxcheck.py. Adding something to Documentation/process/license-rules.html would be a nice bonus.

Thanks,

jon