Skip to contentSkip to conversation

scripts/spdxcheck: Better statistics and exclude handling

PATCH In review Thomas Gleixner 5 participants 20 messages May 16, 2022 – May 23, 2022 Licenses and spdx stuff

lore
Share & tools
lore
#0 lore raw

cover scripts/spdxcheck: Better statistics and exclude handling

spdxcheck -v output is just providing basic statistics, but lacks per directory statistics.

Finding files without SPDX identifiers is cumbersome with spdxcheck, though it has all the information required.

The exclude of files and directories is hardcoded in the script which makes it hard to maintain and the information cannot be accessed by external tools.

The following series addresses this by adding:

1) Directory statistics

Incomplete directories: SPDX in Files
./ : 6 of 13 46%
./Documentation : 4096 of 8451 48%
./arch : 13476 of 16402 82%
./block : 100 of 101 99%
./certs : 11 of 14 78%
./crypto : 145 of 176 82%
./drivers : 24682 of 30745 80%

2) The ability to show files without SPDX

    Files without SPDX:
    ./kernel/cpu.c
    ./kernel/kmod.c
    ./kernel/relay.c

3) A file based handling for exclude patterns

Thanks,

tglx

#1 lore raw

PATCH 1/9 scripts/spdxcheck: Add percentage to statistics

Files checked:            75856
Lines checked:           294516
Files with SPDX:          59410  78%
Files with errors:            0
---
 scripts/spdxcheck.py |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/spdxcheck.py +3 -1
@@ -285,7 +285,9 @@ import os
                 sys.stderr.write('\n')
                 sys.stderr.write('Files checked:     %12d\n' %parser.checked)
                 sys.stderr.write('Lines checked:     %12d\n' %parser.lines_checked)
-                sys.stderr.write('Files with SPDX:   %12d\n' %parser.spdx_valid)
+                if parser.checked:
+                    pc = int(100 * parser.spdx_valid / parser.checked)
+                    sys.stderr.write('Files with SPDX:   %12d %3d%%\n' %(parser.spdx_valid, pc))
                 sys.stderr.write('Files with errors: %12d\n' %parser.spdx_errors)
 
             sys.exit(0)
#2 lore raw

PATCH 2/9 scripts/spdxcheck: Add directory statistics

In reply to Thomas Gleixner

For better insights.

Directories accounted:     4646
Directories complete:      2565  55%
---
 scripts/spdxcheck.py |   27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
scripts/spdxcheck.py +27 -0
@@ -28,6 +28,15 @@ import os
         self.licenses = [ ]
         self.exceptions = { }
 
+class dirinfo(object):
+    def __init__(self):
+        self.missing = 0
+        self.total = 0
+
+    def update(self, miss):
+        self.total += 1
+        self.missing += miss
+
 # Read the spdx data from the LICENSES directory
 def read_spdxdata(repo):
 
@@ -93,6 +102,7 @@ import os
         self.checked = 0
         self.spdx_valid = 0
         self.spdx_errors = 0
+        self.spdx_dirs = {}
         self.curline = 0
         self.deepest = 0
 
@@ -167,6 +177,7 @@ import os
     def parse_lines(self, fd, maxlines, fname):
         self.checked += 1
         self.curline = 0
+        fail = 1
         try:
             for line in fd:
                 line = line.decode(locale.getpreferredencoding(False), errors='ignore')
@@ -192,6 +203,7 @@ import os
                 # Should we check for more SPDX ids in the same file and
                 # complain if there are any?
                 #
+                fail = 0
                 break
 
         except ParserException as pe:
@@ -203,6 +215,11 @@ import os
                 sys.stdout.write('%s: %d:0 %s\n' %(fname, self.curline, pe.txt))
             self.spdx_errors += 1
 
+        base = os.path.dirname(fname)
+        di = self.spdx_dirs.get(base, dirinfo())
+        di.update(fail)
+        self.spdx_dirs[base] = di
+
 def scan_git_tree(tree):
     for el in tree.traverse():
         # Exclude stuff which would make pointless noise
@@ -289,6 +306,16 @@ import os
                     pc = int(100 * parser.spdx_valid / parser.checked)
                     sys.stderr.write('Files with SPDX:   %12d %3d%%\n' %(parser.spdx_valid, pc))
                 sys.stderr.write('Files with errors: %12d\n' %parser.spdx_errors)
+                ndirs = len(parser.spdx_dirs)
+                dirsok = 0
+                if ndirs:
+                    sys.stderr.write('\n')
+                    sys.stderr.write('Directories accounted: %8d\n' %ndirs)
+                    for di in parser.spdx_dirs.values():
+                        if not di.missing:
+                            dirsok += 1
+                    pc = int(100 * dirsok / ndirs)
+                    sys.stderr.write('Directories complete:  %8d %3d%%\n' %(dirsok, pc))
 
             sys.exit(0)
 
#3 lore raw

PATCH 3/9 scripts/spdxcheck: Add [sub]directory statistics

In reply to Thomas Gleixner

Add functionality to display [sub]directory statistics. This is enabled by adding '-d' to the command line. The optional -D parameter allows to limit the directory depth. If supplied the subdirectories are accumulated

# scripts/spdxcheck.py -d kernel/
Incomplete directories: SPDX in Files
    ./kernel                         :   111 of   114   97%
    ./kernel/bpf                     :    43 of    45   95%
    ./kernel/bpf/preload             :     4 of     5   80%
    ./kernel/bpf/preload/iterators   :     4 of     5   80%
    ./kernel/cgroup                  :    10 of    13   76%
    ./kernel/configs                 :     0 of     9    0%
    ./kernel/debug                   :     3 of     4   75%
    ./kernel/debug/kdb               :     1 of    11    9%
    ./kernel/locking                 :    29 of    32   90%
    ./kernel/sched                   :    38 of    39   97%

The result can be accumulated by restricting the depth via the new command line option '-d $DEPTH':

# scripts/spdxcheck.py -d -D1
Incomplete directories: SPDX in Files
    ./                               :     6 of    13   46%
    ./Documentation                  :  4096 of  8451   48%
    ./arch                           : 13476 of 16402   82%
    ./block                          :   100 of   101   99%
    ./certs                          :    11 of    14   78%
    ./crypto                         :   145 of   176   82%
    ./drivers                        : 24682 of 30745   80%
    ./fs                             :  1876 of  2110   88%
    ./include                        :  5175 of  5757   89%
    ./ipc                            :    12 of    13   92%
    ./kernel                         :   493 of   527   93%
    ./lib                            :   393 of   524   75%
    ./mm                             :   151 of   159   94%
    ./net                            :  1713 of  1900   90%
    ./samples                        :   211 of   273   77%
    ./scripts                        :   341 of   435   78%
    ./security                       :   241 of   250   96%
    ./sound                          :  2438 of  2503   97%
    ./tools                          :  3810 of  5462   69%
    ./usr                            :     9 of    10   90%
---
 scripts/spdxcheck.py |   67 +++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 57 insertions(+), 10 deletions(-)
scripts/spdxcheck.py +57 -10
@@ -103,9 +103,21 @@ import os
         self.spdx_valid = 0
         self.spdx_errors = 0
         self.spdx_dirs = {}
+        self.dirdepth = -1
+        self.basedir = '.'
         self.curline = 0
         self.deepest = 0
 
+    def set_dirinfo(self, basedir, dirdepth):
+        if dirdepth >= 0:
+            self.basedir = basedir
+            bdir = basedir.lstrip('./').rstrip('/')
+            if bdir != '':
+                parts = bdir.split('/')
+            else:
+                parts = []
+            self.dirdepth = dirdepth + len(parts)
+
     # Validate License and Exception IDs
     def validate(self, tok):
         id = tok.value.upper()
@@ -215,12 +227,29 @@ import os
                 sys.stdout.write('%s: %d:0 %s\n' %(fname, self.curline, pe.txt))
             self.spdx_errors += 1
 
+        if fname == '-':
+            return
+
         base = os.path.dirname(fname)
+        if self.dirdepth > 0:
+            parts = base.split('/')
+            i = 0
+            base = '.'
+            while i < self.dirdepth and i < len(parts) and len(parts[i]):
+                base += '/' + parts[i]
+                i += 1
+        elif self.dirdepth == 0:
+            base = self.basedir
+        else:
+            base = './' + base.rstrip('/')
+        base += '/'
+
         di = self.spdx_dirs.get(base, dirinfo())
         di.update(fail)
         self.spdx_dirs[base] = di
 
-def scan_git_tree(tree):
+def scan_git_tree(tree, basedir, dirdepth):
+    parser.set_dirinfo(basedir, dirdepth)
     for el in tree.traverse():
         # Exclude stuff which would make pointless noise
         # FIXME: Put this somewhere more sensible
@@ -233,15 +262,19 @@ import os
         with open(el.path, 'rb') as fd:
             parser.parse_lines(fd, args.maxlines, el.path)
 
-def scan_git_subtree(tree, path):
+def scan_git_subtree(tree, path, dirdepth):
     for p in path.strip('/').split('/'):
         tree = tree[p]
-    scan_git_tree(tree)
+    scan_git_tree(tree, path.strip('/'), dirdepth)
 
 if __name__ == '__main__':
 
     ap = ArgumentParser(description='SPDX expression checker')
     ap.add_argument('path', nargs='*', help='Check path or file. If not given full git tree scan. For stdin use "-"')
+    ap.add_argument('-d', '--dirs', action='store_true',
+                    help='Show [sub]directory statistics.')
+    ap.add_argument('-D', '--depth', type=int, default=-1,
+                    help='Directory depth for -d statistics. Default: unlimited')
     ap.add_argument('-m', '--maxlines', type=int, default=15,
                     help='Maximum number of lines to scan in a file. Default 15')
     ap.add_argument('-v', '--verbose', action='store_true', help='Verbose statistics output')
@@ -285,13 +318,21 @@ import os
                     if os.path.isfile(p):
                         parser.parse_lines(open(p, 'rb'), args.maxlines, p)
                     elif os.path.isdir(p):
-                        scan_git_subtree(repo.head.reference.commit.tree, p)
+                        scan_git_subtree(repo.head.reference.commit.tree, p,
+                                         args.depth)
                     else:
                         sys.stderr.write('path %s does not exist\n' %p)
                         sys.exit(1)
             else:
                 # Full git tree scan
-                scan_git_tree(repo.head.commit.tree)
+                scan_git_tree(repo.head.commit.tree, '.', args.depth)
+
+            ndirs = len(parser.spdx_dirs)
+            dirsok = 0
+            if ndirs:
+                for di in parser.spdx_dirs.values():
+                    if not di.missing:
+                        dirsok += 1
 
             if args.verbose:
                 sys.stderr.write('\n')
@@ -306,17 +347,23 @@ import os
                     pc = int(100 * parser.spdx_valid / parser.checked)
                     sys.stderr.write('Files with SPDX:   %12d %3d%%\n' %(parser.spdx_valid, pc))
                 sys.stderr.write('Files with errors: %12d\n' %parser.spdx_errors)
-                ndirs = len(parser.spdx_dirs)
-                dirsok = 0
                 if ndirs:
                     sys.stderr.write('\n')
                     sys.stderr.write('Directories accounted: %8d\n' %ndirs)
-                    for di in parser.spdx_dirs.values():
-                        if not di.missing:
-                            dirsok += 1
                     pc = int(100 * dirsok / ndirs)
                     sys.stderr.write('Directories complete:  %8d %3d%%\n' %(dirsok, pc))
 
+            if ndirs and ndirs != dirsok and args.dirs:
+                if args.verbose:
+                    sys.stderr.write('\n')
+                sys.stderr.write('Incomplete directories: SPDX in Files\n')
+                for f in sorted(parser.spdx_dirs.keys()):
+                    di = parser.spdx_dirs[f]
+                    if di.missing:
+                        valid = di.total - di.missing
+                        pc = int(100 * valid / di.total)
+                        sys.stderr.write('    %-80s: %5d of %5d  %3d%%\n' %(f, valid, di.total, pc))
+
             sys.exit(0)
 
     except Exception as ex:
#4 lore raw

PATCH 4/9 scripts/spdxcheck: Add option to display files without SPDX

In reply to Thomas Gleixner

Makes life easier when chasing the missing ones. Is activated with '-f' on the command line.

# scripts/spdxcheck.py -f kernel/
Files without SPDX:
    ./kernel/cpu.c
    ./kernel/kmod.c
    ./kernel/relay.c
    ./kernel/bpf/offload.c
    ./kernel/bpf/preload/.gitignore
    ./kernel/bpf/preload/iterators/README
    ./kernel/bpf/ringbuf.c
    ./kernel/cgroup/cgroup.c
    ./kernel/cgroup/cpuset.c
    ./kernel/cgroup/legacy_freezer.c
    ./kernel/debug/debug_core.h
    ./kernel/debug/kdb/Makefile
    ./kernel/debug/kdb/kdb_bp.c
    ./kernel/debug/kdb/kdb_bt.c
    ./kernel/debug/kdb/kdb_cmds
    ./kernel/debug/kdb/kdb_debugger.c
    ./kernel/debug/kdb/kdb_io.c
    ./kernel/debug/kdb/kdb_keyboard.c
    ./kernel/debug/kdb/kdb_main.c
    ./kernel/debug/kdb/kdb_private.h
    ./kernel/debug/kdb/kdb_support.c
    ./kernel/locking/lockdep_states.h
    ./kernel/locking/mutex-debug.c
    ./kernel/locking/spinlock_debug.c
    ./kernel/sched/pelt.h

With the optional -D parameter the directory depth can be limited:

# scripts/spdxcheck.py -f -D 0 kernel/
Files without SPDX:
    ./kernel/cpu.c
    ./kernel/kmod.c
    ./kernel/relay.c
---
 scripts/spdxcheck.py |   21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
scripts/spdxcheck.py +19 -2
@@ -32,10 +32,16 @@ import os
     def __init__(self):
         self.missing = 0
         self.total = 0
+        self.files = []
 
-    def update(self, miss):
+    def update(self, fname, basedir, miss):
         self.total += 1
         self.missing += miss
+        if miss:
+            fname = './' + fname
+            bdir = os.path.dirname(fname)
+            if bdir == basedir.rstrip('/'):
+                self.files.append(fname)
 
 # Read the spdx data from the LICENSES directory
 def read_spdxdata(repo):
@@ -245,7 +251,7 @@ import os
         base += '/'
 
         di = self.spdx_dirs.get(base, dirinfo())
-        di.update(fail)
+        di.update(fname, base, fail)
         self.spdx_dirs[base] = di
 
 def scan_git_tree(tree, basedir, dirdepth):
@@ -275,6 +281,8 @@ import os
                     help='Show [sub]directory statistics.')
     ap.add_argument('-D', '--depth', type=int, default=-1,
                     help='Directory depth for -d statistics. Default: unlimited')
+    ap.add_argument('-f', '--files', action='store_true',
+                    help='Show files without SPDX.')
     ap.add_argument('-m', '--maxlines', type=int, default=15,
                     help='Maximum number of lines to scan in a file. Default 15')
     ap.add_argument('-v', '--verbose', action='store_true', help='Verbose statistics output')
@@ -364,6 +372,15 @@ import os
                         pc = int(100 * valid / di.total)
                         sys.stderr.write('    %-80s: %5d of %5d  %3d%%\n' %(f, valid, di.total, pc))
 
+            if ndirs and ndirs != dirsok and args.files:
+                if args.verbose or args.dirs:
+                    sys.stderr.write('\n')
+                sys.stderr.write('Files without SPDX:\n')
+                for f in sorted(parser.spdx_dirs.keys()):
+                    di = parser.spdx_dirs[f]
+                    for f in sorted(di.files):
+                        sys.stderr.write('    %s\n' %f)
+
             sys.exit(0)
 
     except Exception as ex:
#5 lore raw

PATCH 5/9 scripts/spdxcheck: Put excluded files and directories into a separate file

In reply to Thomas Gleixner

The files and directories which are excluded from scanning are currently hard coded in the script. That's not maintainable and not accessible for external tools.

Move the files and directories which should be excluded into a file. The default file is scripts/spdxexclude. This can be overridden with the '-e $FILE' command line option.

The file format and syntax is similar to the .gitignore file.

---
 scripts/spdxcheck.py |   70 ++++++++++++++++++++++++++++++++++++++++++++++-----
 scripts/spdxexclude  |    8 +++++
 2 files changed, 72 insertions(+), 6 deletions(-)
scripts/spdxcheck.py +64 -6
@@ -6,6 +6,7 @@ from argparse import ArgumentParser
 from ply import lex, yacc
 import locale
 import traceback
+import fnmatch
 import sys
 import git
 import re
@@ -106,6 +107,7 @@ import os
         self.parser = yacc.yacc(module = self, write_tables = False, debug = False)
         self.lines_checked = 0
         self.checked = 0
+        self.excluded = 0
         self.spdx_valid = 0
         self.spdx_errors = 0
         self.spdx_dirs = {}
@@ -254,17 +256,47 @@ import os
         di.update(fname, base, fail)
         self.spdx_dirs[base] = di
 
+class pattern(object):
+    def __init__(self, line):
+        self.pattern = line
+        self.match = self.match_file
+        if line == '.*':
+            self.match = self.match_dot
+        elif line.endswith('/'):
+            self.pattern = line[:-1]
+            self.match = self.match_dir
+        elif line.startswith('/'):
+            self.pattern = line[1:]
+            self.match = self.match_fn
+
+    def match_dot(self, fpath):
+        return os.path.basename(fpath).startswith('.')
+
+    def match_file(self, fpath):
+        return os.path.basename(fpath) == self.pattern
+
+    def match_fn(self, fpath):
+        return fnmatch.fnmatchcase(fpath, self.pattern)
+
+    def match_dir(self, fpath):
+        if self.match_fn(os.path.dirname(fpath)):
+            return True
+        return fpath.startswith(self.pattern)
+
+def exclude_file(fpath):
+    for rule in exclude_rules:
+        if rule.match(fpath):
+            return True
+    return False
+
 def scan_git_tree(tree, basedir, dirdepth):
     parser.set_dirinfo(basedir, dirdepth)
     for el in tree.traverse():
-        # Exclude stuff which would make pointless noise
-        # FIXME: Put this somewhere more sensible
-        if el.path.startswith("LICENSES"):
-            continue
-        if el.path.find("license-rules.rst") >= 0:
-            continue
         if not os.path.isfile(el.path):
             continue
+        if exclude_file(el.path):
+            parser.excluded += 1
+            continue
         with open(el.path, 'rb') as fd:
             parser.parse_lines(fd, args.maxlines, el.path)
 
@@ -273,6 +305,20 @@ import os
         tree = tree[p]
     scan_git_tree(tree, path.strip('/'), dirdepth)
 
+def read_exclude_file(fname):
+    rules = []
+    if not fname:
+        return rules
+    with open(fname) as fd:
+        for line in fd:
+            line = line.strip()
+            if line.startswith('#'):
+                continue
+            if not len(line):
+                continue
+            rules.append(pattern(line))
+    return rules
+
 if __name__ == '__main__':
 
     ap = ArgumentParser(description='SPDX expression checker')
@@ -281,6 +327,8 @@ import os
                     help='Show [sub]directory statistics.')
     ap.add_argument('-D', '--depth', type=int, default=-1,
                     help='Directory depth for -d statistics. Default: unlimited')
+    ap.add_argument('-e', '--exclude',
+                    help='File containing file patterns to exclude. Default: scripts/spdxexclude')
     ap.add_argument('-f', '--files', action='store_true',
                     help='Show files without SPDX.')
     ap.add_argument('-m', '--maxlines', type=int, default=15,
@@ -317,6 +365,15 @@ import os
         sys.exit(1)
 
     try:
+        fname = args.exclude
+        if not fname:
+            fname = os.path.join(os.path.dirname(__file__), 'spdxexclude')
+        exclude_rules = read_exclude_file(fname)
+    except Exception as ex:
+        sys.stderr.write('FAIL: Reading exclude file %s: %s\n' %(fname, ex))
+        sys.exit(1)
+
+    try:
         if len(args.path) and args.path[0] == '-':
             stdin = os.fdopen(sys.stdin.fileno(), 'rb')
             parser.parse_lines(stdin, args.maxlines, '-')
@@ -349,6 +406,7 @@ import os
                 sys.stderr.write('License IDs        %12d\n' %len(spdx.licenses))
                 sys.stderr.write('Exception IDs      %12d\n' %len(spdx.exceptions))
                 sys.stderr.write('\n')
+                sys.stderr.write('Files excluded:    %12d\n' %parser.excluded)
                 sys.stderr.write('Files checked:     %12d\n' %parser.checked)
                 sys.stderr.write('Lines checked:     %12d\n' %parser.lines_checked)
                 if parser.checked:
scripts/spdxexclude new file +8 -0
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Patterns for excluding files and directories
+
+# Ignore the license directory and the licensing documentation which would
+# create lots of noise for no value
+LICENSES/
+license-rules.rst
#6 lore raw

PATCH 6/9 scripts/spdxcheck: Exclude config directories

In reply to Thomas Gleixner

Kernel configuration files like default configs are machine generated and pretty useless outside of the kernel context.

---
 scripts/spdxexclude |    5 +++++
 1 file changed, 5 insertions(+)
scripts/spdxexclude +5 -0
@@ -6,3 +6,8 @@
 # create lots of noise for no value
 LICENSES/
 license-rules.rst
+
+# Ignore config files and snippets. The majority is generated
+# by the Kconfig tools
+kernel/configs/
+arch/*/configs/
#7 lore raw

PATCH 7/9 scripts/spdxcheck: Exclude MAINTAINERS/CREDITS

In reply to Thomas Gleixner

Listings of maintainers and people who deserve credits are not really interesting in terms of copyright. The usage of these files outside of the kernel is pointless and the file format is trivial. No point in chasing them or slapping a SPDX identifier into them just because.

---
 scripts/spdxexclude |    4 ++++
 1 file changed, 4 insertions(+)
scripts/spdxexclude +4 -0
@@ -11,3 +11,7 @@ license-rules.rst
 # by the Kconfig tools
 kernel/configs/
 arch/*/configs/
+
+# Other files without copyrightable content
+/CREDITS
+/MAINTAINERS
#8 lore raw

PATCH 8/9 scripts/spdxcheck: Exclude dot files

In reply to Thomas Gleixner

None of these files

.clang-format, .cocciconfig, .get_maintainer.ignore, .gitattributes, .gitignore, .mailmap

have copyrightable content. They are configuration files which use a publicly documented format.

---
 scripts/spdxexclude |    6 ++++++
 1 file changed, 6 insertions(+)
scripts/spdxexclude +6 -0
@@ -2,6 +2,12 @@
 #
 # Patterns for excluding files and directories
 
+# Ignore dot files:
+# .clang-format, .cocciconfig, .get_maintainer.ignore
+# .gitattributes, .gitignore, .mailmap
+# do not really have copyrightable content
+.*
+
 # Ignore the license directory and the licensing documentation which would
 # create lots of noise for no value
 LICENSES/
#9 lore raw

Re: [patch 8/9] scripts/spdxcheck: Exclude dot files

Quoted 8 lines · Thomas Gleixner

None of these files

.clang-format, .cocciconfig, .get_maintainer.ignore, .gitattributes, .gitignore, .mailmap

have copyrightable content. They are configuration files which use a publicly documented format.

Should this files remove their SPDX-License-Identifier? If yes, we should do that for .clang-format.

As another suggestion, we should check that the ignored files actually do _not_ have the SPDX-License-Identifier (i.e. so the above case would trigger a diagnostic).

Cheers,
Miguel

#10 lore raw

Re: [patch 8/9] scripts/spdxcheck: Exclude dot files

Quoted 10 lines · Miguel Ojeda
Quoted 8 lines · Thomas Gleixner

None of these files

.clang-format, .cocciconfig, .get_maintainer.ignore, .gitattributes, .gitignore, .mailmap

have copyrightable content. They are configuration files which use a publicly documented format.

Quoted 6 lines

Should this files remove their SPDX-License-Identifier? If yes, we should do that for .clang-format.

As another suggestion, we should check that the ignored files actually do _not_ have the SPDX-License-Identifier (i.e. so the above case would trigger a diagnostic).

Good questions. I'm happy to drop this patch for now until this discussion has been settled.

Thanks,

tglx

#11 lore raw

Re: [patch 8/9] scripts/spdxcheck: Exclude dot files

Quoted 14 lines · Thomas Gleixner
Quoted 13 lines · Miguel Ojeda
Quoted 8 lines · Thomas Gleixner

None of these files

.clang-format, .cocciconfig, .get_maintainer.ignore, .gitattributes, .gitignore, .mailmap

have copyrightable content. They are configuration files which use a publicly documented format.

Should this files remove their SPDX-License-Identifier? If yes, we should do that for .clang-format.

Quoted 6 lines
Quoted 3 lines

As another suggestion, we should check that the ignored files actually do _not_ have the SPDX-License-Identifier (i.e. so the above case would trigger a diagnostic).

Good questions. I'm happy to drop this patch for now until this discussion has been settled.

I've now taken all patches in this series except for this one.

thanks for doing this work,

greg k-h

#13 lore raw
In reply to Thomas Gleixner

Thank you for picking up the effort to add license (and perhaps also copyright) info to all files in the Kernel. This, as you know, is also working towards making the whole repo REUSE compliant [^1].

~ Thomas Gleixner [2022-05-16 12:27 +0200]:

Quoted 5 lines

Finding files without SPDX identifiers is cumbersome with spdxcheck, though it has all the information required.

The exclude of files and directories is hardcoded in the script which makes it hard to maintain and the information cannot be accessed by external tools.

Unfortunately, excluding files (i.e. not adding machine-readable license/copyright information to it) would also block reaching full compliance with the REUSE best practices. Have you considered making them available under GPL-2.0-only or a license similar to public domain [^2]?

Regarding false-positives, e.g. in license-rules.rst, you could use the brand-new feature that allows to ignore blocks of code (to be released later this week) [^3]. I am aware that spdxcheck would not be able to detect this, but using the REUSE helper tool [^4] could also be a solution to scan for missing files.

Best,
Max

[^1]: https://reuse.software

[^2]: https://reuse.software/faq/#exclude-file

[^3]: https://github.com/fsfe/reuse-docs/pull/104/files

[^4]: https://github.com/fsfe/reuse-tool

Signature · 3 lines
Max Mehl - Programme Manager -- Free Software Foundation Europe
Contact and information: https://fsfe.org/about/mehl -- @mxmehl
The FSFE is a charity that empowers users to control technology
#14 lore raw
Quoted 2 lines · Max Mehl

Thank you for picking up the effort to add license (and perhaps also copyright) info to all files in the Kernel.

Adding copyright notices retroactively is not going to happen ever. That's just impossible.

Quoted 8 lines
Quoted 2 lines

The exclude of files and directories is hardcoded in the script which makes it hard to maintain and the information cannot be accessed by external tools.

Unfortunately, excluding files (i.e. not adding machine-readable license/copyright information to it) would also block reaching full compliance with the REUSE best practices. Have you considered making them available under GPL-2.0-only or a license similar to public domain [^2]?

The LICENSE directory is already handled by spdxcheck as the license information is read from there. And no, we cannot add a GPL-2.0-only identifier to all of the files under the LICENSE directory for obvious reasons.

license-rules.rst is not longer a problem as all incarnations have a proper SPDX identifier today.

Thanks,

tglx

#15 lore raw
Quoted 16 lines · Thomas Gleixner
Quoted 2 lines · Max Mehl

Thank you for picking up the effort to add license (and perhaps also copyright) info to all files in the Kernel.

Adding copyright notices retroactively is not going to happen ever. That's just impossible.

Quoted 8 lines
Quoted 2 lines

The exclude of files and directories is hardcoded in the script which makes it hard to maintain and the information cannot be accessed by external tools.

Unfortunately, excluding files (i.e. not adding machine-readable license/copyright information to it) would also block reaching full compliance with the REUSE best practices. Have you considered making them available under GPL-2.0-only or a license similar to public domain [^2]?

Quoted 7 lines

The LICENSE directory is already handled by spdxcheck as the license information is read from there. And no, we cannot add a GPL-2.0-only identifier to all of the files under the LICENSE directory for obvious reasons.

license-rules.rst is not longer a problem as all incarnations have a proper SPDX identifier today.

There is also an argument to be made whether we really need to have SPDX identifiers on trivial files:

#include <someheader.h>
<EOF>

Such files are not copyrightable by any means. So what's the value of doubling the line count to add an SPDX identifier? Just to make nice statistics?

Thanks,

tglx

#16 lore raw

~ Thomas Gleixner [2022-05-16 20:59 +0200]:

Quoted 11 lines · Thomas Gleixner
Quoted 10 lines
Quoted 5 lines

Unfortunately, excluding files (i.e. not adding machine-readable license/copyright information to it) would also block reaching full compliance with the REUSE best practices. Have you considered making them available under GPL-2.0-only or a license similar to public domain [^2]?

The LICENSE directory is already handled by spdxcheck as the license information is read from there. And no, we cannot add a GPL-2.0-only identifier to all of the files under the LICENSE directory for obvious reasons.

Absolutely. REUSE obviously also ignores this directory, as well as e.g. zero-length files, symlinks, submodules, or .git directory.

Quoted 9 lines

There is also an argument to be made whether we really need to have SPDX identifiers on trivial files:

#include <someheader.h>
<EOF>

Such files are not copyrightable by any means. So what's the value of doubling the line count to add an SPDX identifier? Just to make nice statistics?

We agree that such files are not copyrightable. But where is the threshold? Lines of code? Creativity? Number of used functions? And how to embed this threshold in tooling? So instead of fuzzy exclusion of such files in tools like spdxcheck or REUSE, it makes sense to treat them as every other file with the cost of adding two comment lines.

This clear-cut rule eases maintaining and growing the effort you and others did because developers would know exactly what to add to a new file (license + copyright) without requiring looking up the thresholds or a manual review by maintainers who can interpret them.

Best,
Max

Signature · 3 lines
Max Mehl - Programme Manager -- Free Software Foundation Europe
Contact and information: https://fsfe.org/about/mehl -- @mxmehl
The FSFE is a charity that empowers users to control technology
#17 lore raw
Quoted 17 lines · Max Mehl

~ Thomas Gleixner [2022-05-16 20:59 +0200]:

Quoted 9 lines

There is also an argument to be made whether we really need to have SPDX identifiers on trivial files:

#include <someheader.h>
<EOF>

Such files are not copyrightable by any means. So what's the value of doubling the line count to add an SPDX identifier? Just to make nice statistics?

We agree that such files are not copyrightable. But where is the threshold? Lines of code? Creativity? Number of used functions? And how to embed this threshold in tooling? So instead of fuzzy exclusion of such files in tools like spdxcheck or REUSE, it makes sense to treat them as every other file with the cost of adding two comment lines.

Quoted 4 lines

This clear-cut rule eases maintaining and growing the effort you and others did because developers would know exactly what to add to a new file (license + copyright) without requiring looking up the thresholds or a manual review by maintainers who can interpret them.

Seriously no. I'm outright refusing to add my copyright to a trivial file with one or two includes or a silly comment like '/* empty because */.

There is nothing copyrightable there.

I'm not going to make myself a fool just to make tools happy, which can figure out on their own whether there is reasonable content in the vast majority of cases.

Also you need some exclude rules in any case. Why?

- How do you tell a tool that a file is generated, e.g. in the kernel the default configuration files?

Yes, the file content depends on human input to the generator tool, but I'm looking forward for the explanation how this is copyrightable especially with multiple people updating this file over time where some of the updates are just done by invoking the generator tool itself.

- How do you tell a tool that a file contains licensing documentation?

Go and look what license scanners make out of all the various license-rules.rst files.

- ....

Do all scanners have to grow heuristics for ignoring the content past the topmost SPDX License identifier in certain files or for figuring out what might be generated content?

You also might need to add information about binary blobs, which obviously cannot be part of the binary blobs themself.

The exclude rules I added are lazy and mostly focussed on spdxcheck, but I'm happy to make them more useful and let them carry information about the nature of the exclude or morph them into a general scanner info which also contains binary blob info and other helpful information. But that needs a larger discussion about the format and rules for such a file.

That said, I'm all for clear cut rules, but rules just for the rules sake are almost as bad as no rules at all.

As always you have to apply common sense and look at the bigger picture and come up with solutions which are practicable, enforcable and useful for the larger eco-system.

Your goal of having SPDX ids and copyright notices in every file of a project is honorable, but impractical for various reasons.

See above.

Aside of that you cannot replace a full blown license scanner by REUSE even if your project is SPDX and Copyright notice clean at the top level of a file. You still need to verify that there is no other information in a 'clean' file which might be contradicting or supplemental. You cannot add all of this functionality to REUSE or whatever.

Thanks,

tglx

#18 lore raw
Quoted 72 lines · Thomas Gleixner
Quoted 16 lines · Max Mehl

~ Thomas Gleixner [2022-05-16 20:59 +0200]:

Quoted 9 lines

There is also an argument to be made whether we really need to have SPDX identifiers on trivial files:

#include <someheader.h>
<EOF>

Such files are not copyrightable by any means. So what's the value of doubling the line count to add an SPDX identifier? Just to make nice statistics?

We agree that such files are not copyrightable. But where is the threshold? Lines of code? Creativity? Number of used functions? And how to embed this threshold in tooling? So instead of fuzzy exclusion of such files in tools like spdxcheck or REUSE, it makes sense to treat them as every other file with the cost of adding two comment lines.

Quoted 4 lines

This clear-cut rule eases maintaining and growing the effort you and others did because developers would know exactly what to add to a new file (license + copyright) without requiring looking up the thresholds or a manual review by maintainers who can interpret them.

Seriously no. I'm outright refusing to add my copyright to a trivial file with one or two includes or a silly comment like '/* empty because */.

There is nothing copyrightable there.

I'm not going to make myself a fool just to make tools happy, which can figure out on their own whether there is reasonable content in the vast majority of cases.

Also you need some exclude rules in any case. Why?

- How do you tell a tool that a file is generated, e.g. in the kernel the default configuration files?

Yes, the file content depends on human input to the generator tool, but I'm looking forward for the explanation how this is copyrightable especially with multiple people updating this file over time where some of the updates are just done by invoking the generator tool itself.

- How do you tell a tool that a file contains licensing documentation?

Go and look what license scanners make out of all the various license-rules.rst files.

- ....

Do all scanners have to grow heuristics for ignoring the content past the topmost SPDX License identifier in certain files or for figuring out what might be generated content?

You also might need to add information about binary blobs, which obviously cannot be part of the binary blobs themself.

The exclude rules I added are lazy and mostly focussed on spdxcheck, but I'm happy to make them more useful and let them carry information about the nature of the exclude or morph them into a general scanner info which also contains binary blob info and other helpful information. But that needs a larger discussion about the format and rules for such a file.

That said, I'm all for clear cut rules, but rules just for the rules sake are almost as bad as no rules at all.

As always you have to apply common sense and look at the bigger picture and come up with solutions which are practicable, enforcable and useful for the larger eco-system.

Your goal of having SPDX ids and copyright notices in every file of a project is honorable, but impractical for various reasons.

Quoted 8 lines

See above.

Aside of that you cannot replace a full blown license scanner by REUSE even if your project is SPDX and Copyright notice clean at the top level of a file. You still need to verify that there is no other information in a 'clean' file which might be contradicting or supplemental. You cannot add all of this functionality to REUSE or whatever.

Max, Thomas,

I think the discussion here is hitting upon the "inconvenience" of the lack of black/white rules in the law (as to what is copyrightable) versus the convenience of downstream recipients of code who want to be sure they have proper rights (which mixes in the guidance/rules of Reuse, tooling, etc.).

I think some rules in terms of files that are clearly not copyrightable can be implemented in various tooling (hopefully, with the guidance of a lawyer steeped in copyright law), and I agree that putting a license (by way of an SPDX identifier or any other way for that matter) on such files is neither a good use of time nor a good idea (from the perspective of being inaccurate as to the need for a license and thus sending the wrong impression). That being said, there will not be a way to make clear cut rules for everything, without involving a judge. Sorry! That's just how the law works (and we actually often don't want black/white lines in the law, actually).

I can see a policy of, "when it's not clear (as to copyrightability), then add a license", though.

Thanks,
Jilayne

#19 lore raw
Quoted 6 lines · J. Lovejoy

On 5/17/22 3:43 PM, Thomas Gleixner wrote:
I think the discussion here is hitting upon the "inconvenience" of the lack of black/white rules in the law (as to what is copyrightable) versus the convenience of downstream recipients of code who want to be sure they have proper rights (which mixes in the guidance/rules of Reuse, tooling, etc.).

Correct.

Quoted 11 lines

I think some rules in terms of files that are clearly not copyrightable can be implemented in various tooling (hopefully, with the guidance of a lawyer steeped in copyright law), and I agree that putting a license (by way of an SPDX identifier or any other way for that matter) on such files is neither a good use of time nor a good idea (from the perspective of being inaccurate as to the need for a license and thus sending the wrong impression). That being said, there will not be a way to make clear cut rules for everything, without involving a judge. Sorry! That's just how the law works (and we actually often don't want black/white lines in the law, actually).

Quoted 2 lines

I can see a policy of, "when it's not clear (as to copyrightability), then add a license", though.

No argument here, but trivial things like an include which file includes another include file are pretty clear IMO and we really should make our mind up on those. Even a header file which contains a single function declaration is questionable at best, but yes it's hard to put a hard line on those.

Thanks,

tglx