Skip to contentSkip to conversation

scripts: mandate use of SPDX-License-Identifier tags in new files

PATCH Superseded Daniel P. Berrangé 4 participants 10 messages Oct 7, 2024 Checkpatch

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover scripts: mandate use of SPDX-License-Identifier tags in new files

One of the items raised at the QEMU maintainers meeting at KVM Forum 2024 was adoption of SPDX-License-Identifier for licensing of newly contributed source files, for which there were no dissenting voices.

Thus, to bring the proposal to the wider community, here is a series illustrating a way to put the decision into action by extending checkpatch.pl to mandate SPDX-License-Identifier in all new files.

Furthermore, anytime it sees SPDX-License-Identifier in any patch, whether a new file or pre-existing, it validates the declared license name. If it is not one of the commonly used QEMU licenses (the GPL variants, MIT, & a few BSD variants), it will report an error. To encourage sticking with GPL-2.0-or-later by default, it will issue a warning even if it is one of the common licenses, encouraging the contributor to double check their choice. This should reduce (usually accidental) license proliferation in QEMU code.

Finally, I've seen a few other random SPDX tags such as:

* SPDX-FileCopyrightText - replacing "Copyright ..."
* SPDX-FileContributor - replacing "Authors: ..."
* SPDX-URL - a link to the link license text
* SPDX-sourceInfo - arbitrary free form text about the file

These may or may not be worth considering in QEMU, but this series discourages their usage by raising an error in checkpatch for now.

If we feel we want to adopt any of these, I think it should be through a concious decision applied universally. Inconsistent & adhoc usage of other SPDX tags by a subset of contributors feels like it doesn't seem to give a clear win, and could even be a net loss through making practices inconsistent across the code.

Daniel P. Berrangé (3):
scripts: mandate that new files have SPDX-License-Identifier
scripts: validate SPDX license choices
scripts: forbid use of arbitrary SPDX tags besides license identifiers

 scripts/checkpatch.pl | 104 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)
Signature · 1 line
2.46.0
#1 lore raw

PATCH 1/3 scripts: mandate that new files have SPDX-License-Identifier

Going forward we want all newly created source files to have an SPDX-License-Identifier tag present.

Initially mandate this for C, Python and Perl source files, and encourage this for other file types.

---
 scripts/checkpatch.pl | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
scripts/checkpatch.pl +26 -0
index 1b21249c91..cc266abdcd 100755
@@ -1378,6 +1378,8 @@ sub process {
 	my $in_imported_file = 0;
 	my $in_no_imported_file = 0;
 	my $non_utf8_charset = 0;
+	my $expect_spdx = 0;
+	my $expect_spdx_file;
 
 	our @report = ();
 	our $cnt_lines = 0;
@@ -1615,6 +1617,30 @@ sub process {
 			WARN("added, moved or deleted file(s), does MAINTAINERS need updating?\n" . $herecurr);
 		}
 
+# All new files should have a SPDX-License-Identifier tag
+		if ($line =~ /^new file mode\s*\d+\s*$/) {
+		    if ($expect_spdx) {
+			if ($expect_spdx_file =~ /\.(c|h|py|pl|c\.inc)$/) {
+			    # source code files MUST have SPDX license declared
+			    ERROR("expected 'SPDX-License-Identifer' in new file $expect_spdx_file");
+			} else {
+			    # Other files MAY have SPDX license if appropriate
+			    WARNING("Does new file $expect_spdx_file need 'SPDX-License-Identifer'?");
+			}
+		    }
+		    $expect_spdx = 1;
+		    $expect_spdx_file = undef;
+		} elsif ($expect_spdx) {
+		    $expect_spdx_file = $realfile unless defined $expect_spdx_file;
+
+		    # SPDX tagsd may occurr in comments which were
+		    # stripped from '$line', so use '$rawline'
+		    if ($rawline =~ /SPDX-License-Identifier/) {
+			$expect_spdx = 0;
+			$expect_spdx_file = undef;
+		    }
+		}
+
 # Check for wrappage within a valid hunk of the file
 		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
 			ERROR("patch seems to be corrupt (line wrapped?)\n" .
Signature · 1 line
2.46.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] scripts: mandate that new files have SPDX-License-Identifier

Quoted 45 lines · Daniel P. Berrangé

Going forward we want all newly created source files to have an SPDX-License-Identifier tag present.

Initially mandate this for C, Python and Perl source files, and encourage this for other file types.

---
  scripts/checkpatch.pl | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
scripts/checkpatch.pl +19 -0
index 1b21249c91..cc266abdcd 100755
@@ -1378,6 +1378,8 @@ sub process {
  	my $in_imported_file = 0;
  	my $in_no_imported_file = 0;
  	my $non_utf8_charset = 0;
+	my $expect_spdx = 0;
+	my $expect_spdx_file;
 
  	our @report = ();
  	our $cnt_lines = 0;
@@ -1615,6 +1617,30 @@ sub process {
  			WARN("added, moved or deleted file(s), does MAINTAINERS need updating?\n" . $herecurr);
  		}
 
+# All new files should have a SPDX-License-Identifier tag
+		if ($line =~ /^new file mode\s*\d+\s*$/) {
+		    if ($expect_spdx) {
+			if ($expect_spdx_file =~ /\.(c|h|py|pl|c\.inc)$/) {
+			    # source code files MUST have SPDX license declared
+			    ERROR("expected 'SPDX-License-Identifer' in new file $expect_spdx_file");
+			} else {
+			    # Other files MAY have SPDX license if appropriate
+			    WARNING("Does new file $expect_spdx_file need 'SPDX-License-Identifer'?");
+			}
+		    }
+		    $expect_spdx = 1;
+		    $expect_spdx_file = undef;
+		} elsif ($expect_spdx) {
+		    $expect_spdx_file = $realfile unless defined $expect_spdx_file;
+
+		    # SPDX tagsd may occurr in comments which were

scripts/checkpatch.pl sub process { · line 1646

Quoted 10 lines
scripts/checkpatch.pl +7 -0
+		    # stripped from '$line', so use '$rawline'
+		    if ($rawline =~ /SPDX-License-Identifier/) {
+			$expect_spdx = 0;
+			$expect_spdx_file = undef;
+		    }
+		}
+
  # Check for wrappage within a valid hunk of the file
  		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
  			ERROR("patch seems to be corrupt (line wrapped?)\n" .

This change makes sense to me, thanks for suggesting it.

Reviewed-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] scripts: mandate that new files have SPDX-License-Identifier

In reply to Daniel P. Berrangé
Quoted 25 lines · Daniel P. Berrangé

Going forward we want all newly created source files to have an SPDX-License-Identifier tag present.

Initially mandate this for C, Python and Perl source files, and encourage this for other file types.

---
  scripts/checkpatch.pl | 26 ++++++++++++++++++++++++++
  1 file changed, 26 insertions(+)
scripts/checkpatch.pl +2 -0
index 1b21249c91..cc266abdcd 100755
@@ -1378,6 +1378,8 @@ sub process {
  	my $in_imported_file = 0;
  	my $in_no_imported_file = 0;
  	my $non_utf8_charset = 0;
+	my $expect_spdx = 0;
+	my $expect_spdx_file;
 
  	our @report = ();
  	our $cnt_lines = 0;

scripts/checkpatch.pl sub process { · line 1623

Quoted 7 lines
scripts/checkpatch.pl +4 -0
@@ -1615,6 +1617,30 @@ sub process {
  			WARN("added, moved or deleted file(s), does MAINTAINERS need updating?\n" . $herecurr);
  		}
 
+# All new files should have a SPDX-License-Identifier tag
+		if ($line =~ /^new file mode\s*\d+\s*$/) {
+		    if ($expect_spdx) {
+			if ($expect_spdx_file =~ /\.(c|h|py|pl|c\.inc)$/) {

[ch]\.inc to also mandat for *.h.inc?

Otherwise, thanks!

Quoted 23 lines
scripts/checkpatch.pl +20 -0
+			    # source code files MUST have SPDX license declared
+			    ERROR("expected 'SPDX-License-Identifer' in new file $expect_spdx_file");
+			} else {
+			    # Other files MAY have SPDX license if appropriate
+			    WARNING("Does new file $expect_spdx_file need 'SPDX-License-Identifer'?");
+			}
+		    }
+		    $expect_spdx = 1;
+		    $expect_spdx_file = undef;
+		} elsif ($expect_spdx) {
+		    $expect_spdx_file = $realfile unless defined $expect_spdx_file;
+
+		    # SPDX tagsd may occurr in comments which were
+		    # stripped from '$line', so use '$rawline'
+		    if ($rawline =~ /SPDX-License-Identifier/) {
+			$expect_spdx = 0;
+			$expect_spdx_file = undef;
+		    }
+		}
+
  # Check for wrappage within a valid hunk of the file
  		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
  			ERROR("patch seems to be corrupt (line wrapped?)\n" .
#4 lore raw

PATCH 2/3 scripts: validate SPDX license choices

In reply to Daniel P. Berrangé

We expect all new code to be contributed with the "GPL-2.0-or-later" license tag. Divergance is permitted if the new file is derived from pre-existing code under a different license, whether from elsewhere in QEMU codebase, or outside.

Issue a warning if the declared license is not "GPL-2.0-or-later", and an error if the license is not one of the handful of the expected licenses to prevent unintended proliferation.

---
 scripts/checkpatch.pl | 66 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
scripts/checkpatch.pl +66 -0
index cc266abdcd..cd1ed90f4c 100755
@@ -1353,6 +1353,67 @@ sub checkfilename {
 	}
 }
 
+sub checkspdx {
+    my ($file, $expr) = @_;
+
+    # Imported Linux headers probably have SPDX tags, but if they
+    # don't we're not requiring contributors to fix this
+    if ($file =~ m,include/standard-headers, ||
+	$file =~ m,linux-headers,) {
+	return;
+    }
+
+    my $origexpr = $expr;
+
+    # Flatten sub-expressions
+    $expr =~ s/\(|\)/ /g;
+    $expr =~ s/OR|AND/ /g;
+
+    # Merge WITH exceptions to the license
+    $expr =~ s/\s+WITH\s+/-WITH-/g;
+
+    # Cull more leading/trailing whitespace
+    $expr =~ s/^\s*//g;
+    $expr =~ s/\s*$//g;
+
+    my @bits = split / +/, $expr;
+
+    my $prefer = "GPL-2.0-or-later";
+    my @valid = qw(
+	LGPL-2.0-or-later
+	LGPL-2.1-or-later
+	GPL-2.0-only
+	LGPL-2.0-only
+	LGPL-2.0-only
+	BSD-2-Clause
+	BSD-3-Clause
+	MIT
+	);
+
+    my $nonpreferred = 0;
+    my @unknown = ();
+    foreach my $bit (@bits) {
+	if ($bit eq $prefer) {
+	    next;
+	}
+	if (grep /^$bit$/, @valid) {
+	    $nonpreferred = 1;
+	} else {
+	    push @unknown, $bit;
+	}
+    }
+    if (@unknown) {
+	ERROR("Saw unacceptable licenses '" . join(',', @unknown) .
+	      "', valid choices for QEMU are:\n" . join("\n", $prefer, @valid));
+    }
+
+    if ($nonpreferred) {
+	WARN("Saw acceptable license '$origexpr' but note '$prefer' is preferred " .
+	     "for new files unless the code is derived from a source with an" .
+	     "existed declared license that must be followed.");
+    }
+}
+
 sub process {
 	my $filename = shift;
 
@@ -1641,6 +1702,11 @@ sub process {
 		    }
 		}
 
+# Check SPDX-License-Identifier references a permitted license
+		if ($rawline =~ m,SPDX-License-Identifier: (.*?)(\*/)?\s*$,) {
+		    &checkspdx($realfile, $1);
+		}
+
 # Check for wrappage within a valid hunk of the file
 		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
 			ERROR("patch seems to be corrupt (line wrapped?)\n" .
Signature · 1 line
2.46.0
#5 lore raw

Re: [PATCH 2/3] scripts: validate SPDX license choices

Quoted 70 lines · Daniel P. Berrangé

We expect all new code to be contributed with the "GPL-2.0-or-later" license tag. Divergance is permitted if the new file is derived from pre-existing code under a different license, whether from elsewhere in QEMU codebase, or outside.

Issue a warning if the declared license is not "GPL-2.0-or-later", and an error if the license is not one of the handful of the expected licenses to prevent unintended proliferation.

---
  scripts/checkpatch.pl | 66 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 66 insertions(+)
scripts/checkpatch.pl +48 -0
index cc266abdcd..cd1ed90f4c 100755
@@ -1353,6 +1353,67 @@ sub checkfilename {
  	}
  }
 
+sub checkspdx {
+    my ($file, $expr) = @_;
+
+    # Imported Linux headers probably have SPDX tags, but if they
+    # don't we're not requiring contributors to fix this
+    if ($file =~ m,include/standard-headers, ||
+	$file =~ m,linux-headers,) {
+	return;
+    }
+
+    my $origexpr = $expr;
+
+    # Flatten sub-expressions
+    $expr =~ s/\(|\)/ /g;
+    $expr =~ s/OR|AND/ /g;
+
+    # Merge WITH exceptions to the license
+    $expr =~ s/\s+WITH\s+/-WITH-/g;
+
+    # Cull more leading/trailing whitespace
+    $expr =~ s/^\s*//g;
+    $expr =~ s/\s*$//g;
+
+    my @bits = split / +/, $expr;
+
+    my $prefer = "GPL-2.0-or-later";
+    my @valid = qw(
+	LGPL-2.0-or-later
+	LGPL-2.1-or-later
+	GPL-2.0-only
+	LGPL-2.0-only
+	LGPL-2.0-only
+	BSD-2-Clause
+	BSD-3-Clause
+	MIT
+	);
+
+    my $nonpreferred = 0;
+    my @unknown = ();
+    foreach my $bit (@bits) {
+	if ($bit eq $prefer) {
+	    next;
+	}
+	if (grep /^$bit$/, @valid) {
+	    $nonpreferred = 1;
+	} else {
+	    push @unknown, $bit;
+	}

scripts/checkpatch.pl sub checkfilename { · line 1413

Quoted 10 lines
scripts/checkpatch.pl +10 -0
+    }
+    if (@unknown) {
+	ERROR("Saw unacceptable licenses '" . join(',', @unknown) .
+	      "', valid choices for QEMU are:\n" . join("\n", $prefer, @valid));
+    }
+
+    if ($nonpreferred) {
+	WARN("Saw acceptable license '$origexpr' but note '$prefer' is preferred " .
+	     "for new files unless the code is derived from a source with an" .
+	     "existed declared license that must be followed.");

Is it not preferred to contribute code under the BSD-3-clause? Based on other items in the project, I was expecting we could make contributions for hexagon w/BSD.  But those are exceptional cases and not to be followed in the general case?

Quoted 18 lines
scripts/checkpatch.pl +8 -0
+    }
+}
+
  sub process {
  	my $filename = shift;
 
@@ -1641,6 +1702,11 @@ sub process {
  		    }
  		}
 
+# Check SPDX-License-Identifier references a permitted license
+		if ($rawline =~ m,SPDX-License-Identifier: (.*?)(\*/)?\s*$,) {
+		    &checkspdx($realfile, $1);
+		}
+
  # Check for wrappage within a valid hunk of the file
  		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
  			ERROR("patch seems to be corrupt (line wrapped?)\n" .
#6 lore raw

Re: [PATCH 2/3] scripts: validate SPDX license choices

Quoted 78 lines · Brian Cain
Quoted 76 lines · Daniel P. Berrangé

We expect all new code to be contributed with the "GPL-2.0-or-later" license tag. Divergance is permitted if the new file is derived from pre-existing code under a different license, whether from elsewhere in QEMU codebase, or outside.

Issue a warning if the declared license is not "GPL-2.0-or-later", and an error if the license is not one of the handful of the expected licenses to prevent unintended proliferation.

---
  scripts/checkpatch.pl | 66 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 66 insertions(+)
scripts/checkpatch.pl +55 -0
index cc266abdcd..cd1ed90f4c 100755
@@ -1353,6 +1353,67 @@ sub checkfilename {
  	}
  }
+sub checkspdx {
+    my ($file, $expr) = @_;
+
+    # Imported Linux headers probably have SPDX tags, but if they
+    # don't we're not requiring contributors to fix this
+    if ($file =~ m,include/standard-headers, ||
+	$file =~ m,linux-headers,) {
+	return;
+    }
+
+    my $origexpr = $expr;
+
+    # Flatten sub-expressions
+    $expr =~ s/\(|\)/ /g;
+    $expr =~ s/OR|AND/ /g;
+
+    # Merge WITH exceptions to the license
+    $expr =~ s/\s+WITH\s+/-WITH-/g;
+
+    # Cull more leading/trailing whitespace
+    $expr =~ s/^\s*//g;
+    $expr =~ s/\s*$//g;
+
+    my @bits = split / +/, $expr;
+
+    my $prefer = "GPL-2.0-or-later";
+    my @valid = qw(
+	LGPL-2.0-or-later
+	LGPL-2.1-or-later
+	GPL-2.0-only
+	LGPL-2.0-only
+	LGPL-2.0-only
+	BSD-2-Clause
+	BSD-3-Clause
+	MIT
+	);
+
+    my $nonpreferred = 0;
+    my @unknown = ();
+    foreach my $bit (@bits) {
+	if ($bit eq $prefer) {
+	    next;
+	}
+	if (grep /^$bit$/, @valid) {
+	    $nonpreferred = 1;
+	} else {
+	    push @unknown, $bit;
+	}
+    }
+    if (@unknown) {
+	ERROR("Saw unacceptable licenses '" . join(',', @unknown) .
+	      "', valid choices for QEMU are:\n" . join("\n", $prefer, @valid));
+    }
+
+    if ($nonpreferred) {
Quoted 8 lines

scripts/checkpatch.pl sub checkfilename { · line 1413

Quoted 3 lines
scripts/checkpatch.pl +3 -0
+	WARN("Saw acceptable license '$origexpr' but note '$prefer' is preferred " .
+	     "for new files unless the code is derived from a source with an" .
+	     "existed declared license that must be followed.");

Is it not preferred to contribute code under the BSD-3-clause? Based on other items in the project, I was expecting we could make contributions for hexagon w/BSD.  But those are exceptional cases and not to be followed in the general case?

I'm not saying people can't contribute under BSD, but if that is required, a explanation of why is desired.

IMHO we don't want contributors choosing an non-GPL-2.0-or-later license merely out of personal (or corporate) preferences.

We want GPL-2.0-or-later to be used for everything, unless there is a compelling reason to diverge. The need to remain compliant with existing code that has been incorporated is the most common & obvious reason for this.

Other reasons likely exist - which could be explained in the commit message, to aid reviewers who are likely to see the checkpatch warnings about the unusal license choices.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

PATCH 3/3 scripts: forbid use of arbitrary SPDX tags besides license identifiers

In reply to Daniel P. Berrangé

While SPDX-License-Identifier is a well known SPDX tag, there are a great many more besides that[1]. This are around making machine readable metadata available to the 'reuse' tool and similar for things like author names, copyright owners, and much more. It is even possible to define source file line groups and apply different SPDX tags to just that region of code.

At this time we're only interested in adopting SPDX for recording the licensing info, so detect & reject any other SPDX metadata. If we want to explicitly collect extra data in SPDX format, we can evaluate each case on its merits.

[1] https://spdx.github.io/spdx-spec/v2.2.2/file-tags/

    https://spdx.github.io/spdx-spec/v2.2.2/file-information/
---
 scripts/checkpatch.pl | 12 ++++++++++++
 1 file changed, 12 insertions(+)
scripts/checkpatch.pl +12 -0
index cd1ed90f4c..5ad2d4ca2e 100755
@@ -1707,6 +1707,18 @@ sub process {
 		    &checkspdx($realfile, $1);
 		}
 
+		if ($rawline =~ m,(SPDX-[a-zA-Z0-9-_]+):,) {
+		    my $tag = $1;
+		    my @permitted = qw(
+			SPDX-License-Identifier
+		    );
+
+		    unless (grep { /^$tag$/ } @permitted) {
+			ERROR("Tag $tag not permitted in QEMU code, valid " .
+			      "choices are: " . join(", ", @permitted));
+		    }
+		}
+
 # Check for wrappage within a valid hunk of the file
 		if ($realcnt != 0 && $line !~ m{^(?:\+|-| |\\ No newline|$)}) {
 			ERROR("patch seems to be corrupt (line wrapped?)\n" .
Signature · 1 line
2.46.0
Reviewed-by on v1 3/3
#8 lore raw

Re: [PATCH 3/3] scripts: forbid use of arbitrary SPDX tags besides license identifiers

Quoted 11 lines · Daniel P. Berrangé

While SPDX-License-Identifier is a well known SPDX tag, there are a great many more besides that[1]. This are around making machine readable metadata available to the 'reuse' tool and similar for things like author names, copyright owners, and much more. It is even possible to define source file line groups and apply different SPDX tags to just that region of code.

At this time we're only interested in adopting SPDX for recording the licensing info, so detect & reject any other SPDX metadata. If we want to explicitly collect extra data in SPDX format, we can evaluate each case on its merits.

50 uses, a third of them are mine, 10 different contributors. Can be cleaned.

Quoted 8 lines

[1] https://spdx.github.io/spdx-spec/v2.2.2/file-tags/

     https://spdx.github.io/spdx-spec/v2.2.2/file-information/
---
  scripts/checkpatch.pl | 12 ++++++++++++
  1 file changed, 12 insertions(+)
#9 lore raw
In reply to Daniel P. Berrangé
Quoted 9 lines · Daniel P. Berrangé

One of the items raised at the QEMU maintainers meeting at KVM Forum 2024 was adoption of SPDX-License-Identifier for licensing of newly contributed source files, for which there were no dissenting voices.

Thus, to bring the proposal to the wider community, here is a series illustrating a way to put the decision into action by extending checkpatch.pl to mandate SPDX-License-Identifier in all new files.

Quoted 8 lines

Furthermore, anytime it sees SPDX-License-Identifier in any patch, whether a new file or pre-existing, it validates the declared license name. If it is not one of the commonly used QEMU licenses (the GPL variants, MIT, & a few BSD variants), it will report an error. To encourage sticking with GPL-2.0-or-later by default, it will issue a warning even if it is one of the common licenses, encouraging the contributor to double check their choice. This should reduce (usually accidental) license proliferation in QEMU code.

For the record, I am in favour of this because it will (hopefully) catch some of the typically accidental issues like "user refers to a non-existent license or a license that's probably not what they meant like LGPLv2" or "user forgets to say 'or later' for GPL code" or "user forgets to put in license comment at all" or "user uses a license that is GPL-compatible but which we don't use at all at the moment, with no strong reason why they couldn't use some license we do already use" (to list some which have come up this year). These are trivially easy to fix if we can do it before commit when the author is still around to clarify, and potentially a real pain to try to fix after the fact, especially if multiple people have subsequently contributed to the file. We often do already catch this kind of mistake in code review, but having the checkpatch check should reduce the human-error factor.

(Conversely, I don't see much benefit to the project in trying to retrofit SPDX tags to already existing source files, though I am not in principle opposed to that.)

thanks
-- PMM