Skip to contentSkip to conversation

devtools: add new SPDX license compliance checker

PATCH Superseded Stephen Hemminger 4 participants 24 messages Jan 29, 2020 – Aug 26, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 devtools: add new SPDX license compliance checker

Simple script to look for drivers and scripts that are missing requires SPDX header.

---
 devtools/spdx-check.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100755 devtools/spdx-check.sh
devtools/spdx-check.sh new file +23 -0
new file mode 100755
index 000000000000..6713d556d224
@@ -0,0 +1,23 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Microsoft Corporation
+#
+# Produce a list of files without SPDX license identifiers
+
+echo "Files without SPDX License"
+echo "--------------------------"
+
+git grep -L SPDX-License-Identifier -- \
+    ':^.git*' ':^.ci/*' ':^.travis.yml' \
+    ':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+    ':^*/Kbuild' ':^*/README' \
+    ':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+    ':^devtools/cocci/' \
+    ':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt'
+
+echo
+echo "Files with redundant BSD boilerplate"
+echo "------------------------------------"
+
+git grep -l SPDX-License-Identifier | \
+    xargs grep -l 'Redistribution'
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 1/1 devtools: add new SPDX license compliance checker

In reply to Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

---
v3 - pickup more places with boilerplate text
avoid false positive for cocci scripts or abignore

 devtools/spdx-check.sh | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100755 devtools/spdx-check.sh
devtools/spdx-check.sh new file +24 -0
new file mode 100755
index 000000000000..a5be10b26f44
@@ -0,0 +1,24 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Microsoft Corporation
+#
+# Produce a list of files with incorrect license
+# information
+
+echo "Files without SPDX License"
+echo "--------------------------"
+
+git grep -L SPDX-License-Identifier -- \
+    ':^.git*' ':^.ci/*' ':^.travis.yml' \
+    ':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+    ':^*/Kbuild' ':^*/README' \
+    ':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+    ':^*.cocci' ':^*.abignore' \
+    ':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt'
+
+echo
+echo "Files with additional license text"
+echo "----------------------------------"
+
+git grep -l Redistribution -- \
+    ':^license/' ':^/devtools/spdx-check.sh'
Signature · 1 line
2.20.1
#3 lore raw

07/02/2020 18:52, Stephen Hemminger:

Quoted 6 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

---
 devtools/spdx-check.sh | 24 ++++++++++++++++++++++++

For consistency with other scripts, it should be named check-spdx.sh

Please add the new file in MAINTAINERS file in the section "Developers and Maintainers Tools".

I think it should be mentioned also in the contributors guide.

devtools/spdx-check.sh line 3

Quoted 6 lines
devtools/spdx-check.sh new file +3 -0
@@ -0,0 +1,24 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Microsoft Corporation

While talking about license, are you sure the (c) is required?

devtools/spdx-check.sh line 6

Quoted 3 lines
devtools/spdx-check.sh new file +3 -0
+#
+# Produce a list of files with incorrect license
+# information

In order to add this script in a CI, it should return an error code. Based on the error code, the CI can report it as success, warning or failure to patchwork.

devtools/spdx-check.sh line 17

Quoted 11 lines
devtools/spdx-check.sh new file +11 -0
+
+echo "Files without SPDX License"
+echo "--------------------------"
+
+git grep -L SPDX-License-Identifier -- \
+    ':^.git*' ':^.ci/*' ':^.travis.yml' \
+    ':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+    ':^*/Kbuild' ':^*/README' \
+    ':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+    ':^*.cocci' ':^*.abignore' \
+    ':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt'

This should be considered a critical error.

devtools/spdx-check.sh line 24

Quoted 7 lines
devtools/spdx-check.sh new file +7 -0
+
+echo
+echo "Files with additional license text"
+echo "----------------------------------"
+
+git grep -l Redistribution -- \
+    ':^license/' ':^/devtools/spdx-check.sh'

This should be considered as a warning.

#4 lore raw
In reply to Stephen Hemminger

07/02/2020 18:52, Stephen Hemminger:

Quoted 4 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

[...]

devtools/spdx-check.sh line 17

Quoted 7 lines
devtools/spdx-check.sh new file +7 -0
+git grep -L SPDX-License-Identifier -- \
+    ':^.git*' ':^.ci/*' ':^.travis.yml' \
+    ':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+    ':^*/Kbuild' ':^*/README' \
+    ':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+    ':^*.cocci' ':^*.abignore' \
+    ':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt'

I think we should plan to add SPDX tag to the doc (rst and svg files).

#5 lore raw

PATCH v4 1/1 devtools: add new SPDX license compliance checker

In reply to Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

---
v4 - add MAINTAINERS entry
update coding style document
change name of script

 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 77 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh
MAINTAINERS +1 -0
index 3d5e8d1104b2..6b0e042c5fbb 100644
@@ -96,6 +96,7 @@ F: devtools/check-maintainers.sh
 F: devtools/check-forbidden-tokens.awk
 F: devtools/check-git-log.sh
 F: devtools/check-includes.sh
+F: devtools/check-spdx-tag.sh
 F: devtools/check-symbol-maps.sh
 F: devtools/checkpatches.sh
 F: devtools/get-maintainer.sh
devtools/check-spdx-tag.sh new file +77 -0

77 lines not on this page · raw message

doc/guides/contributing/coding_style.rst +7 -2
index 841ef6d5c829..04626667dc18 100644
@@ -54,8 +54,13 @@ To document a public API, a doxygen-like format must be used: refer to :ref:`dox
 License Header
 ~~~~~~~~~~~~~~
 
-Each file should begin with a special comment containing the appropriate copyright and license for the file.
-Generally this is the BSD License, except for code for Linux Kernel modules.
+Each file must begin with a special comment containing the
+`Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>`_.
+
+Generally this is the BSD License, except for code granted special exceptions.
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
+
 After any copyright header, a blank line should be left before any other contents, e.g. include statements in a C file.
 
 C Preprocessor Directives
Signature · 1 line
2.20.1
#6 lore raw
Quoted 12 lines · On Mon, 24 Feb 2020 13:01:30 -0800 Stephen Hemminger…

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

---
v4 - add MAINTAINERS entry
update coding style document
change name of script

Quoted 5 lines
 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 77 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh

Still waiting for this script all requested changes were done months ago.

Current output is:

$ ./devtools/check-spdx-tag.sh -v
Files without SPDX License
--------------------------

Files with redundant license text
---------------------------------
app/test/test_timer_racecond.c
lib/librte_ethdev/rte_ethdev_pci.h
lib/librte_ethdev/rte_ethdev_vdev.h

total: 0 errors, 3 warnings

#7 lore raw
In reply to Stephen Hemminger
Quoted 12 lines · On Mon, 24 Feb 2020 13:01:30 -0800 Stephen Hemminger…

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

---
v4 - add MAINTAINERS entry
update coding style document
change name of script

Quoted 6 lines
 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 77 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh

Going through the patch backlog.

Why is this not merged yet?
Why is not in DPDK 20.05?

#8 lore raw

11/06/2020 20:46, Stephen Hemminger:

Quoted 9 lines
Quoted 6 lines · On Mon, 24 Feb 2020 13:01:30 -0800 Stephen Hemminger…

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

Quoted 6 lines
Quoted 1 line

Going through the patch backlog.

Why is this not merged yet?
Why is not in DPDK 20.05?

I would like to see some reviews on such patch
which looks like a policy.

#9 lore raw
In reply to Stephen Hemminger

24/02/2020 22:01, Stephen Hemminger:

Quoted 1 line

+tmpfile=$(mktemp)

Please check how other temp files are created in other scripts for consitency.

devtools/check-spdx-tag.sh line 21

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \

I think doc/ should be part of the license check,
same for buildtools/.

devtools/check-spdx-tag.sh line 29

Quoted 8 lines
devtools/check-spdx-tag.sh new file +8 -0
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile
+
+    errors=0
+    while read -r line
+    do $quiet || echo $line
+       errors=$((errors + 1))

I'm surprised this works for you.
In general, "while" creates a subshell which makes impossible updating a variable.
I recommend using "for" with IFS=$'\n'.

Quoted 6 lines
devtools/check-spdx-tag.sh new file +6 -0
+    done < $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo

devtools/check-spdx-tag.sh line 45

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' |
+	while read line
+	do $quiet || echo $line
+	   warnings=$((warnings + 1))
+	done

Same comment about "while" subshell.

devtools/check-spdx-tag.sh line 50

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+
+    warnings=0
+    while read -r line
+    do $quiet || echo $line
+       warnings=$((errors + 1))

Here too

Quoted 2 lines
(quoted diff) +2 -0
+    done < $tmpfile
+}

[...]

Quoted 2 lines

+Each file must begin with a special comment containing the
+Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>_.

Typo: Identifier

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 60

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+
+Generally this is the BSD License, except for code granted special exceptions.

Is a verb missing?

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
#10 lore raw

Hi Thomas, Stephen,

Quoted 15 lines · Thomas Monjalon

24/02/2020 22:01, Stephen Hemminger:

Quoted 1 line

+tmpfile=$(mktemp)

Please check how other temp files are created in other scripts for consitency.

devtools/check-spdx-tag.sh line 21

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \

I think doc/ should be part of the license check,
same for buildtools/.

Quoted 14 lines

devtools/check-spdx-tag.sh line 29

Quoted 8 lines
devtools/check-spdx-tag.sh new file +8 -0
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile
+
+    errors=0
+    while read -r line
+    do $quiet || echo $line
+       errors=$((errors + 1))

I'm surprised this works for you.
In general, "while" creates a subshell which makes impossible updating a variable.
I recommend using "for" with IFS=$'\n'.

No it should work, while will only spawn a subshell if you use a pipe with it. Ex:

   err=0; while true; do err=$((err + 1)); done            # $err changes

   err=0; yes | while read -r y; do err=$((err + 1)); done # $err is always 0

Using for could be an issue, as you are then limited by the number of parameters allowed. Additionally, the script is written for POSIX shell, using $'\n' is forbidden and the POSIX equivalent is very ugly:

   IFS="$(printf '%b_' '\n')"; IFS="${IFS%_}" # protect trailing \n
Quoted 5 lines
Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+    done < $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
Quoted 8 lines

devtools/check-spdx-tag.sh line 42

Quoted 8 lines
devtools/check-spdx-tag.sh new file +8 -0
+	echo
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' |
+	while read line

Missing a -r to read here: https://www.shellcheck.net/wiki/SC2162
Generally, best to use shellcheck when writing a script, especially if using /bin/sh.

Quoted 33 lines

devtools/check-spdx-tag.sh line 45

Quoted 3 lines
devtools/check-spdx-tag.sh new file +3 -0
+	do $quiet || echo $line
+	   warnings=$((warnings + 1))
+	done

Same comment about "while" subshell.

devtools/check-spdx-tag.sh line 50

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+
+    warnings=0
+    while read -r line
+    do $quiet || echo $line
+       warnings=$((errors + 1))

Here too

Quoted 2 lines
(quoted diff) +2 -0
+    done < $tmpfile
+}

[...]

Quoted 2 lines

+Each file must begin with a special comment containing the
+Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>_.

Typo: Identifier

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 60

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+
+Generally this is the BSD License, except for code granted special exceptions.

Is a verb missing?

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
Signature · 1 line
Gaëtan
#11 lore raw
In reply to Thomas Monjalon
Quoted 65 lines · On Thu, 11 Jun 2020 23:39:55 +0200 Thomas Monjalon <[email protected]> wrote:

24/02/2020 22:01, Stephen Hemminger:

Quoted 1 line

+tmpfile=$(mktemp)

Please check how other temp files are created in other scripts for consitency.

devtools/check-spdx-tag.sh line 21

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \  

I think doc/ should be part of the license check,
same for buildtools/.

devtools/check-spdx-tag.sh line 29

Quoted 8 lines
devtools/check-spdx-tag.sh new file +8 -0
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile
+
+    errors=0
+    while read -r line
+    do $quiet || echo $line
+       errors=$((errors + 1))  

I'm surprised this works for you.
In general, "while" creates a subshell which makes impossible updating a variable.
I recommend using "for" with IFS=$'\n'.

Quoted 6 lines
devtools/check-spdx-tag.sh new file +6 -0
+    done < $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo

devtools/check-spdx-tag.sh line 45

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' |
+	while read line
+	do $quiet || echo $line
+	   warnings=$((warnings + 1))
+	done  

Same comment about "while" subshell.

devtools/check-spdx-tag.sh line 50

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+
+    warnings=0
+    while read -r line
+    do $quiet || echo $line
+       warnings=$((errors + 1))  

Here too

Quoted 2 lines
(quoted diff) +2 -0
+    done < $tmpfile
+}  

[...]

Quoted 2 lines

+Each file must begin with a special comment containing the
+Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>_.

Typo: Identifier

Quoted 7 lines

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 60

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+
+Generally this is the BSD License, except for code granted special exceptions.  

Is a verb missing?

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 62

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).  

Thanks for the feedback.
Text processing is simpler in python, will rewrite in next version?

#12 lore raw

12/06/2020 16:53, Stephen Hemminger:

Quoted 73 lines
Quoted 70 lines · On Thu, 11 Jun 2020 23:39:55 +0200 Thomas Monjalon <[email protected]> wrote:

24/02/2020 22:01, Stephen Hemminger:

Quoted 1 line

+tmpfile=$(mktemp)

Please check how other temp files are created in other scripts for consitency.

devtools/check-spdx-tag.sh line 21

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \  

I think doc/ should be part of the license check,
same for buildtools/.

devtools/check-spdx-tag.sh line 29

Quoted 8 lines
devtools/check-spdx-tag.sh new file +8 -0
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile
+
+    errors=0
+    while read -r line
+    do $quiet || echo $line
+       errors=$((errors + 1))  

I'm surprised this works for you.
In general, "while" creates a subshell which makes impossible updating a variable.
I recommend using "for" with IFS=$'\n'.

Quoted 6 lines
devtools/check-spdx-tag.sh new file +6 -0
+    done < $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo

devtools/check-spdx-tag.sh line 45

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' |
+	while read line
+	do $quiet || echo $line
+	   warnings=$((warnings + 1))
+	done  

Same comment about "while" subshell.

devtools/check-spdx-tag.sh line 50

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+
+    warnings=0
+    while read -r line
+    do $quiet || echo $line
+       warnings=$((errors + 1))  

Here too

Quoted 2 lines
(quoted diff) +2 -0
+    done < $tmpfile
+}  

[...]

Quoted 2 lines

+Each file must begin with a special comment containing the
+Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>_.

Typo: Identifier

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 60

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+
+Generally this is the BSD License, except for code granted special exceptions.  

Is a verb missing?

Quoted 5 lines
Quoted 2 lines

doc/guides/contributing/coding_style.rst To document a public API, a doxygen-like format must be used: refer to :ref:`dox · line 62

Quoted 2 lines
doc/guides/contributing/coding_style.rst +2 -0
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).  

Thanks for the feedback.
Text processing is simpler in python, will rewrite in next version?

I don't see a need for rewrite.
I think addressing the comments is simpler.

#13 lore raw
In reply to Stephen Hemminger
Quoted 50 lines · Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

---
v4 - add MAINTAINERS entry
update coding style document
change name of script

 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 77 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh
MAINTAINERS +1 -0
index 3d5e8d1104b2..6b0e042c5fbb 100644
@@ -96,6 +96,7 @@ F: devtools/check-maintainers.sh
 F: devtools/check-forbidden-tokens.awk
 F: devtools/check-git-log.sh
 F: devtools/check-includes.sh
+F: devtools/check-spdx-tag.sh
 F: devtools/check-symbol-maps.sh
 F: devtools/checkpatches.sh
 F: devtools/get-maintainer.sh
devtools/check-spdx-tag.sh new file +14 -0
new file mode 100755
index 000000000000..b1b8cdba4e4e
@@ -0,0 +1,77 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"

devtools/check-spdx-tag.sh line 24

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile

I find it easier to maintain an exclude list by setting a variable and generating the relevant parameters:

    excludes='.git* .ci/* .travis.yml */Kbuild */README'
    exclude_opt=""
    set -f
    for pattern in $excludes; do
        exclude_opt="$exclude_opt ':^${pattern}'"
    done
    set +f
    printf "\"%s\"\n" "$exclude_opt"

However I recognize that means dealing with contrarian globbing issues in shells, so it comes at a price. But I find changing the exclude list much easier that way.

Quoted 76 lines
devtools/check-spdx-tag.sh new file +53 -0
+
+    errors=0
+    while read -r line
+    do $quiet || echo $line
+       errors=$((errors + 1))
+    done < $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' |
+	while read line
+	do $quiet || echo $line
+	   warnings=$((warnings + 1))
+	done
+
+    warnings=0
+    while read -r line
+    do $quiet || echo $line
+       warnings=$((errors + 1))
+    done < $tmpfile
+}
+
+quiet=false
+verbose=false
+
+while getopts qvh ARG ; do
+	case $ARG in
+		q ) quiet=true ;;
+		v ) verbose=true ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+
+tmpfile=$(mktemp)
+trap 'rm -f -- "$tmpfile"' INT TERM HUP EXIT
+
+check_spdx
+$quiet || echo
+
+check_boilerplate
+
+$quiet || echo
+echo "total: $errors errors, $warnings warnings"
+exit $errors
doc/guides/contributing/coding_style.rst +7 -2
index 841ef6d5c829..04626667dc18 100644
@@ -54,8 +54,13 @@ To document a public API, a doxygen-like format must be used: refer to :ref:`dox
 License Header
 ~~~~~~~~~~~~~~
 
-Each file should begin with a special comment containing the appropriate copyright and license for the file.
-Generally this is the BSD License, except for code for Linux Kernel modules.
+Each file must begin with a special comment containing the
+`Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>`_.
+
+Generally this is the BSD License, except for code granted special exceptions.
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
+
 After any copyright header, a blank line should be left before any other contents, e.g. include statements in a C file.
 
 C Preprocessor Directives
Signature · 1 line
2.20.1
Signature · 1 line
Gaëtan
#14 lore raw
Quoted 74 lines · On Fri, 12 Jun 2020 11:05:27 +0200 Gaëtan Rivet <[email protected]> wrote:
Quoted 50 lines · Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

Update the contribution guidelines to indicate that SPDX license identfier is required for this project.

---
v4 - add MAINTAINERS entry
update coding style document
change name of script

 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 77 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh
MAINTAINERS +1 -0
index 3d5e8d1104b2..6b0e042c5fbb 100644
@@ -96,6 +96,7 @@ F: devtools/check-maintainers.sh
 F: devtools/check-forbidden-tokens.awk
 F: devtools/check-git-log.sh
 F: devtools/check-includes.sh
+F: devtools/check-spdx-tag.sh
 F: devtools/check-symbol-maps.sh
 F: devtools/checkpatches.sh
 F: devtools/get-maintainer.sh
devtools/check-spdx-tag.sh new file +14 -0
new file mode 100755
index 000000000000..b1b8cdba4e4e
@@ -0,0 +1,77 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"

devtools/check-spdx-tag.sh line 24

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^doc/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	> $tmpfile  

I find it easier to maintain an exclude list by setting a variable and generating the relevant parameters:

    excludes='.git* .ci/* .travis.yml */Kbuild */README'
    exclude_opt=""
    set -f
    for pattern in $excludes; do
        exclude_opt="$exclude_opt ':^${pattern}'"
    done
    set +f
    printf "\"%s\"\n" "$exclude_opt"
Quoted 2 lines

However I recognize that means dealing with contrarian globbing issues in shells, so it comes at a price. But I find changing the exclude list much easier that way.

This is no easier to maintain. Doing more string stuff in shell is not worth it. If gets to be too much trouble, will consider moving to python3.

#15 lore raw

PATCH 1/1 devtools: add new SPDX license compliance checker

In reply to Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

---
v5 - fix issue in counting warnings
update Copyright date

 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 73 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 81 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh
MAINTAINERS +1 -0
index f4e0ed8e056c..75e0dc75f887 100644
@@ -96,6 +96,7 @@ F: devtools/check-maintainers.sh
 F: devtools/check-forbidden-tokens.awk
 F: devtools/check-git-log.sh
 F: devtools/check-includes.sh
+F: devtools/check-spdx-tag.sh
 F: devtools/check-symbol-maps.sh
 F: devtools/checkpatches.sh
 F: devtools/get-maintainer.sh
devtools/check-spdx-tag.sh new file +73 -0

73 lines not on this page · raw message

doc/guides/contributing/coding_style.rst +7 -2
index 841ef6d5c829..04626667dc18 100644
@@ -54,8 +54,13 @@ To document a public API, a doxygen-like format must be used: refer to :ref:`dox
 License Header
 ~~~~~~~~~~~~~~
 
-Each file should begin with a special comment containing the appropriate copyright and license for the file.
-Generally this is the BSD License, except for code for Linux Kernel modules.
+Each file must begin with a special comment containing the
+`Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>`_.
+
+Generally this is the BSD License, except for code granted special exceptions.
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
+
 After any copyright header, a blank line should be left before any other contents, e.g. include statements in a C file.
 
 C Preprocessor Directives
Signature · 1 line
2.20.1
#16 lore raw

PATCH v5 1/1 devtools: add new SPDX license compliance checker

In reply to Stephen Hemminger

Simple script to look for drivers and scripts that are missing requires SPDX header.

---

v5 - address review comments
simplify the display logic
scan doc and buildtools as well

 MAINTAINERS                              |  1 +
 devtools/check-spdx-tag.sh               | 70 ++++++++++++++++++++++++
 doc/guides/contributing/coding_style.rst |  9 ++-
 3 files changed, 78 insertions(+), 2 deletions(-)
 create mode 100755 devtools/check-spdx-tag.sh
MAINTAINERS +1 -0
index 3cd402b34c91..a9e01330a71e 100644
@@ -90,6 +90,7 @@ F: devtools/check-maintainers.sh
 F: devtools/check-forbidden-tokens.awk
 F: devtools/check-git-log.sh
 F: devtools/check-includes.sh
+F: devtools/check-spdx-tag.sh
 F: devtools/check-symbol-maps.sh
 F: devtools/checkpatches.sh
 F: devtools/get-maintainer.sh
devtools/check-spdx-tag.sh new file +70 -0

70 lines not on this page · raw message

doc/guides/contributing/coding_style.rst +7 -2
index 4efde93f6af0..b55075eaa240 100644
@@ -54,8 +54,13 @@ To document a public API, a doxygen-like format must be used: refer to :ref:`dox
 License Header
 ~~~~~~~~~~~~~~
 
-Each file should begin with a special comment containing the appropriate copyright and license for the file.
-Generally this is the BSD License, except for code for Linux Kernel modules.
+Each file must begin with a special comment containing the
+`Software Package Data Exchange (SPDX) License Identfier <https://spdx.org/using-spdx-license-identifier>`_.
+
+Generally this is the BSD License, except for code granted special exceptions.
+The SPDX licences identifier is sufficient, a file should not contain
+an additional text version of the license (boilerplate).
+
 After any copyright header, a blank line should be left before any other contents, e.g. include statements in a C file.
 
 C Preprocessor Directives
Signature · 1 line
2.27.0
#17 lore raw
Quoted 4 lines · On Tue, 14 Jul 2020 16:21:01 -0700 Stephen Hemminger…

Simple script to look for drivers and scripts that are missing requires SPDX header.

Preprocessor Directives

Because of the long delay in merging this tool new files have come in that are not following the required guidelines!

 $ ./devtools/check-spdx-tag.sh -v
Files without SPDX License
--------------------------
doc/guides/custom.css
doc/guides/linux_gsg/nic_perf_intel_platform.rst
doc/guides/prog_guide/build-sdk-meson.rst
drivers/net/bnxt/tf_core/cfa_resource_types.h
drivers/net/qede/base/ecore_hsi_func_common.h
lib/librte_eal/windows/eal_hugepages.c

Files with redundant license text
---------------------------------
app/test/test_timer_racecond.c
lib/librte_ethdev/rte_ethdev_pci.h
lib/librte_ethdev/rte_ethdev_vdev.h

total: 6 errors, 3 warnings

#18 lore raw

15/07/2020 01:25, Stephen Hemminger:

Quoted 11 lines
Quoted 4 lines · On Tue, 14 Jul 2020 16:21:01 -0700 Stephen Hemminger…

Simple script to look for drivers and scripts that are missing requires SPDX header.

Preprocessor Directives

Because of the long delay in merging this tool new files have come in that are not following the required guidelines!

Probably it would have helped if you Cc'ed some persons known to be interested in this topic. Getting reviews is harder recently.

#20 lore raw
In reply to Stephen Hemminger

15/07/2020 01:21, Stephen Hemminger:

Quoted 4 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

[...]

Quoted 19 lines
devtools/check-spdx-tag.sh new file +19 -0
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+errors=0
+warnings=0
+quiet=false
+verbose=false
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"

devtools/check-spdx-tag.sh line 29

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	':^*.svg' ':^*.png'\

I don't agree with this list of files.
But I guess we can start with that and be more strict in future.

Quoted 21 lines
devtools/check-spdx-tag.sh new file +21 -0
+	> $tmpfile
+
+    errors=$(wc -l < $tmpfile)
+    $quiet || cat $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' > $tmpfile
+
+    warnings=$(wc -l <$tmpfile)
+    $quiet || cat $tmpfile
+}
+
+while getopts qvh ARG ; do

devtools/check-spdx-tag.sh line 60

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	case $ARG in
+		q ) quiet=true ;;
+		v ) verbose=true ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+
+tmpfile=$(mktemp)

Should be mktemp -t dpdk.checkspdx.XXXXXX
to keep namespace of our temp files. Will fix.

Quoted 1 line

+trap 'rm -f -- "$tmpfile"' INT TERM HUP EXIT

Why catching HUP signal?

Applied, thanks

#21 lore raw
Quoted 74 lines · On Fri, 31 Jul 2020 00:06:23 +0200 Thomas Monjalon <[email protected]> wrote:

15/07/2020 01:21, Stephen Hemminger:

Quoted 4 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

[...]

Quoted 19 lines
devtools/check-spdx-tag.sh new file +19 -0
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+errors=0
+warnings=0
+quiet=false
+verbose=false
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"

devtools/check-spdx-tag.sh line 29

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	':^*.svg' ':^*.png'\  

I don't agree with this list of files.
But I guess we can start with that and be more strict in future.

Quoted 21 lines
devtools/check-spdx-tag.sh new file +21 -0
+	> $tmpfile
+
+    errors=$(wc -l < $tmpfile)
+    $quiet || cat $tmpfile
+}
+
+check_boilerplate() {
+    if $verbose ; then
+	echo
+	echo "Files with redundant license text"
+	echo "---------------------------------"
+    fi
+
+    git grep -l Redistribution -- \
+	':^license/' ':^/devtools/check-spdx-tag.sh' > $tmpfile
+
+    warnings=$(wc -l <$tmpfile)
+    $quiet || cat $tmpfile
+}
+
+while getopts qvh ARG ; do

devtools/check-spdx-tag.sh line 60

Quoted 10 lines
devtools/check-spdx-tag.sh new file +10 -0
+	case $ARG in
+		q ) quiet=true ;;
+		v ) verbose=true ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+
+tmpfile=$(mktemp)  

Should be mktemp -t dpdk.checkspdx.XXXXXX
to keep namespace of our temp files. Will fix.

Quoted 3 lines
Quoted 1 line

+trap 'rm -f -- "$tmpfile"' INT TERM HUP EXIT

Why catching HUP signal?

General practice to have a script cleanup if user logs out. Back in the old days, connections were lost sometimes :-)_

#22 lore raw
In reply to Thomas Monjalon
Quoted 30 lines · On Fri, 31 Jul 2020 00:06:23 +0200 Thomas Monjalon <[email protected]> wrote:

15/07/2020 01:21, Stephen Hemminger:

Quoted 4 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

[...]

Quoted 24 lines
devtools/check-spdx-tag.sh new file +24 -0
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+errors=0
+warnings=0
+quiet=false
+verbose=false
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
Quoted 8 lines

devtools/check-spdx-tag.sh line 29

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	':^*.svg' ':^*.png'\  

I don't agree with this list of files.
But I guess we can start with that and be more strict in future.

I suppose README, MAINTAINERS, .ini and .txt files could be removed from the exception list.

#23 lore raw
Quoted 42 lines · Stephen Hemminger
Quoted 30 lines · On Fri, 31 Jul 2020 00:06:23 +0200 Thomas Monjalon <[email protected]> wrote:

15/07/2020 01:21, Stephen Hemminger:

Quoted 4 lines

Simple script to look for drivers and scripts that are missing requires SPDX header.

[...]

Quoted 24 lines
devtools/check-spdx-tag.sh new file +24 -0
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Microsoft Corporation
+#
+# Produce a list of files with incorrect license tags
+
+errors=0
+warnings=0
+quiet=false
+verbose=false
+
+print_usage () {
+    echo "usage: $(basename $0) [-q] [-v]"
+    exit 1
+}
+
+check_spdx() {
+    if  $verbose;  then
+	echo "Files without SPDX License"
+	echo "--------------------------"
+    fi
+    git grep -L SPDX-License-Identifier -- \
+	':^.git*' ':^.ci/*' ':^.travis.yml' \
+	':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
Quoted 8 lines

devtools/check-spdx-tag.sh line 29

Quoted 5 lines
devtools/check-spdx-tag.sh new file +5 -0
+	':^*/Kbuild' ':^*/README' \
+	':^license/' ':^config/' ':^buildtools/' \
+	':^*.cocci' ':^*.abignore' \
+	':^*.def' ':^*.map' ':^*.ini' ':^*.data' ':^*.cfg' ':^*.txt' \
+	':^*.svg' ':^*.png'\  

I don't agree with this list of files.
But I guess we can start with that and be more strict in future.

Quoted 1 line

I suppose README, MAINTAINERS, .ini and .txt files could be removed from the exception list.

We also want to scan the files in config and buildtools.