Skip to contentSkip to conversation

SPDX in the kernel: State of the union

DISCUSSION Thomas Gleixner 2 participants 5 messages May 17, 2022 – May 22, 2022 Licenses and spdx stuff

lore
Share & tools
lore
#0 lore raw

Folks!

After the initial SPDX effort which ended about three years ago there was not really much progress neither in terms of file statistics nor in terms of activity on this list... I'm refraining from asking the obvious questions...

Nevertheless I'm trying to cut myself some cycles to get this rolling again.

As a first step I tried to resurrect my old scripts. That was not really an enjoyable experience due to the python2 -> python3 fallout and the changes in scancode since then.

Though after quite some cursing I was able to gather at least initial statistics and to analyze patches based on the scancode detection rules.

I surely have to say quite some words about the 'improved' scancode detection rules too, but I sort that out with Philippe off-list.

So here is where we are:

Files without SPDX identifier: 16410 ~78% of total files

Files without any license hint: 7131 ~43% of !SPDX'ed files
Files with one license hint: 6673 ~40% of !SPDX'ed files

Files with two license hints:            2267   ~13% of !SPDX'ed files
Files with more than two hints:           339   ~ 2% of !SPDX'ed files

Files with less than 4 lines content:

        0 length:	   33   (some can be removed)
	1 line:		  276
	2 lines:	  109
	3 lines:	  135

Files without any license hint:

        arch                 774
	block		       1
	certs		       2
	crypto		      10
	Documentation	    4266
	drivers		     320
	fs		      26
	include		     124
	init		       0
	ipc		       0
	kernel		      14
	lib		      26
	mm		       3
	net		      15
	samples		       7
	scripts		      63
	security	       8
	sound		       9
	tools		    1457
	usr		       0
	virt		       0

Files with one license hint:

        arch		    1405
	block		       0
	certs		       1
	crypto		       1
	Documentation	      65
	drivers		    4369
	fs		     126
	include		     356
	init		       0
	ipc		       1
	kernel		      18
	lib		      35
	mm		       4
	net		      69
	samples		      14
	scripts		      26
	security	       0
	sound		      40
	tools		     141
	usr		       1
	virt		       0

Files with two license hints:

        arch		     731
	block		       0
	certs		       0
	crypto		       3
	Documentation	      13
	drivers		    1114
	fs		      66
	include		     101
	init		       0
	ipc		       0
	kernel		       0
	lib		      54
	mm		       0
	net		      91
	samples		      39
	scripts		       5
	security	       1
	sound		      14
	tools		      35
	usr		       0
	virt		       0

Script-able files with reasonable effort:

No hint: 6501 ~90% of no-hint files
One hint: 5129 ~76% of one-hint files
Two hints: 584 ~25% of two-hint files
Total: 12213 ~75% of !SPDX'ed file

Remaining: 4197 ~5% of total files

Scancode rules involved:     561
Scancode rules validated:    117

My plan is to focus on the 'low hanging' fruit of reasonably easy script-able files first.

For the files with zero hints that requires a few questions to be answered upfront:

1) What's the approach for files with obviously not copyright-able content:

- Files which just include other file[s] (one or two lines)

- Files which have just a more or less useful comment why they are otherwise empty (one to three lines)

- Files which just contain a #define FOO and an include of another file to compile the included file with some other functionality (two or three lines)

2) What's the approach for machine generated files:

- Primarily kernel configuration files

3) What's the approach for 'hidden' dot-files like .gitignore:

Those files are just providing information to tools. The file format is defined by the tool (git, clang, coccinelle....) and the creative content is exactly zero...

4) What's the approch for binary blobs or other files which cannot carry license information in the file itself?

Which is related to the discussion in this thread:

https://lore.kernel.org/all/[email protected]

The other question for these files with zero hints is which license to chose. Sure you can argue that all files w/o any hint fall under the project license, but especially the Documentation directory is interesting as it's not clear for all of the various content what the preferred and assumed license should be. That needs some thoughts and clarifications. For the kernel code itself that's not a real question, but the tools directory might need some care too.

For the files which have a licensing hint in whatever form, I think resuming the work where we left off, i.e. mainly reviewing per scancode match rules based patterns, makes a lot of sense.

Based on my cursory validation of those patterns I'm confident that we can reach a 95% coverage within a reasonable amount of time.

Finally here is another round of important questions:

#1 Is there still interest to get this done? The silence on this list
after the initial effort is deafening.

#2 Are there still enough interested and comptent people on this list to
handle the legal questions?

#3 Was there any progress on the outstanding questions on this list where
discussion dried out almost 3 years ago?

I'm willing to pull the cart again, but if the interest and support stays around zero, I surely have other things to do.

Thanks,

Thomas

#1 lore raw
Quoted 5 lines · Thomas Gleixner

Finally here is another round of important questions:

#1 Is there still interest to get this done? The silence on this list
after the initial effort is deafening.

This list had 210 messages in 2021, and 64 so far in 2022, which may be silence compared to LKML, but is reasonably respectable ongoing traffic for a small cleanup project.

I'm still reviewing all patches as they flow through this list. I haven't been actively marking them as reviewed-by me, but I would raise any problems I saw, and I've seen others raising problems.

So, yes, there's still interest, and if you want to start generating more patches, I'll happily contribute to the review process.

I actually thought you just ran out of easily scriptable fixes, but it's nice to hear that there's still substantially more we can do with scancode rules.

Quoted 2 lines

#2 Are there still enough interested and comptent people on this list to
handle the legal questions?

I think so, yes. If we've lost some of our reviewers, we can recruit new ones.

With the auto-generated patches, you will probably need to rate-limit like you did in 2019, since the tools can generate patches far more rapidly than the humans can review them.

Quoted 2 lines

#3 Was there any progress on the outstanding questions on this list where
discussion dried out almost 3 years ago?

ISTR reaching conclusions on all the questions before, but if there are some lingering, we can revive them. And, you raised good new questions too.

Quoted 2 lines

I'm willing to pull the cart again, but if the interest and support stays around zero, I surely have other things to do.

If you have the time and energy to do another burst, go for it. I don't know that we'll ever get to 100%, but every file we clean up is helpful, so it's worth continuing.

Allison

#2 lore raw
Quoted 4 lines · Allison Randal

On 5/17/22 7:31 PM, Thomas Gleixner wrote:
I actually thought you just ran out of easily scriptable fixes, but it's nice to hear that there's still substantially more we can do with scancode rules.

I ran out of cycles :)

Quoted 3 lines

With the auto-generated patches, you will probably need to rate-limit like you did in 2019, since the tools can generate patches far more rapidly than the humans can review them.

Sure.

Quoted 3 lines

If you have the time and energy to do another burst, go for it. I don't know that we'll ever get to 100%, but every file we clean up is helpful, so it's worth continuing.

I started to get some structure into this mess. For the first step I excluded the Documentation directory unless files in that, which fit into match rules applying to source files. I'll tend to the Documentation directory in a seperate step.

Then I categorized the remaining match rules into the following:

Nr   Category        Rules     Files affected
 1   GPLv2[+]	    141	         1607
 2   GPL unknown     84	         1663
 3   MIT	     28	         3275
 4   GPLv2/MIT	      2	           36
 5   BSD	     20	          114
 6   GPL/BSD	     32          1004
 7   ISC	      4           343
 8   X11	      1             3
 9   Other	      9            50
10   Unclear	     63	          916
11   Unknown	     78	          321
12   Nasty	     16	           48
13   Bogus	     21	          861

#1 Pretty clear GPLv2[or later] and LGPL matches.

#2 The nasty 'under GPL' ones. Quite some of them reference COPYING

#3-9 Pretty clear matches for MIT/BSD/ISC/X11/ZLIB and GPL combos of
those

#10 The unclear (at least to me) ones

#11 Licenses the kernel does not have (yet) in the LICENSES
directory, but some of them are not really clear to me

#12 GPL version 1 and version 3, reiserfs and some proprietary

#13 A set of bogosities in scancode which I need to discuss
with Philippe.

I probably made some mistakes here and there, but that's what I have now.

I've generated static HTML pages from the data, which are available here:

https://tglx.de/~tglx/spdx/index.html

so you can get a taste of what is coming to you sooner than later. The categories link to pages with rules and the rules to a per rule details page. The latter has links to a Linux cross reference site in case you want to look at the real think instead of the 'normalized' match patterns on the rule page.

My plan is to start with categories #1 and #3-9 and send out batches of patches to the list.

Which size of batches and what rate do you folks prefer?

Thanks,

tglx

#3 lore raw
Quoted 2 lines · Thomas Gleixner

I ran out of cycles :)

Nod, it happens. Really, it's a planet-wide phenomenon the past couple of years. :)

Quoted 1 line

I started to get some structure into this mess.

[...]

Quoted 4 lines

I've generated static HTML pages from the data, which are available here:

https://tglx.de/~tglx/spdx/index.html

Makes sense, and a large number of them look like they'll be easy to review and approve.

Quoted 1 line

Which size of batches and what rate do you folks prefer?

Looking back to 2019, you generally sent batches of 10-25 patches per day, where each patch was one match rule. Seems reasonable to start again there, and tune up or down as needed.

Allison

#4 lore raw
Quoted 8 lines · Allison Randal
Quoted 4 lines · Thomas Gleixner

I've generated static HTML pages from the data, which are available here:

https://tglx.de/~tglx/spdx/index.html

Makes sense, and a large number of them look like they'll be easy to review and approve.

I hope so.

Quoted 5 lines
Quoted 1 line

Which size of batches and what rate do you folks prefer?

Looking back to 2019, you generally sent batches of 10-25 patches per day, where each patch was one match rule. Seems reasonable to start again there, and tune up or down as needed.

Sounds like a plan.

Thanks,

Thomas