Misc Coccinelle-related improvements

The 'coccicheck' target doesn't create a file with the same name, so mark it as .PHONY.

Signed-off-by SZEDER Gábor <[email protected]>

---
  Makefile | 2 ++
  1 file changed, 2 insertions(+)

Makefile +0 -0

index e4b503d259..27bfc196dd 100644

The 'coccicheck' target doesn't create a file with the same name, so mark it as .PHONY.

Signed-off-by SZEDER Gábor <[email protected]>

---

 Makefile | 2 ++
 1 file changed, 2 insertions(+)

Makefile +2 -0

index e4b503d259..27bfc196dd 100644
@@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
 	fi
 coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
 
+.PHONY: coccicheck
+
 ### Installation rules
 
 ifneq ($(filter /%,$(firstword $(template_dir))),)

The dependencies of the 'coccicheck' make target are listed with the help of the $(patsubst) make function, which in this case doesn't do any pattern substitution, but only adds the '.patch' suffix.

Use the shorter and more idiomatic $(addsuffix) make function instead.

Signed-off-by SZEDER Gábor <[email protected]>

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +1 -1

index 27bfc196dd..8f509576e9 100644
@@ -2683,7 +2683,7 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
 	then \
 		echo '    ' SPATCH result: $@; \
 	fi
-coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
+coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
 
 .PHONY: coccicheck

sha1dc is an external library, that we carry in-tree for convenience or grab as a submodule, so there is no use in applying our semantic patches to its source files.

Therefore, exclude sha1dc's source files from Coccinelle's static analysis.

Signed-off-by SZEDER Gábor <[email protected]>

---

Quoted 7 lines · SZEDER Gábor

sha1dc is an external library, that we carry in-tree for convenience or grab as a submodule, so there is no use in applying our semantic patches to its source files.

Therefore, exclude sha1dc's source files from Coccinelle's static analysis.

Makefile check: command-list.h · line 2670

Quoted 9 lines

Signed-off-by SZEDER Gábor <[email protected]>

---

Makefile +5 -0

@@ -2666,10 +2666,16 @@ check: command-list.h
+ifdef DC_SHA1_SUBMODULE
+COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
+else
+COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
+endif

Can't you just filter out both of these unconditionally without worrying about DC_SHA1_SUBMODULE?

Quoted 9 lines · Eric Sunshine

Makefile check: command-list.h · line 2673

Quoted 5 lines · SZEDER Gábor

Makefile +5 -0

+ifdef DC_SHA1_SUBMODULE
+COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
+else
+COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
+endif

Can't you just filter out both of these unconditionally without worrying about DC_SHA1_SUBMODULE?

I'm not sure what you mean by that. Like this perhaps?

COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(filter-out
sha1dc/%,$(C_SOURCES)))

While it's only a single line, I don't think it's any easier on the eyes.

Coccinelle outputs its suggested transformations as patches, whose header looks something like this:

   --- commit.c
   +++ /tmp/cocci-output-19250-7ae78a-commit.c

Note the lack of 'diff --opts <old> <new>' line, the differing number of path components on the --- and +++ lines, and the nonsensical filename on the +++ line. 'patch -p0' can still apply these patches, as it takes the filename to be modified from the --- line. Alas, 'git apply' can't, because it takes the filename from the +++ line, and then complains about the nonexisting file.

Pass the '--patch .' options to Coccinelle via the SPATCH_FLAGS 'make' variable, as it seems to make it generate proper context diff patches, with the header starting with a 'diff ...' line and containing sane filenames. The resulting 'contrib/coccinelle/*.cocci.patch' files then can be applied both with 'git apply' and 'patch' (even without '-p0').

Signed-off-by SZEDER Gábor <[email protected]>

---
  Makefile | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +0 -0

index 73e2d16926..72ea29df4e 100644

Just a couple of minor Coccinelle-related improvements:

- The first two patches are small cleanups.

- The last three could make life perhaps just a tad bit easier for devs running 'make coccicheck'.

Just a couple of minor Coccinelle-related improvements:

- The first two patches are small cleanups.

- The last three could make life perhaps just a tad bit easier for devs running 'make coccicheck'.

Just a couple of minor Coccinelle-related improvements:

- The first two patches are small cleanups.

- The last three could make life perhaps just a tad bit easier for devs running 'make coccicheck'.

SZEDER Gábor (5):
coccinelle: mark the 'coccicheck' make target as .PHONY
coccinelle: use $(addsuffix) in 'coccicheck' make target
coccinelle: exclude sha1dc source files from static analysis
coccinelle: put sane filenames into output patches
coccinelle: extract dedicated make target to clean Coccinelle's
results

Just a couple of minor Coccinelle-related improvements:

- The first two patches are small cleanups.

- The last three could make life perhaps just a tad bit easier for devs running 'make coccicheck'.

Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files. The only remaining sequential part is aggregating the suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions).

This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run

touch apply.c ; time make -j4 coccicheck

went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running

touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck

went from 56.364s to 35.883s, which is ~36% speedup.

Unfortunately, this new approach has some disadvantages compared to the current situation:

- [RFC]
With this patch 'make coccicheck's output will look like this ('make' apparently doesn't iterate over semantic patches in lexicographical order):

SPATCH commit.cocci abspath.c
SPATCH commit.cocci advice.c
<... lots of lines ...>
SPATCH array.cocci http-walker.c
SPATCH array.cocci remote-curl.c

which means that the number of lines in the output grows from "Nr-of-semantic-patches" to "Nr-of-semantic-patches * Nr-of-source-files".

---
  Makefile                      | 52 ++++++++++++++++++++++++-----------
  contrib/coccinelle/.gitignore |  3 +-
  2 files changed, 38 insertions(+), 17 deletions(-)

Makefile +14 -10

index d616c04125..f516dd93d1 100644
@@ -2674,25 +2674,44 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
  else
  COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
  endif
+COCCI_SEM_PATCHES = $(wildcard contrib/coccinelle/*.cocci)
 
-%.cocci.patch: %.cocci $(COCCI_SOURCES)
-	@echo '    ' SPATCH $<; \
-	ret=0; \
-	for f in $(COCCI_SOURCES); do \
-		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
-			{ ret=$$?; break; }; \
-	done >$@+ 2>[email protected]; \
-	if test $$ret != 0; \
+define cocci_template
+$(cocci_sem_patch)_dirs := $$(addprefix $(cocci_sem_patch).patches/,$$(sort $$(dir $$(COCCI_SOURCES))))
+
+$$($(cocci_sem_patch)_dirs):
+	@mkdir -p $$@
+
+# e.g. 'contrib/coccinelle/strbuf.cocci.patches/builtin/commit.c.patch'
+# Applies one semantic patch to _one_ source file.
+$(cocci_sem_patch).patches/%.patch: % $(cocci_sem_patch)
+	@printf '     SPATCH %-25s %s\n' $$(notdir $(cocci_sem_patch)) $$<; \
+	if ! $$(SPATCH) --sp-file $(cocci_sem_patch) $$< $$(SPATCH_FLAGS) >$$@ 2>[email protected]; \
  	then \
-		cat [email protected]; \
+		rm -f $$@; \
+		cat [email protected]; \
  		exit 1; \
-	fi; \

Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files. The only remaining sequential part is aggregating the suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions).

This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run

touch apply.c ; time make -j4 coccicheck

went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running

touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck

went from 56.364s to 35.883s, which is ~36% speedup.

All the above timings are best-of-five on a machine with 2 physical and 2 logical cores. I don't have the hardware to bring any numbers for point (3). The time needed to run 'make -j4 coccicheck' in a clean state didn't change, it's ~3m42s both with and without this patch.

- [RFC]
With this patch 'make coccicheck's output will look like this ('make' apparently doesn't iterate over semantic patches in lexicographical order):

SPATCH commit.cocci abspath.c
SPATCH commit.cocci advice.c
<... lots of lines ...>
SPATCH array.cocci http-walker.c
SPATCH array.cocci remote-curl.c

which means that the number of lines in the output grows from "Nr-of-semantic-patches" to "Nr-of-semantic-patches * Nr-of-source-files".

- [RFC]
The overhead of applying a semantic patch to all source files became larger. 'make coccicheck' currently runs only one shell process and creates two output files for each semantic patch. With this patch it will run approx. "Nr-of-semantic-patches * Nr-of-source-files" shell processes and create twice as many output files.

- [RFC]
This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated.

I suspect if we go with the one-spatch-per-source route, though, that we could do this just with regular make rules.

Quoted 2 lines · Jeff King

I suspect if we go with the one-spatch-per-source route, though, that we could do this just with regular make rules.

Yeah, it's pretty straightforward:

Makefile +0 -0

index d616c0412..86fdcf567 100644

I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in most of the targets, and that would let people do individual:

make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck

Makefile COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) · line 2677

Quoted 9 lines

Makefile +1 -0

index d616c0412..86fdcf567 100644
@@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
 else
 COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
 endif
+COCCI_COMBINED = contrib/coccinelle/combined.cocci

I like this approach.

Quoted 4 lines

I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in most of the targets, and that would let people do individual:

make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck

The issue here is that the dependencies for foo.cocci become unreliable, so I'd rather have a separate target for that (e.g. depending on FORCE) if we go that way.

Quoted 7 lines · Jonathan Nieder

Quoted 4 lines · Jeff King

I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in most of the targets, and that would let people do individual:

make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck

Quoted 3 lines

The issue here is that the dependencies for foo.cocci become unreliable, so I'd rather have a separate target for that (e.g. depending on FORCE) if we go that way.

Can you be more specific? I don't see how it's unreliable, unless you mean that anything relying on "coccicheck" would depend on the exact value of COCCI_PATCH.

But it may all be moot anyway, based no the responses elsewhere in the thread.

Quoted 3 lines · SZEDER Gábor

This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated.

I looked into this a long time ago for another set of Makefile patches I was considering. $(eval) was added to GNU make in 3.80, released in 2002. Which is probably fine by now.

Quoted 8 lines · Jeff King

Quoted 3 lines · SZEDER Gábor

This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated.

I looked into this a long time ago for another set of Makefile patches I was considering. $(eval) was added to GNU make in 3.80, released in 2002. Which is probably fine by now.

For the record, MacOS developer tools are stuck at GNU make 3.81 (from 2006).

Quoted 21 lines · SZEDER Gábor

Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files. The only remaining sequential part is aggregating the suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions).

This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run

touch apply.c ; time make -j4 coccicheck

went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running

Quoted 3 lines

touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck

went from 56.364s to 35.883s, which is ~36% speedup.

I really like this direction. The slowness of coccicheck is one of the things that has prevented me from running it more frequently. And I'm a big fan of breaking steps down as much as possible into make targets. It lets make do its job (avoiding repeated work and parallelizing).

Quoted 5 lines

All the above timings are best-of-five on a machine with 2 physical and 2 logical cores. I don't have the hardware to bring any numbers for point (3). The time needed to run 'make -j4 coccicheck' in a clean state didn't change, it's ~3m42s both with and without this patch.

On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch went from:

real 1m25.520s
user 5m41.492s
sys 0m26.776s

to:

real 0m24.300s
user 14m35.084s
sys 0m50.108s

I was surprised by the jump in CPU times. Doing "make -j1 coccicheck" with your patch results in:

real 5m34.887s
user 5m5.620s
sys 0m19.908s

so it's really the parallelizing that seems to be to blame (possibly because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8 threads in the first example).

Quoted 11 lines

- [RFC]
With this patch 'make coccicheck's output will look like this ('make' apparently doesn't iterate over semantic patches in lexicographical order):

SPATCH commit.cocci abspath.c
SPATCH commit.cocci advice.c
<... lots of lines ...>
SPATCH array.cocci http-walker.c
SPATCH array.cocci remote-curl.c

Quoted 3 lines

which means that the number of lines in the output grows from "Nr-of-semantic-patches" to "Nr-of-semantic-patches * Nr-of-source-files".

IMHO this is not really that big a deal. We are doing useful work for each line, so to me it's just more eye candy (and it's really satisfying to watch it zip by on the 40-core machine ;) ).

What if we inverted the current loop? That is, right now we iterate over the cocci patches at the Makefile level, and then for each target we iterate over the giant list of source files. Instead, we could teach the Makefile to iterate over the source files, with a target for each, and then hit each cocci patch inside there.

That would give roughly the same output as a normal build. But moreover, I wonder if we could make things faster by actually combining the cocci files into a single set of rules to be applied to each source file. That would mean invoking spatch 1/8 as much. It does give fewer opportunities for "make" to reuse work, but that only matters when the cocci files change (which is much less frequent than source files changing).

Doing:

cat contrib/coccinelle/*.cocci >mega.cocci make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

gives me:

real 0m17.573s
user 10m56.724s
sys 0m11.548s

And that should show an improvement on more normal-sized systems, too, because we really are eliminating some of the startup overhead.

The other obvious downside is that you don't get individual patches for each class of transformation. Do we care? Certainly for a routine "make coccicheck" I primarily want to know:

- is there something that needs fixing?

- give me the patch for all fixes

So I wonder if we'd want to have that be the default, and then perhaps optionally give some targets to let people run single scripts (or not; they could probably just run spatch themselves).

Quoted 7 lines

- [RFC]
The overhead of applying a semantic patch to all source files became larger. 'make coccicheck' currently runs only one shell process and creates two output files for each semantic patch. With this patch it will run approx. "Nr-of-semantic-patches * Nr-of-source-files" shell processes and create twice as many output files.

Doing the "one big .cocci" would help with this, too (and again puts us in the same ballpark as a compile).

Quoted 4 lines

- [RFC]
This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated.

I looked into this a long time ago for another set of Makefile patches I was considering. $(eval) was added to GNU make in 3.80, released in 2002. Which is probably fine by now.

If it isn't, one more exotic option would be to push the coccicheck stuff into its own Makefile, and just run it via recursive make. Then anybody doing a vanilla build can do so even with an antique make, but you could only "make coccicheck" with something from the last 16 years (but good luck getting ocaml running there).

I suspect if we go with the one-spatch-per-source route, though, that we could do this just with regular make rules.

-Peff

Quoted 2 lines · Jeff King

On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER G?bor wrote:

Quoted 7 lines

Quoted 7 lines

Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files.

The only remaining sequential part is aggregating the

Quoted 18 lines

Quoted 17 lines

suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions).

This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run

touch apply.c ; time make -j4 coccicheck

went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running

touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck

went from 56.364s to 35.883s, which is ~36% speedup.

Quoted 4 lines

I really like this direction. The slowness of coccicheck is one of the things that has prevented me from running it more frequently. And I'm a big fan of breaking steps down as much as possible into make targets. It lets make do its job (avoiding repeated work and parallelizing).

Yeah, this is great. Also, CC-ing some of the recent contributors to linux.git's coccinelle, perhaps they're interested / have comments.

Quoted 14 lines

Quoted 6 lines

All the above timings are best-of-five on a machine with 2 physical and 2 logical cores. I don't have the hardware to bring any numbers for point (3). The time needed to run 'make -j4 coccicheck' in a clean state didn't change, it's ~3m42s both with and without this patch.

On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch went from:

real 1m25.520s
user 5m41.492s
sys 0m26.776s

Quoted 5 lines

to:

real 0m24.300s
user 14m35.084s
sys 0m50.108s

Quoted 27 lines

I was surprised by the jump in CPU times. Doing "make -j1 coccicheck" with your patch results in:

real 5m34.887s
user 5m5.620s
sys 0m19.908s

so it's really the parallelizing that seems to be to blame (possibly because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8 threads in the first example).

Quoted 14 lines

- [RFC]
With this patch 'make coccicheck's output will look like this ('make' apparently doesn't iterate over semantic patches in lexicographical order):

SPATCH commit.cocci abspath.c
SPATCH commit.cocci advice.c
<... lots of lines ...>
SPATCH array.cocci http-walker.c
SPATCH array.cocci remote-curl.c

which means that the number of lines in the output grows from "Nr-of-semantic-patches" to "Nr-of-semantic-patches * Nr-of-source-files".

Quoted 3 lines

IMHO this is not really that big a deal. We are doing useful work for each line, so to me it's just more eye candy (and it's really satisfying to watch it zip by on the 40-core machine ;) ).

FWIW on the 8-cpu box I usually test on this went from 2m30s to 1m50s, so not a huge improvement in time, but nice to have the per-file progress.

Quoted 13 lines

What if we inverted the current loop? That is, right now we iterate over the cocci patches at the Makefile level, and then for each target we iterate over the giant list of source files. Instead, we could teach the Makefile to iterate over the source files, with a target for each, and then hit each cocci patch inside there.

That would give roughly the same output as a normal build. But moreover, I wonder if we could make things faster by actually combining the cocci files into a single set of rules to be applied to each source file. That would mean invoking spatch 1/8 as much. It does give fewer opportunities for "make" to reuse work, but that only matters when the cocci files change (which is much less frequent than source files changing).

Quoted 4 lines

Doing:

cat contrib/coccinelle/*.cocci >mega.cocci make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

Quoted 18 lines

gives me:

real 0m17.573s
user 10m56.724s
sys 0m11.548s

And that should show an improvement on more normal-sized systems, too, because we really are eliminating some of the startup overhead.

The other obvious downside is that you don't get individual patches for each class of transformation. Do we care? Certainly for a routine "make coccicheck" I primarily want to know:

- is there something that needs fixing?

- give me the patch for all fixes

Quoted 8 lines

So I wonder if we'd want to have that be the default, and then perhaps optionally give some targets to let people run single scripts (or not; they could probably just run spatch themselves).

Quoted 4 lines

- [RFC]
The overhead of applying a semantic patch to all source files became larger. 'make coccicheck' currently runs only one shell process

and creates two output files for each semantic patch.

Quoted 26 lines

Quoted 3 lines

With this patch it will run approx. "Nr-of-semantic-patches * Nr-of-source-files" shell processes and create twice as many output files.

Doing the "one big .cocci" would help with this, too (and again puts us in the same ballpark as a compile).

Quoted 4 lines

- [RFC]
This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated.

I looked into this a long time ago for another set of Makefile patches I was considering. $(eval) was added to GNU make in 3.80, released in 2002. Which is probably fine by now.

If it isn't, one more exotic option would be to push the coccicheck stuff into its own Makefile, and just run it via recursive make. Then anybody doing a vanilla build can do so even with an antique make, but you could only "make coccicheck" with something from the last 16 years (but good luck getting ocaml running there).

I suspect if we go with the one-spatch-per-source route, though, that we could do this just with regular make rules.

-Peff

This was already possible. Make coccicheck is not supposed to be used with -j, but rather with J=n. That tells Coccinelle to parallelize the treatment of the files internally. In this case, the semantic patch is only parsed once, and then n worker processes are forked to treat the different files.

Quoted 4 lines · Jeff King

Quoted 3 lines

cat contrib/coccinelle/*.cocci >mega.cocci make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

There was already a COCCI=foo.cocci argument to focus on a single semantic patch.

I'm surprised that the above cat command would work. Semantic patch rules have names, and Coccinelle will not be happy isf two rules have the same name.

Some may also have variables declared in initializers, although
perhaps the ones in the kernel don't do this. Causing these variables to be shared would not have a good effect.

Hi,

Quoted 5 lines · Julia Lawall

This was already possible. Make coccicheck is not supposed to be used with -j, but rather with J=n. That tells Coccinelle to parallelize the treatment of the files internally. In this case, the semantic patch is only parsed once, and then n worker processes are forked to treat the different files.

Thanks for this hint.

I wonder if we can make this happen automatically under suitable conditions. We can parse -j<num> out of MAKEFLAGS and convert it into a J=<num> argument,

but that only solves half the problem: the "make"
parent process would think that the coccinelle run only deserves to occupy one job slot.

Technically we could do all this using a wrapper that pretends to be a submake <https://www.gnu.org/software/make/manual/html_node/Job-Slots.html>
(prefixing the command with '+', parsing jobserver options from the MAKEFLAGS envvar) but it gets ugly.

It's likely that the best we can do is just to advertise J more prominently.

[...]

Quoted 11 lines

Quoted 4 lines · Jeff King

Quoted 3 lines

cat contrib/coccinelle/*.cocci >mega.cocci make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

There was already a COCCI=foo.cocci argument to focus on a single semantic patch.

I'm surprised that the above cat command would work. Semantic patch rules have names, and Coccinelle will not be happy isf two rules have the same name.

Yes, Git's semantic patches (in contrib/coccinelle) tend to be relatively undemanding.

Quoted 3 lines

Some may also have variables declared in initializers, although
perhaps the ones in the kernel don't do this. Causing these variables to be shared would not have a good effect.

... oh! You're thinking of the Linux kernel.

It looks like Linux's scripts/coccicheck has a lot we can crib from. That's where the J envvar and automatic parallelism you mentioned are implemented, too.

So it sounds to me like at a minimum we should use all of that. ;-)