scripts/coccicheck: add paramap and indexing options

Also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it.

Quoted 2 lines

Also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it.

Is this functionality influenced by the parameter "chunksize"?

Glimpse is a tool you can use to index the kernel. The tool was recently open sourced under the ISC license and can be obtained at:

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Signed-off-by Luis R. Rodriguez <[email protected]>

---
 scripts/coccicheck | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

scripts/coccicheck +24 -0

index eeb5fdc142ca..f31c9a152559 100755
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then

@@ -15,6 +17,20 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# 0. --use-glimpse currently outperforms all. Refer
+#    to scripts/glimpse.sh for details.
+# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
+# 2. Use --use-coccigrep if no indexing options are available and your
+#    version of coccinelle supports it
+USE_GLIMPSE="no"
+$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+USE_GITGREP="no"
+$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+
+USE_COCCIGREP="no"
+$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 

@@ -89,6 +105,14 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+if [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+elif [ "$USE_COCCIGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-coccigrep"
+fi
+
 run_cmd_paramap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"

Signature · 1 line

2.8.2

Quoted 43 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

Quoted 3 lines

real    16m14.794s
user    128m42.356s
sys     0m1.880s

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

I'm puzzled why glimpse gives no benefit.

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Quoted 4 lines

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ?

Quoted 1 line

I'm puzzled why glimpse gives no benefit.

Well, slightly better.

Quoted 1 line

Well, slightly better.

No, it should be much better. You would have to look at the standard

Quoted 3 lines

Quoted 1 line

Well, slightly better.

No, it should be much better. You would have to look at the standard

I use id-utils regularly and it is indeed at least a magnitude better. The indexing often pays off already with the first coccinelle run for me. Highly recommended.

AFAICT coccinelle does not have integration support for id-utils though.

Quoted 1 line

AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

Quoted 6 lines · Wolfram Sang

Quoted 1 line

AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

Sweet, testing that now.

real    16m11.692s
user    127m50.388s
sys     0m2.168s

Quoted 4 lines

real    16m11.692s
user    127m50.388s
sys     0m2.168s

That's better but not a magnitude, I wonder.

works pretty well, and there are quite a lot of files (7514) that contain kfree.

Quoted 5 lines · Wolfram Sang

Quoted 4 lines

real    16m11.692s
user    127m50.388s
sys     0m2.168s

Quoted 1 line

That's better but not a magnitude, I wonder.

I think that it is because the filtering that Coccinelle does already works pretty well, and there are quite a lot of files (7514) that contain kfree.

Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best?

Quoted 3 lines

Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best?

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.

Quoted 4 lines · Wolfram Sang

Quoted 3 lines

Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best?

Quoted 5 lines

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.

Seems like a reasonable choice.

Quoted 8 lines · Julia Lawall

Quoted 4 lines · Wolfram Sang

Quoted 3 lines

Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best?

Quoted 7 lines

Quoted 5 lines

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.

Seems like a reasonable choice.

With this one on a 32-core system, I get:

glimpse:

real    0m6.549s
user    0m49.136s
sys     0m3.076s

idutils:

real    0m6.749s
user    0m51.936s
sys     0m3.876s

gitgrep:

real    0m6.805s
user    0m51.572s
sys     0m4.432s

coccigrep:

real    0m16.369s
user    0m58.712s
sys     0m5.064s

I redirected stderr to stdout, and verified glimpse output has:

glimpse request = request_threaded_irq

Does this match expectations?

Quoted 6 lines · Wolfram Sang

Quoted 1 line

AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

Sweet, testing that now.

It's not as efficient as glimpse because the query language is simpler.

So more filtering has to be done at the ocaml level. But it's probably fine in most cases.

Quoted 2 lines

It's not as efficient as glimpse because the query language is simpler.

Interesting, what is missing compared to glimpse?

Quoted 2 lines

So more filtering has to be done at the ocaml level. But it's probably fine in most cases.

For me, it has two advantages over glimpse:

a) it is in the debian package repository
b) the same database can be used with the code browser 'seascope' which can do nice things by feeding ctags on the fly with data from id-utils.

Mileages vary, of course, just wanted to mention it to give pointers.

Quoted 1 line

AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

Quoted 2 lines · Wolfram Sang

Quoted 1 line

AFAICT coccinelle does not have integration support for id-utils though.

Quoted 3 lines

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

Coccinelle includes a script scripts/idutils_index.sh

This does mkid -i C --output .id-utils.index *

Quoted 7 lines · Wolfram Sang

Quoted 3 lines

Quoted 1 line

Well, slightly better.

No, it should be much better. You would have to look at the standard

I use id-utils regularly and it is indeed at least a magnitude better. The indexing often pays off already with the first coccinelle run for me. Highly recommended.

AFAICT coccinelle does not have integration support for id-utils though.

Quoted 52 lines · Luis R. Rodriguez

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Quoted 7 lines

Quoted 4 lines

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

Quoted 2 lines

Quoted 1 line

I'm puzzled why glimpse gives no benefit.

Quoted 1 line

Well, slightly better.

No, it should be much better. You would have to look at the standard error to see if you are getting any benefit. There should be very few occurrences of Skipping if you are really using glimpse. In any case, if you asked for glimpse and it was not able to provide it, there should be warning messages at the top of stderr.

Quoted 70 lines · Julia Lawall

Quoted 52 lines · Luis R. Rodriguez

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Quoted 7 lines

Quoted 4 lines

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

Quoted 3 lines

Quoted 1 line

I'm puzzled why glimpse gives no benefit.

Well, slightly better.

Quoted 5 lines

No, it should be much better. You would have to look at the standard error to see if you are getting any benefit. There should be very few occurrences of Skipping if you are really using glimpse. In any case, if you asked for glimpse and it was not able to provide it, there should be warning messages at the top of stderr.

I'll redirect stderr to stdout by default when parmap support is used then.

Quoted 77 lines · Luis R. Rodriguez

Quoted 75 lines · Julia Lawall

Quoted 52 lines · Luis R. Rodriguez

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Quoted 7 lines

Quoted 4 lines

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

Quoted 3 lines

Quoted 1 line

I'm puzzled why glimpse gives no benefit.

Well, slightly better.

No, it should be much better. You would have to look at the standard error to see if you are getting any benefit. There should be very few occurrences of Skipping if you are really using glimpse. In any case, if you asked for glimpse and it was not able to provide it, there should be warning messages at the top of stderr.

Quoted 1 line

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

Quoted 81 lines · Julia Lawall

Quoted 77 lines · Luis R. Rodriguez

Quoted 75 lines · Julia Lawall

Quoted 52 lines · Luis R. Rodriguez

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Quoted 7 lines

Quoted 4 lines

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

Quoted 3 lines

Quoted 1 line

I'm puzzled why glimpse gives no benefit.

Well, slightly better.

No, it should be much better. You would have to look at the standard error to see if you are getting any benefit. There should be very few occurrences of Skipping if you are really using glimpse. In any case, if you asked for glimpse and it was not able to provide it, there should be warning messages at the top of stderr.

Quoted 3 lines

Quoted 1 line

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 7 lines · Luis R. Rodriguez

Quoted 6 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Quoted 5 lines

Quoted 1 line

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

I don't understand the connection with parmap.

Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

Quoted 17 lines · Julia Lawall

Quoted 12 lines · Luis R. Rodriguez

Quoted 7 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 1 line

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Quoted 5 lines

Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

Quoted 18 lines · Luis R. Rodriguez

Quoted 17 lines · Julia Lawall

Quoted 12 lines · Luis R. Rodriguez

Quoted 7 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 5 lines

Quoted 1 line

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

Quoted 28 lines · Julia Lawall

Quoted 23 lines · Luis R. Rodriguez

Quoted 11 lines · Julia Lawall

Quoted 7 lines · Luis R. Rodriguez

Quoted 6 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Quoted 2 lines

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

Quoted 1 line

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

$$ is the PID of the bash script.

Quoted 32 lines · Luis R. Rodriguez

Quoted 30 lines · Julia Lawall

Quoted 18 lines · Luis R. Rodriguez

Quoted 17 lines · Julia Lawall

Quoted 12 lines · Luis R. Rodriguez

Quoted 7 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 5 lines

Quoted 1 line

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

Quoted 3 lines

OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of interest when using parmap either.

I'm not sure to understan the issue about appending. Appending what to what?
While Coccinelle is running, you have a directory with the same name as your semantic patch (or a directory name that you specify with --tmp-dir) that has separate files for each core's standard output and standard error. At the end, when all the code has been treated, Coccinelle writes the successive stdouts to standard output, and the sucessive stderrs to standard error.

Quoted 14 lines

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output, specia files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

Quoted 1 line

$$ is the PID of the bash script.

OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones.

Quoted 40 lines · Julia Lawall

Quoted 35 lines · Luis R. Rodriguez

Quoted 22 lines · Julia Lawall

Quoted 18 lines · Luis R. Rodriguez

Quoted 17 lines · Julia Lawall

Quoted 12 lines · Luis R. Rodriguez

Quoted 7 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle.

Quoted 2 lines

If coccicheck is using --very-quiet, there will not be much stderr of interest when using parmap either.

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet is used ? Or does using --very-quiet suppress stderr ?

Quoted 14 lines

Quoted 14 lines

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output, specia files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

Quoted 5 lines

Quoted 1 line

$$ is the PID of the bash script.

OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones.

Which is why I would much prefer to instead just redirect in coccicheck case stderr to stdout from coccinelle. Is that preferred?

Quoted 44 lines · Luis R. Rodriguez

Quoted 42 lines · Julia Lawall

Quoted 29 lines · Luis R. Rodriguez

Quoted 28 lines · Julia Lawall

Quoted 23 lines · Luis R. Rodriguez

Quoted 11 lines · Julia Lawall

Quoted 7 lines · Luis R. Rodriguez

Quoted 6 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Quoted 6 lines

Quoted 2 lines

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of interest when using parmap either.

Quoted 2 lines

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet is used ? Or does using --very-quiet suppress stderr ?

--very-quiet suppresses most administrative messages that go to stderr. There are still actual errors. Bu you don't see eg what file is being currently processed.

Quoted 20 lines

Quoted 19 lines

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output, specia files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

Quoted 7 lines

Quoted 1 line

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

$$ is the PID of the bash script.

OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones.

Quoted 2 lines

Which is why I would much prefer to instead just redirect in coccicheck case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Why not just let the user decide what to do with these things?

Quoted 51 lines · Julia Lawall

Quoted 46 lines · Luis R. Rodriguez

Quoted 36 lines · Julia Lawall

Quoted 32 lines · Luis R. Rodriguez

Quoted 30 lines · Julia Lawall

Quoted 18 lines · Luis R. Rodriguez

Quoted 17 lines · Julia Lawall

Quoted 12 lines · Luis R. Rodriguez

Quoted 7 lines · Julia Lawall

Quoted 2 lines · Luis R. Rodriguez

I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script?

Quoted 5 lines

Quoted 1 line

I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error.

Quoted 6 lines

Quoted 3 lines

OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of interest when using parmap either.

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet is used ? Or does using --very-quiet suppress stderr ?

Quoted 3 lines

--very-quiet suppresses most administrative messages that go to stderr. There are still actual errors. Bu you don't see eg what file is being currently processed.

OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can specify extra options that the script doesn't specifically support.

Quoted 20 lines

Quoted 20 lines

Quoted 19 lines

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output, specia files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

Quoted 7 lines

Quoted 1 line

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

$$ is the PID of the bash script.

OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones.

Quoted 6 lines

Quoted 2 lines

Which is why I would much prefer to instead just redirect in coccicheck case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Why not just let the user decide what to do with these things?

Sure, what should be the default?

Quoted 3 lines · Luis R. Rodriguez

OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can specify extra options that the script doesn't specifically support.

If it is more convenient, you don't actually have to remove --very-quiet. You can just put --quiet before --show-trying or --profile. --quiet will override --very-quiet.

Quoted 25 lines

Quoted 25 lines

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

Originally our use of parmap made output, specia files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

$$ is the PID of the bash script.

OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones.

Which is why I would much prefer to instead just redirect in coccicheck case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Quoted 3 lines

Quoted 1 line

Why not just let the user decide what to do with these things?

Sure, what should be the default?

I would normally just expect standard output and standard error to appear randomly on the screen. That is, no management effort from the tool at all.

in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available.

we should be able to also take advantage of with coccicheck, the most important one is paramap support.