Skip to contentSkip to conversation

promisor-remote: fix xcalloc() argument order

PATCH Superseded SZEDER Gábor 4 participants 14 messages Aug 22, 2022 – Aug 25, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 promisor-remote: fix xcalloc() argument order

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with 'contrib/coccinelle/xcalloc.cocci' and Coccinelle v1.0.7 or later (previous Coccinelle versions don't notice this).

---
 promisor-remote.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
promisor-remote.c +1 -1
index 5b33f88bca..68f46f5ec7 100644
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct repository *r)
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;
 
 	repo_config(r, promisor_remote_config, config);
Signature · 1 line
2.37.2.880.g3b8fb2ce68
#1 lore raw
Quoted 5 lines · SZEDER Gábor

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with 'contrib/coccinelle/xcalloc.cocci' and Coccinelle v1.0.7 or later (previous Coccinelle versions don't notice this).

One thing is that Coccinelle is way too slow on our codebase, compared to the usual compilation, to run every time we make changes. Combined with the fact that our codebase is mostly clean, the cycles feel huge waste of time only to find something small like what this patch fixes.

That sadly discourages us from doing "make coccicheck" more often as we should.

I _think_ Googlers have 1.1.1 on their linux boxes, so even if our GitHub Actions CI is fixed to Ubuntu 18.04 and does not run more recent Coccinelle, we theoretically should have been able to catch it before it hit the public list, if "1.0.7 or later" was the condition. FWIW, "make coccicheck" with what I happen to have notices it.

$ spatch version
spatch version 1.1.1 compiled with OCaml version 4.13.1
Flags passed to the configure script: --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --enable-ocaml --enable-python --enable-opt
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expressions: PCRE

Anyway, the patch is correct.

Thanks, will queue.

Quoted 18 lines
---
 promisor-remote.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
promisor-remote.c +1 -1
index 5b33f88bca..68f46f5ec7 100644
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct repository *r)
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;
 
 	repo_config(r, promisor_remote_config, config);
#2 lore raw
Quoted 2 lines · Junio C Hamano

... FWIW, "make coccicheck" with what I happen to have notices it.

Oops, that was a serious typo. "notices" -> "fails to notice".

Quoted 29 lines
$ spatch version
spatch version 1.1.1 compiled with OCaml version 4.13.1
Flags passed to the configure script: --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --enable-ocaml --enable-python --enable-opt
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expressions: PCRE

Anyway, the patch is correct.

Thanks, will queue.

Quoted 18 lines
---
 promisor-remote.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
promisor-remote.c +1 -1
index 5b33f88bca..68f46f5ec7 100644
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct repository *r)
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;
 
 	repo_config(r, promisor_remote_config, config);
#3 lore raw
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Junio C Hamano

... FWIW, "make coccicheck" with what I happen to have notices it.

Oops, that was a serious typo. "notices" -> "fails to notice".

Hm, that's strange. 1.1.1 did notice this transformation for me, as did 1.1.0, 1.0.8, 1.0.7, and, contrary to what I wrote in the commit message, even 1.0.6 as well. (IIRC Ubuntu 18.04 has 1.0.4, which I didn't try, but if that could notice it, then we should have heard about it from CI.)

$ spatch --version
spatch version 1.1.1 compiled with OCaml version 4.08.1
Flags passed to the configure script: [none]
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expressions: PCRE
Quoted 6 lines
Quoted 6 lines
$ spatch version
spatch version 1.1.1 compiled with OCaml version 4.13.1
Flags passed to the configure script: --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --enable-ocaml --enable-python --enable-opt
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expressions: PCRE
#4 lore raw
Quoted 9 lines · SZEDER Gábor
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Junio C Hamano

... FWIW, "make coccicheck" with what I happen to have notices it.

Oops, that was a serious typo. "notices" -> "fails to notice".

Hm, that's strange. 1.1.1 did notice this transformation for me,

Hang on, it's not quite that simple.
Watch this, it will get weird:

# Batched spatch for faster processing, see 960154b9c1 (coccicheck:

# optionally batch spatch invocations, 2019-05-06)
$ grep SPATCH_BATCH_SIZE config.mak
SPATCH_BATCH_SIZE = 32
$ time make contrib/coccinelle/xcalloc.cocci.patch    
    SPATCH contrib/coccinelle/xcalloc.cocci
     SPATCH result: contrib/coccinelle/xcalloc.cocci.patch

real 0m35.902s
user 0m34.840s
sys 0m1.030s
# Found it, good.

$ make -s cocciclean
# Turn off batched processing (this is the default)
$ sed -i -e '/SPATCH_BATCH_SIZE/ s/= .*/= 1/' config.mak
$ time make contrib/coccinelle/xcalloc.cocci.patch    
    SPATCH contrib/coccinelle/xcalloc.cocci

real 0m24.553s
user 0m21.468s
sys 0m3.099s
# Not only missed the transformation, but it's faster than batched # processing?!

# Let's invoke spatch directly.

$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c                               
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c
# Nope.
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c lockfile.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c lockfile.c
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c git.c     
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c git.c
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c usage.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c usage.c
# Nope, nope, nope.
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c builtin/*.c
[...]
# Nope!

# But watch this!

$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c config.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c config.c
diff = 
diff -u -p a/promisor-remote.c b/promisor-remote.c
--- a/promisor-remote.c
+++ b/promisor-remote.c
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;

repo_config(r, promisor_remote_config, config);

# Huh?! (;

FWIW, I see this with Coccinelle 1.1.1, 1.0.8 and 1.0.6 as well.

#5 lore raw
Quoted 54 lines · SZEDER Gábor
Quoted 9 lines · SZEDER Gábor
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Junio C Hamano

... FWIW, "make coccicheck" with what I happen to have notices it.

Oops, that was a serious typo. "notices" -> "fails to notice".

Hm, that's strange. 1.1.1 did notice this transformation for me,

# Let's invoke spatch directly.

$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c                               
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c
# Nope.
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c lockfile.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c lockfile.c
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c git.c     
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c git.c
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c usage.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c usage.c
# Nope, nope, nope.
$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c builtin/*.c
[...]
# Nope!

# But watch this!

$ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c config.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c config.c
diff = 
diff -u -p a/promisor-remote.c b/promisor-remote.c
promisor-remote.c +1 -1
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;
 
 	repo_config(r, promisor_remote_config, config);
Quoted 3 lines

# Huh?! (;

FWIW, I see this with Coccinelle 1.1.1, 1.0.8 and 1.0.6 as well.

Ok, I think I got this.

The transformation involves 'r->promisor_remote_config', where 'r' is a 'struct repository*'. However, 'promisor-remote.c' doesn't include 'repository.h' directly:

  $ grep '^#include' promisor-remote.c 
  #include "cache.h"
  #include "object-store.h"
  #include "promisor-remote.h"
  #include "config.h"
  #include "transport.h"
  #include "strvec.h"

If I add an '#include "repository.h" to 'promisor-remote.c', then Coccinelle finds the transformation even when it processes this source file on is own:

  $ spatch --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c
  init_defs_builtins: /usr/local/lib/coccinelle/standard.h
  HANDLING: promisor-remote.c
  diff = 
  diff -u -p a/promisor-remote.c b/promisor-remote.c
  --- a/promisor-remote.c
  +++ b/promisor-remote.c
  @@ -149,7 +149,7 @@ static void promisor_remote_init(struct
   	if (r->promisor_remote_config)
   		return;
   	config = r->promisor_remote_config =
  -		xcalloc(sizeof(*r->promisor_remote_config), 1);
  +		xcalloc(1, sizeof(*r->promisor_remote_config));
   	config->promisors_tail = &config->promisors;
   	repo_config(r, promisor_remote_config, config);
#6 lore raw

PATCH v2 1/1 promisor-remote: fix xcalloc() argument order

In reply to SZEDER Gábor

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Our default SPATCH_FLAGS ('--all-includes') doesn't catch this transformation by default, unless used in combination with a large-ish SPATCH_BATCH_SIZE which happens to put 'promisor-remote.c' with a file that includes 'repository.h' directly in the same batch.

---
 promisor-remote.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
promisor-remote.c +1 -1
index 5b33f88bca..68f46f5ec7 100644
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct repository *r)
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;
 
 	repo_config(r, promisor_remote_config, config);
Signature · 1 line
2.37.2.880.g3b8fb2ce68
#7 lore raw
Quoted 6 lines · SZEDER Gábor

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Thanks for digging here. I think it probably explains a lot of "wait, why did this result change" head-scratching I did back when we started batching a few years ago.

Is there any reason we wouldn't want --recursive-includes to be added to the default SPATCH_FLAGS?

I suspect we'd still want to leave --all-includes there to get as much type information as possible. If I understand correctly, the two are orthogonal (one is "follow includes of includes" and the other is "follow system includes in angle brackets").

Doing so doesn't seem to find any other missed entries in the current codebase, but I'm pretty sure there are some it would have caught in a less mysterious fashion over the years.

-Peff

#8 lore raw
Quoted 7 lines · Jeff King
Quoted 5 lines · SZEDER Gábor

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with:

Quoted 8 lines
Quoted 1 line

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Thanks for digging here. I think it probably explains a lot of "wait, why did this result change" head-scratching I did back when we started batching a few years ago.

Is there any reason we wouldn't want --recursive-includes to be added to the default SPATCH_FLAGS?

I ran some runtime measurements yesterday, and there is a considerable runtime increase in the unbatched case:

    The tables below show the runtimes of applying each of our semantic
    patches separately with Coccinelle v1.1.1, with the '--all-includes'
    or '--recursive-includes' flags and with batch sizes 1 (no batching)
    or 32, i.e.:
    
      time make SPATCH_FLAGS=$flag SPATCH_BATCH_SIZE=$size $cocci.patch
    
    SPATCH_BATCH_SIZE=1:
    
       semantic patch  |    all    | recursive |
      -----------------+-----------+-----------+--------
       array           |   69.90s  |  140.12s  |  +100%
       commit          |   94.44s  |  223.63s  |  +137%
       equals-null     |   86.93s  |  205.40s  |  +136%
       flex_alloc      |   11.32s  |   16.45s  |   +45%
       free            |   70.47s  |  159.75s  |  +127%
       hashmap         |   83.48s  |  199.70s  |  +139%
       object_id       |  107.83s  |  241.69s  |  +124%
       preincr         |   79.33s  |  202.98s  |  +156%
       qsort           |   16.20s  |   33.86s  |  +109%
       strbuf          |   60.54s  |  129.93s  |  +115%
       swap            |   81.70s  |  200.75s  |  +146%
       unused          |  499.19s  |  626.35s  |   +25%
       xcalloc         |   26.71s  |   57.63s  |  +116%
       xopen           |   30.92s  |   59.26s  |   +92%
       xstrdup_or_null |    5.05s  |    6.94s  |   +37%
      -----------------+-----------+-----------+--------
       all             |    1324s  |    2504s  |   +89%
    
    SPATCH_BATCH_SIZE=32:
    
       semantic patch  |    all    | recursive |
      -----------------+-----------+-----------+--------
       array           |   43.81s  |   52.83s  |   +21%
       commit          |   50.16s  |   52.76s  |    +5%
       equals-null     |   47.77s  |   50.86s  |    +6%
       flex_alloc      |   41.00s  |   43.64s  |    +6%
       free            |   43.12s  |   46.68s  |    +8%
       hashmap         |   42.76s  |   46.12s  |    +8%
       object_id       |   56.17s  |   60.00s  |    +7%
       preincr         |   39.82s  |   42.57s  |    +7%
       qsort           |   39.48s  |   53.09s  |   +34%
       strbuf          |   51.27s  |   49.38s  |    -4%
       swap            |   41.93s  |   58.17s  |   +39%
       unused          |  440.86s  |  445.47s  |    +1%
       xcalloc         |   39.90s  |   42.22s  |    +6%
       xopen           |   40.26s  |   43.19s  |    +7%
       xstrdup_or_null |   39.14s  |   41.72s  |    +7%
      -----------------+-----------+-----------+--------
       all             |    1057s  |    1129s  |    +7%
    
    I don't have meaningful numbers about the impact of
    '--recursive-includes' on the runtime of a parallel 'make -j<N>
    coccicheck', because they're severely skewed by 'unused.cocci' and
    'make's scheduler: applying 'unused.cocci' takes 5-10x as long as
    other semantic patches (accounting for about 38-42% of the total
    sequential runtime), and 'make' on my system usually schedules it near
    the end, meaning that for a significant part there is no parallel
    processing at all.

So I'm not sure about making '--recursive-includes' the default, at least as long as we don't batch by default. However, we could still use this flag in CI...

Quoted 4 lines

I suspect we'd still want to leave --all-includes there to get as much type information as possible. If I understand correctly, the two are orthogonal (one is "follow includes of includes" and the other is "follow system includes in angle brackets").

'spatch --help' tells me:

--recursive-includes causes all available include files, both
those included in the C file(s) and those included in header files, to be used
--all-includes causes all available include files included
in the C file(s) to be used

So I think the difference is not about system includes, but rather about "consider includes of includes" vs. "consider only direct includes in the C files", so it seems '--recursive-includes' is a superset of '--all-includes'. Oh, and let's not forget the rather surprising behavior that if spatch processes multiple C files at once, then for each of the C files it considers all includes from all C files; this explains why '--all-includes promisor-remote.c config.c' works, because 'config.c' does include 'repository.h', supplying the necessary type information to catch the previously missed transformation in 'promisor-remote.c'.

These descriptions don't make it clear (to me) whether the two options are orthogonal, exclusive, or something else. However, trying out various combinations indicates that "last one wins", e.g.:

$ spatch --recursive-includes --all-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c
$ spatch --all-includes --recursive-includes --sp-file contrib/coccinelle/xcalloc.cocci --patch . promisor-remote.c
warning: Can't find macro file: /usr/local/bin/lib/coccinelle/standard.h
warning: Can't find default iso file: /usr/local/bin/lib/coccinelle/standard.iso
HANDLING: promisor-remote.c
diff = 
diff -u -p a/promisor-remote.c b/promisor-remote.c
--- a/promisor-remote.c
+++ b/promisor-remote.c
@@ -146,7 +146,7 @@ static void promisor_remote_init(struct
 	if (r->promisor_remote_config)
 		return;
 	config = r->promisor_remote_config =
-		xcalloc(sizeof(*r->promisor_remote_config), 1);
+		xcalloc(1, sizeof(*r->promisor_remote_config));
 	config->promisors_tail = &config->promisors;

repo_config(r, promisor_remote_config, config);

#9 lore raw
Quoted 2 lines · SZEDER Gábor

I ran some runtime measurements yesterday, and there is a considerable runtime increase in the unbatched case:

Let me condense and rearrange this a bit to the part I think is important:

Quoted 6 lines
       semantic patch  |    all    | recursive |
      -----------------+-----------+-----------+--------
    SPATCH_BATCH_SIZE=1:
       all             |    1324s  |    2504s  |   +89%
    SPATCH_BATCH_SIZE=32:
       all             |    1057s  |    1129s  |    +7%

Right, that's not surprising. "recursive" means we're looking at more header files than before. For the batch case it has less impact because it's more likely that some other file we're processing in the batch _also_ was looking at that header, so we saw it anyway.

With an infinite bath size, adding --recursive-includes should make no difference (or almost none; it's possible there are headers which are _only_ included recursively across the whole code base).

My numbers for array.cocci are:

SPATCH_BATCH_SIZE=1, 133s
SPATCH_BATCH_SIZE=32, 34s
SPATCH_BATCH_SIZE=0, 31s

Of course there we run into potential memory problems. Based on glancing at top occasionally, the first one peaked around 100MB of memory, the second around 200MB, and the third one around 2GB. So "32" is probably still a reasonable middle ground for CPU vs memory.

Spending extra CPU, especially for people stuck BATCH_SIZE=1, does suck. But the tool is already terribly slow. Even if we spend twice as much CPU (which is what your numbers above show), isn't it worth it to get the _correct_ result? To me there is not much point in running it at all if it misses cases.

It would be nice if there was some way to tell coccinelle not to actually _analyze_ included header files, but just to include them to pick up types, etc (and then we'd add the headers themselves to COCCI_SOURCES to check them). But I don't see a way to do that. And I wouldn't be surprised if the code simply isn't written in a way to make that easy internally.

Another alternative would be --no-includes (and again, listing the headers manually via COCCI_SOURCES). But as the example you found shows, it really does need the headers in order to properly process the code in the C files themselves.

Quoted 8 lines

I don't have meaningful numbers about the impact of '--recursive-includes' on the runtime of a parallel 'make -j<N> coccicheck', because they're severely skewed by 'unused.cocci' and 'make's scheduler: applying 'unused.cocci' takes 5-10x as long as other semantic patches (accounting for about 38-42% of the total sequential runtime), and 'make' on my system usually schedules it near the end, meaning that for a significant part there is no parallel processing at all.

It may be that unused.cocci can be optimized. Once upon a time I spent some effort optimizing some of the rules files. I found that if you relied on having python support, it helped quite a bit, because without it you are stuck with the "..." operator, which can be very expensive. More details in this thread:

https://lore.kernel.org/git/[email protected]/

Back then the debian package wasn't being actively updated, and it wasn't clear if python support would remain. But these days it seems a bit more stable. And with the CI job available, I'd prefer to optimize over making the tool pleasant to use for people who do use it, rather than worrying that everybody on every platform/build can use it.

All that said, I am not about to get sucked into the rabbit hole of coccinelle optimization anytime soon. So good luck to you, if you're interested. ;)

Quoted 13 lines
Quoted 4 lines

I suspect we'd still want to leave --all-includes there to get as much type information as possible. If I understand correctly, the two are orthogonal (one is "follow includes of includes" and the other is "follow system includes in angle brackets").

'spatch --help' tells me:

--recursive-includes causes all available include files, both
those included in the C file(s) and those included in header files, to be used
--all-includes causes all available include files included
in the C file(s) to be used

Quoted 4 lines

So I think the difference is not about system includes, but rather about "consider includes of includes" vs. "consider only direct includes in the C files", so it seems '--recursive-includes' is a superset of '--all-includes'.

I'm not sure, because there's also --local-includes and --no-includes, which are grouped with --all-includes. (The grouping is doubly confusing because my copy of the manpage has _two_ options sections, and only some of the options are mentioned together in the first one).

Quoted 6 lines

Oh, and let's not forget the rather surprising behavior that if spatch processes multiple C files at once, then for each of the C files it considers all includes from all C files; this explains why '--all-includes promisor-remote.c config.c' works, because 'config.c' does include 'repository.h', supplying the necessary type information to catch the previously missed transformation in 'promisor-remote.c'.

Right. I have a feeling it is sucking in as much data as possible to a global symbol table, and then doing the analysis. At least that's how I'd implement it. ;)

Quoted 3 lines

These descriptions don't make it clear (to me) whether the two options are orthogonal, exclusive, or something else. However, trying out various combinations indicates that "last one wins", e.g.:

OK, that's compelling evidence. In that case it wouldn't hurt to say:

  --all-includes --recursive-includes

versus just

--recursive-includes

but it also probably isn't doing anything. ;)

-Peff

#10 lore raw
In reply to Jeff King
Quoted 16 lines · Jeff King
Quoted 6 lines · SZEDER Gábor

Pass the number of elements first and their size second, as expected by xcalloc().

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Thanks for digging here. I think it probably explains a lot of "wait, why did this result change" head-scratching I did back when we started batching a few years ago.

Is there any reason we wouldn't want --recursive-includes to be added to the default SPATCH_FLAGS?

Quoted 8 lines

I suspect we'd still want to leave --all-includes there to get as much type information as possible. If I understand correctly, the two are orthogonal (one is "follow includes of includes" and the other is "follow system includes in angle brackets").

Doing so doesn't seem to find any other missed entries in the current codebase, but I'm pretty sure there are some it would have caught in a less mysterious fashion over the years.

This feels to me like hacks around other issues we should fix the root cause of.

So, first of all, I think this is a perfectly good fix, and something we should do more of in general. It'll apply the wanted change *and* speed up the run:

	diff --git a/contrib/coccinelle/xcalloc.cocci b/contrib/coccinelle/xcalloc.cocci
	index c291011607e..bd51e33af83 100644
	--- a/contrib/coccinelle/xcalloc.cocci
	+++ b/contrib/coccinelle/xcalloc.cocci
	@@ -1,10 +1,8 @@
	 @@
	-type T;
	-T *ptr;
	 expression n;
	 @@
	   xcalloc(
	 + n,
	-  \( sizeof(T) \| sizeof(*ptr) \)
	+  sizeof(...)
	 - , n
	   )

But it's *not* functionally the same thing, pedantically speaking, but I think it's fine. I pointed this out at the the time in [1].

Also, more generally we could avoid includes in headers, and use forward decls. This would make e.g. the "iwyu" tool report the missing include.

Unfortunately it seems we added it in this case due to "the_repository". I wonder if this hack would be better:

	diff --git a/Makefile b/Makefile
	index 9410a587fc0..92d5726b392 100644
	--- a/Makefile
	+++ b/Makefile
	@@ -3119,6 +3119,7 @@ HCC = $(HCO:hco=hcc)
	 
	 %.hcc: %.h
	 	@echo '#include "git-compat-util.h"' >$@
	+	@echo "extern struct repository *the_repository;" >>$@
	 	@echo '#include "$<"' >>$@
	 
	 $(HCO): %.hco: %.hcc FORCE
	diff --git a/promisor-remote.h b/promisor-remote.h
	index edc45ab0f5f..e864c093a44 100644
	--- a/promisor-remote.h
	+++ b/promisor-remote.h
	@@ -1,7 +1,7 @@
	 #ifndef PROMISOR_REMOTE_H
	 #define PROMISOR_REMOTE_H
	 
	-#include "repository.h"
	+struct repository;
	 
	 struct object_id;
	 
	@@ -23,6 +23,7 @@ static inline void promisor_remote_reinit(void)
	 	repo_promisor_remote_reinit(the_repository);
	 }
	 
	+struct promisor_remote_config;
	 void promisor_remote_clear(struct promisor_remote_config *config);
	 
	 struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);

So yeah, one way to deal with this is --recursive-includes, but I think we're better off heading in the direction of having the includes for a given *.c file be what it actually needs, and not rely on includes by proxy, or in this case to pollute the namespace of everyone using promisor-remote.h with remote.h (probably doesn't matter in that case, but in the general case...).

1. https://lore.kernel.org/git/[email protected]/

#11 lore raw
In reply to SZEDER Gábor
Quoted 8 lines · SZEDER Gábor

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Our default SPATCH_FLAGS ('--all-includes') doesn't catch this transformation by default, unless used in combination with a large-ish SPATCH_BATCH_SIZE which happens to put 'promisor-remote.c' with a file that includes 'repository.h' directly in the same batch.

Our default SPATCH_FLAGS is actually

SPATCH_FLAGS = --all-includes --patch .

and I am wondering how "--patch ." part (or droppage thereof due to overriding it from the command line) affects the outcome.

In any case, good finding how the header file inclusion was affected with batch-size and this option that solved a mystery. Thanks.

#12 lore raw
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · SZEDER Gábor

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Our default SPATCH_FLAGS ('--all-includes') doesn't catch this transformation by default, unless used in combination with a large-ish SPATCH_BATCH_SIZE which happens to put 'promisor-remote.c' with a file that includes 'repository.h' directly in the same batch.

Quoted 6 lines

Our default SPATCH_FLAGS is actually

SPATCH_FLAGS = --all-includes --patch .

and I am wondering how "--patch ." part (or droppage thereof due to overriding it from the command line) affects the outcome.

'--patch .' is not part of SPATCH_FLAGS anymore, and for a good reason, see the recent 7b63ea5750 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05).

#13 lore raw
Quoted 14 lines · SZEDER Gábor
Quoted 13 lines · Junio C Hamano
Quoted 8 lines · SZEDER Gábor

Patch generated with:

make SPATCH_FLAGS=--recursive-includes contrib/coccinelle/xcalloc.cocci.patch

Our default SPATCH_FLAGS ('--all-includes') doesn't catch this transformation by default, unless used in combination with a large-ish SPATCH_BATCH_SIZE which happens to put 'promisor-remote.c' with a file that includes 'repository.h' directly in the same batch.

Our default SPATCH_FLAGS is actually

Quoted 8 lines
Quoted 4 lines

SPATCH_FLAGS = --all-includes --patch .

and I am wondering how "--patch ." part (or droppage thereof due to overriding it from the command line) affects the outcome.

'--patch .' is not part of SPATCH_FLAGS anymore, and for a good reason, see the recent 7b63ea5750 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05).

Ahh, that makes my life a bit more cumbersome, as I was looking at how we did things on the 'maint' branch.

Thanks, mystery solved.