range-diff: show submodule changes irrespective of diff.submodule

From: Philippe Blain <[email protected]>

After generating diffs for each range to be compared using a 'git log' invocation, range-diff.c::read_patches looks for the "diff --git" header in those diffs to recognize the beginning of a new change.

In a project with submodules, and with 'diff.submodule=log' set in the config, this header is missing for the diff of a changed submodule, so any submodule changes are quietly ignored in the range-diff.

When 'diff.submodule=diff' is set in the config, the "diff --git" header is also missing for the submodule itself, but is shown for submodule content changes, which can easily confuse 'git range-diff' and lead to errors such as:

error: git apply: bad git-diff - inconsistent old filename on line 1
error: could not parse git header 'diff --git path/to/submodule/and/some/file/within '
error: could not parse log for '@{u}..@{1}'

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Note that the test must use '--creation-factor=100' to force the second commit in the range not to be considered a complete rewrite.

Signed-off-by Philippe Blain <[email protected]>

---
range-diff: show submodule changes irrespective of diff.submodule

This fixes a bug that I reported last summer [1].

[1]
https://lore.kernel.org/git/[email protected]/

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1244%2Fphil-blain%2Frange-diff-submodule-diff-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1244/phil-blain/range-diff-submodule-diff-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1244

 range-diff.c          |  2 +-
 t/t3206-range-diff.sh | 44 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

range-diff.c +1 -1

index b72eb9fdbee..068bf214544 100644
@@ -44,7 +44,7 @@ static int read_patches(const char *range, struct string_list *list,
 
 	strvec_pushl(&cp.args, "log", "--no-color", "-p", "--no-merges",
 		     "--reverse", "--date-order", "--decorate=no",
-		     "--no-prefix",
+		     "--no-prefix", "--submodule=short",
 		     /*
 		      * Choose indicators that are not used anywhere
 		      * else in diffs, but still look reasonable

t/t3206-range-diff.sh +34 -0

index e30bc48a290..ac848c42536 100755
@@ -772,4 +772,48 @@ test_expect_success '--left-only/--right-only' '
 	test_cmp expect actual
 '
 
+test_expect_success 'submodule changes are shown irrespective of diff.submodule' '
+	git init sub-repo &&
+	test_commit -C sub-repo sub-first &&
+	sub_oid1=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-second &&
+	sub_oid2=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-third &&
+	sub_oid3=$(git -C sub-repo rev-parse HEAD) &&
+
+	git checkout -b main-sub topic &&
+	git submodule add ./sub-repo sub &&
+	git -C sub checkout --detach sub-first &&
+	git add sub &&
+	git commit -m "add sub" &&
+	sup_oid1=$(git rev-parse --short HEAD) &&
+	git checkout -b topic-sub &&
+	git -C sub checkout sub-second &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid2=$(git rev-parse --short HEAD) &&
+	git checkout -b modified-sub main-sub &&
+	git -C sub checkout sub-third &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid3=$(git rev-parse --short HEAD) &&
+
+	test_config diff.submodule log &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	cat >expect <<-EOF &&
+	1:  $sup_oid1 = 1:  $sup_oid1 add sub
+	2:  $sup_oid2 ! 2:  $sup_oid3 change sub
+	    @@ Commit message
+	      ## sub ##
+	     @@

Quoted 41 lines

From: Philippe Blain <[email protected]>

After generating diffs for each range to be compared using a 'git log' invocation, range-diff.c::read_patches looks for the "diff --git" header in those diffs to recognize the beginning of a new change.

In a project with submodules, and with 'diff.submodule=log' set in the config, this header is missing for the diff of a changed submodule, so any submodule changes are quietly ignored in the range-diff.

When 'diff.submodule=diff' is set in the config, the "diff --git" header is also missing for the submodule itself, but is shown for submodule content changes, which can easily confuse 'git range-diff' and lead to errors such as:

error: git apply: bad git-diff - inconsistent old filename on line 1
error: could not parse git header 'diff --git path/to/submodule/and/some/file/within '
error: could not parse log for '@{u}..@{1}'

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Note that the test must use '--creation-factor=100' to force the second commit in the range not to be considered a complete rewrite.

Signed-off-by Philippe Blain <[email protected]>

---
range-diff: show submodule changes irrespective of diff.submodule

This fixes a bug that I reported last summer [1].

[1]
https://lore.kernel.org/git/[email protected]/

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1244%2Fphil-blain%2Frange-diff-submodule-diff-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1244/phil-blain/range-diff-submodule-diff-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1244

Quoted 3 lines

 range-diff.c          |  2 +-
 t/t3206-range-diff.sh | 44 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

Thanks for picking this up again, and nice to have a test on this iteration!

Quoted 56 lines

range-diff.c +1 -1

index b72eb9fdbee..068bf214544 100644
@@ -44,7 +44,7 @@ static int read_patches(const char *range, struct string_list *list,
 
 	strvec_pushl(&cp.args, "log", "--no-color", "-p", "--no-merges",
 		     "--reverse", "--date-order", "--decorate=no",
-		     "--no-prefix",
+		     "--no-prefix", "--submodule=short",
 		     /*
 		      * Choose indicators that are not used anywhere
 		      * else in diffs, but still look reasonable

t/t3206-range-diff.sh +34 -0

index e30bc48a290..ac848c42536 100755
@@ -772,4 +772,48 @@ test_expect_success '--left-only/--right-only' '
 	test_cmp expect actual
 '
 
+test_expect_success 'submodule changes are shown irrespective of diff.submodule' '
+	git init sub-repo &&
+	test_commit -C sub-repo sub-first &&
+	sub_oid1=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-second &&
+	sub_oid2=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-third &&
+	sub_oid3=$(git -C sub-repo rev-parse HEAD) &&
+
+	git checkout -b main-sub topic &&
+	git submodule add ./sub-repo sub &&
+	git -C sub checkout --detach sub-first &&
+	git add sub &&
+	git commit -m "add sub" &&
+	sup_oid1=$(git rev-parse --short HEAD) &&
+	git checkout -b topic-sub &&
+	git -C sub checkout sub-second &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid2=$(git rev-parse --short HEAD) &&
+	git checkout -b modified-sub main-sub &&
+	git -C sub checkout sub-third &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid3=$(git rev-parse --short HEAD) &&
+
+	test_config diff.submodule log &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	cat >expect <<-EOF &&
+	1:  $sup_oid1 = 1:  $sup_oid1 add sub
+	2:  $sup_oid2 ! 2:  $sup_oid3 change sub
+	    @@ Commit message
+	      ## sub ##
+	     @@

t/t3206-range-diff.sh test_expect_success '--left-only/--right-only' ' · line 818

Quoted 10 lines

t/t3206-range-diff.sh +10 -0

+	     -Subproject commit $sub_oid1
+	    -+Subproject commit $sub_oid2
+	    ++Subproject commit $sub_oid3
+	EOF
+	test_cmp expect actual &&
+	test_config diff.submodule diff &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	test_cmp expect actual
+'
+

I'd find this much easier to follow if this were a two-part where we do most of this test code in the 1st commit, and assert the current (failing) behavior with a test_expect_failure.

Then this commit would narrowly be the bugfix itself.

I also see that the --creation-factor=100 isn't necessary and seems somewhat orthagonal, i.e. we'd like to test this *without* that option and see how we behave, i.e. we'll emit the "full replacement".

Why not compare the output without --creation-factor=100, and then just have another --creation-factor=100 test to show what we emit if we "look into" those commits and diff their contents?

From: Philippe Blain <[email protected]>

After generating diffs for each range to be compared using a 'git log' invocation, range-diff.c::read_patches looks for the "diff --git" header in those diffs to recognize the beginning of a new change.

In a project with submodules, and with 'diff.submodule=log' set in the config, this header is missing for the diff of a changed submodule, so any submodule changes are quietly ignored in the range-diff.

When 'diff.submodule=diff' is set in the config, the "diff --git" header is also missing for the submodule itself, but is shown for submodule content changes, which can easily confuse 'git range-diff' and lead to errors such as:

error: git apply: bad git-diff - inconsistent old filename on line 1
error: could not parse git header 'diff --git path/to/submodule/and/some/file/within '
error: could not parse log for '@{u}..@{1}'

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Note that the test must use '--creation-factor=100' to force the second commit in the range not to be considered a complete rewrite.

Signed-off-by Philippe Blain <[email protected]>

---
range-diff: show submodule changes irrespective of diff.submodule

This fixes a bug that I reported last summer [1].

[1]
https://lore.kernel.org/git/[email protected]/

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1244%2Fphil-blain%2Frange-diff-submodule-diff-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1244/phil-blain/range-diff-submodule-diff-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1244

 range-diff.c          |  2 +-
 t/t3206-range-diff.sh | 44 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

range-diff.c +0 -0

index b72eb9fdbee..068bf214544 100644

range-diff.c +0 -0

 		     /*
 		      * Choose indicators that are not used anywhere
 		      * else in diffs, but still look reasonable

t/t3206-range-diff.sh +4 -0

index e30bc48a290..ac848c42536 100755
@@ -772,4 +772,48 @@ test_expect_success '--left-only/--right-only' '
 	test_cmp expect actual
 '
 
+test_expect_success 'submodule changes are shown irrespective of diff.submodule' '
+	git init sub-repo &&
+	test_commit -C sub-repo sub-first &&
+	sub_oid1=$(git -C sub-repo rev-parse HEAD) &&

t/t3206-range-diff.sh +12 -0

+	git -C sub checkout sub-third &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid3=$(git rev-parse --short HEAD) &&
+
+	test_config diff.submodule log &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	cat >expect <<-EOF &&
+	1:  $sup_oid1 = 1:  $sup_oid1 add sub
+	2:  $sup_oid2 ! 2:  $sup_oid3 change sub
+	    @@ Commit message
+	      ## sub ##

t/t3206-range-diff.sh +1 -0

+
 test_done

base-commit: 7a3eb286977746bc09a5de7682df0e5a7085e17c --
gitgitgadget

Quoted 3 lines

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Full disclosure: I do not see much value in range-diffs in the presence of submodules. Nothing in the design of range-diffs is prepared for submodules.

But since --submodules=short does not change anything when running range-diff in repositories without submodules, I don't mind this change.

This test case is very clear and concise, even without my suggested changes. Therefore, if you want to keep the patch as-is, I am fine with that, too.

Acked-by Johannes Schindelin <[email protected]>

Hi Philippe,

Quoted 9 lines · Philippe Blain via GitGitGadget

From: Philippe Blain <[email protected]>

After generating diffs for each range to be compared using a 'git log' invocation, range-diff.c::read_patches looks for the "diff --git" header in those diffs to recognize the beginning of a new change.

In a project with submodules, and with 'diff.submodule=log' set in the config, this header is missing for the diff of a changed submodule, so any submodule changes are quietly ignored in the range-diff.

This means that we can go two ways here: either we explicitly disable diff.submodule for the invocation that is spawned from range-diff, or we allow it but then handle the diff header as expected.

Quoted 6 lines

When 'diff.submodule=diff' is set in the config, the "diff --git" header is also missing for the submodule itself, but is shown for submodule content changes, which can easily confuse 'git range-diff' and lead to errors such as:

Quoted 8 lines

error: git apply: bad git-diff - inconsistent old filename on line 1
error: could not parse git header 'diff --git path/to/submodule/and/some/file/within '
error: could not parse log for '@{u}..@{1}'

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Full disclosure: I do not see much value in range-diffs in the presence of submodules. Nothing in the design of range-diffs is prepared for submodules.

But since --submodules=short does not change anything when running range-diff in repositories without submodules, I don't mind this change.

Quoted 3 lines

Note that the test must use '--creation-factor=100' to force the second commit in the range not to be considered a complete rewrite.

Thank you for this considerate note!

Quoted 23 lines

Signed-off-by Philippe Blain <[email protected]>

---
range-diff: show submodule changes irrespective of diff.submodule

This fixes a bug that I reported last summer [1].

[1]
https://lore.kernel.org/git/[email protected]/

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1244%2Fphil-blain%2Frange-diff-submodule-diff-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1244/phil-blain/range-diff-submodule-diff-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1244

 range-diff.c          |  2 +-
 t/t3206-range-diff.sh | 44 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

range-diff.c +0 -0

index b72eb9fdbee..068bf214544 100644

range-diff.c static int read_patches(const char *range, struct string_list *list, · line 47

Quoted 5 lines

range-diff.c +1 -1

@@ -44,7 +44,7 @@ static int read_patches(const char *range, struct string_list *list,
 
 	strvec_pushl(&cp.args, "log", "--no-color", "-p", "--no-merges",
 		     "--reverse", "--date-order", "--decorate=no",
-		     "--no-prefix",
+		     "--no-prefix", "--submodule=short",

As I mentioned above, since this does not change anything in the intended scenarios (i.e. without submodules), I am fine with it.

Quoted 20 lines

range-diff.c +0 -0

 		     /*
 		      * Choose indicators that are not used anywhere
 		      * else in diffs, but still look reasonable

t/t3206-range-diff.sh +9 -0

index e30bc48a290..ac848c42536 100755
@@ -772,4 +772,48 @@ test_expect_success '--left-only/--right-only' '
 	test_cmp expect actual
 '
 
+test_expect_success 'submodule changes are shown irrespective of diff.submodule' '
+	git init sub-repo &&
+	test_commit -C sub-repo sub-first &&
+	sub_oid1=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-second &&
+	sub_oid2=$(git -C sub-repo rev-parse HEAD) &&
+	test_commit -C sub-repo sub-third &&
+	sub_oid3=$(git -C sub-repo rev-parse HEAD) &&
+

t/t3206-range-diff.sh test_expect_success '--left-only/--right-only' ' · line 788

Quoted 5 lines

t/t3206-range-diff.sh +5 -0

+	git checkout -b main-sub topic &&
+	git submodule add ./sub-repo sub &&
+	git -C sub checkout --detach sub-first &&
+	git add sub &&
+	git commit -m "add sub" &&

Just a suggestion: use git commit -m sub-first sub instead (one git invocation instead of two).

t/t3206-range-diff.sh test_expect_success '--left-only/--right-only' ' · line 795

Quoted 8 lines

t/t3206-range-diff.sh +7 -0

+	sup_oid1=$(git rev-parse --short HEAD) &&
+	git checkout -b topic-sub &&
+	git -C sub checkout sub-second &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid2=$(git rev-parse --short HEAD) &&
+	git checkout -b modified-sub main-sub &&

Another suggestion: instead of naming the branches, use the sup_oid* variables directly.

Quoted 12 lines

t/t3206-range-diff.sh +12 -0

+	git -C sub checkout sub-third &&
+	git add sub &&
+	git commit -m "change sub" &&
+	sup_oid3=$(git rev-parse --short HEAD) &&
+
+	test_config diff.submodule log &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	cat >expect <<-EOF &&
+	1:  $sup_oid1 = 1:  $sup_oid1 add sub
+	2:  $sup_oid2 ! 2:  $sup_oid3 change sub
+	    @@ Commit message
+	      ## sub ##

t/t3206-range-diff.sh test_expect_success '--left-only/--right-only' ' · line 817

Quoted 10 lines

t/t3206-range-diff.sh +10 -0

+	     @@
+	     -Subproject commit $sub_oid1
+	    -+Subproject commit $sub_oid2
+	    ++Subproject commit $sub_oid3
+	EOF
+	test_cmp expect actual &&
+	test_config diff.submodule diff &&
+	git range-diff --creation-factor=100 topic topic-sub modified-sub >actual &&
+	test_cmp expect actual
+'

This test case is very clear and concise, even without my suggested changes. Therefore, if you want to keep the patch as-is, I am fine with that, too.

Acked-by Johannes Schindelin <[email protected]>

Thank you,

Quoted 10 lines · Johannes Schindelin

Quoted 3 lines

Force the submodule diff format to its default ("short") when invoking 'git log' to generate the patches for each range, such that submodule changes are always shown.

Full disclosure: I do not see much value in range-diffs in the presence of submodules. Nothing in the design of range-diffs is prepared for submodules.

But since --submodules=short does not change anything when running range-diff in repositories without submodules, I don't mind this change.

IOW, "I wrote it for the purpose of doing X, I do not care those who have been using it for doing Y, I am OK with changing behaviour on them".

Philippe, do you have a good guess on other users and workflows that may benefit from the current behaviour? I suspect in the longer term this might have to become configurable, and I am having a hard time judging if (1) a temporary regression (to them) is acceptable or (2) the new feature to also show submodule changes is not urgent enough that it may be better to make it configurable from day one, instead of using a different hardcoded and only setting like this patch does.

Just to be clear: the "out of the box" behaviour (i.e. nothing in the config) is correct, i.e. submodule changes are detected and shown by 'git range-diff'.

It's only if someone has 'diff.submodule={log,diff}' in their config that submodule changes are quietly ignored (log) or might crash 'git range-diff' (diff). So I do not think of any user or workflow that benefit from the current behaviour, no. If you have diff.submodule={log,diff} set in your config, it's most probably because you work in projects that involve submodules and you do care about submodule changes. So having these changes "hidden" by range-diff (or having range-diff crash) just because the output format of 'git -c diff.submodule={log,diff} log' does not use a 'diff --git' header for submodules is really not expected. So I do not think we need to make that configurable. I think hardcoding '--submodule=short' is an easy fix and a good first step in making 'git range-diff' more useful for submodule users.