merge-ort: avoid assuming all renames detected

From: Elijah Newren <[email protected]>

In commit 8b09a900a1 ("merge-ort: restart merge with cached renames to reduce process entry cost", 2021-07-16), we noted that in the merge-ort steps of
collect_merge_info()
detect_and_process_renames()
process_entries()
that process_entries() was expensive, and we could often make it cheaper by changing this to
collect_merge_info()
detect_and_process_renames()
<cache all the renames, and restart> collect_merge_info()
detect_and_process_renames()
process_entries()
because the second collect_merge_info() would be cheaper (we could avoid traversing into some directories), the second detect_and_process_renames() would be free since we had already detected all renames, and then process_entries() has far fewer entries to handle.

However, this was built on the assumption that the first detect_and_process_renames() actually detected all potential renames. If someone has merge.renameLimit set to some small value, that assumption is violated which manifests later with the following message:

    $ git -c merge.renameLimit=1 rebase upstream
    ...
    git: merge-ort.c:546: clear_or_reinit_internal_opts: Assertion
    `renames->cached_pairs_valid_side == 0' failed.

Turn off this cache-renames-and-restart whenever we cannot detect all renames, and add a testcase that would have caught this problem.

Reported-by Taylor Blau <[email protected]>

Signed-off-by Elijah Newren <[email protected]>

---

merge-ort.c +3 -0

+		renames->cached_pairs_valid_side = 0;
+		renames->redo_after_renames = 0;
+	}
 	if (renames->redo_after_renames && detection_run) {
 		int i, side;
 		struct diff_filepair *p;

From: Elijah Newren <[email protected]>

In commit 8b09a900a1 ("merge-ort: restart merge with cached renames to reduce process entry cost", 2021-07-16), we noted that in the merge-ort steps of
collect_merge_info()
detect_and_process_renames()
process_entries()
that process_entries() was expensive, and we could often make it cheaper by changing this to
collect_merge_info()
detect_and_process_renames()
<cache all the renames, and restart> collect_merge_info()
detect_and_process_renames()
process_entries()
because the second collect_merge_info() would be cheaper (we could avoid traversing into some directories), the second detect_and_process_renames() would be free since we had already detected all renames, and then process_entries() has far fewer entries to handle.

However, this was built on the assumption that the first detect_and_process_renames() actually detected all potential renames. If someone has merge.renameLimit set to some small value, that assumption is violated which manifests later with the following message:

    $ git -c merge.renameLimit=1 rebase upstream
    ...
    git: merge-ort.c:546: clear_or_reinit_internal_opts: Assertion
    `renames->cached_pairs_valid_side == 0' failed.

Turn off this cache-renames-and-restart whenever we cannot detect all renames, and add a testcase that would have caught this problem.

Reported-by Taylor Blau <[email protected]>

Signed-off-by Elijah Newren <[email protected]>

---

merge-ort: avoid assuming all renames detected

Fixes https://lore.kernel.org/git/[email protected]/

Changes since v1:

* Fixed a small style issue

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1194%2Fnewren%2Favoid-assertion-assuming-renames-found-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1194/newren/avoid-assertion-assuming-renames-found-v2
Pull-Request: https://github.com/git/git/pull/1194

Range-diff vs v1:

1: f1e9901ae67 ! 1: 239d3ba08c1 merge-ort: avoid assuming all renames detected

     @@ merge-ort.c: static int detect_and_process_renames(struct merge_options *opt,
       	trace2_region_enter("merge", "regular renames", opt->repo);
       	detection_run |= detect_regular_renames(opt, MERGE_SIDE1);
       	detection_run |= detect_regular_renames(opt, MERGE_SIDE2);
     -+	if (renames->needed_limit != 0) {
     ++	if (renames->needed_limit) {
      +		renames->cached_pairs_valid_side = 0;
      +		renames->redo_after_renames = 0;
      +	}

 merge-ort.c                              |  4 ++
 t/t6429-merge-sequence-rename-caching.sh | 67 ++++++++++++++++++++++++
 2 files changed, 71 insertions(+)

merge-ort.c +4 -0

index c3197970219..b0ff9a72879 100644
@@ -3060,6 +3060,10 @@ static int detect_and_process_renames(struct merge_options *opt,
 	trace2_region_enter("merge", "regular renames", opt->repo);
 	detection_run |= detect_regular_renames(opt, MERGE_SIDE1);
 	detection_run |= detect_regular_renames(opt, MERGE_SIDE2);
+	if (renames->needed_limit) {
+		renames->cached_pairs_valid_side = 0;
+		renames->redo_after_renames = 0;
+	}
 	if (renames->redo_after_renames && detection_run) {
 		int i, side;
 		struct diff_filepair *p;

t/t6429-merge-sequence-rename-caching.sh +67 -0

index 035edc40b1e..f2bc8a7d2a2 100755
@@ -697,4 +697,71 @@ test_expect_success 'caching renames only on upstream side, part 2' '
 	)
 '
 
+#
+# The following testcase just creates two simple renames (slightly modified
+# on both sides but without conflicting changes), and a directory full of
+# files that are otherwise uninteresting.  The setup is as follows:
+#
+#   base:     unrelated/<BUNCH OF FILES>
+#             numbers
+#             values
+#   upstream: modify: numbers
+#             modify: values
+#   topic:    add: unrelated/foo
+#             modify: numbers
+#             modify: values
+#             rename: numbers -> sequence
+#             rename: values -> progression
+#
+# This is a trivial rename case, but we're curious what happens with a very
+# low renameLimit interacting with the restart optimization trying to notice
+# that unrelated/ looks like a trivial merge candidate.
+#
+test_expect_success 'avoid assuming we detected renames' '
+	git init redo-weirdness &&
+	(
+		cd redo-weirdness &&
+
+		mkdir unrelated &&
+		for i in $(test_seq 1 10)
+		do
+			>unrelated/$i
+		done &&
+		test_seq  2 10 >numbers &&
+		test_seq 12 20 >values &&
+		git add numbers values unrelated/ &&
+		git commit -m orig &&
+
+		git branch upstream &&
+		git branch topic &&
+
+		git switch upstream &&
+		test_seq  1 10 >numbers &&
+		test_seq 11 20 >values &&
+		git add numbers &&
+		git commit -m "Some tweaks" &&
+
+		git switch topic &&
+
+		>unrelated/foo &&
+		test_seq  2 12 >numbers &&
+		test_seq 12 22 >values &&
+		git add numbers values unrelated/ &&
+		git mv numbers sequence &&
+		git mv values progression &&
+		git commit -m A &&
+
+		#
+		# Actual testing
+		#
+
+		git switch --detach topic^0 &&
+
+		test_must_fail git -c merge.renameLimit=1 rebase upstream &&
+
+		git ls-files -u >actual &&
+		! test_file_is_empty actual
+	)
+'
+
 test_done

base-commit: 1ffcbaa1a5f10c9f706314d77f88de20a4a498c2

Quoted 31 lines · Elijah Newren via GitGitGadget

From: Elijah Newren <[email protected]>

In commit 8b09a900a1 ("merge-ort: restart merge with cached renames to reduce process entry cost", 2021-07-16), we noted that in the merge-ort steps of
collect_merge_info()
detect_and_process_renames()
process_entries()
that process_entries() was expensive, and we could often make it cheaper by changing this to
collect_merge_info()
detect_and_process_renames()
<cache all the renames, and restart> collect_merge_info()
detect_and_process_renames()
process_entries()
because the second collect_merge_info() would be cheaper (we could avoid traversing into some directories), the second detect_and_process_renames() would be free since we had already detected all renames, and then process_entries() has far fewer entries to handle.

However, this was built on the assumption that the first detect_and_process_renames() actually detected all potential renames. If someone has merge.renameLimit set to some small value, that assumption is violated which manifests later with the following message:

    $ git -c merge.renameLimit=1 rebase upstream
    ...
    git: merge-ort.c:546: clear_or_reinit_internal_opts: Assertion
    `renames->cached_pairs_valid_side == 0' failed.

Quoted 6 lines

Turn off this cache-renames-and-restart whenever we cannot detect all renames, and add a testcase that would have caught this problem.

Reported-by Taylor Blau <[email protected]>

Signed-off-by Elijah Newren <[email protected]>

---

Thanks. An Ack?

Quoted 1 line · Junio C Hamano

Thanks. An Ack?

Taylor told me the code change fixed his case, and that he'd review my full patch with the testcase when I posted it. Let's wait to hear from him.

Quoted 41 lines · Junio C Hamano

Quoted 37 lines · Elijah Newren via GitGitGadget

From: Elijah Newren <[email protected]>

In commit 8b09a900a1 ("merge-ort: restart merge with cached renames to reduce process entry cost", 2021-07-16), we noted that in the merge-ort steps of
collect_merge_info()
detect_and_process_renames()
process_entries()
that process_entries() was expensive, and we could often make it cheaper by changing this to
collect_merge_info()
detect_and_process_renames()
<cache all the renames, and restart> collect_merge_info()
detect_and_process_renames()
process_entries()
because the second collect_merge_info() would be cheaper (we could avoid traversing into some directories), the second detect_and_process_renames() would be free since we had already detected all renames, and then process_entries() has far fewer entries to handle.

However, this was built on the assumption that the first detect_and_process_renames() actually detected all potential renames. If someone has merge.renameLimit set to some small value, that assumption is violated which manifests later with the following message:

    $ git -c merge.renameLimit=1 rebase upstream
    ...
    git: merge-ort.c:546: clear_or_reinit_internal_opts: Assertion
    `renames->cached_pairs_valid_side == 0' failed.

Turn off this cache-renames-and-restart whenever we cannot detect all renames, and add a testcase that would have caught this problem.

Reported-by Taylor Blau <[email protected]>

Signed-off-by Elijah Newren <[email protected]>

---

Quoted 1 line

Thanks. An Ack?

Taylor told me the code change fixed his case, and that he'd review my full patch with the testcase when I posted it. Let's wait to hear from him.

Quoted 7 lines · Elijah Newren

Quoted 1 line · Junio C Hamano

Thanks. An Ack?

Taylor told me the code change fixed his case, and that he'd review my full patch with the testcase when I posted it. Let's wait to hear from him.

Ack. I can't vouch for the ort-specific details, but I trust Elijah's judgement (obviously). Running a version of Git with this patch applied fixes the issue I originally reported.

Thanks, Elijah!

t/t6429-merge-sequence-rename-caching.sh +54 -0

index 035edc40b1e..f2bc8a7d2a2 100755
@@ -697,4 +697,71 @@ test_expect_success 'caching renames only on upstream side, part 2' '
 	)
 '
 
+#
+# The following testcase just creates two simple renames (slightly modified
+# on both sides but without conflicting changes), and a directory full of
+# files that are otherwise uninteresting.  The setup is as follows:
+#
+#   base:     unrelated/<BUNCH OF FILES>
+#             numbers
+#             values
+#   upstream: modify: numbers
+#             modify: values
+#   topic:    add: unrelated/foo
+#             modify: numbers
+#             modify: values
+#             rename: numbers -> sequence
+#             rename: values -> progression
+#
+# This is a trivial rename case, but we're curious what happens with a very
+# low renameLimit interacting with the restart optimization trying to notice
+# that unrelated/ looks like a trivial merge candidate.
+#
+test_expect_success 'avoid assuming we detected renames' '
+	git init redo-weirdness &&
+	(
+		cd redo-weirdness &&
+
+		mkdir unrelated &&
+		for i in $(test_seq 1 10)
+		do
+			>unrelated/$i
+		done &&
+		test_seq  2 10 >numbers &&
+		test_seq 12 20 >values &&
+		git add numbers values unrelated/ &&
+		git commit -m orig &&
+
+		git branch upstream &&
+		git branch topic &&
+
+		git switch upstream &&
+		test_seq  1 10 >numbers &&
+		test_seq 11 20 >values &&
+		git add numbers &&
+		git commit -m "Some tweaks" &&
+
+		git switch topic &&
+
+		>unrelated/foo &&
+		test_seq  2 12 >numbers &&
+		test_seq 12 22 >values &&
+		git add numbers values unrelated/ &&
+		git mv numbers sequence &&
+		git mv values progression &&
+		git commit -m A &&
+

t/t6429-merge-sequence-rename-caching.sh +3 -0

+	)
+'
+
 test_done

base-commit: 1ffcbaa1a5f10c9f706314d77f88de20a4a498c2

Signature · 1 line

gitgitgadget

Quoted 67 lines · Elijah Newren via GitGitGadget

t/t6429-merge-sequence-rename-caching.sh +59 -0

index 035edc40b1e..f2bc8a7d2a2 100755
@@ -697,4 +697,71 @@ test_expect_success 'caching renames only on upstream side, part 2' '
      )
 '
 
+#
+# The following testcase just creates two simple renames (slightly modified
+# on both sides but without conflicting changes), and a directory full of
+# files that are otherwise uninteresting.  The setup is as follows:
+#
+#   base:     unrelated/<BUNCH OF FILES>
+#             numbers
+#             values
+#   upstream: modify: numbers
+#             modify: values
+#   topic:    add: unrelated/foo
+#             modify: numbers
+#             modify: values
+#             rename: numbers -> sequence
+#             rename: values -> progression
+#
+# This is a trivial rename case, but we're curious what happens with a very
+# low renameLimit interacting with the restart optimization trying to notice
+# that unrelated/ looks like a trivial merge candidate.
+#
+test_expect_success 'avoid assuming we detected renames' '
+     git init redo-weirdness &&
+     (
+             cd redo-weirdness &&
+
+             mkdir unrelated &&
+             for i in $(test_seq 1 10)
+             do
+                     >unrelated/$i
+             done &&
+             test_seq  2 10 >numbers &&
+             test_seq 12 20 >values &&
+             git add numbers values unrelated/ &&
+             git commit -m orig &&
+
+             git branch upstream &&
+             git branch topic &&
+
+             git switch upstream &&
+             test_seq  1 10 >numbers &&
+             test_seq 11 20 >values &&
+             git add numbers &&
+             git commit -m "Some tweaks" &&
+
+             git switch topic &&
+
+             >unrelated/foo &&
+             test_seq  2 12 >numbers &&
+             test_seq 12 22 >values &&
+             git add numbers values unrelated/ &&
+             git mv numbers sequence &&
+             git mv values progression &&
+             git commit -m A &&
+
+             #
+             # Actual testing
+             #
+
+             git switch --detach topic^0 &&

t/t6429-merge-sequence-rename-caching.sh test_expect_success 'caching renames only on upstream side, part 2' ' · line 763

Quoted 5 lines

t/t6429-merge-sequence-rename-caching.sh +5 -0

+
+             test_must_fail git -c merge.renameLimit=1 rebase upstream &&
+
+             git ls-files -u >actual &&
+             ! test_file_is_empty actual

There is no 'test_file_is_empty' function, but because of the ! at the beginning of the line it didn't fail the test.

The minimal fix would be to use 'test_file_not_empty' instead, but I wonder whether we should use 'test_line_count = 2' instead for a tad tighter check.

Quoted 9 lines

t/t6429-merge-sequence-rename-caching.sh +3 -0

+     )
+'
+
 test_done

base-commit: 1ffcbaa1a5f10c9f706314d77f88de20a4a498c2 --
gitgitgadget

Quoted 6 lines · SZEDER Gábor

Quoted 4 lines · Elijah Newren via GitGitGadget

t/t6429-merge-sequence-rename-caching.sh +0 -0

index 035edc40b1e..f2bc8a7d2a2 100755

Quoted 5 lines

t/t6429-merge-sequence-rename-caching.sh test_expect_success 'caching renames only on upstream side, part 2' ' · line 763

Quoted 2 lines

t/t6429-merge-sequence-rename-caching.sh +2 -0

+             git ls-files -u >actual &&
+             ! test_file_is_empty actual

There is no 'test_file_is_empty' function, but because of the ! at the beginning of the line it didn't fail the test.

Oops, looks like I meant test_must_be_empty.

Quoted 3 lines

The minimal fix would be to use 'test_file_not_empty' instead, but I wonder whether we should use 'test_line_count = 2' instead for a tad tighter check.

Makes sense; since this merged about half a year ago, I'll submit a new patch to fix this. Thanks for catching and pointing it out!