rebase no longer omits local commits

Starting with git 1.9.0, rebase no longer omits local commits that appear in both the upstream and local branches.

I've bisected this down to commit bb3f458: "rebase: fix fork-point with zero arguments". The attached script reproduces the problem. Reverting the aforementioned commit fixes the problem.

A failed run of this script will result in conflicts. A successful run against master with bb3f458 reverted ends as follows:

From /tmp/rebase-issue/maint
fe401cd..955af04 master -> origin/master
fatal: Not a valid object name: ''
First, rewinding head to replay your work on top of it...
Applying: Third change

(I'm not sure if that "fatal: Not a valid object name: ''" is of any concern. It started appearing for me at some point during the bisect.)

Let me know if there's more I can do to help.

#!/bin/bash

# git-rebase is supposed to drop commits that it finds in both the 
# local and upstream branches.  As of 1.9.0, this isn't happening.
# This script reproduces the problem.

# I've bisected the issue down to commit bb3f458. Reverting that commit

# solves the problem.

# Run this in a directory where you have create privs. # At the end, if there are conflicts, then the test has failed.

# Create a repo.
mkdir rebase-issue
cd rebase-issue
mkdir maint
cd maint
git init

# Create a README file and put some text in it echo "Hi there!" >README
git add README
git commit -a -m "Initial commit"

# Clone the repo for "dev"
cd ..
git clone maint dev

# Dev makes *two* changes to the *same* area. cd dev

# edit something, make some typos
echo "Freekwently Mispeled Werdz" >README
git commit -a -m "First change"
# edit same thing, fix those typos
echo "Frequently Misspelled Words" >README
git commit -a -m "Second change"

# Create patches to send to maintainer... git format-patch -M origin/master
mv *.patch ../maint

# Add a third change that should make it through for completeness. echo "Frequently Misspelled Words version 2" >README git commit -a -m "Third change"

# We have to sleep (to make sure the times do not match?). # If we don't, this script will succeed on fast machines. # This can probably be reduced to 2 which should guarantee that

# the seconds will turn over on the clock.
echo
echo "Waiting 5 seconds to make sure apply time is different from patch time..."
sleep 5

echo
echo "Maint applies patches..."
cd ../maint
git am -3 *.patch

echo
echo "Dev does the fetch/rebase..."
cd ../dev
git fetch
git rebase

echo
git --version

It is the problem that bb3f458 fixes. The change in behaviour is actually introduced by ad8261d (rebase: use reflog to find common base with upstream).

In your example, I think this is working as designed. You can restore the previous behaviour either with git rebase --no-fork-point or with git rebase @{u}.

The change is designed to help users recover from an upstream rebase, as described in the "DISCUSSION ON FORK-POINT MODE" section of git-merge-base(1) and makes git rebase match the behaviour of git pull --rebase so that:

	git fetch &&
	git rebase

really is equivalent to:

	git pull --rebase

Having thought about this a bit more, I think the case you've identified is an unexpected side effect of that commit.

Perhaps we shuld do something like this (which passes the test suite):

-- >8 --

index 06c810b..0c6c5d3 100755
@@ -544,7 +544,8 @@ if test "$fork_point" = t
 then
 	new_upstream=$(git merge-base --fork-point "$upstream_name" \
 			"${switch_to:-HEAD}")
-	if test -n "$new_upstream"
+	if test -n "$new_upstream" &&
+	   ! git merge-base --is-ancestor "$new_upstream" "$upstream_name"
 	then
 		upstream=$new_upstream
 	fi

-- 8< --

Since the intent of --fork-point is to find the best starting point for the "$upstream...$orig_head" range, if the fork point is behind the new location of the upstream then should we leave the upstream as it was?

I haven't thought through this completely, but it seems like we should be doing a check like the above, at least when we're in "$fork_point=auto" mode.

Probably; but the check to avoid giving worse fork-point should be in the implementation of "merge-base --fork-point" itself, so that we do not have to do the above to both "rebase" and "pull --rebase", no?

I don't think so, since in that case we're not actually finding the fork point as defined in the documentation, we're finding the upstream rebase wants.

Having played with this a bit, I think we shouldn't be replacing the upstream with the fork point but should instead add the fork point as an additional negative ref:

	$upstream...$orig_head ^$fork_point

Here's a script that creates a repository showing this:

-- >8 --
#!/bin/sh
git init rebase-test &&
cd rebase-test &&
echo one >file &&
git add file &&
git commit -m one &&
echo frist >file2 &&
git add file2 &&
git commit -m first &&
git branch --track dev &&
echo first >file2 &&
git commit -a --amend --no-edit &&
echo two >file &&
git commit -a -m two &&
echo three >file &&
git commit -a -m three &&
echo second >file2 &&
git commit -a -m second &&
git checkout dev &&
git cherry-pick -2 master &&
echo four >file &&
git commit -a -m four &&
printf '\nWithout fork point (old behaviour)\n' &&
git rev-list --oneline --cherry @{u}... &&
printf '\nFork point as upstream (current behaviour)\n' &&
git rev-list --oneline --cherry $(git merge-base --fork-point master HEAD)... && printf '\nWith fork point\n' &&
git rev-list --oneline --cherry @{u}... ^$(git merge-base --fork-point master HEAD) -- 8< --

In this case the rebase should be clean since the only applicable patch changes "three" to "four" in "file", but the current rebase code fails both with --fork-point and with --no-fork-point.

With --fork-point we try to apply "two" and "three" which have already been cherry-picked across (as Ted originally reported) and with --no-fork-point, we try to apply "first" which conflicts because we have the version prior to it being fixed up on master.

I hacked up git-rebase to test this and the change to use the fork point as in the last line of the script above does indeed make the rebase go through cleanly, but I have not yet looked at how to cleanly patch in that behaviour.

I haven't tested git-pull, but it looks like it has always (since 2009) behaved in the way git rebase --fork-point does now, failing to detect cherry-picked commits that are now in the upstream.

When using git format-patch --ignore-if-in-upstream we are only allowed to give a single revision range. In the next commit we will want to add an additional exclusion revision in order to handle fork points correctly, so convert git-rebase--am to use a symmetric difference with --cherry-pick --right-only.

This does not change the result of the format-patch invocation, just how we spell the arguments.

---
 git-rebase--am.sh | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

index ca20e1e..902bf2d 100644
@@ -29,7 +29,13 @@ skip)
 	;;
 esac
 
-test -n "$rebase_root" && root_flag=--root
+if test -z "$rebase_root"
+	# this is now equivalent to ! -z "$upstream"
+then
+	revisions=$upstream...$orig_head
+else
+	revisions=$onto...$orig_head
+fi
 
 ret=0
 if test -n "$keep_empty"

@@ -38,14 +44,15 @@ then
 	# empty commits and even if it didn't the format doesn't really lend
 	# itself well to recording empty patches.  fortunately, cherry-pick
 	# makes this easy
-	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty "$revisions"
+	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty \
+		--right-only "$revisions"
 	ret=$?
 else
 	rm -f "$GIT_DIR/rebased-patches"
 
-	git format-patch -k --stdout --full-index --ignore-if-in-upstream \
+	git format-patch -k --stdout --full-index --cherry-pick --right-only \
 		--src-prefix=a/ --dst-prefix=b/ --no-renames --no-cover-letter \
-		$root_flag "$revisions" >"$GIT_DIR/rebased-patches"
+		"$revisions" >"$GIT_DIR/rebased-patches"
 	ret=$?
 
 	if test 0 != $ret

2.0.1.472.g6f92e5f.dirty

When the --fork-point argument was added to git rebase, we changed the value of $upstream to be the fork point instead of the point from which we want to rebase. When $orig_head..$upstream is empty this does not change the behaviour, but when there are new changes in the upstream we are no longer checking if any of them are patch-identical with changes in $upstream..$orig_head.

Fix this by introducing a new variable to hold the fork point and using this to restrict the range as an extra (negative) revision argument so that the set of desired revisions becomes (in fork-point mode):

	git rev-list --cherry-pick --right-only \
		$upstream...$orig_head ^$fork_point

This allows us to correctly handle the scenario where we have the following topology:

	    C --- D --- E  <- dev
	   /
	  B  <- master@{1}
	 /
	o --- B' --- C* --- D*  <- master

where:
- B' is a fixed-up version of B that is not patch-identical with B;
- C* and D* are patch-identical to C and D respectively and conflict textually if applied in the wrong order;
- E depends textually on D.

The correct result of git rebase master dev is that B is identified as the fork-point of dev and master, so that C, D, E are the commits that need to be replayed onto master; but C and D are patch-identical with C* and D* and so can be dropped, so that the end result is:

o --- B' --- C* --- D* --- E <- dev

If the fork-point is not identified, then picking B onto a branch containing B' results in a conflict and if the patch-identical commits are not correctly identified then picking C onto a branch containing D (or equivalently D*) results in a conflict.

This change allows us to handle both of these cases, where previously we either identified the fork-point (with --fork-point) but not the patch-identical commits *or* (with --no-fork-point) identified the patch-identical commits but not the fact that master had been rewritten.

---
 git-rebase--am.sh          | 6 ++++--
 git-rebase--interactive.sh | 2 +-
 git-rebase.sh              | 7 ++++---
 3 files changed, 9 insertions(+), 6 deletions(-)

index 902bf2d..f923732 100644
@@ -45,14 +45,16 @@ then
 	# itself well to recording empty patches.  fortunately, cherry-pick
 	# makes this easy
 	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty \
-		--right-only "$revisions"
+		--right-only "$revisions" \
+		${restrict_revision+^$restrict_revision}
 	ret=$?
 else
 	rm -f "$GIT_DIR/rebased-patches"
 
 	git format-patch -k --stdout --full-index --cherry-pick --right-only \
 		--src-prefix=a/ --dst-prefix=b/ --no-renames --no-cover-letter \
-		"$revisions" >"$GIT_DIR/rebased-patches"
+		"$revisions" ${restrict_revision+^$restrict_revision} \
+		>"$GIT_DIR/rebased-patches"
 	ret=$?
 
 	if test 0 != $ret

index 7e1eda0..b64dd28 100644
@@ -963,7 +963,7 @@ else
 fi
 git rev-list $merges_option --pretty=oneline --abbrev-commit \
 	--abbrev=7 --reverse --left-right --topo-order \
-	$revisions | \
+	$revisions ${restrict_revision+^$restrict_revision} | \
 	sed -n "s/^>//p" |
 while read -r shortsha1 rest
 do

index 06c810b..55da9db 100755
@@ -59,6 +59,7 @@ If you prefer to skip this patch, run "git rebase --skip" instead.
 To check out the original branch and stop rebasing, run "git rebase --abort".')
 "
 unset onto
+unset restrict_revision
 cmd=
 strategy=
 strategy_opts=

@@ -546,7 +547,7 @@ then
 			"${switch_to:-HEAD}")
 	if test -n "$new_upstream"
 	then
-		upstream=$new_upstream
+		restrict_revision=$new_upstream
 	fi
 fi
 

@@ -572,7 +573,7 @@ require_clean_work_tree "rebase" "$(gettext "Please commit or stash them.")"
 # and if this is not an interactive rebase.
 mb=$(git merge-base "$onto" "$orig_head")
 if test "$type" != interactive && test "$upstream" = "$onto" &&
-	test "$mb" = "$onto" &&
+	test "$mb" = "$onto" && test -z "$restrict_revision" &&
 	# linear history?
 	! (git rev-list --parents "$onto".."$orig_head" | sane_grep " .* ") > /dev/null
 then

@@ -626,7 +627,7 @@ if test -n "$rebase_root"
 then
 	revisions="$onto..$orig_head"
 else
-	revisions="$upstream..$orig_head"
+	revisions="${restrict_revision-$upstream}..$orig_head"
 fi
 
 run_specific_rebase

2.0.1.472.g6f92e5f.dirty

Thanks, John. My test script is working fine for me now with these
patches.

Can we have some tests, too?

The changes look correct from a cursory reading.

Thanks.

When using git format-patch --ignore-if-in-upstream we are only allowed to give a single revision range. In the next commit we will want to add an additional exclusion revision in order to handle fork points correctly, so convert git-rebase--am to use a symmetric difference with --cherry-pick --right-only.

This does not change the result of the format-patch invocation, just how we spell the arguments.

---

Unchanged from v1.

 git-rebase--am.sh | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

index ca20e1e..902bf2d 100644
@@ -29,7 +29,13 @@ skip)
 	;;
 esac
 
-test -n "$rebase_root" && root_flag=--root
+if test -z "$rebase_root"
+	# this is now equivalent to ! -z "$upstream"
+then
+	revisions=$upstream...$orig_head
+else
+	revisions=$onto...$orig_head
+fi
 
 ret=0
 if test -n "$keep_empty"

@@ -38,14 +44,15 @@ then
 	# empty commits and even if it didn't the format doesn't really lend
 	# itself well to recording empty patches.  fortunately, cherry-pick
 	# makes this easy
-	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty "$revisions"
+	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty \
+		--right-only "$revisions"
 	ret=$?
 else
 	rm -f "$GIT_DIR/rebased-patches"
 
-	git format-patch -k --stdout --full-index --ignore-if-in-upstream \
+	git format-patch -k --stdout --full-index --cherry-pick --right-only \
 		--src-prefix=a/ --dst-prefix=b/ --no-renames --no-cover-letter \
-		$root_flag "$revisions" >"$GIT_DIR/rebased-patches"
+		"$revisions" >"$GIT_DIR/rebased-patches"
 	ret=$?
 
 	if test 0 != $ret

2.0.1.472.g6f92e5f.dirty

When the --fork-point argument was added to git rebase, we changed the value of $upstream to be the fork point instead of the point from which we want to rebase. When $orig_head..$upstream is empty this does not change the behaviour, but when there are new changes in the upstream we are no longer checking if any of them are patch-identical with changes in $upstream..$orig_head.

Fix this by introducing a new variable to hold the fork point and using this to restrict the range as an extra (negative) revision argument so that the set of desired revisions becomes (in fork-point mode):

	git rev-list --cherry-pick --right-only \
		$upstream...$orig_head ^$fork_point

This allows us to correctly handle the scenario where we have the following topology:

	    C --- D --- E  <- dev
	   /
	  B  <- master@{1}
	 /
	o --- B' --- C* --- D*  <- master

where:
- B' is a fixed-up version of B that is not patch-identical with B;
- C* and D* are patch-identical to C and D respectively and conflict textually if applied in the wrong order;
- E depends textually on D.

The correct result of git rebase master dev is that B is identified as the fork-point of dev and master, so that C, D, E are the commits that need to be replayed onto master; but C and D are patch-identical with C* and D* and so can be dropped, so that the end result is:

o --- B' --- C* --- D* --- E <- dev

If the fork-point is not identified, then picking B onto a branch containing B' results in a conflict and if the patch-identical commits are not correctly identified then picking C onto a branch containing D (or equivalently D*) results in a conflict.

This change allows us to handle both of these cases, where previously we either identified the fork-point (with --fork-point) but not the patch-identical commits *or* (with --no-fork-point) identified the patch-identical commits but not the fact that master had been rewritten.

---

Change from v1:
- add a test case

 git-rebase--am.sh          |  6 ++++--
 git-rebase--interactive.sh |  2 +-
 git-rebase.sh              |  7 ++++---
 t/t3400-rebase.sh          | 23 +++++++++++++++++++++++
 4 files changed, 32 insertions(+), 6 deletions(-)

index 902bf2d..f923732 100644
@@ -45,14 +45,16 @@ then
 	# itself well to recording empty patches.  fortunately, cherry-pick
 	# makes this easy
 	git cherry-pick ${gpg_sign_opt:+"$gpg_sign_opt"} --allow-empty \
-		--right-only "$revisions"
+		--right-only "$revisions" \
+		${restrict_revision+^$restrict_revision}
 	ret=$?
 else
 	rm -f "$GIT_DIR/rebased-patches"
 
 	git format-patch -k --stdout --full-index --cherry-pick --right-only \
 		--src-prefix=a/ --dst-prefix=b/ --no-renames --no-cover-letter \
-		"$revisions" >"$GIT_DIR/rebased-patches"
+		"$revisions" ${restrict_revision+^$restrict_revision} \
+		>"$GIT_DIR/rebased-patches"
 	ret=$?
 
 	if test 0 != $ret

index 7e1eda0..b64dd28 100644
@@ -963,7 +963,7 @@ else
 fi
 git rev-list $merges_option --pretty=oneline --abbrev-commit \
 	--abbrev=7 --reverse --left-right --topo-order \
-	$revisions | \
+	$revisions ${restrict_revision+^$restrict_revision} | \
 	sed -n "s/^>//p" |
 while read -r shortsha1 rest
 do

index 06c810b..55da9db 100755
@@ -59,6 +59,7 @@ If you prefer to skip this patch, run "git rebase --skip" instead.
 To check out the original branch and stop rebasing, run "git rebase --abort".')
 "
 unset onto
+unset restrict_revision
 cmd=
 strategy=
 strategy_opts=

@@ -546,7 +547,7 @@ then
 			"${switch_to:-HEAD}")
 	if test -n "$new_upstream"
 	then
-		upstream=$new_upstream
+		restrict_revision=$new_upstream
 	fi
 fi
 

@@ -572,7 +573,7 @@ require_clean_work_tree "rebase" "$(gettext "Please commit or stash them.")"
 # and if this is not an interactive rebase.
 mb=$(git merge-base "$onto" "$orig_head")
 if test "$type" != interactive && test "$upstream" = "$onto" &&
-	test "$mb" = "$onto" &&
+	test "$mb" = "$onto" && test -z "$restrict_revision" &&
 	# linear history?
 	! (git rev-list --parents "$onto".."$orig_head" | sane_grep " .* ") > /dev/null
 then

@@ -626,7 +627,7 @@ if test -n "$rebase_root"
 then
 	revisions="$onto..$orig_head"
 else
-	revisions="$upstream..$orig_head"
+	revisions="${restrict_revision-$upstream}..$orig_head"
 fi
 
 run_specific_rebase

index 80e0a95..47b5682 100755
@@ -169,6 +169,29 @@ test_expect_success 'default to common base in @{upstream}s reflog if no upstrea
 	test_cmp expect actual
 '
 
+test_expect_success 'cherry-picked commits and fork-point work together' '
+	git checkout default-base &&
+	echo Amended >A &&
+	git commit -a --no-edit --amend &&
+	test_commit B B &&
+	test_commit new_B B "New B" &&
+	test_commit C C &&
+	git checkout default &&
+	git reset --hard default-base@{4} &&
+	test_commit D D &&
+	git cherry-pick -2 default-base^ &&
+	test_commit final_B B "Final B" &&
+	git rebase &&
+	echo Amended >expect &&
+	test_cmp A expect &&
+	echo "Final B" >expect &&
+	test_cmp B expect &&
+	echo C >expect &&
+	test_cmp C expect &&
+	echo D >expect &&
+	test_cmp D expect
+'
+
 test_expect_success 'rebase -q is quiet' '
 	git checkout -b quiet topic &&
 	git rebase -q master >output.out 2>&1 &&

2.0.1.472.g6f92e5f.dirty

No, I didn't consider the log message when adding the test; it simply uses the next letters that are unused in the test repository.

Yes, that's right. The idea is to make sure that we skip the cherry-picked commits, which requires two commits that will conflict if we try to apply the first on top of the second and to make sure that we don't lose the new commit on the upstream (C).

Test case is working fine for me. It passes with the patch and fails without. However, it does seem to cause all the rest of the test cases to fail if it fails. Is there some cleanup missing?

Ted.

The individual test cases in the script don't run in isolation, so I don't think it's surprising the the remainder fail if this one ends up stuck in rebase-in-progress state. I think the same will happen with most of the other test cases in this script.