different git-merge behavior with regard to submodules in 1.6.2.4 vs. 1.6.2.1

Hello,

I noticed some weirdness lately when trying to merge branches in a repository containing submodules. I get the following behavior with git 1.6.2.4:

$ git merge origin/deployed
fatal: cannot read object 83055ffdddde60d41d9811aae77e78be50b329f8
'rubydav': It is a submodule!

Nothing in my history suggests that rubydav was at one point not a submodule.

I looked at the 1.6.2.4 release notes and noticed the following:

* "git-merge-recursive" was broken when a submodule entry was involved
in a criss-cross merge situation.

So then I downgraded to the last debian package of git which is 1.6.2.1.
Now I get a result which is more approachable:

$ git merge origin/deployed
Auto-merging rubydav
CONFLICT (submodule): Merge conflict in rubydav - needs
167a344227c4745031d50a210869e6fb59a5ac03
Auto-merging server
CONFLICT (submodule): Merge conflict in server - needs
82a74ae791c8563ca65f29187d2fe5ebfbc167ea
Automatic merge failed; fix conflicts and then commit the result.

Both merges are from freshly checked out clones.

Is this a bug in 1.6.2.4? Please let me know what other information I can provide to help debug the problem.

Thanks,
Tim

Thanks for a report. I think the following commits are involved.

39d8e27 simplify output of conflicting merge 0eb6574 update cache for conflicting submodule entries f37ae35 add tests for merging with submodules

Clemens, these seem to be yours. Thoughts?

[...]

The current error message is not an improvement I think, it should say that merge does not support merging submodules, not complain about being unable to read some object because it is a submodule.

I added the "CONFLICT (submodule) Merge conflict .. needs <SHA-1>" messages when I tried to work with submodules a while (1-2 years?) ago. The intention was that you could enter the submodule(s), write "git merge <SHA-1>", and resolve the conflict that way.

git is unfortunately not capable of merging submodules at all, so I added these error messages to give me a hint about what I needed to do in conflicting submodules to get something useful. I have used git a lot more now, so maybe it is time to pick this up again and implement proper recursive sub-module merging.

- Finn Arne

From: Clemens Buchacher <[email protected]>
Date: Tue, 28 Apr 2009 21:16:02 +0200
Subject: [PATCH] fix recursive merge with submodule add/add conflict

In case of a submodule we should not attempt to update the working copy, but we do have to update the index.

---

The fatal error is indeed caused by 0eb6574 (update cache for conflicting submodule entries). The problem is also documented by t7405. The test exposes a problem even previous to this commit, wherein "git diff" aborts with

fatal: read error 'sub'

which is why I thought this was broken anyways. Only I see now that I have made things worse.

The appended patch should fix both problems for now. I'm still not satisfied with the result, because the conflicting submodule path is not marked as unmerged. I modified t7405 to reflect this failure. I think we should be able to handle this in the same way we handle symlinks.

I think you should get that information from git diff instead.

Are you sure it's always the right thing to merge conflicting submodule versions? The user could easily commit two versions, which you would never want merge -- due to changed history, for example. On the other hand, if a fast-forward merge is possible I suppose this could be considered a non-conflicting change.

Clemens

---
 merge-recursive.c          |   11 ++++++-----
 t/t7405-submodule-merge.sh |    3 ++-
 2 files changed, 8 insertions(+), 6 deletions(-)

index d6f0582..b14b3fe 100644
@@ -1116,13 +1116,14 @@ static int process_entry(struct merge_options *o,
 				 o->branch1, o->branch2);
 
 		clean_merge = mfi.clean;
-		if (!mfi.clean) {
-			if (S_ISGITLINK(mfi.mode))
-				reason = "submodule";
+		if (S_ISGITLINK(mfi.mode)) {
+			reason = "submodule";
+			add_cacheinfo(mfi.mode, mfi.sha, path, 0, 0, ADD_CACHE_OK_TO_ADD);
+		} else
+			update_file(o, mfi.clean, mfi.sha, mfi.mode, path);
+		if (!mfi.clean)
 			output(o, 1, "CONFLICT (%s): Merge conflict in %s",
 					reason, path);
-		}
-		update_file(o, mfi.clean, mfi.sha, mfi.mode, path);
 	} else if (!o_sha && !a_sha && !b_sha) {
 		/*
 		 * this entry was deleted altogether. a_mode == 0 means

index aa6c44c..b881370 100755
@@ -59,7 +59,8 @@ test_expect_failure 'merging with modify/modify conflict' '
 	git checkout -b test1 a &&
 	test_must_fail git merge b &&
 	test -f .git/MERGE_MSG &&
-	git diff
+	git diff &&
+	test -n "`git ls-files -u`"
 
 '
 

1.6.3.rc2

On Wed, Apr 29, 2009 at 10:42:09AM +0200, Clemens Buchacher wrote:
[...]

I would _like_ git to be able to merge submodules. However, if we want to keep limiting sub-modules to only track external independent 3rdparty modules, it isn't so useful I guess. But I also think that seriously reduces the usefulness of sub-modules.

Maybe a per-submodule flag that indicates whether or not it wants to be automatically merged from the supermodule? Closely coupled vs loosely coupled sub-modules.

- Finn Arne

The combine diff logic knew only about blobs (and their checked-out form in the work tree, either regular files or symlinks), and barfed when fed submodules. This "externalizes" gitlinks in the same way as the normal patch generation codepath does (i.e. "Subproject commit Xxx\n") to fix the issue.

---

I think we should share the implementation betweeen grab_blob() and diff.c::diff_populate_gitlink() to keep their submodule representation the same, but I am doing this as a "fix" patch and I am lazy...

 combine-diff.c |   31 +++++++++++++++++++++++--------
 1 files changed, 23 insertions(+), 8 deletions(-)

index 066ce84..0f192e1 100644
@@ -6,6 +6,7 @@
 #include "quote.h"
 #include "xdiff-interface.h"
 #include "log-tree.h"
+#include "refs.h"
 
 static struct combine_diff_path *intersect_paths(struct combine_diff_path *curr, int n, int num_parent)
 {

@@ -90,7 +91,7 @@ struct sline {
 	unsigned long *p_lno;
 };
 
-static char *grab_blob(const unsigned char *sha1, unsigned long *size)
+static char *grab_blob(const unsigned char *sha1, unsigned int mode, unsigned long *size)
 {
 	char *blob;
 	enum object_type type;

@@ -98,10 +99,15 @@ static char *grab_blob(const unsigned char *sha1, unsigned long *size)
 		/* deleted blob */
 		*size = 0;
 		return xcalloc(1, 1);
+	} else if (S_ISGITLINK(mode)) {
+		blob = xmalloc(100);
+		*size = snprintf(blob, 100,
+				 "Subproject commit %s\n", sha1_to_hex(sha1));
+	} else {
+		blob = read_sha1_file(sha1, &type, size);
+		if (type != OBJ_BLOB)
+			die("object '%s' is not a blob!", sha1_to_hex(sha1));
 	}
-	blob = read_sha1_file(sha1, &type, size);
-	if (type != OBJ_BLOB)
-		die("object '%s' is not a blob!", sha1_to_hex(sha1));
 	return blob;
 }
 

@@ -195,7 +201,8 @@ static void consume_line(void *state_, char *line, unsigned long len)
 	}
 }
 
-static void combine_diff(const unsigned char *parent, mmfile_t *result_file,
+static void combine_diff(const unsigned char *parent, unsigned int mode,
+			 mmfile_t *result_file,
 			 struct sline *sline, unsigned int cnt, int n,
 			 int num_parent)
 {

@@ -211,7 +218,7 @@ static void combine_diff(const unsigned char *parent, mmfile_t *result_file,
 	if (!cnt)
 		return; /* result deleted */
 
-	parent_file.ptr = grab_blob(parent, &sz);
+	parent_file.ptr = grab_blob(parent, mode, &sz);
 	parent_file.size = sz;
 	memset(&xpp, 0, sizeof(xpp));
 	xpp.flags = XDF_NEED_MINIMAL;

@@ -692,7 +699,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 
 	/* Read the result of merge first */
 	if (!working_tree_file)
-		result = grab_blob(elem->sha1, &result_size);
+		result = grab_blob(elem->sha1, elem->mode, &result_size);
 	else {
 		/* Used by diff-tree to read from the working tree */
 		struct stat st;

@@ -712,6 +719,12 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 			result_size = buf.len;
 			result = strbuf_detach(&buf, NULL);
 			elem->mode = canon_mode(st.st_mode);
+		} else if (S_ISDIR(st.st_mode)) {
+			unsigned char sha1[20];
+			if (resolve_gitlink_ref(elem->path, "HEAD", sha1) < 0)
+				result = grab_blob(elem->sha1, elem->mode, &result_size);
+			else
+				result = grab_blob(sha1, elem->mode, &result_size);
 		} else if (0 <= (fd = open(elem->path, O_RDONLY))) {
 			size_t len = xsize_t(st.st_size);
 			ssize_t done;

@@ -804,7 +817,9 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 			}
 		}
 		if (i <= j)
-			combine_diff(elem->parent[i].sha1, &result_file, sline,
+			combine_diff(elem->parent[i].sha1,
+				     elem->parent[i].mode,
+				     &result_file, sline,
 				     cnt, i, num_parent);
 		if (elem->parent[i].mode != elem->mode)
 			mode_differs = 1;

The combine diff logic knew only about blobs (and their checked-out form in the work tree, either regular files or symlinks), and barfed when fed submodules. This "externalizes" gitlinks in the same way as the normal patch generation codepath does (i.e. "Subproject commit Xxx\n") to fix the issue.

---

This is a re-roll of the earlier "[PATCH] Teach gitlinks to combine-diff", meant to apply to maint-1.6.0 and later.

 combine-diff.c            |   38 ++++++++++++++++++++++++++------------
 t/t4027-diff-submodule.sh |   39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 12 deletions(-)

index aa9d79e..f617e9d 100644
@@ -6,6 +6,7 @@
 #include "quote.h"
 #include "xdiff-interface.h"
 #include "log-tree.h"
+#include "refs.h"
 
 static struct combine_diff_path *intersect_paths(struct combine_diff_path *curr, int n, int num_parent)
 {

@@ -90,18 +91,24 @@ struct sline {
 	unsigned long *p_lno;
 };
 
-static char *grab_blob(const unsigned char *sha1, unsigned long *size)
+static char *grab_blob(const unsigned char *sha1, unsigned int mode, unsigned long *size)
 {
 	char *blob;
 	enum object_type type;
-	if (is_null_sha1(sha1)) {
+
+	if (S_ISGITLINK(mode)) {
+		blob = xmalloc(100);
+		*size = snprintf(blob, 100,
+				 "Subproject commit %s\n", sha1_to_hex(sha1));
+	} else if (is_null_sha1(sha1)) {
 		/* deleted blob */
 		*size = 0;
 		return xcalloc(1, 1);
+	} else {
+		blob = read_sha1_file(sha1, &type, size);
+		if (type != OBJ_BLOB)
+			die("object '%s' is not a blob!", sha1_to_hex(sha1));
 	}
-	blob = read_sha1_file(sha1, &type, size);
-	if (type != OBJ_BLOB)
-		die("object '%s' is not a blob!", sha1_to_hex(sha1));
 	return blob;
 }
 

@@ -197,7 +204,8 @@ static void consume_line(void *state_, char *line, unsigned long len)
 	}
 }
 
-static void combine_diff(const unsigned char *parent, mmfile_t *result_file,
+static void combine_diff(const unsigned char *parent, unsigned int mode,
+			 mmfile_t *result_file,
 			 struct sline *sline, unsigned int cnt, int n,
 			 int num_parent)
 {

@@ -213,7 +221,7 @@ static void combine_diff(const unsigned char *parent, mmfile_t *result_file,
 	if (!cnt)
 		return; /* result deleted */
 
-	parent_file.ptr = grab_blob(parent, &sz);
+	parent_file.ptr = grab_blob(parent, mode, &sz);
 	parent_file.size = sz;
 	xpp.flags = XDF_NEED_MINIMAL;
 	memset(&xecfg, 0, sizeof(xecfg));

@@ -692,7 +700,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 	context = opt->context;
 	/* Read the result of merge first */
 	if (!working_tree_file)
-		result = grab_blob(elem->sha1, &result_size);
+		result = grab_blob(elem->sha1, elem->mode, &result_size);
 	else {
 		/* Used by diff-tree to read from the working tree */
 		struct stat st;

@@ -712,9 +720,13 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 			}
 			result[len] = 0;
 			elem->mode = canon_mode(st.st_mode);
-		}
-		else if (0 <= (fd = open(elem->path, O_RDONLY)) &&
-			 !fstat(fd, &st)) {
+		} else if (S_ISDIR(st.st_mode)) {
+			unsigned char sha1[20];
+			if (resolve_gitlink_ref(elem->path, "HEAD", sha1) < 0)
+				result = grab_blob(elem->sha1, elem->mode, &result_size);
+			else
+				result = grab_blob(sha1, elem->mode, &result_size);
+		} else if (0 <= (fd = open(elem->path, O_RDONLY))) {
 			size_t len = xsize_t(st.st_size);
 			ssize_t done;
 			int is_file, i;

@@ -807,7 +819,9 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,
 			}
 		}
 		if (i <= j)
-			combine_diff(elem->parent[i].sha1, &result_file, sline,
+			combine_diff(elem->parent[i].sha1,
+				     elem->parent[i].mode,
+				     &result_file, sline,
 				     cnt, i, num_parent);
 		if (elem->parent[i].mode != elem->mode)
 			mode_differs = 1;

index ba6679c..718efe8 100755
@@ -57,4 +57,43 @@ test_expect_success 'git diff (empty submodule dir)' '
 	test_cmp empty actual.empty
 '
 
+test_expect_success 'conflicted submodule setup' '
+
+	# 39 efs
+	c=fffffffffffffffffffffffffffffffffffffff
+	(
+		echo "000000 $_z40 0	sub"
+		echo "160000 1$c 1	sub"
+		echo "160000 2$c 2	sub"
+		echo "160000 3$c 3	sub"
+	) | git update-index --index-info &&
+	echo >expect.nosub '\''diff --cc sub
+index 2ffffff,3ffffff..0000000
+--- a/sub
++++ b/sub
+@@@ -1,1 -1,1 +1,1 @@@
+- Subproject commit 2fffffffffffffffffffffffffffffffffffffff
+ -Subproject commit 3fffffffffffffffffffffffffffffffffffffff
+++Subproject commit 0000000000000000000000000000000000000000'\'' &&
+
+	hh=$(git rev-parse HEAD) &&
+	sed -e "s/$_z40/$hh/" expect.nosub >expect.withsub
+
+'
+
+test_expect_success 'combined (empty submodule)' '
+	rm -fr sub && mkdir sub &&
+	git diff >actual &&
+	test_cmp expect.nosub actual
+'
+
+test_expect_success 'combined (with submodule)' '
+	rm -fr sub &&
+	git clone --no-checkout . sub &&
+	git diff >actual &&
+	test_cmp expect.withsub actual
+'
+
+
+
 test_done

1.6.3.rc3.16.gb37759

2009/4/29 Junio C Hamano <[email protected]>:

snprintf returns a signed value. It also has a bad record of returning negative values for obscure reasons (on obscure platforms, admittedly).

For this particular case,

  strcpy(blob, "Subproject commit ");
  strcat(blob, sha1_to_hex(sha1));
  strcat(blob, "\n");
  *size = strlen(blob); /* that's a constant */

could be considered.

The arena is sufficiently large that there is no way any broken snprintf can return negative here.

This is a copy from Linus's diff_populate_gitlink(), that dates back to 0478675 (Expose subprojects as special files to "git diff" machinery, 2007-04-15), and you have never seen any breakage, which should tell you something.

As I mentioned in the original patch, the codepath that reads one side of diff (either from a blob or from a work tree entity) in show_patch_diff() and grab_blob() in combine-diff.c should do the same thing as what diff_populate_filespec() in diff.c does, and these three functions need some refactoring to share more code. The patch however is about fixing the existing breakage without invasive refactoring.

Hi,

Actually, we know _exactly_ the size of the thing. It is 18+40+1. But I think that *size wants to have the size, not the length. So add 1.

In any case, I don't think that we have to jump through hoops here:
snprintf() is _most_ unlikely to return something negative here. So I'd say that readability trumps paranoia here.

Ciao,
Dscho

2009/4/30 Johannes Schindelin <[email protected]>:

http://www.google.com/search?q=snprintf+negative+return+value

First link: http://bytes.com/groups/c/590845-snprintf-return-value

Look for "(Windows, mingw)"

Hi,

No, I will not.

This is a _lousy_ time balance you propose: yourself saving a few minutes, everybody spending more time, trying to figure out what your point is, only so that you do not have to summarize in a concise manner why you thing that I am wrong.

I fully disagree with that, and refuse to obey your command, Dscho

2009/4/30 Johannes Schindelin <[email protected]>:

"The following are the results from different systems:
gcc 3.3 (Mac OS X): 10
gcc 3.4.5 (Windows, mingw): -1
BC++ 5.5.1 (Windows): 10
MS VC++ 6.0 and 2005 (Windows, _snprintf() used): -1 DigitalMars 8.42n (Windows): -1"

snprintf always (on all these arechitectures) returns the number of characters written if the buffer is large enough. In the code we are discussing here the buffer is always large enough, so this is not a problem.

- Finn Arne

Your "git diff" calls into the combine-diff logic, and it was not updated when the "gitlink" was added. With your change you are merely hiding the issue under the rug. I've sent a fix as a separate patch.

How about doing this to fix the merge-recursive part? Looks a lot simpler; I haven't tested it very much, though.

-- >8 --
From: Junio C Hamano <[email protected]>
Date: Wed, 29 Apr 2009 11:08:18 -0700
Subject: [PATCH] merge-recursive: do not die on a conflicting submodule

We cannot represent the 3-way conflicted state in the work tree for these entries, but it is normal not to have commit objects for them in our repository. Just update the index and the life will be good.

---
 merge-recursive.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

index d6f0582..a3721ef 100644
@@ -520,8 +520,12 @@ static void update_file_flags(struct merge_options *o,
 		unsigned long size;
 
 		if (S_ISGITLINK(mode))
-			die("cannot read object %s '%s': It is a submodule!",
-			    sha1_to_hex(sha), path);
+			/*
+			 * We may later decide to recursively descend into
+			 * the submodule directory and update its index
+			 * and/or work tree, but we do not do that now.
+			 */
+			goto update_index;
 
 		buf = read_sha1_file(sha, &type, &size);
 		if (!buf)

1.6.3.rc3.16.gb37759