Skip to contentSkip to conversation

msysgit git-submodule: "Unable to fetch in submodule path ..."

PATCH In review Peter Krefting 6 participants 28 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

Hi!

This is with msysGit 1.6.3.2.1299.gee46c

I am having some problem when checking out a large project with many submodules. After cloning the superproject, doing a git submodule --init will fail with a "Unable to fetch in submodule path '...'" for one specific module (does this on all machines that we have tried it on). I cannot see any error messages from git-fetch itself:

[...]
Initialized empty Git repository in c:/Users/peter/src/foo/bar/modules/foo/.git/
remote: Counting objects: 14752, done.
remote: Compressing objects: 100% (5036/5036), done.
remote: Total 14752 (delta 9278), reused 14752 (delta 9278)Receiving objects: 9

Receiving objects: 100% (14752/14752), 5.07 MiB | 3501 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Unable to fetch in submodule path 'modules/foo'

If I enter the modules/foo manually and enter

  git reset --hard

or similar, everything works fine, and I do have the complete history.

Looking at the code for git-submodule, it seems to suggest that git submodule is calling git-fetch without parameters, and checking the return value from it. It does, as indicated above, not seem to return any errors.

I tried adding a "-v -v" to the git-fetch command line in git-submodule, and I see that it does terminate early. With the other modules, I get a list of all the cloned branches and tags. But for this module, it stops as above. I get some additional debug output, but can't quite say it helps me much:

Initialized empty Git repository in c:/Users/peter/src/foo/baz/modules/foo/.git/
remote: Counting objects: 14752, done.
remote: Compressing objects: 100% (5036/5036), done.
emote: Total 14752 (delta 9278), reused 14752 (delta 9278) Receiving objects: 100% (14752/14752), 5.07 MiB | 3629 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Server supports multi_ack
Server supports side-band-64k
Server supports ofs-delta
Marking 76b96bfecc0d47013dd1fca1a555f12074eca814 as complete
Unable to fetch in submodule path 'modules/foo'

The string "Marking %s as complete" seems to stem from mark_recent_complete_commits() in builtin-fetch-pack.c. The other messages seems to stem from do_fetch_pack() in the same file, so it gets there and not further. I cannot seem to find any exit point.

Does anyone know how to continue debugging, or know what might be going wrong?

Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
#1 lore raw

Peter Krefting:

Quoted 7 lines

Receiving objects: 100% (14752/14752), 5.07 MiB | 3629 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Server supports multi_ack
Server supports side-band-64k
Server supports ofs-delta
Marking 76b96bfecc0d47013dd1fca1a555f12074eca814 as complete Unable to fetch in submodule path 'modules/foo'

I added some extra debugging output to the code in builtin-fetch.c and builtin-fetch-pack.c, and ended up with this:

Receiving objects: 100% (14752/14752), 5.07 MiB | 3383 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
cmd_fetch(): calling do_fetch()
do_fetch(): calling get_ref_map()
do_fetch(): get_ref_map() done
do_fetch(): check_not_current_branch() done
do_fetch(): read_ref() loop done
fetch_refs(): quickfetch() returned -10001
Server supports multi_ack
Server supports side-band-64k
Server supports ofs-delta
Entering everything_local()
everything_local() after 1st ref loop
Marking 76b96bfecc0d47013dd1fca1a555f12074eca814 as complete everything_local() after mark_recent_complete_commits() everything_local() after 2nd ref loop
everything_local() after filter_refs() everything_local() done with retval = 1 everything_local() returned true
do_fetch_pack() done
fetch_pack(): calling reprepare_packed_git() fetch_pack(): done
fetch_refs(): transport_fetch_refs() returned -1 do_fetch(): fetch_refs(transport, ref_map) returned non-zero cmd_fetch(): do_fetch() return with exit_code = 1
Unable to fetch in submodule path 'modules/foo'

This seems to indicate that fetch_refs() seems to think that the fetch (which is done over ssh) fails, whereas the regular trace output ("Receiving objects", etc.) indicates that it succeeds.

Is there anything obvious that I should have a look at here?

My next step otherwise is adding trace output to the transport_fetch_refs() and whatever it is it calls and that calls the code in builtin-fetch-pack.c. Anywhere in particular I should have a look at?

Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
#2 lore raw

PATCH 1/1 quickfetch(): Prevent overflow of the rev-list command line

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.

---

Hi,

It seems the git fetch failure described by Peter earlier in this thread is caused by a long ref list overflowing the command line buffer on Windows (32K I am told), when calling rev-list from quickfetch(). AFAICS this overflow will trigger on any fetch from msysgit with more than ~800 (32K / 40) refs.

According to Peter, this patch fixes the submodule update failure.

CC-ing Shawn since he is the original author of quickfetch().

Have fun! :)

...Johan

 builtin-fetch.c |   63 ++++++++++++++++++++++++++++++------------------------
 1 files changed, 35 insertions(+), 28 deletions(-)
builtin-fetch.c +35 -28
index cd5eb9a..52febc6 100644
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 
 /*
  * We would want to bypass the object transfer altogether if
- * everything we are going to fetch already exists and connected
+ * everything we are going to fetch already exists and is connected
  * locally.
  *
- * The refs we are going to fetch are in to_fetch (nr_heads in
- * total).  If running
+ * The refs we are going to fetch are in ref_map.  If running
  *
- *  $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all
+ *  $ git rev-list --objects --stdin --not --all
  *
+ * (feeding all the refs in ref_map on its standard input)
  * does not error out, that means everything reachable from the
  * refs we are going to fetch exists and is connected to some of
  * our existing refs.
@@ -416,9 +416,10 @@ static int quickfetch(struct ref *ref_map)
 {
 	struct child_process revlist;
 	struct ref *ref;
-	char **argv;
-	int i, err;
-
+	int err;
+	const char *argv[] = {
+		"rev-list", "--quiet", "--objects", "--stdin", "--not", "--all", NULL
+	};
 	/*
 	 * If we are deepening a shallow clone we already have these
 	 * objects reachable.  Running rev-list here will return with
@@ -429,34 +430,40 @@ static int quickfetch(struct ref *ref_map)
 	if (depth)
 		return -1;
 
-	for (i = 0, ref = ref_map; ref; ref = ref->next)
-		i++;
-	if (!i)
+	if (!ref_map)
 		return 0;
 
-	argv = xmalloc(sizeof(*argv) * (i + 6));
-	i = 0;
-	argv[i++] = xstrdup("rev-list");
-	argv[i++] = xstrdup("--quiet");
-	argv[i++] = xstrdup("--objects");
-	for (ref = ref_map; ref; ref = ref->next)
-		argv[i++] = xstrdup(sha1_to_hex(ref->old_sha1));
-	argv[i++] = xstrdup("--not");
-	argv[i++] = xstrdup("--all");
-	argv[i++] = NULL;
-
 	memset(&revlist, 0, sizeof(revlist));
-	revlist.argv = (const char**)argv;
+	revlist.argv = argv;
 	revlist.git_cmd = 1;
-	revlist.no_stdin = 1;
 	revlist.no_stdout = 1;
 	revlist.no_stderr = 1;
-	err = run_command(&revlist);
+	revlist.in = -1;
+
+	/* If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below. */
+	signal(SIGPIPE, SIG_IGN);
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			error("failed write to rev-list");
+			err = errno;
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		error("failed to close rev-list's stdin");
+		err = errno;
+	}
+	return finish_command(&revlist) || err;
 }
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
Signature · 1 line
1.6.3.2.316.gda4e
#3 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 444

Quoted 2 lines · Johan Herland
builtin-fetch.c +2 -0
+	/* If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below. */

Under the conditions you describe here...

builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 445

Quoted 1 line
builtin-fetch.c +1 -0
+	signal(SIGPIPE, SIG_IGN);

... and SIGPIPE being ignored...

Quoted 6 lines
builtin-fetch.c +6 -0
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}

builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 457

Quoted 10 lines
builtin-fetch.c +5 -4
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			error("failed write to rev-list");
+			err = errno;

... don't you get this error message with errno set to EPIPE? Previously, there was no error message.

-- Hannes

#4 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 18 lines · Johannes Sixt
Quoted 3 lines · Johan Herland
(quoted diff) +1 -0
+	/* If rev-list --stdin encounters an unknown commit, it

terminates, + * which will cause SIGPIPE in the write loop below.
*/

Under the conditions you describe here...

builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 445

Quoted 1 line
builtin-fetch.c +1 -0
+	signal(SIGPIPE, SIG_IGN);

... and SIGPIPE being ignored...

Quoted 7 lines
builtin-fetch.c +6 -0
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
Quoted 12 lines

Review

Quoted 9 lines
builtin-fetch.c +2 -4
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) <

0 || + write_in_full(revlist.in, "\n", 1) < 0) {

(quoted diff) +2 -0
+			error("failed write to rev-list");
+			err = errno;

... don't you get this error message with errno set to EPIPE? Previously, there was no error message.

Indeed, you are correct. I guess the following should be added to the patch:

if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
write_in_full(revlist.in, "\n", 1) < 0) {
- error("failed write to rev-list");
- err = errno;
+ if (errno != EPIPE) {
+ error("failed write to rev-list");
+ err = errno;
+ }
break;
}

Maybe I need to do something to the close() call as well? What happens on close() after EPIPE?

Thanks,

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#5 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 2 lines · Johan Herland

Maybe I need to do something to the close() call as well? What happens on close() after EPIPE?

You should be OK (you could try this).

-- >8 --
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>

int main(int ac, char **av)

{
	int pipefd[2];
	int child;

	if (pipe(pipefd) < 0) {
		fprintf(stderr, "pipe failed: %s\n", strerror(errno));
		exit(1);
	}
	child = fork();
	if (child < 0) {
		fprintf(stderr, "fork failed: %s\n", strerror(errno));
		exit(1);
	} else if (child == 0) {
		char buf[1024];
		ssize_t sz;

		/* the child reads from the parent but does not talk back */
		close(pipefd[1]);

		/* emulate reading a bit, then dying without cleaning up */
		sz = read(pipefd[0], buf, sizeof(buf));
		fprintf(stderr, "read %lu bytes, and will die\n",
			(unsigned long) sz);
		exit(1);
	} else {
		const char data[] = "abcdefg";
		size_t len = sizeof(data);
		size_t written = 0;

		/* the parent writes to the child but does not listen */
		close(pipefd[0]);

		/* we will rite to the pipe even after the child is gone */
		signal(SIGPIPE, SIG_IGN);

		/* write, write, write, ... */
		while (1) {
			ssize_t sz = write(pipefd[1], data, len);
			if (sz < 0) {
				/* error */
				fprintf(stderr,
					"write failed (%s) after writing"
					" %lu bytes\n",
					strerror(errno),
					(unsigned long) written);
				break;
			}
			written += sz;
		}
		errno = 0;
		if (close(pipefd[1]))
			fprintf(stderr, "close failed: %s\n", strerror(errno));
		else
			fprintf(stderr, "close ok\n");
	}
	exit(0);
}
#6 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 7 lines · Junio C Hamano
Quoted 2 lines · Johan Herland

Maybe I need to do something to the close() call as well? What happens on close() after EPIPE?

You should be OK (you could try this).

-- >8 --

Thanks! The programs works well on my Linux box (close() succeeds), but it does not run at all in Windows/MSYS (lacks pipe() and fork()).

Does anybody with Windows/MSYS experience know how this scenario (write() to a terminated process, followed by close()) would play out in msysGit?

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#7 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 2 lines · Johan Herland

Does anybody with Windows/MSYS experience know how this scenario (write() to a terminated process, followed by close()) would play out in msysGit?

The first write() sometimes fails with EPIPE, otherwise it fails with EINVAL. All subsequent write()s fail with EINVAL. The setting of SIGPIPE is irrelevant because it is unknown to Windows.

There's precedent already in write_or_die.c. You should not write the error message for both EPIPE and EINVAL.

-- Hannes

#8 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 11 lines · Johannes Sixt
Quoted 3 lines · Johan Herland

Does anybody with Windows/MSYS experience know how this scenario (write() to a terminated process, followed by close()) would play out in msysGit?

The first write() sometimes fails with EPIPE, otherwise it fails with EINVAL. All subsequent write()s fail with EINVAL. The setting of SIGPIPE is irrelevant because it is unknown to Windows.

There's precedent already in write_or_die.c. You should not write the error message for both EPIPE and EINVAL.

Thanks, but what about the subsequent close()? Will it fail with EINVAL? EBADF? or will is succeed (like on Linux)?

I will send an updated patch with all fixes, as soon as I know what to do about close().

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#9 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 2 lines · Johan Herland

Thanks, but what about the subsequent close()? Will it fail with EINVAL? EBADF? or will is succeed (like on Linux)?

I guess it succeeds, but I don't have a quick test. Send your patch, and we'll see ;)

-- Hannes

#10 lore raw

PATCH 1/1 quickfetch(): Prevent overflow of the rev-list command line

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.

---
 builtin-fetch.c |   62 +++++++++++++++++++++++++++++++------------------------
 1 files changed, 35 insertions(+), 27 deletions(-)
builtin-fetch.c +35 -27
index cd5eb9a..20bcbdd 100644
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 
 /*
  * We would want to bypass the object transfer altogether if
- * everything we are going to fetch already exists and connected
+ * everything we are going to fetch already exists and is connected
  * locally.
  *
- * The refs we are going to fetch are in to_fetch (nr_heads in
- * total).  If running
+ * The refs we are going to fetch are in ref_map.  If running
  *
- *  $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all
+ *  $ git rev-list --objects --stdin --not --all
  *
+ * (feeding all the refs in ref_map on its standard input)
  * does not error out, that means everything reachable from the
  * refs we are going to fetch exists and is connected to some of
  * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map)
 {
 	struct child_process revlist;
 	struct ref *ref;
-	char **argv;
-	int i, err;
+	int err;
+	const char *argv[] = {"rev-list",
+		"--quiet", "--objects", "--stdin", "--not", "--all", NULL};
 
 	/*
 	 * If we are deepening a shallow clone we already have these
@@ -429,34 +430,41 @@ static int quickfetch(struct ref *ref_map)
 	if (depth)
 		return -1;
 
-	for (i = 0, ref = ref_map; ref; ref = ref->next)
-		i++;
-	if (!i)
+	if (!ref_map)
 		return 0;
 
-	argv = xmalloc(sizeof(*argv) * (i + 6));
-	i = 0;
-	argv[i++] = xstrdup("rev-list");
-	argv[i++] = xstrdup("--quiet");
-	argv[i++] = xstrdup("--objects");
-	for (ref = ref_map; ref; ref = ref->next)
-		argv[i++] = xstrdup(sha1_to_hex(ref->old_sha1));
-	argv[i++] = xstrdup("--not");
-	argv[i++] = xstrdup("--all");
-	argv[i++] = NULL;
-
 	memset(&revlist, 0, sizeof(revlist));
-	revlist.argv = (const char**)argv;
+	revlist.argv = argv;
 	revlist.git_cmd = 1;
-	revlist.no_stdin = 1;
 	revlist.no_stdout = 1;
 	revlist.no_stderr = 1;
-	err = run_command(&revlist);
+	revlist.in = -1;
+
+	/* If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below. */
+	signal(SIGPIPE, SIG_IGN);
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			err = errno;
+			if (err != EPIPE && err != EINVAL)
+				error("failed write to rev-list");
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		err = errno;
+		error("failed to close rev-list's stdin");
+	}
+	return finish_command(&revlist) || err;
 }
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
Signature · 1 line
1.6.3.rc0.1.gf800
#11 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 4 lines · Johan Herland

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.

I removed the "signal(SIGPIPE, SIG_IGN)", but the test suite still passes. IOW, there is no test case that has the configuration that you describe here. Would you please add such a test (perhaps in t5502)? It would also help me verify the patch works as intended on Windows.

Quoted 2 lines

Please make this <[email protected]> despite the email address I'm using right now.

Quoted 8 lines
builtin-fetch.c +5 -0
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			err = errno;
+			if (err != EPIPE && err != EINVAL)

builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 467

Quoted 10 lines
builtin-fetch.c +10 -0
+				error("failed write to rev-list");
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		err = errno;
+		error("failed to close rev-list's stdin");
+	}
+	return finish_command(&revlist) || err;

The call site of quickfetch() is not interested in the errno, only on whether the return value is non-zero: You can just assign -1 to err (that's our convention for failure). OTOH, it would be helpful to include strerror(errno) in the error message.

Shouldn't you reset signal(SIGPIPE) to its previous value?

-- Hannes

#12 lore raw

PATCH v3 1/1 quickfetch(): Prevent overflow of the rev-list command line

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.

The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).

---

Quoted 2 lines · Johannes Sixt

Would you please add such a test (perhaps in t5502)? It would also help me verify the patch works as intended on Windows.

Done (although somewhat naively). I don't have an msysgit setup to test this, but faking the failure condition in quickfetch() (return -1 if #refs > 800) does trigger the selftest (the second git fetch fails).

I could add a separate pre-patch introducing the selftest with test_expect_failure, but that would only confuse non-msysgit users where the test succeeds both before and after the fix.

Quoted 2 lines

Please make this <[email protected]> despite the email address I'm using right now.

Ok.

Quoted 4 lines

The call site of quickfetch() is not interested in the errno, only on whether the return value is non-zero: You can just assign -1 to err (that's our convention for failure). OTOH, it would be helpful to include strerror(errno) in the error message.

Fixed.

Quoted 1 line

Shouldn't you reset signal(SIGPIPE) to its previous value?

Done (provided that the sigchain_push/pop infrastructure works the way I expect).

Thanks a lot for your review and suggestions.

Have fun! :)

...Johan

 builtin-fetch.c       |   65 ++++++++++++++++++++++++++++--------------------
 t/t5502-quickfetch.sh |   20 +++++++++++++++
 2 files changed, 58 insertions(+), 27 deletions(-)
builtin-fetch.c +38 -27
index cd5eb9a..2e3c609 100644
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 
 /*
  * We would want to bypass the object transfer altogether if
- * everything we are going to fetch already exists and connected
+ * everything we are going to fetch already exists and is connected
  * locally.
  *
- * The refs we are going to fetch are in to_fetch (nr_heads in
- * total).  If running
+ * The refs we are going to fetch are in ref_map.  If running
  *
- *  $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all
+ *  $ git rev-list --objects --stdin --not --all
  *
+ * (feeding all the refs in ref_map on its standard input)
  * does not error out, that means everything reachable from the
  * refs we are going to fetch exists and is connected to some of
  * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map)
 {
 	struct child_process revlist;
 	struct ref *ref;
-	char **argv;
-	int i, err;
+	int err;
+	const char *argv[] = {"rev-list",
+		"--quiet", "--objects", "--stdin", "--not", "--all", NULL};
 
 	/*
 	 * If we are deepening a shallow clone we already have these
@@ -429,34 +430,44 @@ static int quickfetch(struct ref *ref_map)
 	if (depth)
 		return -1;
 
-	for (i = 0, ref = ref_map; ref; ref = ref->next)
-		i++;
-	if (!i)
+	if (!ref_map)
 		return 0;
 
-	argv = xmalloc(sizeof(*argv) * (i + 6));
-	i = 0;
-	argv[i++] = xstrdup("rev-list");
-	argv[i++] = xstrdup("--quiet");
-	argv[i++] = xstrdup("--objects");
-	for (ref = ref_map; ref; ref = ref->next)
-		argv[i++] = xstrdup(sha1_to_hex(ref->old_sha1));
-	argv[i++] = xstrdup("--not");
-	argv[i++] = xstrdup("--all");
-	argv[i++] = NULL;
-
 	memset(&revlist, 0, sizeof(revlist));
-	revlist.argv = (const char**)argv;
+	revlist.argv = argv;
 	revlist.git_cmd = 1;
-	revlist.no_stdin = 1;
 	revlist.no_stdout = 1;
 	revlist.no_stderr = 1;
-	err = run_command(&revlist);
+	revlist.in = -1;
+
+	/* If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below. */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
+
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			if (err != EPIPE && err != EINVAL)
+				error("failed write to rev-list: %s", strerror(errno));
+			err = -1;
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		error("failed to close rev-list's stdin: %s", strerror(errno));
+		err = -1;
+	}
+
+	sigchain_pop(SIGPIPE);
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	return finish_command(&revlist) || err;
 }
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
index 16eadd6..1037a72 100755
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' '
 
 '
 
+test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' '
+
+	git gc &&
+	head=$(git rev-parse HEAD) &&
+	branchprefix="$head refs/heads/branch" &&
+	for i in 0 1 2 3 4 5 6 7 8 9; do
+		for j in 0 1 2 3 4 5 6 7 8 9; do
+			for k in 0 1 2 3 4 5 6 7 8 9; do
+				echo "$branchprefix$i$j$k" >> .git/packed-refs
+			done
+		done
+	done &&
+	(
+		cd cloned &&
+		git fetch &&
+		git fetch
+	)
+
+'
+
 test_done
Signature · 1 line
1.6.3.rc0.1.gf800
#13 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

Quoted 5 lines · Johan Herland
Quoted 1 line · Johannes Sixt

Shouldn't you reset signal(SIGPIPE) to its previous value?

Done (provided that the sigchain_push/pop infrastructure works the way I expect).

I'm not sure, either. Peff?

Quoted 9 lines
t/t5502-quickfetch.sh +9 -0
+test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' '
+
+	git gc &&
+	head=$(git rev-parse HEAD) &&
+	branchprefix="$head refs/heads/branch" &&
+	for i in 0 1 2 3 4 5 6 7 8 9; do
+		for j in 0 1 2 3 4 5 6 7 8 9; do
+			for k in 0 1 2 3 4 5 6 7 8 9; do
+				echo "$branchprefix$i$j$k" >> .git/packed-refs

t/t5502-quickfetch.sh test_expect_success 'quickfetch should not copy from alternate' ' · line 140

Quoted 10 lines
t/t5502-quickfetch.sh +10 -0
+			done
+		done
+	done &&
+	(
+		cd cloned &&
+		git fetch &&
+		git fetch
+	)
+
+'

This test fails on Windows without the code change and passes with the code change. So, it's a good test.

But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.

-- Hannes

#14 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

Quoted 8 lines · Johannes Sixt
Quoted 5 lines · Johan Herland
Quoted 1 line · Johannes Sixt

Shouldn't you reset signal(SIGPIPE) to its previous value?

Done (provided that the sigchain_push/pop infrastructure works the way I expect).

I'm not sure, either. Peff?

I don't think I ever tried explicitly pushing SIG_IGN, but the infrastructure was designed so that it would Just Work. So yes, I think it's right, but you may want to test it. :)

That being said, in the patch in question there is an early return after the push that misses the corresponding pop. That should be fixed.

-Peff

#15 lore raw

PATCH v4 1/1 quickfetch(): Prevent overflow of the rev-list command line

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.

The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).

---

Quoted 2 lines · Jeff King

That being said, in the patch in question there is an early return after the push that misses the corresponding pop. That should be fixed.

Thanks, fixed.

...Johan

Signature · 6 lines
Johan Herland, <[email protected]>
www.herland.net

 builtin-fetch.c       |   65 ++++++++++++++++++++++++++++--------------------
 t/t5502-quickfetch.sh |   20 +++++++++++++++
 2 files changed, 58 insertions(+), 27 deletions(-)
builtin-fetch.c +38 -27
index cd5eb9a..34ba878 100644
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 
 /*
  * We would want to bypass the object transfer altogether if
- * everything we are going to fetch already exists and connected
+ * everything we are going to fetch already exists and is connected
  * locally.
  *
- * The refs we are going to fetch are in to_fetch (nr_heads in
- * total).  If running
+ * The refs we are going to fetch are in ref_map.  If running
  *
- *  $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all
+ *  $ git rev-list --objects --stdin --not --all
  *
+ * (feeding all the refs in ref_map on its standard input)
  * does not error out, that means everything reachable from the
  * refs we are going to fetch exists and is connected to some of
  * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map)
 {
 	struct child_process revlist;
 	struct ref *ref;
-	char **argv;
-	int i, err;
+	int err;
+	const char *argv[] = {"rev-list",
+		"--quiet", "--objects", "--stdin", "--not", "--all", NULL};
 
 	/*
 	 * If we are deepening a shallow clone we already have these
@@ -429,34 +430,44 @@ static int quickfetch(struct ref *ref_map)
 	if (depth)
 		return -1;
 
-	for (i = 0, ref = ref_map; ref; ref = ref->next)
-		i++;
-	if (!i)
+	if (!ref_map)
 		return 0;
 
-	argv = xmalloc(sizeof(*argv) * (i + 6));
-	i = 0;
-	argv[i++] = xstrdup("rev-list");
-	argv[i++] = xstrdup("--quiet");
-	argv[i++] = xstrdup("--objects");
-	for (ref = ref_map; ref; ref = ref->next)
-		argv[i++] = xstrdup(sha1_to_hex(ref->old_sha1));
-	argv[i++] = xstrdup("--not");
-	argv[i++] = xstrdup("--all");
-	argv[i++] = NULL;
-
 	memset(&revlist, 0, sizeof(revlist));
-	revlist.argv = (const char**)argv;
+	revlist.argv = argv;
 	revlist.git_cmd = 1;
-	revlist.no_stdin = 1;
 	revlist.no_stdout = 1;
 	revlist.no_stderr = 1;
-	err = run_command(&revlist);
+	revlist.in = -1;
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
+
+	/* If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below. */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			if (err != EPIPE && err != EINVAL)
+				error("failed write to rev-list: %s", strerror(errno));
+			err = -1;
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		error("failed to close rev-list's stdin: %s", strerror(errno));
+		err = -1;
+	}
+
+	sigchain_pop(SIGPIPE);
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	return finish_command(&revlist) || err;
 }
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
index 16eadd6..1037a72 100755
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' '
 
 '
 
+test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' '
+
+	git gc &&
+	head=$(git rev-parse HEAD) &&
+	branchprefix="$head refs/heads/branch" &&
+	for i in 0 1 2 3 4 5 6 7 8 9; do
+		for j in 0 1 2 3 4 5 6 7 8 9; do
+			for k in 0 1 2 3 4 5 6 7 8 9; do
+				echo "$branchprefix$i$j$k" >> .git/packed-refs
+			done
+		done
+	done &&
+	(
+		cd cloned &&
+		git fetch &&
+		git fetch
+	)
+
+'
+
 test_done
Signature · 1 line
1.6.3.rc0.1.gf800
#16 lore raw

Re: [PATCH v4] quickfetch(): Prevent overflow of the rev-list command line

With this fixup the patch is good, I think.

-- Hannes

builtin-fetch.c +5 -3
index 34ba878..7547631 100644
@@ -446,14 +446,16 @@ static int quickfetch(struct ref *ref_map)
 		return err;
 	}
 
-	/* If rev-list --stdin encounters an unknown commit, it terminates,
-	 * which will cause SIGPIPE in the write loop below. */
+	/*
+	 * If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which would cause SIGPIPE in the write loop below.
+	 */
 	sigchain_push(SIGPIPE, SIG_IGN);
 
 	for (ref = ref_map; ref; ref = ref->next) {
 		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
 		    write_in_full(revlist.in, "\n", 1) < 0) {
-			if (err != EPIPE && err != EINVAL)
+			if (errno != EPIPE && errno != EINVAL)
 				error("failed write to rev-list: %s", strerror(errno));
 			err = -1;
 			break;
#17 lore raw

PATCH v5 1/1 quickfetch(): Prevent overflow of the rev-list command line

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.

The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).

---

Quoted 1 line · Johannes Sixt

With this fixup the patch is good, I think.

Again, thanks for all your help!

Here's the final (*crossing fingers*) iteration of the patch.

Have fun! :)

...Johan

 builtin-fetch.c       |   67 +++++++++++++++++++++++++++++-------------------
 t/t5502-quickfetch.sh |   20 ++++++++++++++
 2 files changed, 60 insertions(+), 27 deletions(-)
builtin-fetch.c +40 -27
index cd5eb9a..817dd6b 100644
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 
 /*
  * We would want to bypass the object transfer altogether if
- * everything we are going to fetch already exists and connected
+ * everything we are going to fetch already exists and is connected
  * locally.
  *
- * The refs we are going to fetch are in to_fetch (nr_heads in
- * total).  If running
+ * The refs we are going to fetch are in ref_map.  If running
  *
- *  $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all
+ *  $ git rev-list --objects --stdin --not --all
  *
+ * (feeding all the refs in ref_map on its standard input)
  * does not error out, that means everything reachable from the
  * refs we are going to fetch exists and is connected to some of
  * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map)
 {
 	struct child_process revlist;
 	struct ref *ref;
-	char **argv;
-	int i, err;
+	int err;
+	const char *argv[] = {"rev-list",
+		"--quiet", "--objects", "--stdin", "--not", "--all", NULL};
 
 	/*
 	 * If we are deepening a shallow clone we already have these
@@ -429,34 +430,46 @@ static int quickfetch(struct ref *ref_map)
 	if (depth)
 		return -1;
 
-	for (i = 0, ref = ref_map; ref; ref = ref->next)
-		i++;
-	if (!i)
+	if (!ref_map)
 		return 0;
 
-	argv = xmalloc(sizeof(*argv) * (i + 6));
-	i = 0;
-	argv[i++] = xstrdup("rev-list");
-	argv[i++] = xstrdup("--quiet");
-	argv[i++] = xstrdup("--objects");
-	for (ref = ref_map; ref; ref = ref->next)
-		argv[i++] = xstrdup(sha1_to_hex(ref->old_sha1));
-	argv[i++] = xstrdup("--not");
-	argv[i++] = xstrdup("--all");
-	argv[i++] = NULL;
-
 	memset(&revlist, 0, sizeof(revlist));
-	revlist.argv = (const char**)argv;
+	revlist.argv = argv;
 	revlist.git_cmd = 1;
-	revlist.no_stdin = 1;
 	revlist.no_stdout = 1;
 	revlist.no_stderr = 1;
-	err = run_command(&revlist);
+	revlist.in = -1;
+
+	err = start_command(&revlist);
+	if (err) {
+		error("could not run rev-list");
+		return err;
+	}
+
+	/*
+	 * If rev-list --stdin encounters an unknown commit, it terminates,
+	 * which will cause SIGPIPE in the write loop below.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	for (ref = ref_map; ref; ref = ref->next) {
+		if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
+		    write_in_full(revlist.in, "\n", 1) < 0) {
+			if (errno != EPIPE && errno != EINVAL)
+				error("failed write to rev-list: %s", strerror(errno));
+			err = -1;
+			break;
+		}
+	}
+
+	if (close(revlist.in)) {
+		error("failed to close rev-list's stdin: %s", strerror(errno));
+		err = -1;
+	}
+
+	sigchain_pop(SIGPIPE);
 
-	for (i = 0; argv[i]; i++)
-		free(argv[i]);
-	free(argv);
-	return err;
+	return finish_command(&revlist) || err;
 }
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
index 16eadd6..1037a72 100755
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' '
 
 '
 
+test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' '
+
+	git gc &&
+	head=$(git rev-parse HEAD) &&
+	branchprefix="$head refs/heads/branch" &&
+	for i in 0 1 2 3 4 5 6 7 8 9; do
+		for j in 0 1 2 3 4 5 6 7 8 9; do
+			for k in 0 1 2 3 4 5 6 7 8 9; do
+				echo "$branchprefix$i$j$k" >> .git/packed-refs
+			done
+		done
+	done &&
+	(
+		cd cloned &&
+		git fetch &&
+		git fetch
+	)
+
+'
+
 test_done
Signature · 1 line
1.6.3.rc0.1.gf800
#18 lore raw

Re: [PATCH v5] quickfetch(): Prevent overflow of the rev-list command line

Quoted 11 lines · Johan Herland

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

I feel uneasy with that "somehow" at the end of the first paragraph, but nevertheless this is the right thing to do. Since it is a very isolated change, I'd queue this directly on 'master' and see if anybody notices a breakage, as it would be relatively pain-free to revert if it turns out to be necessary.

Thanks.

#19 lore raw

Re: [PATCH v5] quickfetch(): Prevent overflow of the rev-list command line

Quoted 10 lines · Junio C Hamano
Quoted 9 lines · Johan Herland

quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.

Quoted 7 lines
Quoted 4 lines

By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.

I feel uneasy with that "somehow" at the end of the first paragraph, but nevertheless this is the right thing to do.

Yes, it feels wrong that transport_fetch_refs() returns error when there are no objects to be fetched. After all, the quickfetch() routine is only meant to be an optimization (to skip the object fetching when not needed). Only if quickfetch() returned a false positive (indicating that all objects are present when they're really not) would I expect it to have adverse effects on the result of the fetch. But even then, as you indicate, fixing quickfetch() itself is the right thing to do, and looking into transport_fetch_refs() is a separate issue.

Quoted 4 lines

Since it is a very isolated
change, I'd queue this directly on 'master' and see if anybody notices a breakage, as it would be relatively pain-free to revert if it turns out to be necessary.

Thanks.

Have fun! :)

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#20 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

In reply to Johannes Sixt
Quoted 3 lines · Johannes Sixt

But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.

AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#21 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

Quoted 10 lines · Johan Herland
Quoted 3 lines · Johannes Sixt

But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.

AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.

I know that, of course. But try this: Remove the signal(SIGPIPE, SIG_IGN) and run the test suite. There is not a single failure. IOW, we don't have a single test case that verifies that the signal(SIGPIPE, SIG_IGN) is needed. I would like to have that test case, and you seem to know how to construct it (otherwise there wouldn't be so much buzz about it).

-- Hannes

#22 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

Quoted 13 lines · Johannes Sixt
Quoted 11 lines · Johan Herland
Quoted 3 lines · Johannes Sixt

But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.

AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.

Quoted 2 lines

I know that, of course. But try this: Remove the signal(SIGPIPE, SIG_IGN) and run the test suite. There is not a single failure.

That's not what I'm seeing. When I don't ignore the signal, the testsuite fails intermittently for me (on Linux). I see the following tests fail:

- t3409-rebase-preserve-merges.sh (subtest #2)
- t5503-tagfollow.sh (subtests #4, #6, #7)
- t5505-remote.sh (subtests #10, #12, #14 - #20, #27)
- t5510-fetch.sh (subtest #6 or #25)
- probably more (I seldom get this far...)

I assume the intermittent failures are caused by git rev-list sometimes terminate before git fetch is finished writing objects to its standard input (because of scheduling differences).

When i enable the signal handling, all selftests pass every time.

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#23 lore raw

Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line

Quoted 11 lines · Johan Herland
Quoted 2 lines · Johannes Sixt

But try this: Remove the signal(SIGPIPE, SIG_IGN) and run the test suite. There is not a single failure.

That's not what I'm seeing. When I don't ignore the signal, the testsuite fails intermittently for me (on Linux). I see the following tests fail:

- t3409-rebase-preserve-merges.sh (subtest #2)
- t5503-tagfollow.sh (subtests #4, #6, #7)
- t5505-remote.sh (subtests #10, #12, #14 - #20, #27)
- t5510-fetch.sh (subtest #6 or #25)

I see. If I insert sched_yield() in the for loop, I see many failures (not 100% reproducible, but almost). On Windows, I have to insert Sleep(10) so that the error exit is taken, and the error code is EINVAL :-/

-- Hannes

#24 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

In reply to Johan Herland
Quoted 8 lines · Johan Herland
Quoted 3 lines · Junio C Hamano
Quoted 2 lines · Johan Herland

Maybe I need to do something to the close() call as well? What happens on close() after EPIPE?

Does anybody with Windows/MSYS experience know how this scenario (write() to a terminated process, followed by close()) would play out in msysGit?

It fails with ERROR_BROKEN_PIPE. See MSDN for WriteFile:

http://msdn.microsoft.com/en-us/library/aa365747%28VS.85%29.aspx

(look for the error above).

Well, sometimes it just fails, so you can hardly use the error code to detect if the other process is truly gone or something broke in Windows.

#25 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

In reply to Johan Herland
Quoted 7 lines · Johan Herland
Quoted 2 lines · Johannes Sixt

... don't you get this error message with errno set to EPIPE? Previously, there was no error message.

Indeed, you are correct. I guess the following should be added to the patch:

Quoted 7 lines

       if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 || write_in_full(revlist.in, "\n", 1) < 0) {
-               error("failed write to rev-list");
-               err = errno;
+               if (errno != EPIPE) {
+                       error("failed write to rev-list");
+                       err = errno;

You'll loose errno this way: error() does not save it.

#26 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 16 lines · Alex Riesen
Quoted 9 lines · Johan Herland
Quoted 2 lines · Johannes Sixt

... don't you get this error message with errno set to EPIPE? Previously, there was no error message.

Indeed, you are correct. I guess the following should be added to the patch:

       if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) <

0 || write_in_full(revlist.in, "\n", 1) < 0) {

Review

Quoted 5 lines
(quoted diff) +3 -2
-               error("failed write to rev-list");
-               err = errno;
+               if (errno != EPIPE) {
+                       error("failed write to rev-list");
+                       err = errno;
Quoted 1 line

You'll loose errno this way: error() does not save it.

Not sure what you mean here. Should I move "err = errno;" outside the innermost "if"?

Quoted 1 line

From my POV, if errno != EPIPE, we save it into err, and return that

(overridden by finish_command()'s return value, if non-zero). If errno == EPIPE, we're not interested in saving it, because we expect finish_command() to return non-zero in any case.

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#27 lore raw

Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line

Quoted 17 lines · Johan Herland
Quoted 16 lines · Alex Riesen
Quoted 14 lines · Johan Herland
Quoted 2 lines · Johannes Sixt

... don't you get this error message with errno set to EPIPE? Previously, there was no error message.

Indeed, you are correct. I guess the following should be added to the patch:

       if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 || write_in_full(revlist.in, "\n", 1) < 0) {
-               error("failed write to rev-list");
-               err = errno;
+               if (errno != EPIPE) {
+                       error("failed write to rev-list");
+                       err = errno;

Quoted 4 lines
Quoted 1 line

You'll loose errno this way: error() does not save it.

Not sure what you mean here. Should I move "err = errno;" outside the innermost "if"?

put it before error("failed write to rev-list"); or even before the "if (err != EPIPE)".
Otherwise it is 0 after fprintf to stderr (which is the error() call).

Quoted 4 lines

From my POV, if errno != EPIPE, we save it into err, and return that (overridden by finish_command()'s return value, if non-zero). If errno == EPIPE, we're not interested in saving it, because we expect finish_command() to return non-zero in any case.

And you think this expectation makes the code simpler to understand?