msysgit git-submodule: "Unable to fetch in submodule path ..."
Hi!
This is with msysGit 1.6.3.2.1299.gee46c
I am having some problem when checking out a large project with many submodules. After cloning the superproject, doing a git submodule --init will fail with a "Unable to fetch in submodule path '...'" for one specific module (does this on all machines that we have tried it on). I cannot see any error messages from git-fetch itself:
[...]
Initialized empty Git repository in c:/Users/peter/src/foo/bar/modules/foo/.git/
remote: Counting objects: 14752, done.
remote: Compressing objects: 100% (5036/5036), done.
remote: Total 14752 (delta 9278), reused 14752 (delta 9278)Receiving objects: 9
Receiving objects: 100% (14752/14752), 5.07 MiB | 3501 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Unable to fetch in submodule path 'modules/foo'
If I enter the modules/foo manually and enter
git reset --hard
or similar, everything works fine, and I do have the complete history.
Looking at the code for git-submodule, it seems to suggest that git submodule is calling git-fetch without parameters, and checking the return value from it. It does, as indicated above, not seem to return any errors.
I tried adding a "-v -v" to the git-fetch command line in git-submodule, and I see that it does terminate early. With the other modules, I get a list of all the cloned branches and tags. But for this module, it stops as above. I get some additional debug output, but can't quite say it helps me much:
Initialized empty Git repository in c:/Users/peter/src/foo/baz/modules/foo/.git/
remote: Counting objects: 14752, done.
remote: Compressing objects: 100% (5036/5036), done.
emote: Total 14752 (delta 9278), reused 14752 (delta 9278) Receiving objects: 100% (14752/14752), 5.07 MiB | 3629 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Server supports multi_ack
Server supports side-band-64k
Server supports ofs-delta
Marking 76b96bfecc0d47013dd1fca1a555f12074eca814 as complete
Unable to fetch in submodule path 'modules/foo'
The string "Marking %s as complete" seems to stem from mark_recent_complete_commits() in builtin-fetch-pack.c. The other messages seems to stem from do_fetch_pack() in the same file, so it gets there and not further. I cannot seem to find any exit point.
Does anyone know how to continue debugging, or know what might be going wrong?
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
Peter Krefting:
Quoted 7 lines
Receiving objects: 100% (14752/14752), 5.07 MiB | 3629 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
Server supportsmulti_ack
Server supports side-band-64k
Server supports ofs-delta
Marking76b96bfecc0d47013dd1fca1a555f12074eca814as complete Unable to fetch in submodule path 'modules/foo'
I added some extra debugging output to the code in builtin-fetch.c and builtin-fetch-pack.c, and ended up with this:
Receiving objects: 100% (14752/14752), 5.07 MiB | 3383 KiB/s, done. Resolving deltas: 100% (9278/9278), done.
cmd_fetch(): calling do_fetch()
do_fetch(): calling get_ref_map()
do_fetch(): get_ref_map() done
do_fetch(): check_not_current_branch() done
do_fetch(): read_ref() loop done
fetch_refs(): quickfetch() returned -10001
Server supports multi_ack
Server supports side-band-64k
Server supports ofs-delta
Entering everything_local()
everything_local() after 1st ref loop
Marking 76b96bfecc0d47013dd1fca1a555f12074eca814 as complete everything_local() after mark_recent_complete_commits() everything_local() after 2nd ref loop
everything_local() after filter_refs() everything_local() done with retval = 1 everything_local() returned true
do_fetch_pack() done
fetch_pack(): calling reprepare_packed_git() fetch_pack(): done
fetch_refs(): transport_fetch_refs() returned -1 do_fetch(): fetch_refs(transport, ref_map) returned non-zero cmd_fetch(): do_fetch() return with exit_code = 1
Unable to fetch in submodule path 'modules/foo'
This seems to indicate that fetch_refs() seems to think that the fetch (which is done over ssh) fails, whereas the regular trace output ("Receiving objects", etc.) indicates that it succeeds.
Is there anything obvious that I should have a look at here?
My next step otherwise is adding trace output to the transport_fetch_refs() and whatever it is it calls and that calls the code in builtin-fetch-pack.c. Anywhere in particular I should have a look at?
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
PATCH 1/1 quickfetch(): Prevent overflow of the rev-list command line
quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.
---
Hi,
It seems the git fetch failure described by Peter earlier in this thread is caused by a long ref list overflowing the command line buffer on Windows (32K I am told), when calling rev-list from quickfetch(). AFAICS this overflow will trigger on any fetch from msysgit with more than ~800 (32K / 40) refs.
According to Peter, this patch fixes the submodule update failure.
CC-ing Shawn since he is the original author of quickfetch().
Have fun! :)
...Johan
builtin-fetch.c | 63 ++++++++++++++++++++++++++++++------------------------ 1 files changed, 35 insertions(+), 28 deletions(-)
builtin-fetch.c +35 -28
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,/* * We would want to bypass the object transfer altogether if -* everything we are going to fetch already exists and connected +* everything we are going to fetch already exists and is connected * locally. * -* The refs we are going to fetch are in to_fetch (nr_heads in -* total). If running +* The refs we are going to fetch are in ref_map. If running * -* $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all +* $ git rev-list --objects --stdin --not --all * +* (feeding all the refs in ref_map on its standard input) * does not error out, that means everything reachable from the * refs we are going to fetch exists and is connected to some of * our existing refs.
@@ -416,9 +416,10 @@ static int quickfetch(struct ref *ref_map) {struct child_process revlist;struct ref *ref; -char **argv; -int i, err; - +int err; +const char *argv[] = { +"rev-list" ,"--quiet" ,"--objects" ,"--stdin" ,"--not" ,"--all" ,NULL + };/* * If we are deepening a shallow clone we already have these * objects reachable. Running rev-list here will return with
@@ -429,34 +430,40 @@ static int quickfetch(struct ref *ref_map)if (depth)return -1 ; -for (i =0 , ref = ref_map; ref; ref = ref->next) - i++; -if (!i) +if (!ref_map)return 0 ; - argv =xmalloc (sizeof (*argv) * (i +6 )); - i =0 ; - argv[i++] =xstrdup ("rev-list" ); - argv[i++] =xstrdup ("--quiet" ); - argv[i++] =xstrdup ("--objects" ); -for (ref = ref_map; ref; ref = ref->next) - argv[i++] =xstrdup (sha1_to_hex (ref->old_sha1)); - argv[i++] =xstrdup ("--not" ); - argv[i++] =xstrdup ("--all" ); - argv[i++] =NULL ; -memset (&revlist,0 ,sizeof (revlist)); - revlist.argv = (const char **)argv; + revlist.argv = argv; revlist.git_cmd =1 ; - revlist.no_stdin =1 ; revlist.no_stdout =1 ; revlist.no_stderr =1 ; - err =run_command (&revlist); + revlist.in =-1 ; + +/* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. */ +signal (SIGPIPE ,SIG_IGN ); + + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + } -for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { +error ("failed write to rev-list" ); + err = errno; +break ; + } + } + +if (close (revlist.in)) { +error ("failed to close rev-list's stdin" ); + err = errno; + } +return finish_command (&revlist) || err; }static int fetch_refs (struct transport *transport,struct ref *ref_map)
Signature · 1 line
1.6.3.2.316.gda4e
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 444
Quoted 2 lines · Johan Herland
builtin-fetch.c +2
-0+/* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. */
Under the conditions you describe here...
builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 445
Quoted 1 line
builtin-fetch.c +1
-0+signal (SIGPIPE ,SIG_IGN );
... and SIGPIPE being ignored...
Quoted 6 lines
builtin-fetch.c +6
-0+ + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + }
builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 457
Quoted 10 lines
builtin-fetch.c +5
-4-for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { +error ("failed write to rev-list" ); + err = errno;
... don't you get this error message with errno set to EPIPE? Previously, there was no error message.
-- Hannes
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 18 lines · Johannes Sixt
Quoted 3 lines · Johan Herland
(quoted diff) +1
-0+ /* If rev-list --stdin encounters an unknown commit, itterminates, + * which will cause SIGPIPE in the write loop below.
*/Under the conditions you describe here...
builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 445
Quoted 1 line
builtin-fetch.c +1
-0+signal (SIGPIPE ,SIG_IGN );... and SIGPIPE being ignored...
Quoted 7 lines
builtin-fetch.c +6
-0+ + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + }
Quoted 12 lines
Review
Quoted 9 lines
builtin-fetch.c +2
-4-for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +for (ref = ref_map; ref; ref = ref->next) { + if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) <0 || +
write_in_full(revlist.in, "\n", 1)< 0) {(quoted diff) +2
-0+ error("failed write to rev-list"); + err = errno;... don't you get this error message with errno set to EPIPE? Previously, there was no error message.
Indeed, you are correct. I guess the following should be added to the patch:
if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||
write_in_full(revlist.in, "\n", 1) < 0) {
- error("failed write to rev-list");
- err = errno;
+ if (errno != EPIPE) {
+ error("failed write to rev-list");
+ err = errno;
+ }
break;
}
Maybe I need to do something to the close() call as well? What happens on close() after EPIPE?
Thanks,
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 2 lines · Johan Herland
Maybe I need to do something to the
close()call as well? What happens onclose()after EPIPE?
You should be OK (you could try this).
-- >8 --
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>
int main(int ac, char **av)
{
int pipefd[2];
int child;
if (pipe(pipefd) < 0) {
fprintf(stderr, "pipe failed: %s\n", strerror(errno));
exit(1);
}
child = fork();
if (child < 0) {
fprintf(stderr, "fork failed: %s\n", strerror(errno));
exit(1);
} else if (child == 0) {
char buf[1024];
ssize_t sz;
/* the child reads from the parent but does not talk back */
close(pipefd[1]);
/* emulate reading a bit, then dying without cleaning up */
sz = read(pipefd[0], buf, sizeof(buf));
fprintf(stderr, "read %lu bytes, and will die\n",
(unsigned long) sz);
exit(1);
} else {
const char data[] = "abcdefg";
size_t len = sizeof(data);
size_t written = 0;
/* the parent writes to the child but does not listen */
close(pipefd[0]);
/* we will rite to the pipe even after the child is gone */
signal(SIGPIPE, SIG_IGN);
/* write, write, write, ... */
while (1) {
ssize_t sz = write(pipefd[1], data, len);
if (sz < 0) {
/* error */
fprintf(stderr,
"write failed (%s) after writing"
" %lu bytes\n",
strerror(errno),
(unsigned long) written);
break;
}
written += sz;
}
errno = 0;
if (close(pipefd[1]))
fprintf(stderr, "close failed: %s\n", strerror(errno));
else
fprintf(stderr, "close ok\n");
}
exit(0);
}Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 7 lines · Junio C Hamano
Quoted 2 lines · Johan Herland
Maybe I need to do something to the
close()call as well? What happens onclose()after EPIPE?You should be OK (you could try this).
-- >8 --
Thanks! The programs works well on my Linux box (close() succeeds), but it does not run at all in Windows/MSYS (lacks pipe() and fork()).
Does anybody with Windows/MSYS experience know how this scenario (write() to a terminated process, followed by close()) would play out in msysGit?
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 2 lines · Johan Herland
Does anybody with Windows/MSYS experience know how this scenario (
write()to a terminated process, followed byclose()) would play out in msysGit?
The first write() sometimes fails with EPIPE, otherwise it fails with EINVAL. All subsequent write()s fail with EINVAL. The setting of SIGPIPE is irrelevant because it is unknown to Windows.
There's precedent already in write_or_die.c. You should not write the error message for both EPIPE and EINVAL.
-- Hannes
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 11 lines · Johannes Sixt
Quoted 3 lines · Johan Herland
Does anybody with Windows/MSYS experience know how this scenario (
write()to a terminated process, followed byclose()) would play out in msysGit?The first
write()sometimes fails with EPIPE, otherwise it fails with EINVAL. All subsequent write()s fail with EINVAL. The setting of SIGPIPE is irrelevant because it is unknown to Windows.There's precedent already in
write_or_die.c. You should not write the error message for both EPIPE and EINVAL.
Thanks, but what about the subsequent close()? Will it fail with EINVAL? EBADF? or will is succeed (like on Linux)?
I will send an updated patch with all fixes, as soon as I know what to do about close().
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 2 lines · Johan Herland
Thanks, but what about the subsequent
close()? Will it fail with EINVAL? EBADF? or will is succeed (like on Linux)?
I guess it succeeds, but I don't have a quick test. Send your patch, and we'll see ;)
-- Hannes
PATCH 1/1 quickfetch(): Prevent overflow of the rev-list command line
quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.
--- builtin-fetch.c | 62 +++++++++++++++++++++++++++++++------------------------ 1 files changed, 35 insertions(+), 27 deletions(-)
builtin-fetch.c +35 -27
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,/* * We would want to bypass the object transfer altogether if -* everything we are going to fetch already exists and connected +* everything we are going to fetch already exists and is connected * locally. * -* The refs we are going to fetch are in to_fetch (nr_heads in -* total). If running +* The refs we are going to fetch are in ref_map. If running * -* $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all +* $ git rev-list --objects --stdin --not --all * +* (feeding all the refs in ref_map on its standard input) * does not error out, that means everything reachable from the * refs we are going to fetch exists and is connected to some of * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map) {struct child_process revlist;struct ref *ref; -char **argv; -int i, err; +int err; +const char *argv[] = {"rev-list" , +"--quiet" ,"--objects" ,"--stdin" ,"--not" ,"--all" ,NULL };/* * If we are deepening a shallow clone we already have these
@@ -429,34 +430,41 @@ static int quickfetch(struct ref *ref_map)if (depth)return -1 ; -for (i =0 , ref = ref_map; ref; ref = ref->next) - i++; -if (!i) +if (!ref_map)return 0 ; - argv =xmalloc (sizeof (*argv) * (i +6 )); - i =0 ; - argv[i++] =xstrdup ("rev-list" ); - argv[i++] =xstrdup ("--quiet" ); - argv[i++] =xstrdup ("--objects" ); -for (ref = ref_map; ref; ref = ref->next) - argv[i++] =xstrdup (sha1_to_hex (ref->old_sha1)); - argv[i++] =xstrdup ("--not" ); - argv[i++] =xstrdup ("--all" ); - argv[i++] =NULL ; -memset (&revlist,0 ,sizeof (revlist)); - revlist.argv = (const char **)argv; + revlist.argv = argv; revlist.git_cmd =1 ; - revlist.no_stdin =1 ; revlist.no_stdout =1 ; revlist.no_stderr =1 ; - err =run_command (&revlist); + revlist.in =-1 ; + +/* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. */ +signal (SIGPIPE ,SIG_IGN ); + + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + } -for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { + err = errno; +if (err !=EPIPE && err !=EINVAL ) +error ("failed write to rev-list" ); +break ; + } + } + +if (close (revlist.in)) { + err = errno; +error ("failed to close rev-list's stdin" ); + } +return finish_command (&revlist) || err; }static int fetch_refs (struct transport *transport,struct ref *ref_map)
Signature · 1 line
1.6.3.rc0.1.gf800
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 4 lines · Johan Herland
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore ignore SIGPIPE so that the fetch process is not terminated.
I removed the "signal(SIGPIPE, SIG_IGN)", but the test suite still passes. IOW, there is no test case that has the configuration that you describe here. Would you please add such a test (perhaps in t5502)? It would also help me verify the patch works as intended on Windows.
Quoted 2 lines
Signed-off-by Johan Herland <[email protected]>Improved-by Johannes Sixt <[email protected]>
Please make this <[email protected]> despite the email address I'm using right now.
Quoted 8 lines
Improved-by Alex Riesen <[email protected]>Tested-by Peter Krefting <[email protected]>builtin-fetch.c +5
-0+for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { + err = errno; +if (err !=EPIPE && err !=EINVAL )
builtin-fetch.c static int quickfetch(struct ref *ref_map) · line 467
Quoted 10 lines
builtin-fetch.c +10
-0+error ("failed write to rev-list" ); +break ; + } + } + +if (close (revlist.in)) { + err = errno; +error ("failed to close rev-list's stdin" ); + } +return finish_command (&revlist) || err;
The call site of quickfetch() is not interested in the errno, only on whether the return value is non-zero: You can just assign -1 to err (that's our convention for failure). OTOH, it would be helpful to include strerror(errno) in the error message.
Shouldn't you reset signal(SIGPIPE) to its previous value?
-- Hannes
PATCH v3 1/1 quickfetch(): Prevent overflow of the rev-list command line
quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.
The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).
---
Quoted 2 lines · Johannes Sixt
Would you please add such a test (perhaps in t5502)? It would also help me verify the patch works as intended on Windows.
Done (although somewhat naively). I don't have an msysgit setup to test this, but faking the failure condition in quickfetch() (return -1 if #refs > 800) does trigger the selftest (the second git fetch fails).
I could add a separate pre-patch introducing the selftest with test_expect_failure, but that would only confuse non-msysgit users where the test succeeds both before and after the fix.
Quoted 2 lines
Please make this <[email protected]> despite the email address I'm using right now.
Ok.
Quoted 4 lines
The call site of
quickfetch()is not interested in the errno, only on whether the return value is non-zero: You can just assign -1 to err (that's our convention for failure). OTOH, it would be helpful to include strerror(errno) in the error message.
Fixed.
Quoted 1 line
Shouldn't you reset signal(SIGPIPE) to its previous value?
Done (provided that the sigchain_push/pop infrastructure works the way I expect).
Thanks a lot for your review and suggestions.
Have fun! :)
...Johan
builtin-fetch.c | 65 ++++++++++++++++++++++++++++-------------------- t/t5502-quickfetch.sh | 20 +++++++++++++++ 2 files changed, 58 insertions(+), 27 deletions(-)
builtin-fetch.c +38 -27
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,/* * We would want to bypass the object transfer altogether if -* everything we are going to fetch already exists and connected +* everything we are going to fetch already exists and is connected * locally. * -* The refs we are going to fetch are in to_fetch (nr_heads in -* total). If running +* The refs we are going to fetch are in ref_map. If running * -* $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all +* $ git rev-list --objects --stdin --not --all * +* (feeding all the refs in ref_map on its standard input) * does not error out, that means everything reachable from the * refs we are going to fetch exists and is connected to some of * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map) {struct child_process revlist;struct ref *ref; -char **argv; -int i, err; +int err; +const char *argv[] = {"rev-list" , +"--quiet" ,"--objects" ,"--stdin" ,"--not" ,"--all" ,NULL };/* * If we are deepening a shallow clone we already have these
@@ -429,34 +430,44 @@ static int quickfetch(struct ref *ref_map)if (depth)return -1 ; -for (i =0 , ref = ref_map; ref; ref = ref->next) - i++; -if (!i) +if (!ref_map)return 0 ; - argv =xmalloc (sizeof (*argv) * (i +6 )); - i =0 ; - argv[i++] =xstrdup ("rev-list" ); - argv[i++] =xstrdup ("--quiet" ); - argv[i++] =xstrdup ("--objects" ); -for (ref = ref_map; ref; ref = ref->next) - argv[i++] =xstrdup (sha1_to_hex (ref->old_sha1)); - argv[i++] =xstrdup ("--not" ); - argv[i++] =xstrdup ("--all" ); - argv[i++] =NULL ; -memset (&revlist,0 ,sizeof (revlist)); - revlist.argv = (const char **)argv; + revlist.argv = argv; revlist.git_cmd =1 ; - revlist.no_stdin =1 ; revlist.no_stdout =1 ; revlist.no_stderr =1 ; - err =run_command (&revlist); + revlist.in =-1 ; + +/* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. */ +sigchain_push (SIGPIPE ,SIG_IGN ); + + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + } + +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { +if (err !=EPIPE && err !=EINVAL ) +error ("failed write to rev-list: %s" ,strerror (errno)); + err =-1 ; +break ; + } + } + +if (close (revlist.in)) { +error ("failed to close rev-list's stdin: %s" ,strerror (errno)); + err =-1 ; + } + +sigchain_pop (SIGPIPE ); -for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +return finish_command (&revlist) || err; }static int fetch_refs (struct transport *transport,struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' ' ' +test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' ' + + git gc && + head=$(git rev-parse HEAD) && + branchprefix="$head refs/heads/branch" && + for i in 0 1 2 3 4 5 6 7 8 9; do + for j in 0 1 2 3 4 5 6 7 8 9; do + for k in 0 1 2 3 4 5 6 7 8 9; do + echo "$branchprefix$i$j$k" >> .git/packed-refs + done + done + done && + ( + cd cloned && + git fetch && + git fetch + ) + +' + test_done
Signature · 1 line
1.6.3.rc0.1.gf800
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
Quoted 5 lines · Johan Herland
Quoted 1 line · Johannes Sixt
Shouldn't you reset signal(SIGPIPE) to its previous value?
Done (provided that the sigchain_push/pop infrastructure works the way I expect).
I'm not sure, either. Peff?
Quoted 9 lines
t/t5502-quickfetch.sh +9
-0+test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' ' + + git gc && + head=$(git rev-parse HEAD) && + branchprefix="$head refs/heads/branch" && + for i in 0 1 2 3 4 5 6 7 8 9; do + for j in 0 1 2 3 4 5 6 7 8 9; do + for k in 0 1 2 3 4 5 6 7 8 9; do + echo "$branchprefix$i$j$k" >> .git/packed-refs
t/t5502-quickfetch.sh test_expect_success 'quickfetch should not copy from alternate' ' · line 140
Quoted 10 lines
t/t5502-quickfetch.sh +10
-0+ done + done + done && + ( + cd cloned && + git fetch && + git fetch + ) + +'
This test fails on Windows without the code change and passes with the code change. So, it's a good test.
But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.
-- Hannes
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
Quoted 8 lines · Johannes Sixt
Quoted 5 lines · Johan Herland
Quoted 1 line · Johannes Sixt
Shouldn't you reset signal(SIGPIPE) to its previous value?
Done (provided that the sigchain_push/pop infrastructure works the way I expect).
I'm not sure, either. Peff?
I don't think I ever tried explicitly pushing SIG_IGN, but the infrastructure was designed so that it would Just Work. So yes, I think it's right, but you may want to test it. :)
That being said, in the patch in question there is an early return after the push that misses the corresponding pop. That should be fixed.
-Peff
PATCH v4 1/1 quickfetch(): Prevent overflow of the rev-list command line
quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.
The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).
---
Quoted 2 lines · Jeff King
That being said, in the patch in question there is an early return after the push that misses the corresponding pop. That should be fixed.
Thanks, fixed.
...Johan
Signature · 6 lines
Johan Herland, <[email protected]> www.herland.net builtin-fetch.c | 65 ++++++++++++++++++++++++++++-------------------- t/t5502-quickfetch.sh | 20 +++++++++++++++ 2 files changed, 58 insertions(+), 27 deletions(-)
builtin-fetch.c +38 -27
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,/* * We would want to bypass the object transfer altogether if -* everything we are going to fetch already exists and connected +* everything we are going to fetch already exists and is connected * locally. * -* The refs we are going to fetch are in to_fetch (nr_heads in -* total). If running +* The refs we are going to fetch are in ref_map. If running * -* $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all +* $ git rev-list --objects --stdin --not --all * +* (feeding all the refs in ref_map on its standard input) * does not error out, that means everything reachable from the * refs we are going to fetch exists and is connected to some of * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map) {struct child_process revlist;struct ref *ref; -char **argv; -int i, err; +int err; +const char *argv[] = {"rev-list" , +"--quiet" ,"--objects" ,"--stdin" ,"--not" ,"--all" ,NULL };/* * If we are deepening a shallow clone we already have these
@@ -429,34 +430,44 @@ static int quickfetch(struct ref *ref_map)if (depth)return -1 ; -for (i =0 , ref = ref_map; ref; ref = ref->next) - i++; -if (!i) +if (!ref_map)return 0 ; - argv =xmalloc (sizeof (*argv) * (i +6 )); - i =0 ; - argv[i++] =xstrdup ("rev-list" ); - argv[i++] =xstrdup ("--quiet" ); - argv[i++] =xstrdup ("--objects" ); -for (ref = ref_map; ref; ref = ref->next) - argv[i++] =xstrdup (sha1_to_hex (ref->old_sha1)); - argv[i++] =xstrdup ("--not" ); - argv[i++] =xstrdup ("--all" ); - argv[i++] =NULL ; -memset (&revlist,0 ,sizeof (revlist)); - revlist.argv = (const char **)argv; + revlist.argv = argv; revlist.git_cmd =1 ; - revlist.no_stdin =1 ; revlist.no_stdout =1 ; revlist.no_stderr =1 ; - err =run_command (&revlist); + revlist.in =-1 ; + + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + } + +/* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. */ +sigchain_push (SIGPIPE ,SIG_IGN ); + +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { +if (err !=EPIPE && err !=EINVAL ) +error ("failed write to rev-list: %s" ,strerror (errno)); + err =-1 ; +break ; + } + } + +if (close (revlist.in)) { +error ("failed to close rev-list's stdin: %s" ,strerror (errno)); + err =-1 ; + } + +sigchain_pop (SIGPIPE ); -for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +return finish_command (&revlist) || err; }static int fetch_refs (struct transport *transport,struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' ' ' +test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' ' + + git gc && + head=$(git rev-parse HEAD) && + branchprefix="$head refs/heads/branch" && + for i in 0 1 2 3 4 5 6 7 8 9; do + for j in 0 1 2 3 4 5 6 7 8 9; do + for k in 0 1 2 3 4 5 6 7 8 9; do + echo "$branchprefix$i$j$k" >> .git/packed-refs + done + done + done && + ( + cd cloned && + git fetch && + git fetch + ) + +' + test_done
Signature · 1 line
1.6.3.rc0.1.gf800
Re: [PATCH v4] quickfetch(): Prevent overflow of the rev-list command line
With this fixup the patch is good, I think.
-- Hannes
builtin-fetch.c +5 -3
@@ -446,14 +446,16 @@ static int quickfetch(struct ref *ref_map)return err; } -/* If rev-list --stdin encounters an unknown commit, it terminates, -* which will cause SIGPIPE in the write loop below. */ +/* +* If rev-list --stdin encounters an unknown commit, it terminates, +* which would cause SIGPIPE in the write loop below. +*/ sigchain_push (SIGPIPE ,SIG_IGN );for (ref = ref_map; ref; ref = ref->next) {if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 ||write_in_full (revlist.in,"\n" ,1 ) <0 ) { -if (err !=EPIPE && err !=EINVAL ) +if (errno !=EPIPE && errno !=EINVAL )error ("failed write to rev-list: %s" ,strerror (errno)); err =-1 ;break ;
PATCH v5 1/1 quickfetch(): Prevent overflow of the rev-list command line
quickfetch() calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, making quickfetch() return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list terminates immediately when encountering an unknown object, which can trigger SIGPIPE if we are still writing object's to its standard input. We therefore temporarily ignore SIGPIPE so that the fetch process is not terminated.
The patch also contains a testcase to verify the fix (note that before the patch, the testcase would only fail on msysGit).
---
Quoted 1 line · Johannes Sixt
With this fixup the patch is good, I think.
Again, thanks for all your help!
Here's the final (*crossing fingers*) iteration of the patch.
Have fun! :)
...Johan
builtin-fetch.c | 67 +++++++++++++++++++++++++++++------------------- t/t5502-quickfetch.sh | 20 ++++++++++++++ 2 files changed, 60 insertions(+), 27 deletions(-)
builtin-fetch.c +40 -27
@@ -400,14 +400,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,/* * We would want to bypass the object transfer altogether if -* everything we are going to fetch already exists and connected +* everything we are going to fetch already exists and is connected * locally. * -* The refs we are going to fetch are in to_fetch (nr_heads in -* total). If running +* The refs we are going to fetch are in ref_map. If running * -* $ git rev-list --objects to_fetch[0] to_fetch[1] ... --not --all +* $ git rev-list --objects --stdin --not --all * +* (feeding all the refs in ref_map on its standard input) * does not error out, that means everything reachable from the * refs we are going to fetch exists and is connected to some of * our existing refs.
@@ -416,8 +416,9 @@ static int quickfetch(struct ref *ref_map) {struct child_process revlist;struct ref *ref; -char **argv; -int i, err; +int err; +const char *argv[] = {"rev-list" , +"--quiet" ,"--objects" ,"--stdin" ,"--not" ,"--all" ,NULL };/* * If we are deepening a shallow clone we already have these
@@ -429,34 +430,46 @@ static int quickfetch(struct ref *ref_map)if (depth)return -1 ; -for (i =0 , ref = ref_map; ref; ref = ref->next) - i++; -if (!i) +if (!ref_map)return 0 ; - argv =xmalloc (sizeof (*argv) * (i +6 )); - i =0 ; - argv[i++] =xstrdup ("rev-list" ); - argv[i++] =xstrdup ("--quiet" ); - argv[i++] =xstrdup ("--objects" ); -for (ref = ref_map; ref; ref = ref->next) - argv[i++] =xstrdup (sha1_to_hex (ref->old_sha1)); - argv[i++] =xstrdup ("--not" ); - argv[i++] =xstrdup ("--all" ); - argv[i++] =NULL ; -memset (&revlist,0 ,sizeof (revlist)); - revlist.argv = (const char **)argv; + revlist.argv = argv; revlist.git_cmd =1 ; - revlist.no_stdin =1 ; revlist.no_stdout =1 ; revlist.no_stderr =1 ; - err =run_command (&revlist); + revlist.in =-1 ; + + err =start_command (&revlist); +if (err) { +error ("could not run rev-list" ); +return err; + } + +/* +* If rev-list --stdin encounters an unknown commit, it terminates, +* which will cause SIGPIPE in the write loop below. +*/ +sigchain_push (SIGPIPE ,SIG_IGN ); + +for (ref = ref_map; ref; ref = ref->next) { +if (write_in_full (revlist.in,sha1_to_hex (ref->old_sha1),40 ) <0 || +write_in_full (revlist.in,"\n" ,1 ) <0 ) { +if (errno !=EPIPE && errno !=EINVAL ) +error ("failed write to rev-list: %s" ,strerror (errno)); + err =-1 ; +break ; + } + } + +if (close (revlist.in)) { +error ("failed to close rev-list's stdin: %s" ,strerror (errno)); + err =-1 ; + } + +sigchain_pop (SIGPIPE ); -for (i =0 ; argv[i]; i++) -free (argv[i]); -free (argv); -return err; +return finish_command (&revlist) || err; }static int fetch_refs (struct transport *transport,struct ref *ref_map)
t/t5502-quickfetch.sh +20 -0
@@ -119,4 +119,24 @@ test_expect_success 'quickfetch should not copy from alternate' ' ' +test_expect_success 'quickfetch should handle ~1000 refs (on Windows)' ' + + git gc && + head=$(git rev-parse HEAD) && + branchprefix="$head refs/heads/branch" && + for i in 0 1 2 3 4 5 6 7 8 9; do + for j in 0 1 2 3 4 5 6 7 8 9; do + for k in 0 1 2 3 4 5 6 7 8 9; do + echo "$branchprefix$i$j$k" >> .git/packed-refs + done + done + done && + ( + cd cloned && + git fetch && + git fetch + ) + +' + test_done
Signature · 1 line
1.6.3.rc0.1.gf800
Re: [PATCH v5] quickfetch(): Prevent overflow of the rev-list command line
Quoted 11 lines · Johan Herland
quickfetch()calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, makingquickfetch()return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes
quickfetch(), and subsequently the overall fetch, to succeed.
I feel uneasy with that "somehow" at the end of the first paragraph, but nevertheless this is the right thing to do. Since it is a very isolated change, I'd queue this directly on 'master' and see if anybody notices a breakage, as it would be relatively pain-free to revert if it turns out to be necessary.
Thanks.
Re: [PATCH v5] quickfetch(): Prevent overflow of the rev-list command line
Quoted 10 lines · Junio C Hamano
Quoted 9 lines · Johan Herland
quickfetch()calls rev-list to check whether the objects we are about to fetch are already present in the repo (if so, we can skip the object fetch). However, when there are many (~1000) refs to be fetched, the rev-list command line grows larger than the maximum command line size on some systems (32K in Windows). This causes rev-list to fail, makingquickfetch()return non-zero, which unnecessarily triggers the transport machinery. This somehow causes fetch to fail with an exit code.
Quoted 7 lines
Quoted 4 lines
By using the --stdin option to rev-list (and feeding the object list to its standard input), we prevent the overflow of the rev-list command line, which causes
quickfetch(), and subsequently the overall fetch, to succeed.I feel uneasy with that "somehow" at the end of the first paragraph, but nevertheless this is the right thing to do.
Yes, it feels wrong that transport_fetch_refs() returns error when there are no objects to be fetched. After all, the quickfetch() routine is only meant to be an optimization (to skip the object fetching when not needed). Only if quickfetch() returned a false positive (indicating that all objects are present when they're really not) would I expect it to have adverse effects on the result of the fetch. But even then, as you indicate, fixing quickfetch() itself is the right thing to do, and looking into transport_fetch_refs() is a separate issue.
Quoted 4 lines
Since it is a very isolated
change, I'd queue this directly on 'master' and see if anybody notices a breakage, as it would be relatively pain-free to revert if it turns out to be necessary.
Thanks.
Have fun! :)
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
In reply to Johannes SixtQuoted 3 lines · Johannes Sixt
But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.
AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
Quoted 10 lines · Johan Herland
Quoted 3 lines · Johannes Sixt
But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.
AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.
I know that, of course. But try this: Remove the signal(SIGPIPE, SIG_IGN) and run the test suite. There is not a single failure. IOW, we don't have a single test case that verifies that the signal(SIGPIPE, SIG_IGN) is needed. I would like to have that test case, and you seem to know how to construct it (otherwise there wouldn't be so much buzz about it).
-- Hannes
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
Quoted 13 lines · Johannes Sixt
Quoted 11 lines · Johan Herland
Quoted 3 lines · Johannes Sixt
But actually I meant you to make a test that triggers the SIGPIPE that would kill git-fetch if it were not ignored. This one doesn't trigger it, either.
AFAIU from earlier in this thread (and a mail from Peter linking to
http://markmail.org/message/dbgdj4csafen65ye), SIGPIPE _never_ triggers on Windows, thus ignoring SIGPIPE is not needed for the fix per se. However, as a side-effect of the fix, we may now get SIGPIPE on Linux (and other POSIX platforms), so although it never triggers on Windows, it's still needed.
Quoted 2 lines
I know that, of course. But try this: Remove the
signal(SIGPIPE, SIG_IGN)and run the test suite. There is not a single failure.
That's not what I'm seeing. When I don't ignore the signal, the testsuite fails intermittently for me (on Linux). I see the following tests fail:
- t3409-rebase-preserve-merges.sh (subtest #2)
- t5503-tagfollow.sh (subtests #4, #6, #7)
- t5505-remote.sh (subtests #10, #12, #14 - #20, #27)
- t5510-fetch.sh (subtest #6 or #25)
- probably more (I seldom get this far...)
I assume the intermittent failures are caused by git rev-list sometimes terminate before git fetch is finished writing objects to its standard input (because of scheduling differences).
When i enable the signal handling, all selftests pass every time.
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH v3] quickfetch(): Prevent overflow of the rev-list command line
Quoted 11 lines · Johan Herland
Quoted 2 lines · Johannes Sixt
But try this: Remove the
signal(SIGPIPE, SIG_IGN)and run the test suite. There is not a single failure.That's not what I'm seeing. When I don't ignore the signal, the testsuite fails intermittently for me (on Linux). I see the following tests fail:
-
t3409-rebase-preserve-merges.sh(subtest #2)
-t5503-tagfollow.sh(subtests #4, #6, #7)
-t5505-remote.sh(subtests #10, #12, #14 - #20, #27)
-t5510-fetch.sh(subtest #6 or #25)
I see. If I insert sched_yield() in the for loop, I see many failures (not 100% reproducible, but almost). On Windows, I have to insert Sleep(10) so that the error exit is taken, and the error code is EINVAL :-/
-- Hannes
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
In reply to Johan HerlandQuoted 8 lines · Johan Herland
Quoted 3 lines · Junio C Hamano
Quoted 2 lines · Johan Herland
Maybe I need to do something to the
close()call as well? What happens onclose()after EPIPE?Does anybody with Windows/MSYS experience know how this scenario (
write()to a terminated process, followed byclose()) would play out in msysGit?
It fails with ERROR_BROKEN_PIPE. See MSDN for WriteFile:
http://msdn.microsoft.com/en-us/library/aa365747%28VS.85%29.aspx
(look for the error above).
Well, sometimes it just fails, so you can hardly use the error code to detect if the other process is truly gone or something broke in Windows.
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
In reply to Johan HerlandQuoted 7 lines · Johan Herland
Quoted 2 lines · Johannes Sixt
... don't you get this error message with errno set to EPIPE? Previously, there was no error message.
Indeed, you are correct. I guess the following should be added to the patch:
Quoted 7 lines
if (
write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40)< 0 ||write_in_full(revlist.in, "\n", 1)< 0) {
-error("failed write to rev-list");
- err = errno;
+ if (errno != EPIPE) {
+error("failed write to rev-list");
+ err = errno;
You'll loose errno this way: error() does not save it.
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 16 lines · Alex Riesen
Quoted 9 lines · Johan Herland
Quoted 2 lines · Johannes Sixt
... don't you get this error message with errno set to EPIPE? Previously, there was no error message.
Indeed, you are correct. I guess the following should be added to the patch:
if (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) <0 ||
write_in_full(revlist.in, "\n", 1)< 0) {Review
Quoted 5 lines
(quoted diff) +3
-2- error("failed write to rev-list"); - err = errno; + if (errno != EPIPE) { + error("failed write to rev-list"); + err = errno;
Quoted 1 line
You'll loose errno this way:
error()does not save it.
Not sure what you mean here. Should I move "err = errno;" outside the innermost "if"?
Quoted 1 line
From my POV, if errno != EPIPE, we save it into err, and return that
(overridden by finish_command()'s return value, if non-zero). If errno == EPIPE, we're not interested in saving it, because we expect finish_command() to return non-zero in any case.
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Re: [PATCH] quickfetch(): Prevent overflow of the rev-list command line
Quoted 17 lines · Johan Herland
Quoted 16 lines · Alex Riesen
Quoted 14 lines · Johan Herland
Quoted 2 lines · Johannes Sixt
... don't you get this error message with errno set to EPIPE? Previously, there was no error message.
Indeed, you are correct. I guess the following should be added to the patch:
if (
write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40)< 0 ||write_in_full(revlist.in, "\n", 1)< 0) {
-error("failed write to rev-list");
- err = errno;
+ if (errno != EPIPE) {
+error("failed write to rev-list");
+ err = errno;
Quoted 4 lines
Quoted 1 line
You'll loose errno this way:
error()does not save it.Not sure what you mean here. Should I move "err = errno;" outside the innermost "if"?
put it before error("failed write to rev-list"); or even before the "if (err != EPIPE)".
Otherwise it is 0 after fprintf to stderr (which is the error() call).
Quoted 4 lines
From my POV, if errno != EPIPE, we save it into err, and return that (overridden by finish_command()'s return value, if non-zero). If errno == EPIPE, we're not interested in saving it, because we expect
finish_command()to return non-zero in any case.
And you think this expectation makes the code simpler to understand?