die routine: change recursion limit from 1 to 1024

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>

---
 usage.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

usage.c +3 -2

index 2f87ca69a8..1c198d4882 100644
@@ -44,7 +44,9 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
        static int dying;
-       return dying++;
+       static int recursion_limit = 1024;
+
+       return dying++ > recursion_limit;
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
--

2.13.1.518.g3df882009

Quoted 3 lines

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

I think we should do that instead (though I have not looked at the downsides of this), because...

Quoted 5 lines

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

how are we handling access to that global variable?
Do we need to hold a mutex to be correct? or rather hope that it works across threads, not counting on it, because each thread individually would count up to 1024?

I would prefer if we kept the number as low as "at most one screen of lines".

Quoted 23 lines

Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>

---
 usage.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

usage.c +3 -2

index 2f87ca69a8..1c198d4882 100644
@@ -44,7 +44,9 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
        static int dying;
-       return dying++;
+       static int recursion_limit = 1024;
+
+       return dying++ > recursion_limit;
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
--

2.13.1.518.g3df882009

On Mon, Jun 19 2017, Stefan Beller jotted:

Quoted 6 lines

Quoted 3 lines

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

I think we should do that instead (though I have not looked at the downsides of this), because...

It makes sense to do that in addition to what I'm doing here (or if the approach I'm taking doesn't make sense, some other patch to fix the general issue in the default handler).

I'm going to try to get around to fixing the grep behavior in a follow-up patch, this is a fix for the overzealous recursion detection in the default handler needlessly causing other issues.

Quoted 10 lines

Quoted 5 lines

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

how are we handling access to that global variable?
Do we need to hold a mutex to be correct? or rather hope that it works across threads, not counting on it, because each thread individually would count up to 1024?

It's not guarded by a mutex and the ++ here and the reads from it are both racy.

However, for its stated purpose that's fine, even if we're racily incrementing it and losing some updates some will get through, which is good enough for an infinite recursion detection. We don't really care if we die at exactly 1 or exactly 1024.

Quoted 2 lines

I would prefer if we kept the number as low as "at most one screen of lines".

In practice this is the case in git, because the programs that would encounter this are going to be spawning less than screenfull of threads, assuming (as is the case) that each thread might print out one error.

The semantics of that aren't changed with this patch, the difference is that you're going to get e.g. N repeats of a meaningful error instead of N repeats of either the meaningful error OR "recursion detected in die handler", depending on your luck.

I.e. in current git (after a few runs to get an unlucky one):

    $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'
    fatal: recursion detected in die handler
    fatal: recursion detected in die handler
    fatal: recursion detected in die handler

Or if you're lucky at least one of these will be the actual error:

    $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'
    fatal: recursion detected in die handler
    fatal: pcre_exec failed with error code -8
    fatal: recursion detected in die handler
    fatal: recursion detected in die handler
    fatal: recursion detected in die handler

But with this change:

    $ ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'
    fatal: pcre2_jit_match failed with error code -47: match limit exceeded
    fatal: pcre2_jit_match failed with error code -47: match limit exceeded
    fatal: pcre2_jit_match failed with error code -47: match limit exceeded

(The error message is different because I compiled with PCRE v2 locally, instead of the system PCRE v1, but that doesn't matter for this example)

Over 1000 runs thish is how that breaks down on my machine, without this patch. I've replaced the recursion error with just "R" and the PCRE error with "P", and shown them in descending order by occurrence, lines without a "P" only printed out the recursion error:

    $ (for i in {1..1000}; do git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10
        247 R R
        136 P R R
        122 P R
        112 R R R
        108 R
         59 R P R
         54 R P
         54 P
         31 P R R R
         21 R R P

There's a long tail I've omitted there of alterations to that. As this shows in >10% of cases we don't print out any meaningful error at all. But with this change:

    $ (for i in {1..1000}; do ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10
        377 P P
        358 P P P
        192 P
         63 P P P P
          8 P P P P P
          2 P P P P P P

We will always show a meaningful error, but may of course do so multiple times, which is a subject for a fix in git-grep in particular, but the point is again, to fix the general case for the default handler.

Something something sorry about the long mail didn't have time to write a shorter one :)

Change the recursion limit for the default die routine from a *very* low 1 to 1024. This ensures that infinite recursions are broken, but doesn't lose the meaningful error messages under threaded execution where threads concurrently start to die.

The intent of the existing code, as explained in commit cd163d4b4e ("usage.c: detect recursion in die routines and bail out immediately", 2012-11-14), is to break infinite recursion in cases where the die routine itself calls die(), and would thus infinitely recurse.

However, doing that very aggressively by immediately printing out "recursion detected in die handler" if we've already called die() once means that threaded invocations of git can end up only printing out the "recursion detected" error, while hiding the meaningful error.

An example of this is running a threaded grep which dies on execution against pretty much any repo, git.git will do:

    git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'

With the current version of git this will print some combination of multiple PCRE failures that caused the abort and multiple "recursion detected", some invocations will print out multiple "recursion detected" errors with no PCRE error at all!

Before this change, running the above grep command 1000 times against git.git[1] and taking the top 20 results will on my system yield the following distribution of actual errors ("E") and recursion errors ("R"):

322 E R
306 E
116 E R R
65 R R
54 R E
49 E E
44 R
15 E R R R
9 R R R
7 R E R
5 R R E
3 E R R R R
2 E E R
1 R R R R
1 R R R E
1 R E R R

The exact results are obviously random and system-dependent, but this shows the race condition in this code. Some small part of the time we're about to print out the actual error ("E") but another thread's recursion error beats us to it, and sometimes we print out nothing but the recursion error.

With this change we get, now with "W" to mean the new warning being emitted indicating that we've called die() many times:

502 E
160 E W E
120 E E
53 E W
35 E W E E
34 W E E
29 W E E E
16 E E W
16 E E E
11 W E E E E
7 E E W E
4 W E
3 W W E E
2 E W E E E
1 W W E
1 W E W E
1 E W W E E E
1 E W W E E
1 E W W E
1 E W E E W

Which still sucks a bit, due to a still present race-condition in this code we're sometimes going to print out several errors still, or several warnings, or two duplicate errors without the warning.

But we will never have a case where we completely hide the actual error as we do now.

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16). There's other threaded code that calls set_die_routine() or set_die_is_recursing_routine().

But this is about fixing the general die() behavior with threading when we don't have such a custom routine yet. Right now the common case is not an infinite recursion in the handler, but us losing error messages by default because we're overly paranoid about our recursion check.

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

There are race conditions in this code itself, in particular the "dying" variable is not thread mutexed, so we e.g. won't be dying at exactly 1024, or for that matter even be able to accurately test "dying == 2", see the cases where we print out more than one "W" above.

But that doesn't really matter, for the recursion guard we just need to die "soon", not at exactly 1024 calls, and for printing the correct error and only one warning most of the time in the face of threaded death this is good enough and a net improvement on the current code.

1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20

Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>

---

This replaces v1 and takes into account the feedback in this thread (thanks everyone!).

The commit message is also much improved and includes more rationale originally in my reply to Stefan in [email protected]

 usage.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

usage.c +17 -2

index 2f87ca69a8..1ea7df9a20 100644
@@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
        static int dying;
-       return dying++;
+       /*
+        * Just an arbitrary number X where "a < x < b" where "a" is
+        * "maximum number of pthreads we'll ever plausibly spawn" and
+        * "b" is "something less than Inf", since the point is to
+        * prevent infinite recursion.
+        */
+       static const int recursion_limit = 1024;
+
+       dying++;
+       if (dying > recursion_limit) {
+               return 1;
+       } else if (dying == 2) {
+               warning("die() called many times. Recursion error or racy threaded death!");
+               return 0;
+       } else {
+               return 0;
+       }
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
--

2.13.1.611.g7e3b11ae1

Change the recursion limit for the default die routine from a *very* low 1 to 1024. This ensures that infinite recursions are broken, but doesn't lose the meaningful error messages under threaded execution where threads concurrently start to die.

The intent of the existing code, as explained in commit cd163d4b4e ("usage.c: detect recursion in die routines and bail out immediately", 2012-11-14), is to break infinite recursion in cases where the die routine itself calls die(), and would thus infinitely recurse.

However, doing that very aggressively by immediately printing out "recursion detected in die handler" if we've already called die() once means that threaded invocations of git can end up only printing out the "recursion detected" error, while hiding the meaningful error.

An example of this is running a threaded grep which dies on execution against pretty much any repo, git.git will do:

    git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'

With the current version of git this will print some combination of multiple PCRE failures that caused the abort and multiple "recursion detected", some invocations will print out multiple "recursion detected" errors with no PCRE error at all!

Before this change, running the above grep command 1000 times against git.git[1] and taking the top 20 results will on my system yield the following distribution of actual errors ("E") and recursion errors ("R"):

322 E R
306 E
116 E R R
65 R R
54 R E
49 E E
44 R
15 E R R R
9 R R R
7 R E R
5 R R E
3 E R R R R
2 E E R
1 R R R R
1 R R R E
1 R E R R

The exact results are obviously random and system-dependent, but this shows the race condition in this code. Some small part of the time we're about to print out the actual error ("E") but another thread's recursion error beats us to it, and sometimes we print out nothing but the recursion error.

With this change we get, now with "W" to mean the new warning being emitted indicating that we've called die() many times:

502 E
160 E W E
120 E E
53 E W
35 E W E E
34 W E E
29 W E E E
16 E E W
16 E E E
11 W E E E E
7 E E W E
4 W E
3 W W E E
2 E W E E E
1 W W E
1 W E W E
1 E W W E E E
1 E W W E E
1 E W W E
1 E W E E W

Which still sucks a bit, due to a still present race-condition in this code we're sometimes going to print out several errors still, or several warnings, or two duplicate errors without the warning.

But we will never have a case where we completely hide the actual error as we do now.

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16). There's other threaded code that calls set_die_routine() or set_die_is_recursing_routine().

But this is about fixing the general die() behavior with threading when we don't have such a custom routine yet. Right now the common case is not an infinite recursion in the handler, but us losing error messages by default because we're overly paranoid about our recursion check.

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

There are race conditions in this code itself, in particular the "dying" variable is not thread mutexed, so we e.g. won't be dying at exactly 1024, or for that matter even be able to accurately test "dying == 2", see the cases where we print out more than one "W" above.

But that doesn't really matter, for the recursion guard we just need to die "soon", not at exactly 1024 calls, and for printing the correct error and only one warning most of the time in the face of threaded death this is good enough and a net improvement on the current code.

1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20

Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>

---

This replaces v1 and takes into account the feedback in this thread (thanks everyone!).

The commit message is also much improved and includes more rationale originally in my reply to Stefan in [email protected]

 usage.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

usage.c +17 -2

index 2f87ca69a8..1ea7df9a20 100644
@@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
        static int dying;
-       return dying++;
+       /*
+        * Just an arbitrary number X where "a < x < b" where "a" is
+        * "maximum number of pthreads we'll ever plausibly spawn" and
+        * "b" is "something less than Inf", since the point is to
+        * prevent infinite recursion.
+        */
+       static const int recursion_limit = 1024;
+
+       dying++;
+       if (dying > recursion_limit) {
+               return 1;
+       } else if (dying == 2) {
+               warning("die() called many times. Recursion error or racy threaded death!");
+               return 0;
+       } else {
+               return 0;
+       }
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
--

2.13.1.611.g7e3b11ae1

If threading is the issue, how do you get meaningful results from reading and updating
"dying" with no use of atomic types or locks? Other than winning the implied race, of
course.

Quoted 158 lines · Ævar Arnfjörð Bjarmason

Change the recursion limit for the default die routine from a *very* low 1 to 1024. This ensures that infinite recursions are broken, but doesn't lose the meaningful error messages under threaded execution where threads concurrently start to die.

The intent of the existing code, as explained in commit cd163d4b4e ("usage.c: detect recursion in die routines and bail out immediately", 2012-11-14), is to break infinite recursion in cases where the die routine itself calls die(), and would thus infinitely recurse.

However, doing that very aggressively by immediately printing out "recursion detected in die handler" if we've already called die() once means that threaded invocations of git can end up only printing out the "recursion detected" error, while hiding the meaningful error.

An example of this is running a threaded grep which dies on execution against pretty much any repo, git.git will do:

    git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'

With the current version of git this will print some combination of multiple PCRE failures that caused the abort and multiple "recursion detected", some invocations will print out multiple "recursion detected" errors with no PCRE error at all!

Before this change, running the above grep command 1000 times against git.git[1] and taking the top 20 results will on my system yield the following distribution of actual errors ("E") and recursion errors ("R"):

322 E R
306 E
116 E R R
65 R R
54 R E
49 E E
44 R
15 E R R R
9 R R R
7 R E R
5 R R E
3 E R R R R
2 E E R
1 R R R R
1 R R R E
1 R E R R

The exact results are obviously random and system-dependent, but this shows the race condition in this code. Some small part of the time we're about to print out the actual error ("E") but another thread's recursion error beats us to it, and sometimes we print out nothing but the recursion error.

With this change we get, now with "W" to mean the new warning being emitted indicating that we've called die() many times:

502 E
160 E W E
120 E E
53 E W
35 E W E E
34 W E E
29 W E E E
16 E E W
16 E E E
11 W E E E E
7 E E W E
4 W E
3 W W E E
2 E W E E E
1 W W E
1 W E W E
1 E W W E E E
1 E W W E E
1 E W W E
1 E W E E W

Which still sucks a bit, due to a still present race-condition in this code we're sometimes going to print out several errors still, or several warnings, or two duplicate errors without the warning.

But we will never have a case where we completely hide the actual error as we do now.

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16). There's other threaded code that calls set_die_routine() or set_die_is_recursing_routine().

But this is about fixing the general die() behavior with threading when we don't have such a custom routine yet. Right now the common case is not an infinite recursion in the handler, but us losing error messages by default because we're overly paranoid about our recursion check.

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

There are race conditions in this code itself, in particular the "dying" variable is not thread mutexed, so we e.g. won't be dying at exactly 1024, or for that matter even be able to accurately test "dying == 2", see the cases where we print out more than one "W" above.

But that doesn't really matter, for the recursion guard we just need to die "soon", not at exactly 1024 calls, and for printing the correct error and only one warning most of the time in the face of threaded death this is good enough and a net improvement on the current code.

1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20

Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>

---

This replaces v1 and takes into account the feedback in this thread (thanks everyone!).

The commit message is also much improved and includes more rationale originally in my reply to Stefan in [email protected]

 usage.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

usage.c +17 -2

index 2f87ca69a8..1ea7df9a20 100644
@@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
        static int dying;
-       return dying++;
+       /*
+        * Just an arbitrary number X where "a < x < b" where "a" is
+        * "maximum number of pthreads we'll ever plausibly spawn" and
+        * "b" is "something less than Inf", since the point is to
+        * prevent infinite recursion.
+        */
+       static const int recursion_limit = 1024;
+
+       dying++;
+       if (dying > recursion_limit) {
+               return 1;
+       } else if (dying == 2) {
+               warning("die() called many times. Recursion error or racy threaded death!");
+               return 0;
+       } else {
+               return 0;
+       }
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
--

2.13.1.611.g7e3b11ae1

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

There are race conditions in this code itself, in particular the "dying" variable is not thread mutexed, so we e.g. won't be dying at exactly 1024, or for that matter even be able to accurately test "dying == 2", see the cases where we print out more than one "W" above.

 usage.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

usage.c +17 -1

index 2f87ca69a8..1ea7df9a20 100644
@@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)
 static int die_is_recursing_builtin(void)
 {
 	static int dying;
-	return dying++;
+	/*
+	 * Just an arbitrary number X where "a < x < b" where "a" is
+	 * "maximum number of pthreads we'll ever plausibly spawn" and
+	 * "b" is "something less than Inf", since the point is to
+	 * prevent infinite recursion.
+	 */
+	static const int recursion_limit = 1024;
+
+	dying++;
+	if (dying > recursion_limit) {
+		return 1;
+	} else if (dying == 2) {
+		warning("die() called many times. Recursion error or racy threaded death!");
+		return 0;
+	} else {
+		return 0;
+	}
 }
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault

Quoted 10 lines · Ævar Arnfjörð Bjarmason

So let's just set the recursion limit to a number higher than the number of threads we're ever likely to spawn. Now we won't lose errors, and if we have a recursing die handler we'll still die within microseconds.

There are race conditions in this code itself, in particular the "dying" variable is not thread mutexed, so we e.g. won't be dying at exactly 1024, or for that matter even be able to accurately test "dying == 2", see the cases where we print out more than one "W" above.

One case I'd be worried about would be that the race is so bad that die-is-recursing-builtin never returns 0 even once. Everybody will just say "recursing" and die, without giving any useful information.

Quoted 3 lines

One case I'd be worried about would be that the race is so bad that die-is-recursing-builtin never returns 0 even once. Everybody will just say "recursing" and die, without giving any useful information.

I was trying to think how that would happen. If nobody's actually recursing indefinitely, then the value in theory peaks at the number of threads (modulo the fact that we're modifying a variable from multiple threads without any locking; I'm not sure how reasonable it is to assume in practice that sheared writes may cause us to lose an increment but not to put nonsense in to the variable). If they are, then one thread may increment it to 1024 before another thread gets a chance to say anything. But in that case, the recursion-die is our expected outcome.

Anyway, it might be reasonable to protect the counter with a mutex. Like:
...
To be honest, I'm not sure if it's worth giving it much more time, though. I'd be fine with Ævar's patch as-is.

Change the recursion limit for the default die routine from a *very* low 1 to 1024. This ensures that infinite recursions are broken, but doesn't lose error messages.

The intent of the existing code, as explained in commit cd163d4b4e ("usage.c: detect recursion in die routines and bail out immediately", 2012-11-14), is to break infinite recursion in cases where the die routine itself dies.

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

Quoted 3 lines

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

Yeah, I think this is a bug in git-grep and should be fixed, independent of this commit. You should be able to use as a template the callbacks added by the child of c19a490e37:

1ece66bc9 (run-command: use thread-aware die_is_recursing routine, 2013-04-16)

Quoted 9 lines

Change the recursion limit for the default die routine from a *very* low 1 to 1024. This ensures that infinite recursions are broken, but doesn't lose error messages.

The intent of the existing code, as explained in commit cd163d4b4e ("usage.c: detect recursion in die routines and bail out immediately", 2012-11-14), is to break infinite recursion in cases where the die routine itself dies.

I agree that was the original intent, but I think it also does something else. Anytime die() recurses, even a single level, we're going to cover up the original failure with the one that happened inside die(), which is almost certainly the less interesting of the two.

E.g., if I

  die_errno("unable to open %s", filename);

and then the die handler calls malloc() and fails, you'd much rather see that first message than "out of memory".

To be fair, "die handler is recursing" is _also_ not helpful, but at least it's clear that this is a bug (and IMHO it should be marked with BUG()). Saying "out of memory" tells you about the second error, but it doesn't tell you that we've masked the first error. So it may lead to more confusion in the long run.

I wonder if we can get the best of both, though. Can we make the logic more like:

  if (!dying) {
	/* ok, normal */
	return 0;
  } else if (dying < 1024) {
	/* only show the warning once */
	if (dying == 1)
		warning("I heard you liked errors, so I put a die() in your die()");
	return 0; /* don't bail yet */
  } else {
	BUG("recursion detected in die handler");

}

Quoted 3 lines

Now, git-grep could make use of the pluggable error facility added in commit c19a490e37 ("usage: allow pluggable die-recursion checks", 2013-04-16).

Yeah, I think this is a bug in git-grep and should be fixed, independent of this commit. You should be able to use as a template the callbacks added by the child of c19a490e37:

1ece66bc9 (run-command: use thread-aware die_is_recursing routine, 2013-04-16)

-Peff

As you can see the third most common case is that we needlessly print out the warning, i.e. we have only one error anyway, but we can't guarantee that, so it probably makes sense to emit it.

To reply to your [email protected] downthread here (where you talk about setting up a custom die handler for grep) yeah that would make sense, but as long as we're supplying this default behavior (and not outlawing using it with pthreads) it makes sense to get out of our own way with this recursion detection.

I think my patch (possibly with the fixup above, depending on what we think about dupe warnings) is just fine to fix this. This is already a super-rare edge case in grep, and to the extent that it would be a problem for anyone it's because our paranoid recursion detector totally hides the error, I don't think it's worth worrying about us printing a few dupe error messages under threading for something that almost never happens.

At least, that's starting to go beyond my bothering to hack on it :)

If I understand you correctly this on top:

    diff --git a/usage.c b/usage.c
    index 1c198d4882..f6d5af2bb4 100644
    --- a/usage.c
    +++ b/usage.c
    @@ -46,7 +46,19 @@ static int die_is_recursing_builtin(void)
     	static int dying;
     	static int recursion_limit = 1024;

    -	return dying++ > recursion_limit;
    +	dying++;
    +
    +	if (!dying) {

    +		/* ok, normal */
    +		return 0;
    +	} else if (dying < recursion_limit) {
    +		/* only show the warning once */
    +		if (dying == 1)
    +			warning("die() called many times. Recursion error or racy threaded death!");
    +		return 0; /* don't bail yet */
    +	} else {
    +		return 1;
    +	}
     }

Quoted 14 lines

If I understand you correctly this on top:

    diff --git a/usage.c b/usage.c
    index 1c198d4882..f6d5af2bb4 100644
    --- a/usage.c
    +++ b/usage.c
    @@ -46,7 +46,19 @@ static int die_is_recursing_builtin(void)
     	static int dying;
     	static int recursion_limit = 1024;

    -	return dying++ > recursion_limit;
    +	dying++;
    +
    +	if (!dying) {

This will never trigger as dying was incremented two lines before. But I think it's already handled by the dying < recursion_limit case so we can just omit it.

Quoted 11 lines

    +		/* ok, normal */
    +		return 0;
    +	} else if (dying < recursion_limit) {
    +		/* only show the warning once */
    +		if (dying == 1)
    +			warning("die() called many times. Recursion error or racy threaded death!");
    +		return 0; /* don't bail yet */
    +	} else {
    +		return 1;
    +	}
     }

Maybe restructure it like this:

    dying++
    if (dying > recursion_limit)
        return 1;
    if (dying == 1)
        warning();
    return 0;

Btw. is there a reason why recursion_limit is a static variable and not a constant/define?