t1401: test symbolic-ref exit codes on a non-symbolic ref

From: Nikolaus Schuetz <[email protected]>

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref exits with a non-zero status, and that --quiet does so silently rather than printing a diagnostic. This was not tested.

Check that querying a non-symbolic ref exits 128 with the usual "is not a symbolic ref" message, and that --quiet instead exits 1 with no output.

---
t1401: test symbolic-ref exit codes on a non-symbolic ref

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref exits with a non-zero status, and that --quiet does so silently rather than printing a diagnostic. This exit-code contract was untested.

This adds two tests: querying a non-symbolic ref exits 128 with the usual "is not a symbolic ref" message, and --quiet instead exits 1 with no output.

Test-only; documents existing behaviour, in the spirit of 919eb8ace (t1402: check for refs ending with a dot).

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2204%2Fnikolauspschuetz%2Fns%2Ft1401-symbolic-ref-quiet-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2204/nikolauspschuetz/ns/t1401-symbolic-ref-quiet-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2204

 t/t1401-symbolic-ref.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

index a2a7e94716..602db6d080 100755
@@ -38,6 +38,16 @@ test_expect_success 'symbolic-ref refuses bare sha1' '
 
 reset_to_sane
 
+test_expect_success 'symbolic-ref reports a non-symbolic ref with exit code 128' '
+	test_expect_code 128 git symbolic-ref refs/heads/foo 2>err &&
+	test_grep "is not a symbolic ref" err
+'
+
+test_expect_success 'symbolic-ref -q is silent and exits 1 on a non-symbolic ref' '
+	test_expect_code 1 git symbolic-ref -q refs/heads/foo 2>err &&
+	test_must_be_empty err
+'
+
 test_expect_success 'HEAD cannot be removed' '
 	test_must_fail git symbolic-ref -d HEAD
 '

base-commit: 745601a9a94110d74769ab605ccd4f61339758d2

gitgitgadget

Out of curiosity, what made you address these gaps in particular? Is there any motivation, or are you just picking random things to work on?

This is testing the status quo, but what I think would be good to research in this context is why the error codes are different in the first place. I personally find that quite a bit puzzling, as my expectation would be that "--quiet" really only impacts whether we print anything or not. That it also changes the error code is weird.

Not random -- I've been going through git commands, checking whether the behavior their man pages promise is actually exercised from t/, and filling the gaps. The idea is to pin the documented contract in a test so a later refactor can't quietly change it. git-symbolic-ref(1) spells out both the exit status and the --quiet silence, but neither was tested, so they stood out.

Agreed it's surprising, though it's not unique to symbolic-ref: git rev-parse --verify --quiet does the same thing (exit 1 and silent, vs a fatal 128 without --quiet). It falls out of how the two paths report in check_symref() (builtin/symbolic-ref.c): the non-quiet path calls die(), which always exits 128, while --quiet can't die() -- that would print -- so it returns 1.

Great idea. I've revised the added tests to redirect stdout and check for empty stdout in both cases.

Thanks,
Nikolaus

From: Nikolaus Schuetz <[email protected]>

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref exits non-zero, and that --quiet does so silently. Tests such as t2020 and t5621 already rely on "symbolic-ref -q HEAD" failing on a detached HEAD, but none pins the exact exit codes or checks that --quiet actually suppresses the diagnostic.

Assert that a non-symbolic ref exits 128 with the "is not a symbolic ref" message, and that --quiet instead exits 1 with no output.

---
t1401: test symbolic-ref exit codes on a non-symbolic ref

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref exits with a non-zero status, and that --quiet does so silently rather than printing a diagnostic. This exit-code contract was untested.

This adds two tests: querying a non-symbolic ref exits 128 with the usual "is not a symbolic ref" message, and --quiet instead exits 1 with no output.

Test-only; documents existing behaviour, in the spirit of 919eb8ace (t1402: check for refs ending with a dot).

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2204%2Fnikolauspschuetz%2Fns%2Ft1401-symbolic-ref-quiet-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2204/nikolauspschuetz/ns/t1401-symbolic-ref-quiet-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2204

Range-diff vs v1:

1: 345e664a52 ! 1: 22694da869 t1401: test symbolic-ref exit codes on a non-symbolic ref
@@ Metadata
Author: Nikolaus Schuetz <[email protected]>

## Commit message ##
- t1401: test symbolic-ref exit codes on a non-symbolic ref
+ t1401: check symbolic-ref exit codes and --quiet silence

git-symbolic-ref(1) documents that reading a name that is not a
- symbolic ref exits with a non-zero status, and that --quiet does so
- silently rather than printing a diagnostic. This was not tested.
+ symbolic ref exits non-zero, and that --quiet does so silently.
+ Tests such as t2020 and t5621 already rely on "symbolic-ref -q HEAD"
+ failing on a detached HEAD, but none pins the exact exit codes or
+ checks that --quiet actually suppresses the diagnostic.

- Check that querying a non-symbolic ref exits 128 with the usual
- "is not a symbolic ref" message, and that --quiet instead exits 1
- with no output.
+ Assert that a non-symbolic ref exits 128 with the "is not a symbolic
+ ref" message, and that --quiet instead exits 1 with no output.

@@ t/t1401-symbolic-ref.sh: test_expect_success 'symbolic-ref refuses bare sha1' '
reset_to_sane

+test_expect_success 'symbolic-ref reports a non-symbolic ref with exit code 128' '
-+ test_expect_code 128 git symbolic-ref refs/heads/foo 2>err &&
++ test_expect_code 128 git symbolic-ref refs/heads/foo >out 2>err &&
++ test_must_be_empty out &&
+ test_grep "is not a symbolic ref" err
+'
+
+test_expect_success 'symbolic-ref -q is silent and exits 1 on a non-symbolic ref' '
-+ test_expect_code 1 git symbolic-ref -q refs/heads/foo 2>err &&
++ test_expect_code 1 git symbolic-ref -q refs/heads/foo >out 2>err &&
++ test_must_be_empty out &&
+ test_must_be_empty err
+'
+

 t/t1401-symbolic-ref.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index a2a7e94716..6fad83ade8 100755
@@ -38,6 +38,18 @@ test_expect_success 'symbolic-ref refuses bare sha1' '
 
 reset_to_sane
 
+test_expect_success 'symbolic-ref reports a non-symbolic ref with exit code 128' '
+	test_expect_code 128 git symbolic-ref refs/heads/foo >out 2>err &&
+	test_must_be_empty out &&
+	test_grep "is not a symbolic ref" err
+'
+
+test_expect_success 'symbolic-ref -q is silent and exits 1 on a non-symbolic ref' '
+	test_expect_code 1 git symbolic-ref -q refs/heads/foo >out 2>err &&
+	test_must_be_empty out &&
+	test_must_be_empty err
+'
+
 test_expect_success 'HEAD cannot be removed' '
 	test_must_fail git symbolic-ref -d HEAD
 '

base-commit: 745601a9a94110d74769ab605ccd4f61339758d2

gitgitgadget

I'm still not completely sold that this difference is intentional. As you mentioned elsewhere, in one code path we use exit(1) and in the other we use die(...) to print an error message, which returs 128. But that to me feels more like nobody noticed that those return different error codes rather than an intentional difference in design, so casting that behaviour into stone with new tests feels wrong to me.

I'd suggest that we either:

- Continue not checking for the exact error code.

- Adapt the exit(1) to instead be die(NULL) so that we have the same error code across both code paths and then verify that both have the same error code.

Thanks!

Patrick

I tend to agree. The choice of 1 would have been more about "we need to exit with non-zero status" than "we need to use something different from what is used by die()".

Both sounds very sensible.

Especially the former to avoid overspecifying what does not matter in the end.

Thanks.

I'd always assumed the difference was to allow scripts to distinguish between an invalid option and the ref not existing, like "git rev-parse --quiet --verify" and "git show-ref --quiet --verify".

Thanks

Phillip

I would favor continuing to not test for the exact error code, and would rather not disturb anything scripts may already rely on.

In either case the git-symbolic-ref.adoc NOTES are out of sync with the actual behavior:

'git symbolic-ref' will exit with status 0 [...], with status 1 if the requested name is not a symbolic ref, or 128 if another error occurs.

To describe the actual behavior:

'git symbolic-ref' will exit with status 0 if the contents of the symbolic ref were printed correctly, with status 1 if the requested name is not a symbolic ref and --quiet was given, or with status 128 if it is not a symbolic ref and --quiet was not given, or another error occurs.

If that sounds OK I can update the adoc and call it good.

Thanks,
Nikolaus

This is the kind of "casting wrong behaviour into stone by either documentation or tests" that I advised against in my earlier message.

From: Nikolaus Schuetz <[email protected]>

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref fails, and that --quiet does so silently. Tests such as t2020 and t5621 already rely on "symbolic-ref -q HEAD" failing on a detached HEAD, but none checks that the plain form reports the error or that --quiet stays silent.

Assert that a non-symbolic ref fails with the "is not a symbolic ref" message, and that --quiet fails with no output. Use test_must_fail rather than pinning the exact exit codes, which are documented but not worth freezing in the test.

---
t1401: test symbolic-ref exit codes on a non-symbolic ref

git-symbolic-ref(1) documents that reading a name that is not a symbolic ref exits with a non-zero status, and that --quiet does so silently rather than printing a diagnostic. This exit-code contract was untested.

This adds two tests: querying a non-symbolic ref exits 128 with the usual "is not a symbolic ref" message, and --quiet instead exits 1 with no output.

Test-only; documents existing behaviour, in the spirit of 919eb8ace (t1402: check for refs ending with a dot).

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2204%2Fnikolauspschuetz%2Fns%2Ft1401-symbolic-ref-quiet-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2204/nikolauspschuetz/ns/t1401-symbolic-ref-quiet-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2204

Range-diff vs v2:

1: 22694da869 ! 1: 0391dcceef t1401: check symbolic-ref exit codes and --quiet silence
@@ Metadata
Author: Nikolaus Schuetz <[email protected]>

## Commit message ##
- t1401: check symbolic-ref exit codes and --quiet silence
+ t1401: check symbolic-ref failure and --quiet silence on a non-symbolic ref

git-symbolic-ref(1) documents that reading a name that is not a
- symbolic ref exits non-zero, and that --quiet does so silently.
- Tests such as t2020 and t5621 already rely on "symbolic-ref -q HEAD"
- failing on a detached HEAD, but none pins the exact exit codes or
- checks that --quiet actually suppresses the diagnostic.
+ symbolic ref fails, and that --quiet does so silently. Tests such as
+ t2020 and t5621 already rely on "symbolic-ref -q HEAD" failing on a
+ detached HEAD, but none checks that the plain form reports the error
+ or that --quiet stays silent.

- Assert that a non-symbolic ref exits 128 with the "is not a symbolic
- ref" message, and that --quiet instead exits 1 with no output.
+ Assert that a non-symbolic ref fails with the "is not a symbolic ref"
+ message, and that --quiet fails with no output. Use test_must_fail
+ rather than pinning the exact exit codes, which are documented but not
+ worth freezing in the test.

@@ t/t1401-symbolic-ref.sh: test_expect_success 'symbolic-ref refuses bare sha1' '

reset_to_sane

-+test_expect_success 'symbolic-ref reports a non-symbolic ref with exit code 128' '
-+ test_expect_code 128 git symbolic-ref refs/heads/foo >out 2>err &&
++test_expect_success 'symbolic-ref reports a non-symbolic ref' '
++ test_must_fail git symbolic-ref refs/heads/foo >out 2>err &&
+ test_must_be_empty out &&
+ test_grep "is not a symbolic ref" err
+'
+
-+test_expect_success 'symbolic-ref -q is silent and exits 1 on a non-symbolic ref' '
-+ test_expect_code 1 git symbolic-ref -q refs/heads/foo >out 2>err &&
++test_expect_success 'symbolic-ref -q is silent on a non-symbolic ref' '
++ test_must_fail git symbolic-ref -q refs/heads/foo >out 2>err &&
+ test_must_be_empty out &&
+ test_must_be_empty err
+'

 t/t1401-symbolic-ref.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index a2a7e94716..fd3aa89a91 100755
@@ -38,6 +38,18 @@ test_expect_success 'symbolic-ref refuses bare sha1' '
 
 reset_to_sane
 
+test_expect_success 'symbolic-ref reports a non-symbolic ref' '
+	test_must_fail git symbolic-ref refs/heads/foo >out 2>err &&
+	test_must_be_empty out &&
+	test_grep "is not a symbolic ref" err
+'
+
+test_expect_success 'symbolic-ref -q is silent on a non-symbolic ref' '
+	test_must_fail git symbolic-ref -q refs/heads/foo >out 2>err &&
+	test_must_be_empty out &&
+	test_must_be_empty err
+'
+
 test_expect_success 'HEAD cannot be removed' '
 	test_must_fail git symbolic-ref -d HEAD
 '

base-commit: 745601a9a94110d74769ab605ccd4f61339758d2

gitgitgadget

Documented sure, but the behaviour does not match the documentation:

git symbolic-ref will exit with status 0 if the contents of the symbolic ref were printed correctly, with status 1 if the requested name is not a symbolic ref, or 128 if another error occurs.

So in theory we should always exit with 1 in case the refname is not a symbolic reference, no matter whether the user passes "--quiet" or not.

Anyway, we're not asserting the current broken behaviour in the new tests anymore, so I think that this version is good enough. Thanks!

Patrick