Skip to contentSkip to conversation

Hardcoded #!/bin/sh in t5532 causes problems on Solaris

DISCUSSION Tom G. Christensen 4 participants 14 messages Jun 16, 2016

lore
Share & tools
lore
#0 lore raw

Hello,

Looking at the testsuite results on Solaris I see a failure in t5532.3.

Running the testsuite with -v -i revealed a shell syntax error:

proxying for example.com 9418
./proxy: syntax error at line 3: `cmd=$' unexpected not ok 3 - fetch through proxy works
#

#               git fetch fake &&
#               echo one >expect &&
#               git log -1 --format=%s FETCH_HEAD >actual &&
#               test_cmp expect actual

#

Looking a t5532-fetch-proxy.sh the problem is obvious, it writes out a helper script which explicitly uses #!/bin/sh but fails to take into account that systems like Solaris has an ancient /bin/sh that knows nothing about POSIX things like $(). Replacing $() with `` was enough to make the test pass.

-tgc

#1 lore raw
Quoted 12 lines · Tom G. Christensen

Looking at the testsuite results on Solaris I see a failure in t5532.3.

Running the testsuite with -v -i revealed a shell syntax error:

proxying for example.com 9418
./proxy: syntax error at line 3: `cmd=$' unexpected not ok 3 - fetch through proxy works
#

#               git fetch fake &&
#               echo one >expect &&
#               git log -1 --format=%s FETCH_HEAD >actual &&
#               test_cmp expect actual
Quoted 8 lines

#

Looking a t5532-fetch-proxy.sh the problem is obvious, it writes out a helper script which explicitly uses #!/bin/sh but fails to take into account that systems like Solaris has an ancient /bin/sh that knows nothing about POSIX things like $().
Replacing $() with `` was enough to make the test pass.

Right, this is a recent regression from Elia's $() series. Rather than just revert, I think the cleanup below is the best fix.

I did some quick grepping around, and I suspect you may run into the same thing in other places (e.g., t3404.40 looks like a similar case). We left a lot of "#!/bin/sh" cases unconverted to write_script because we knew what they were doing was trivial enough not to matter. But the $() conversion made them non-trivial.

-- >8 --
Subject: [PATCH] t5532: use write_script

The recent cleanup in b7cbbff switched t5532's use of backticks to $(). This matches our normal shell style, which is good. But it also breaks the test on Solaris, where /bin/sh does not understand $().

Our normal shell style assumes a modern-ish shell which knows about $(). However, some tests create small helper scripts and just write "#!/bin/sh" into them. These scripts either need to go back to using backticks, or they need to respect $SHELL_PATH. The easiest way to do the latter is to use write_script.

While we're at it, let's also stick the script creation inside a test_expect block (our usual style), and split the perl snippet into its own script (to prevent quoting madness).

---
 t/t5532-fetch-proxy.sh | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
t/t5532-fetch-proxy.sh +12 -9
index d75ef0e..51c9669 100755
@@ -12,10 +12,8 @@ test_expect_success 'setup remote repo' '
 	)
 '
 
-cat >proxy <<'EOF'
-#!/bin/sh
-echo >&2 "proxying for $*"
-cmd=$("$PERL_PATH" -e '
+test_expect_success 'setup proxy script' '
+	write_script proxy-get-cmd "$PERL_PATH" <<-\EOF &&
 	read(STDIN, $buf, 4);
 	my $n = hex($buf) - 4;
 	read(STDIN, $buf, $n);
@@ -23,11 +21,16 @@ cmd=$("$PERL_PATH" -e '
 	# drop absolute-path on repo name
 	$cmd =~ s{ /}{ };
 	print $cmd;
-')
-echo >&2 "Running '$cmd'"
-exec $cmd
-EOF
-chmod +x proxy
+	EOF
+
+	write_script proxy <<-\EOF
+	echo >&2 "proxying for $*"
+	cmd=$(./proxy-get-cmd)
+	echo >&2 "Running $cmd"
+	exec $cmd
+	EOF
+'
+
 test_expect_success 'setup local repo' '
 	git remote add fake git://example.com/remote &&
 	git config core.gitproxy ./proxy
Signature · 1 line
2.8.1.245.g18e0f5c
#2 lore raw
Quoted 3 lines · Jeff King

I did some quick grepping around, and I suspect you may run into the same thing in other places (e.g., t3404.40 looks like a similar case).

There are only a few tests that fail and just t5532.3 seems affected by this issue.

Quoted 19 lines

Subject: [PATCH] t5532: use write_script

The recent cleanup in b7cbbff switched t5532's use of backticks to $(). This matches our normal shell style, which is good. But it also breaks the test on Solaris, where /bin/sh does not understand $().

Our normal shell style assumes a modern-ish shell which knows about $(). However, some tests create small helper scripts and just write "#!/bin/sh" into them. These scripts either need to go back to using backticks, or they need to respect $SHELL_PATH. The easiest way to do the latter is to use write_script.

While we're at it, let's also stick the script creation inside a test_expect block (our usual style), and split the perl snippet into its own script (to prevent quoting madness).

Quoted 5 lines
---
  t/t5532-fetch-proxy.sh | 21 ++++++++++++---------
  1 file changed, 12 insertions(+), 9 deletions(-)

I applied this to 2.8.1 and as expected the test now passes on Solaris.

-tgc

#3 lore raw
Quoted 7 lines · Tom G. Christensen
Quoted 3 lines · Jeff King

I did some quick grepping around, and I suspect you may run into the same thing in other places (e.g., t3404.40 looks like a similar case).

There are only a few tests that fail and just t5532.3 seems affected by this issue.

Hmm. t3404.40 does this:

        echo "#!/bin/sh" > $PRE_COMMIT &&
	echo "test -z \"\$(git diff --cached --check)\"" >>$PRE_COMMIT &&
	chmod a+x $PRE_COMMIT &&

So I'm pretty sure that $PRE_COMMIT script should be barfing each time it is called on Solaris. I think the test itself doesn't notice because "/bin/sh barfed" and "the pre-commit check said no" look the same from git's perspective (both non-zero exits), and we test only cases where we expect the hook to fail.

I think that particular test could simplify its pre-commit hook to just "exit 1".

I didn't dig into any other cases, so that might be the only one. If you're not seeing problems, I'm not inclined to explore each one manually.

Quoted 1 line

I applied this to 2.8.1 and as expected the test now passes on Solaris.

Thanks.

-Peff

#4 lore raw
Quoted 11 lines · Jeff King

Hmm. t3404.40 does this:

        echo "#!/bin/sh" > $PRE_COMMIT &&
	echo "test -z \"\$(git diff --cached --check)\"" >>$PRE_COMMIT &&
	chmod a+x $PRE_COMMIT &&

So I'm pretty sure that $PRE_COMMIT script should be barfing each time it is called on Solaris. I think the test itself doesn't notice because "/bin/sh barfed" and "the pre-commit check said no" look the same from git's perspective (both non-zero exits), and we test only cases where we expect the hook to fail.

I looked at

    $ git grep -c '#! */bin/sh' t | grep -v ':1$'

and did a few just for fun. Doing it fully may be a good microproject for next year ;-)

 t/t1020-subdirectory.sh       |  6 +++---
 t/t2050-git-dir-relative.sh   | 11 ++++++-----
 t/t3404-rebase-interactive.sh |  7 +++----
 3 files changed, 12 insertions(+), 12 deletions(-)
t/t1020-subdirectory.sh +3 -3
index 8e22b03..6dedb1c 100755
@@ -142,9 +142,9 @@ test_expect_success 'GIT_PREFIX for built-ins' '
 	# Use GIT_EXTERNAL_DIFF to test that the "diff" built-in
 	# receives the GIT_PREFIX variable.
 	printf "dir/" >expect &&
-	printf "#!/bin/sh\n" >diff &&
-	printf "printf \"\$GIT_PREFIX\"" >>diff &&
-	chmod +x diff &&
+	write_script diff <<-\EOF &&
+	printf "%s" "$GIT_PREFIX"
+	EOF
 	(
 		cd dir &&
 		printf "change" >two &&
t/t2050-git-dir-relative.sh +6 -5
index 21f4659..7a05b20 100755
@@ -18,11 +18,12 @@ COMMIT_FILE="$(pwd)/output"
 export COMMIT_FILE
 
 test_expect_success 'Setting up post-commit hook' '
-mkdir -p .git/hooks &&
-echo >.git/hooks/post-commit "#!/bin/sh
-touch \"\${COMMIT_FILE}\"
-echo Post commit hook was called." &&
-chmod +x .git/hooks/post-commit'
+	mkdir -p .git/hooks &&
+	write_script .git/hooks/post-commit <<-\EOF
+	>"${COMMIT_FILE}"
+	echo Post commit hook was called.
+	EOF
+'
 
 test_expect_success 'post-commit hook used ordinarily' '
 echo initial >top &&
t/t3404-rebase-interactive.sh +3 -4
index b79f442..d96d0e4 100755
@@ -555,10 +555,9 @@ test_expect_success 'rebase a detached HEAD' '
 test_expect_success 'rebase a commit violating pre-commit' '
 
 	mkdir -p .git/hooks &&
-	PRE_COMMIT=.git/hooks/pre-commit &&
-	echo "#!/bin/sh" > $PRE_COMMIT &&
-	echo "test -z \"\$(git diff --cached --check)\"" >> $PRE_COMMIT &&
-	chmod a+x $PRE_COMMIT &&
+	write_script .git/hooks/pre-commit <<-\EOF &&
+	test -z "$(git diff --cached --check)"
+	EOF
 	echo "monde! " >> file1 &&
 	test_tick &&
 	test_must_fail git commit -m doesnt-verify file1 &&
#5 lore raw
Quoted 5 lines · Junio C Hamano
t/t3404-rebase-interactive.sh +0 -0
index b79f442..d96d0e4 100755

t/t3404-rebase-interactive.sh test_expect_success 'rebase a detached HEAD' ' · line 560

Quoted 10 lines
t/t3404-rebase-interactive.sh +3 -4
@@ -555,10 +555,9 @@ test_expect_success 'rebase a detached HEAD' '
 test_expect_success 'rebase a commit violating pre-commit' '
 
 	mkdir -p .git/hooks &&
-	PRE_COMMIT=.git/hooks/pre-commit &&
-	echo "#!/bin/sh" > $PRE_COMMIT &&
-	echo "test -z \"\$(git diff --cached --check)\"" >> $PRE_COMMIT &&
-	chmod a+x $PRE_COMMIT &&
+	write_script .git/hooks/pre-commit <<-\EOF &&
+	test -z "$(git diff --cached --check)"
+	EOF

Looks good and is the minimal change. I kind of wonder if the example would be more clear, though, as just:

write_script .git/hooks/pre-commit <<-\EOF &&
exit 1
EOF
echo whatever >file1 &&
...

I don't think we ever actually need the pre-commit check to pass, as we simply override it with --no-verify. But I dunno. Maybe people find it easier to read with a pseudo-realistic example (it took me a minute to realize the trailing whitespace in the content was important).

It could also stand to clean up its hook with test_when_finished. The next test resorts to "rm -rf" on the hooks directory at the beginning. Yuck.

-Peff

#6 lore raw
Quoted 27 lines · Jeff King
Quoted 5 lines · Junio C Hamano
t/t3404-rebase-interactive.sh +0 -0
index b79f442..d96d0e4 100755

t/t3404-rebase-interactive.sh test_expect_success 'rebase a detached HEAD' ' · line 560

Quoted 10 lines
t/t3404-rebase-interactive.sh +3 -4
@@ -555,10 +555,9 @@ test_expect_success 'rebase a detached HEAD' '
 test_expect_success 'rebase a commit violating pre-commit' '
 
 	mkdir -p .git/hooks &&
-	PRE_COMMIT=.git/hooks/pre-commit &&
-	echo "#!/bin/sh" > $PRE_COMMIT &&
-	echo "test -z \"\$(git diff --cached --check)\"" >> $PRE_COMMIT &&
-	chmod a+x $PRE_COMMIT &&
+	write_script .git/hooks/pre-commit <<-\EOF &&
+	test -z "$(git diff --cached --check)"
+	EOF

Looks good and is the minimal change. I kind of wonder if the example would be more clear, though, as just:

write_script .git/hooks/pre-commit <<-\EOF &&
exit 1
EOF
echo whatever >file1 &&
...

Quoted 4 lines

I don't think we ever actually need the pre-commit check to pass, as we simply override it with --no-verify. But I dunno. Maybe people find it easier to read with a pseudo-realistic example (it took me a minute to realize the trailing whitespace in the content was important).

I was mostly worried about closing the door for future enhancement where there are multiple commits to be replayed, some of which fail and others pass the test. Unconditional "exit 1" would have to be reverted when it happens.

Quoted 3 lines

It could also stand to clean up its hook with test_when_finished. The next test resorts to "rm -rf" on the hooks directory at the beginning. Yuck.

Yeah, that may be an accident waiting to happen.

#7 lore raw
Quoted 14 lines · Junio C Hamano
Quoted 9 lines

Looks good and is the minimal change. I kind of wonder if the example would be more clear, though, as just:

write_script .git/hooks/pre-commit <<-\EOF &&
exit 1
EOF
echo whatever >file1 &&
...

Quoted 4 lines

I don't think we ever actually need the pre-commit check to pass, as we simply override it with --no-verify. But I dunno. Maybe people find it easier to read with a pseudo-realistic example (it took me a minute to realize the trailing whitespace in the content was important).

Quoted 4 lines

I was mostly worried about closing the door for future enhancement where there are multiple commits to be replayed, some of which fail and others pass the test. Unconditional "exit 1" would have to be reverted when it happens.

Yeah, that's fair. It is at least trying to re-create a real-world situation.

-Peff

#8 lore raw
In reply to Junio C Hamano
Quoted 21 lines · Junio C Hamano

Jeff King <[email protected]> writes:
...

Quoted 9 lines

Looks good and is the minimal change. I kind of wonder if the example would be more clear, though, as just:

write_script .git/hooks/pre-commit <<-\EOF &&
exit 1
EOF
echo whatever >file1 &&
...

Quoted 4 lines

I don't think we ever actually need the pre-commit check to pass, as we simply override it with --no-verify. But I dunno. Maybe people find it easier to read with a pseudo-realistic example (it took me a minute to realize the trailing whitespace in the content was important).

I was mostly worried about closing the door for future enhancement where there are multiple commits to be replayed, some of which fail and others pass the test. Unconditional "exit 1" would have to be reverted when it happens.

Quoted 5 lines
Quoted 3 lines

It could also stand to clean up its hook with test_when_finished. The next test resorts to "rm -rf" on the hooks directory at the beginning. Yuck.

Yeah, that may be an accident waiting to happen.

In any case, lest we forget...

-- >8 --
Subject: [PATCH] t3404: use write_script

The test uses hardcoded #!/bin/sh to create a pre-commit hook script. Because the generated script uses $(command substitution), which is not supported by /bin/sh on some platforms (e.g. Solaris), the resulting pre-commit always fails.

Which is not noticeable as the test that uses the hook is about checking the behaviour of the command when the hook fails ;-), but nevertheless it is not testing what we wanted to test.

Use write_script so that the resulting script is run under the same shell our scripted Porcelain commands are run, which must support the necessary $(construct).

---
 t/t3404-rebase-interactive.sh | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
t/t3404-rebase-interactive.sh +3 -4
index 544f9ad..d6d65a3 100755
@@ -555,10 +555,9 @@ test_expect_success 'rebase a detached HEAD' '
 test_expect_success 'rebase a commit violating pre-commit' '
 
 	mkdir -p .git/hooks &&
-	PRE_COMMIT=.git/hooks/pre-commit &&
-	echo "#!/bin/sh" > $PRE_COMMIT &&
-	echo "test -z \"\$(git diff --cached --check)\"" >> $PRE_COMMIT &&
-	chmod a+x $PRE_COMMIT &&
+	write_script .git/hooks/pre-commit <<-\EOF &&
+	test -z "$(git diff --cached --check)"
+	EOF
 	echo "monde! " >> file1 &&
 	test_tick &&
 	test_must_fail git commit -m doesnt-verify file1 &&
Signature · 1 line
2.8.1-339-gc925d85
#9 lore raw
Quoted 4 lines · Junio C Hamano

In any case, lest we forget...

-- >8 --
Subject: [PATCH] t3404: use write_script

Yep, looks good. Thanks.

-Peff

#10 lore raw
In reply to Junio C Hamano
Quoted 19 lines · Junio C Hamano

I looked at

    $ git grep -c '#! */bin/sh' t | grep -v ':1$'

and did a few just for fun. Doing it fully may be a good microproject for next year ;-)

 t/t1020-subdirectory.sh       |  6 +++---
 t/t2050-git-dir-relative.sh   | 11 ++++++-----
 t/t3404-rebase-interactive.sh |  7 +++----
 3 files changed, 12 insertions(+), 12 deletions(-)
t/t1020-subdirectory.sh +0 -0
index 8e22b03..6dedb1c 100755
@@ -142,9 +142,9 @@ test_expect_success 'GIT_PREFIX for built-ins' '
 	# Use GIT_EXTERNAL_DIFF to test that the "diff" built-in
 	# receives the GIT_PREFIX variable.

t/t1020-subdirectory.sh test_expect_success 'GIT_PREFIX for built-ins' ' · line 150

Quoted 10 lines
t/t1020-subdirectory.sh +3 -3
 	printf "dir/" >expect &&
-	printf "#!/bin/sh\n" >diff &&
-	printf "printf \"\$GIT_PREFIX\"" >>diff &&
-	chmod +x diff &&
+	write_script diff <<-\EOF &&
+	printf "%s" "$GIT_PREFIX"
+	EOF
 	(
 		cd dir &&
 		printf "change" >two &&

Regarding this one, I notice that "expect" and "actual" (produced later in this script by executing "diff" script) are eventually compared by test_cmp, which runs "diff" to show the actual differences. If we are doing this modernization to use write_script more, we probably should make "expect" and "actual" text files that end with a complete line.

I.e.

-- >8 --
Subject: t1020: do not overuse printf and use write_script

The test prepares a sample file "dir/two" with a single incomplete line in it with "printf", and also prepares a small helper script "diff" to create a file with a single incomplete line in it, again with "printf". The output from the latter is compared with an expected output, again prepared with "printf" hance lacking the final LF. There is no reason for this test to be using files with an incomplete line at the end, and these look more like a mistake of not using

printf "%s\n" "string to be written"

and using

printf "string to be written"

Depending on what would be in $GIT_PREFIX, using the latter form could be a bug waiting to happen. Correct them.

Also, the test uses hardcoded #!/bin/sh to create a small helper script. For a small task like what the generated script does, it does not matter too much in that what appears as /bin/sh would not be _so_ broken, but while we are at it, use write_script instead, which happens to make the result easier to read by reducing need of one level of quoting.

---
 t/t1020-subdirectory.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
t/t1020-subdirectory.sh +5 -5
index 8e22b03..df3183e 100755
@@ -141,13 +141,13 @@ test_expect_success 'GIT_PREFIX for !alias' '
 test_expect_success 'GIT_PREFIX for built-ins' '
 	# Use GIT_EXTERNAL_DIFF to test that the "diff" built-in
 	# receives the GIT_PREFIX variable.
-	printf "dir/" >expect &&
-	printf "#!/bin/sh\n" >diff &&
-	printf "printf \"\$GIT_PREFIX\"" >>diff &&
-	chmod +x diff &&
+	echo "dir/" >expect &&
+	write_script diff <<-\EOF &&
+	printf "%s\n" "$GIT_PREFIX"
+	EOF
 	(
 		cd dir &&
-		printf "change" >two &&
+		echo "change" >two &&
 		GIT_EXTERNAL_DIFF=./diff git diff >../actual
 		git checkout -- two
 	) &&
#11 lore raw
Quoted 16 lines · Junio C Hamano
Quoted 16 lines
t/t1020-subdirectory.sh +3 -3
index 8e22b03..6dedb1c 100755
@@ -142,9 +142,9 @@ test_expect_success 'GIT_PREFIX for built-ins' '
 	# Use GIT_EXTERNAL_DIFF to test that the "diff" built-in
 	# receives the GIT_PREFIX variable.
 	printf "dir/" >expect &&
-	printf "#!/bin/sh\n" >diff &&
-	printf "printf \"\$GIT_PREFIX\"" >>diff &&
-	chmod +x diff &&
+	write_script diff <<-\EOF &&
+	printf "%s" "$GIT_PREFIX"
+	EOF
 	(
 		cd dir &&
Quoted 8 lines

t/t1020-subdirectory.sh test_expect_success 'GIT_PREFIX for built-ins' ' · line 150

Quoted 1 line
t/t1020-subdirectory.sh +0 -0
 		printf "change" >two &&

Regarding this one, I notice that "expect" and "actual" (produced later in this script by executing "diff" script) are eventually compared by test_cmp, which runs "diff" to show the actual differences. If we are doing this modernization to use write_script more, we probably should make "expect" and "actual" text files that end with a complete line.

Yeah I wondered about that. And also the fact that the shell script itself doesn't end in newline. But I think that is just an accident, and no shell happened to complain (not that I would expect them to, but we come across enough weirdness around final newlines with tools like sed and tr, I wouldn't have been surprised).

Quoted 21 lines

-- >8 --
Subject: t1020: do not overuse printf and use write_script

The test prepares a sample file "dir/two" with a single incomplete line in it with "printf", and also prepares a small helper script "diff" to create a file with a single incomplete line in it, again with "printf". The output from the latter is compared with an expected output, again prepared with "printf" hance lacking the final LF. There is no reason for this test to be using files with an incomplete line at the end, and these look more like a mistake of not using

printf "%s\n" "string to be written"

and using

printf "string to be written"

Depending on what would be in $GIT_PREFIX, using the latter form could be a bug waiting to happen. Correct them.

Quoted 6 lines

Also, the test uses hardcoded #!/bin/sh to create a small helper script. For a small task like what the generated script does, it does not matter too much in that what appears as /bin/sh would not be _so_ broken, but while we are at it, use write_script instead, which happens to make the result easier to read by reducing need of one level of quoting.

Looks good to me. I suspect you could actually just use:

echo "$GIT_PREFIX"

in the helper script. That is also not completely safe against arbitrary bytes in $GIT_PREFIX (due to unportable backslash escapes), though I suspect it would be fine for the purposes of the test script. Using a proper printf isn't that many more bytes, though.

-Peff

#12 lore raw
In reply to Junio C Hamano
Quoted 7 lines · Junio C Hamano

Subject: t1020: do not overuse printf and use write_script

The test prepares a sample file "dir/two" with a single incomplete line in it with "printf", and also prepares a small helper script "diff" to create a file with a single incomplete line in it, again with "printf". The output from the latter is compared with an expected output, again prepared with "printf" hance lacking the

s/hance/hence/

Quoted 21 lines

final LF. There is no reason for this test to be using files with an incomplete line at the end, and these look more like a mistake of not using

printf "%s\n" "string to be written"

and using

printf "string to be written"

Depending on what would be in $GIT_PREFIX, using the latter form could be a bug waiting to happen. Correct them.

Also, the test uses hardcoded #!/bin/sh to create a small helper script. For a small task like what the generated script does, it does not matter too much in that what appears as /bin/sh would not be _so_ broken, but while we are at it, use write_script instead, which happens to make the result easier to read by reducing need of one level of quoting.

#13 lore raw
Quoted 4 lines · Eric Sunshine
Quoted 2 lines

with "printf". The output from the latter is compared with an expected output, again prepared with "printf" hance lacking the

s/hance/hence/

Thanks