Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1
Subtests t5510.69, 71, 97, 99, 125, 127, 147, 149, 151, 153, 155, 157, 159, 161, 163, 165, 167, 170 failed in rc1 on the NonStop ia64 platform. These were working in rc0, but after analysis, I do not think that is relevant. Here is what we see for subtest 69:
expecting success of 5510.69 'link prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune origin refs/tags/*:refs/tags/*; branch:kept tag:pruned':
...
git fetch "file:///home/jenkins/.jenkins/workspace/Git_Pipeline@2/t/trash directory.t5510-fetch/." "+refs/heads/*:refs/remotes/origin/*" &&
...
git$git_fetch_c fetch --prune "file:///home/jenkins/.jenkins/workspace/Git_Pipeline/t/trash directory.t5510-fetch/." refs/tags/*:refs/tags/* &&What seems to be happening is that the test script is computing the wrong parent directory in the second case, cutting the @2. We use Jenkins for our builds and occasional multiple builds run. The @2 is Jenkins way of running in another place. This seems to be confusing the script parsing of the parent, causing the issue. When I rebuilt and test in the primary directory, the failures disappeared because no @2.
t5562 also seems to have the same issue.
Quoted 9 lines · On 6/20/22 2:04 PM, [email protected] wrote:
Subtests t5510.69, 71, 97, 99, 125, 127, 147, 149, 151, 153, 155, 157, 159, 161, 163, 165, 167, 170 failed in rc1 on the NonStop ia64 platform. These were working in rc0, but after analysis, I do not think that is relevant. Here is what we see for subtest 69:
expecting success of 5510.69 'link prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune origin refs/tags/*:refs/tags/*; branch:kept tag:pruned':
...git fetch "file:///home/jenkins/.jenkins/workspace/Git_Pipeline@2/t/trash directory.t5510-fetch/." "+refs/heads/*:refs/remotes/origin/*" && ... git$git_fetch_c fetch --prune "file:///home/jenkins/.jenkins/workspace/Git_Pipeline/t/trash directory.t5510-fetch/." refs/tags/*:refs/tags/* &&What seems to be happening is that the test script is computing the wrong parent directory in the second case, cutting the @2. We use Jenkins for our builds and occasional multiple builds run. The @2 is Jenkins way of running in another place. This seems to be confusing the script parsing of the parent, causing the issue. When I rebuilt and test in the primary directory, the failures disappeared because no @2.
Yes, the script is incorrectly computing the remote URL. This is not a bug in Git, but a bug in the test script.
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g')
At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).
Thanks,
-Stolee
Quoted 33 lines · Derrick Stolee
Quoted 1 line · On 6/20/22 2:04 PM, [email protected] wrote:
Subtests t5510.69, 71, 97, 99, 125, 127, 147, 149, 151, 153, 155, 157, 159, 161, 163,
165, 167, 170 failed in rc1 on the NonStop ia64 platform. These were working in rc0, but after analysis, I do not think that is relevant. Here is what we see for subtest 69:
Quoted 2 lines
expecting success of 5510.69 'link prune fetch.prune=unset
remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune origin refs/tags/*:refs/tags/*; branch:kept tag:pruned':
Quoted 3 lines
...
git fetch
"file:///home/jenkins/.jenkins/workspace/Git_Pipeline@2/t/trashdirectory.t5510-fetch/." "+refs/heads/*:refs/remotes/origin/*" && ...
Quoted 5 lines
git$git_fetch_c fetch --prune"file:///home/jenkins/.jenkins/workspace/Git_Pipeline/t/trash directory.t5510-fetch/." refs/tags/*:refs/tags/* &&
What seems to be happening is that the test script is computing the wrong
parent directory in the second case, cutting the @2. We use Jenkins for our builds and occasional multiple builds run. The @2 is Jenkins way of running in another place. This seems to be confusing the script parsing of the parent, causing the issue. When I rebuilt and test in the primary directory, the failures disappeared because no @2.
Yes, the script is incorrectly computing the remote URL. This is not a bug in Git, but a bug in the test script.
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')
Quoted 8 lines
At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).
How about using sed instead of perl for this?
Quoted 10 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 8 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
Quoted 7 lines
Quoted 5 lines
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee
--- >8 ---
From 1df4fc66d4a62adc7087d7d22c8d78842b4e9b4d Mon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefully
The many test_configured_prune tests in t5510-fetch.sh test many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced in e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.
However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
t/t5510-fetch.sh +3 -1
@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ '$remote_url'~g") fi if test "$fetch_prune_tags" = 'true' ||
Signature · 1 line
2.36.1.vfs.0.0
Quoted 14 lines · Derrick Stolee
Quoted 13 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 11 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
In a 'sed s/regexp/replacement/' command the replacement part has a few characters with special meaning, and if those characters happen to appear in the path of the trash directory, then:
$ ./t5510-fetch.sh --root='/tmp/foo\1/' ok 64 - name prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --no-prune; branch:kept tag:kept sed: -e expression #1, char 62: invalid reference \1 on `s' command's RHS sed: -e expression #1, char 62: invalid reference \1 on `s' command's RHS ok 65 - link prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --no-prune; branch:kept tag:kept ok 66 - name prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune; branch:pruned tag:kept sed: -e expression #1, char 62: invalid reference \1 on `s' command's RHS sed: -e expression #1, char 62: invalid reference \1 on `s' command's RHS not ok 67 - link prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune; branch:pruned tag:kept [...] # failed 28 among 183 test(s)
Quoted 50 lines
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee--- >8 ---
Quoted 1 line
From
1df4fc66d4a62adc7087d7d22c8d78842b4e9b4dMon Sep 17 00:00:00 2001From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefullyThe many
test_configured_prunetests int5510-fetch.shtest many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced ine1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
Reported-by Randall Becker <[email protected]>Signed-off-by Derrick Stolee <[email protected]>--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)t/t5510-fetch.sh +0
-0
t/t5510-fetch.sh test_configured_prune_type () { · line 858
Quoted 7 lines
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ '$remote_url'~g")
'sed' can run multiple commands at once, e.g. 'sed -e s/// -e s///'.
Quoted 8 lines
t/t5510-fetch.sh +0
-0fi if test "$fetch_prune_tags" = 'true' ||Signature · 1 line
2.36.1.vfs.0.0
Quoted 72 lines · Derrick Stolee
Quoted 10 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 8 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
Quoted 7 lines
Quoted 5 lines
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee--- >8 ---
From
1df4fc66d4a62adc7087d7d22c8d78842b4e9b4dMon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefullyThe many
test_configured_prunetests int5510-fetch.shtest many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced ine1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
Reported-by Randall Becker <[email protected]>Signed-off-by Derrick Stolee <[email protected]>--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)t/t5510-fetch.sh +0
-0
t/t5510-fetch.sh test_configured_prune_type () { · line 861
Quoted 10 lines
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ '$remote_url'~g") fi if test "$fetch_prune_tags" = 'true' ||
Thanks for looking at this. Checking this out again this whole quoting mess is a bit of a ... mess, I wonder if there's some better way to avoid this. Anyway:
So, is this functionally the same as:
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 4620f0ca7fa..9cd8b36f835 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -853,7 +853,9 @@ test_configured_prune_type () {
then
new_cmdline=$cmdline_setup
else
- new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g')
+ new_cmdline=$(printf "%s" "$cmdline" |
+ perl -pe 's[origin ]["'"$remote_url"'" ]g' |
+ perl -pe 's[ origin][ "'"$remote_url"'"]g')
fi
if test "$fetch_prune_tags" = 'true' ||?
I don't mind the migration to "sed", but doing so to fix a bug makes this especially hard to analyze. I.e. you've gotten rid of the (?!/), I haven't re-looked at this enough to see if/how that's important.
I just came up with the above as a quick hack, but for any proper migration to sed can't we do this in one command?
In any case you never need "| \" in your scripts, just end the line with "|".
I was expecting you to use \Q...\E (and passing $remote_url as an argument to perl script) actually. If you let $remote_url interpolated by shell into a script of the host language, whether perl or sed, you'd be responsible for quoting it appropriately for the host language yourself, and use of single-quote pair would not necessarily be sufficient, no?
Quoted 120 lines · Ævar Arnfjörð Bjarmason
Quoted 72 lines · Derrick Stolee
Quoted 10 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 8 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
Quoted 7 lines
Quoted 5 lines
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee--- >8 ---
From
1df4fc66d4a62adc7087d7d22c8d78842b4e9b4dMon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefullyThe many
test_configured_prunetests int5510-fetch.shtest many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced ine1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
Reported-by Randall Becker <[email protected]>Signed-off-by Derrick Stolee <[email protected]>--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)t/t5510-fetch.sh +0
-0t/t5510-fetch.sh test_configured_prune_type () { · line 861
Quoted 10 lines
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ '$remote_url'~g") fi if test "$fetch_prune_tags" = 'true' ||Thanks for looking at this. Checking this out again this whole quoting mess is a bit of a ... mess, I wonder if there's some better way to avoid this. Anyway:
So, is this functionally the same as:
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index 4620f0ca7fa..9cd8b36f835 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | + perl -pe 's[origin ]["'"$remote_url"'" ]g' | + perl -pe 's[ origin][ "'"$remote_url"'"]g') fi if test "$fetch_prune_tags" = 'true' ||?
I don't mind the migration to "sed", but doing so to fix a bug makes this especially hard to analyze. I.e. you've gotten rid of the (?!/), I haven't re-looked at this enough to see if/how that's important.
I just came up with the above as a quick hack, but for any proper migration to sed can't we do this in one command?
In any case you never need "| \" in your scripts, just end the line with "|".
Quoted 5 lines · Junio C Hamano
I was expecting you to use \Q...\E (and passing $remote_url as an argument to perl script) actually. If you let $remote_url interpolated by shell into a script of the host language, whether perl or sed, you'd be responsible for quoting it appropriately for the host language yourself, and use of single-quote pair would not necessarily be sufficient, no?
That would make it easier to debug. In the mean time, I am trying to get the environment to test with the problematic path and am having issues after renaming the directory.
Quoted 130 lines
Quoted 7 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Derrick Stolee
Quoted 1 line · On 6/20/22 2:59 PM, [email protected] wrote:
On June 20, 2022 2:46 PM, Derrick Stolee wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the
@ to the next slash.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing
other issues.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind
of issue.
Quoted 47 lines
Quoted 47 lines
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee--- >8 ---
From
1df4fc66d4a62adc7087d7d22c8d78842b4e9b4dMon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefullyThe many
test_configured_prunetests int5510-fetch.shtest many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced ine1790f9245f(fetch
tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
Reported-by Randall Becker <[email protected]>Signed-off-by Derrick Stolee <[email protected]>--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Quoted 9 lines
t/t5510-fetch.sh test_configured_prune_type () { · line 856
Quoted 9 lines
t/t5510-fetch.sh index renamed from t/t5510-fetch.sh +0
-0
4620f0ca7fa..8ca3aa5e931100755t/t5510-fetch.sh +0
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe's[origin(?!/)]["'"$remote_url"'"]g')
Quoted 14 lines
t/t5510-fetch.sh test_configured_prune_type () { · line 861
Quoted 7 lines
t/t5510-fetch.sh +4
-0+ new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ + '$remote_url'~g") fi if test "$fetch_prune_tags" = 'true' ||Thanks for looking at this. Checking this out again this whole quoting mess is a bit of a ... mess, I wonder if there's some better way to avoid this. Anyway:
So, is this functionally the same as:
Quoted 9 lines
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index 4620f0ca7fa..9cd8b36f835 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe's[origin(?!/)]["'"$remote_url"'"]g')
Quoted 18 lines
+ new_cmdline=$(printf "%s" "$cmdline" |
+ perl -pe 's[origin ]["'"$remote_url"'" ]g' |
+ perl -pe 's[ origin][ "'"$remote_url"'"]g')
fiif test "$fetch_prune_tags" = 'true' ||
?
I don't mind the migration to "sed", but doing so to fix a bug makes this especially hard to analyze. I.e. you've gotten rid of the (?!/), I haven't re-looked at this enough to see if/how that's important.
I just came up with the above as a quick hack, but for any proper migration to sed can't we do this in one command?
In any case you never need "| \" in your scripts, just end the line with "|".
Quoted 6 lines · Junio C Hamano
I was expecting you to use \Q...\E (and passing $remote_url as an argument to perl script) actually. If you let $remote_url interpolated by shell into a script of the host language, whether perl or sed, you'd be responsible for quoting it appropriately for the host language yourself, and use of single-quote pair would not necessarily be sufficient, no?
Yes, I wasn't aiming to try to come up with an ideal Perl version, just wondering why it needed the s/perl/sed/ while-we're-at-it...
Quoted 89 lines
-----Original Message-----
From: Derrick Stolee <[email protected]>
Sent: June 20, 2022 4:00 PM
To: [email protected]; 'Junio C Hamano' <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1Quoted 9 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 7 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @
to the next slash.
Quoted 8 lines
Quoted 6 lines
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
Please see the patch below. I'm currently running CI in a GGG PR [1]
[1] https://github.com/gitgitgadget/git/pull/1267
Thanks,
-Stolee--- >8 ---
Quoted 1 line
From
1df4fc66d4a62adc7087d7d22c8d78842b4e9b4dMon Sep 17 00:00:00 2001From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefullyThe many
test_configured_prunetests int5510-fetch.shtest many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced ine1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
Reported-by Randall Becker <[email protected]>Signed-off-by Derrick Stolee <[email protected]>---
t/t5510-fetch.sh| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)t/t5510-fetch.sh index 4620f0ca7fa..8ca3aa5e931 renamed from t/t5510-fetch.sh +0
-0100755
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | \ + sed "s~origin ~'$remote_url' ~g" | \ + sed "s~ origin~ '$remote_url'~g") fi
Quoted 1 line
if test "$fetch_prune_tags" = 'true' ||
Ok, this seems to work. I have the test environment set up so can retest other options.
PATCH v2 1/1 t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
In reply to Derrick StoleeQuoted 17 lines · Derrick Stolee
Quoted 16 lines · On 6/20/22 2:59 PM, [email protected] wrote:
Quoted 13 lines · Derrick Stolee
The issue is this line (some tabs removed):
new_cmdline=$(printf "%s" "$cmdline" | perl -pe
's[origin(?!/)]["'"$remote_url"'"]g')At this point, $remote_url contains the file path including the @ symbol. However, this perl invocation is dropping everything starting at the @ to the next slash.
I'm not sure of a better way to accomplish what is trying to be done here (replace 'origin' with that specific url) without maybe causing other issues.
This line was introduced by
e1790f9245f(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09).
Quoted 6 lines
Quoted 1 line
How about using sed instead of perl for this?
I wasn't sure if using sed would create a different kind of replacement problem, but using single-quotes seems to get around that kind of issue.
Please see the patch below. I'm currently running CI in a GGG PR [1]
Here is a v2 that updates the sed call based on the feedback so far.
I'm basically switching to sed because I don't know perl and the discussion about how to fix this replacement command is completely over my head. That makes me think that using a simpler find-and-replace tool like sed would be good here.
Thanks,
-Stolee
--- >8 ---
From 7fae22e0ad09af00de5a294f61dfd29cb349feeb Mon Sep 17 00:00:00 2001
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH v2] t5510: replace 'origin' with URL more carefully
The many test_configured_prune tests in t5510-fetch.sh test many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced in e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.
However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name.
You can verify this locally by cloning git.git into a "victim@03" directory and running the test script.
To resolve this issue, replace the perl invocation with two sed commands. These two are used to ensure that we match exactly on the whole word 'origin'. We can guarantee that the word boundaries are spaces in our tests. The reason to use exact words is that sometimes a refspec is supplied, such as "+refs/heads/*:refs/remotes/origin/*" which would cause an incorrect replacement. The two commands are used because there is not a clear POSIX way to match on word boundaries without getting extremely pedantic about what possible characters we could have at the boundaries.
--- t/t5510-fetch.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
t/t5510-fetch.sh +3 -1
@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | + sed -e "s~origin ~'$remote_url' ~g" \ + -e "s~ origin~ '$remote_url'~g") fi if test "$fetch_prune_tags" = 'true' ||
Signature · 1 line
2.36.1.220.g1fae7daf425
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Am 21.06.22 um 00:20 schrieb Derrick Stolee:
t/t5510-fetch.sh test_configured_prune_type () { · line 858
Quoted 8 lines
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | + sed -e "s~origin ~'$remote_url' ~g" \ + -e "s~ origin~ '$remote_url'~g")
I'm not sure if this is still an issue, but I recall that there are seds in the wild that do not grok an incomplete last line. Better make this printf "%s\n" "$cmdline" if echo "$cmdline" is not an option.
t/t5510-fetch.sh test_configured_prune_type () { · line 861
Quoted 3 lines
t/t5510-fetch.sh +0
-0fi if test "$fetch_prune_tags" = 'true' ||
-- Hannes
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
In reply to Derrick Stoleet/t5510-fetch.sh test_configured_prune_type () { · line 859
Quoted 13 lines · Derrick Stolee
t/t5510-fetch.sh +3
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(printf "%s" "$cmdline" | + sed -e "s~origin ~'$remote_url' ~g" \ + -e "s~ origin~ '$remote_url'~g") fi
Doesn't this introduce a new problem if $remote_url contains a tilde? Unlikely, but I thought the point of the exercise was defending against funny paths.
Getting this bullet-proof with sed is tricky, I think. I didn't follow all the other logic that that might need fixed, but handling this in perl is easy by passing the string on the command line, like:
printf "%s" "$cmdline" |
perl -pe '
BEGIN { $url = shift }
s[origin(?!/)]['\''$url'\'']g;
' "$remote_url"For that matter, using printf and "perl -p" is a little silly, since we know there is only a single string to modify. Perhaps:
perl -e '
my ($cmdline, $url) = @ARGV;
$cmdline =~ s[origin(?!/)]['\''$url'\'']g;
print $cmdline;
' -- "$cmdline" "$remote_url"And then further changes are easy:
- you could replace the ad-hoc "I hope single quotes are enough" quoting of $url with a real regex
- you can define $sq inside the perl script to avoid the gross '\'' quoting (or even avoid it entirely with quotemeta)
So perhaps something like:
perl -e '
my ($cmdline, $url) = @ARGV;
$cmdline =~ s[origin(?!/)][quotemeta($url)]ge;
print $cmdline;
' -- "$cmdline" "$remote_url"I don't mean to golf on this forever, but I wanted to show something concrete since you said you don't know perl well. I just think moving to sed introduces more opportunities for errors here, not fewer. :)
-Peff
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Quoted 11 lines · Jeff King
Quoted 9 lines · Derrick Stolee
t/t5510-fetch.sh +0
-1@@ -853,7 +853,9 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g')
Quoted 8 lines
t/t5510-fetch.sh test_configured_prune_type () { · line 859
Quoted 4 lines
t/t5510-fetch.sh +3
-0+ new_cmdline=$(printf "%s" "$cmdline" | + sed -e "s~origin ~'$remote_url' ~g" \ + -e "s~ origin~ '$remote_url'~g") fiDoesn't this introduce a new problem if $remote_url contains a tilde? Unlikely, but I thought the point of the exercise was defending against funny paths.
And it doesn't replace a standalone "origin" word, either, which does occur a few times in the test script.
Quoted 5 lines
So perhaps something like:
perl -e ' my ($cmdline, $url) = @ARGV; $cmdline =~ s[origin(?!/)][quotemeta($url)]ge;
I don't like this "(?!/)" magic, because I haven't got the slightest idea of what it might do by merely looking at it, and these characters are not exactly easy to search for.
The good old "add a space prefix and suffix" trick can help to easily match the "origin" word even when it stands alone, but, alas, the result is still not as simple as I'd like with the \s and the string concatenation:
perl -e '
new_cmdline=$(perl -e '
my ($cmdline, $url) = @ARGV;
$cmdline =~ s[\sorigin\s][" " . quotemeta($url) . " "]ge;
print $cmdline;
' -- " $cmdline " "$remote_url")(The 'sed' variant looks good to me:
printf " %s " "$cmdline" |sed "s~ origin ~ '$remote_url' ~g"
but then again it just trades one set of troublesome characters for an other.)
Quoted 8 lines
print $cmdline;' -- "$cmdline" "$remote_url"
I don't mean to golf on this forever, but I wanted to show something concrete since you said you don't know perl well. I just think moving to sed introduces more opportunities for errors here, not fewer. :)
-Peff
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Quoted 9 lines · SZEDER Gábor
Quoted 5 lines
So perhaps something like:
perl -e ' my ($cmdline, $url) = @ARGV; $cmdline =~ s[origin(?!/)][quotemeta($url)]ge;I don't like this "(?!/)" magic, because I haven't got the slightest idea of what it might do by merely looking at it, and these characters are not exactly easy to search for.
Yeah, I hadn't really dug into the rest of the thread and didn't understand what that part was trying to do. So I left it untouched in my examples as an exercise for the reader. :)
Quoted 11 lines
The good old "add a space prefix and suffix" trick can help to easily match the "origin" word even when it stands alone, but, alas, the result is still not as simple as I'd like with the \s and the string concatenation:
perl -e ' new_cmdline=$(perl -e ' my ($cmdline, $url) = @ARGV; $cmdline =~ s[\sorigin\s][" " . quotemeta($url) . " "]ge; print $cmdline; ' -- " $cmdline " "$remote_url")
If you do:
$url = quotemeta($url);
then you can drop the "e" from the regex, which gets rid of the gross concatenation:
$cmdline =~ s[\sorigin\s][ $url ];
I think "\b" for a word boundary would avoid the whitespace prefix/suffix hackery, but it also matches non-alphabetics (like "/"). You could use alternation, though, like:
$ cmdline='origin notorigin origin originnot origin/foo origin'
$ remote_url=real_url
$ perl -e '
my ($cmdline, $url) = @ARGV;
$url = quotemeta($url);
$cmdline =~ s/(\s|^)origin(\s|$)/$1$url$2/g;
print $cmdline;
' "$cmdline" "$remote_url"
real_url notorigin real_url originnot origin/foo real_urlNegative lookbehind and lookahead get rid of the "$1" and "$2", but they are magical-looking, as you noted above. Possibly "/x" and some whitespace would make the whole thing more readable.
-Peff
Quoted 21 lines · On June 20, 2022 2:04 PM, I wrote:
To: 'Junio C Hamano' <[email protected]>
Cc: [email protected]
Subject: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1Subtests t5510.69, 71, 97, 99, 125, 127, 147, 149, 151, 153, 155, 157, 159, 161, 163, 165, 167, 170 failed in rc1 on the NonStop ia64 platform. These were working in rc0, but after analysis, I do not think that is relevant. Here is what we see for subtest 69:
expecting success of 5510.69 'link prune fetch.prune=unset remote.origin.prune=unset fetch.pruneTags=unset remote.origin.pruneTags=unset --prune origin refs/tags/*:refs/tags/*; branch:kept tag:pruned':
...
git fetch
"file:///home/jenkins/.jenkins/workspace/Git_Pipeline@2/t/trash directory.t5510-fetch/." "+refs/heads/*:refs/remotes/origin/*" && ...git$git_fetch_c fetch --prune"file:///home/jenkins/.jenkins/workspace/Git_Pipeline/t/trash directory.t5510- fetch/." refs/tags/*:refs/tags/* &&
Quoted 8 lines
What seems to be happening is that the test script is computing the wrong parent directory in the second case, cutting the @2. We use Jenkins for our builds and occasional multiple builds run. The @2 is Jenkins way of running in another place. This seems to be confusing the script parsing of the parent, causing the issue. When I rebuilt and test in the primary directory, the failures disappeared because no @2.
t5562 also seems to have the same issue.
A more complete list of tests failing in path with @2 is - and this is likely caused by different constructs:
t5300, t5510, t5516, t5562, t5601, t7528.
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
In reply to Jeff KingQuoted 4 lines · Jeff King
And then further changes are easy:
- you could replace the ad-hoc "I hope single quotes are enough" quoting of $url with a real regex
Yes, this is what I found a bit queasy about in the whole thing.
Quoted 10 lines
- you can define $sq inside the perl script to avoid the gross '\'' quoting (or even avoid it entirely with quotemeta)
So perhaps something like:
perl -e ' my ($cmdline, $url) = @ARGV; $cmdline =~ s[origin(?!/)][quotemeta($url)]ge; print $cmdline; ' -- "$cmdline" "$remote_url"
Yup, a solution along that line was what I expected to see from those who write Perl when I saw the discussion yesterday.
FWIW, there are a few characters that won't be acceptable in the pathname for our source tree, including a single quote (I think some of our build procedures are loose in quoting, unlike parts of the Makefile that use $(FOO_SQ) for that) and a colon (testing the version of Git we just built needs the path of the build directory in GIT_EXEC_PATH, which is prepended to PATH at runtime during the test, but the colon means that the path is split into two there), so "I hope single quotes are enough" is not making things worse in practice, but quotemeta() is so simple---allowing us not having to worry about quoting is a very good thing.
Thanks.
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Quoted 2 lines · Junio C Hamano
Yup, a solution along that line was what I expected to see from those who write Perl when I saw the discussion yesterday.
Here is what I queued tentatively. This is not exactly new; e1790f92 (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) first appeared in v2.17.0 and we can live with the same glitch for few more weeks ;-)
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
From: Derrick Stolee <[email protected]>
Date: Mon, 20 Jun 2022 15:52:09 -0400
Subject: [PATCH] t5510: replace 'origin' with URL more carefully
The many test_configured_prune tests in t5510-fetch.sh test many combinations of --prune, --prune-tags, and using 'origin' or an explicit URL. Some machinery was introduced in e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to replace 'origin' with this explicit URL. This URL is a "file:///" URL for the root of the $TRASH_DIRECTORY.
However, if the current build tree has an '@' symbol, the replacement using perl fails. It drops the '@' as well as anything else in that directory name. You can observe this locally by cloning git.git into a "victim@03" directory and running the test script.
As we are writing in Perl anyway, pass in the shell variables involved to the script as arguments and perform necessary string transformations inside it, instead of assuming that it is sufficient to enclose the $remote_url variable inside a pair of single quotes.
--- t/t5510-fetch.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
t/t5510-fetch.sh +5 -1
@@ -853,7 +853,11 @@ test_configured_prune_type () { then new_cmdline=$cmdline_setup else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') + new_cmdline=$(perl -e ' + my ($cmdline, $url) = @ARGV; + $cmdline =~ s[origin(?!/)][quotemeta($url)]ge; + print $cmdline; + ' -- "$cmdline" "$remote_url") fi if test "$fetch_prune_tags" = 'true' ||
Signature · 1 line
2.37.0-rc1-99-g4626346e14
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Junio C Hamano
Yup, a solution along that line was what I expected to see from those who write Perl when I saw the discussion yesterday.
Here is what I queued tentatively. This is not exactly new;
e1790f92(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) first appeared in v2.17.0 and we can live with the same glitch for few more weeks ;-)
There are also other tests that need similar updates before we consider this topic done.
Quoted 3 lines
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
From: Derrick Stolee <[email protected]>Original-patch-by Derrick Stolee <[email protected]>
I recommend updating the authorship to yourself or Peff, since I was mostly useless in constructing the actual solution.
Thanks,
-Stolee
Re: [PATCH v2] t5510: replace 'origin' with URL more carefully (was Re: Test Failure t5510,t5562 - was RE: [ANNOUNCE] Git v2.37.0-rc1)
Quoted 6 lines · Derrick Stolee
Quoted 5 lines · Junio C Hamano
Quoted 2 lines · Junio C Hamano
Yup, a solution along that line was what I expected to see from those who write Perl when I saw the discussion yesterday.
Quoted 7 lines
Quoted 4 lines
Here is what I queued tentatively. This is not exactly new;
e1790f92(fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) first appeared in v2.17.0 and we can live with the same glitch for few more weeks ;-)There are also other tests that need similar updates before we consider this topic done.
If we can check with single-quotes and dollar-signs in addition to at-mark, that would be interesting.
Quoted 2 lines
I recommend updating the authorship to yourself or Peff, since I was mostly useless in constructing the actual solution.
Sure. That's what "tentatively" means. I didn't do anything (other than cheering from the sidelines), so...
cover t5510: fix the quoting mess
Since the upthread quoting discussion pointed at a commit of mine I felt obligated to look at this in a deeper way.
When I added those tests I carried forward some technical debt where we were passing arguments to a function as a string, instead of as multiple arguments which we could get with "$@", in hindsight that was a big mistake.
As far as I can tell the proposed
https://lore.kernel.org/git/[email protected]/ fixes the issue at hand, but this is the deeper and I think better direction to go in (but I could rebase in top of that fix if needed, although the conflict is trivial, keep this side).
Now we don't use "perl", or "sed", or whatever to re-quote arguments in t/t5510-fetch.sh.
There's still some weirdness in that test, e.g. the whole behavior of set_config_tristate(), but I had to stop somewhere.
Ævar Arnfjörð Bjarmason (10):
fetch tests: remove redundant test_unconfig()
fetch tests: use named, not positional parameters fetch tests: use "local", &&-chaining, style etc. fetch tests: add a helper to avoid boilerplate
fetch tests: pass "mode" parameter first, pave way for "$@" fetch tests: pass a list, not a string of arguments fetch tests: remove lazy variable setup
fetch tests: remove shelling out for previously "lazy" variables fetch tests: stop implicitly adding refspecs
fetch tests: fix needless and buggy re-quoting
t/t5510-fetch.sh | 289 +++++++++++++++++++++++++++++------------------ 1 file changed, 178 insertions(+), 111 deletions(-)
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 1/10 fetch tests: remove redundant test_unconfig()
The test_unconfig() calls here were added as boilerplate in 737c5a9cde7 (fetch: make --prune configurable, 2013-07-13), and then faithfully reproduced in e249ce0ccdb (fetch tests: add scaffolding for the new fetch.pruneTags, 2018-02-09). But they were never necessary, so let's remove them.
This actually improves our test coverage, as we'll now be asserting that whatever configuration we leave here (in the "one" block below) won't affect this particular "git fetch" command.
--- t/t5510-fetch.sh | 4 ---- 1 file changed, 4 deletions(-)
t/t5510-fetch.sh +0 -4
@@ -874,10 +874,6 @@ test_configured_prune_type () { git tag -f newtag && ( cd one && - test_unconfig fetch.prune && - test_unconfig fetch.pruneTags && - test_unconfig remote.origin.prune && - test_unconfig remote.origin.pruneTags && git fetch '"$cmdline_setup"' && git rev-parse --verify refs/remotes/origin/newbranch && git rev-parse --verify refs/tags/newtag
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 4/10 fetch tests: add a helper to avoid boilerplate
In reply to Ævar Arnfjörð BjarmasonAdd a new test_configured_prune_type_branch() helper to avoid the boilerplate introduced in 97716d217c1 (fetch: add a --prune-tags option and fetch.pruneTags config, 2018-02-09).
Back then it was somewhat necessary, but since 6317972cff9 (fetch:
make the --prune-tags work with <url>, 2018-02-09) these tests have been regular enough that we can always pass the "kept" argument for "link", and "pruned" for "name".
--- t/t5510-fetch.sh | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-)
t/t5510-fetch.sh +11 -14
@@ -1006,22 +1006,19 @@ test_configured_prune unset unset unset true pruned kept \ # remote. However, because there's no implicit # +refs/heads/*:refs/remotes/origin/* refspec and supplying it on the # command-line negates --prune-tags, the branches will not be pruned. +test_configured_prune_type_branch () { + test_configured_prune_type "$1" "$2" "$3" "$4" pruned "$6" "$7" "name" + test_configured_prune_type "$1" "$2" "$3" "$4" kept "$6" "$7" "link" +} test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "name" test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "link" -test_configured_prune_type unset unset unset unset pruned pruned "origin --prune --prune-tags" "name" -test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "link" -test_configured_prune_type unset unset unset unset pruned pruned "--prune --prune-tags origin" "name" -test_configured_prune_type unset unset unset unset kept pruned "--prune --prune-tags origin" "link" -test_configured_prune_type unset unset true unset pruned pruned "--prune origin" "name" -test_configured_prune_type unset unset true unset kept pruned "--prune origin" "link" -test_configured_prune_type unset unset unset true pruned pruned "--prune origin" "name" -test_configured_prune_type unset unset unset true kept pruned "--prune origin" "link" -test_configured_prune_type true unset true unset pruned pruned "origin" "name" -test_configured_prune_type true unset true unset kept pruned "origin" "link" -test_configured_prune_type unset true true unset pruned pruned "origin" "name" -test_configured_prune_type unset true true unset kept pruned "origin" "link" -test_configured_prune_type unset true unset true pruned pruned "origin" "name" -test_configured_prune_type unset true unset true kept pruned "origin" "link" +test_configured_prune_type_branch unset unset unset unset - pruned "origin --prune --prune-tags" +test_configured_prune_type_branch unset unset unset unset - pruned "--prune --prune-tags origin" +test_configured_prune_type_branch unset unset true unset - pruned "--prune origin" +test_configured_prune_type_branch unset unset unset true - pruned "--prune origin" +test_configured_prune_type_branch true unset true unset - pruned "origin" +test_configured_prune_type_branch unset true true unset - pruned "origin" +test_configured_prune_type_branch unset true unset true - pruned "origin" # When all remote.origin.fetch settings are deleted a --prune # --prune-tags still implicitly supplies refs/tags/*:refs/tags/* so
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 3/10 fetch tests: use "local", &&-chaining, style etc.
In reply to Ævar Arnfjörð BjarmasonMake use of &&-chaining, the "local" keyword, the more idiomatic "test -z" over comparison against "" etc. in the "test_configured_prune_type()" function.
Let's also move the wrapper function to below the function itself. In e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) placed in there to minimize the diff size. While it's not incorrect to define these in this order (this isn't C), it's less confusing this way.
--- t/t5510-fetch.sh | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-)
t/t5510-fetch.sh +21 -19
@@ -821,20 +821,15 @@ set_config_tristate () { esac } -test_configured_prune () { - test_configured_prune_type "$@" "name" - test_configured_prune_type "$@" "link" -} - test_configured_prune_type () { - fetch_prune=$1 - remote_origin_prune=$2 - fetch_prune_tags=$3 - remote_origin_prune_tags=$4 - expected_branch=$5 - expected_tag=$6 - cmdline=$7 - mode=$8 + local fetch_prune="$1" && + local remote_origin_prune="$2" && + local fetch_prune_tags="$3" && + local remote_origin_prune_tags="$4" && + local expected_branch="$5" && + local expected_tag="$6" && + local cmdline="$7" && + local mode="$8" && if test -z "$cmdline_setup" then
@@ -843,18 +838,18 @@ test_configured_prune_type () { remote_fetch="$(git -C one config remote.origin.fetch)" && cmdline_setup="\"$remote_url\" \"$remote_fetch\"" ' - fi + fi && if test "$mode" = 'link' then - new_cmdline="" + new_cmdline="" && - if test "$cmdline" = "" + if test -z "$cmdline" then new_cmdline=$cmdline_setup else new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') - fi + fi && if test "$fetch_prune_tags" = 'true' || test "$remote_origin_prune_tags" = 'true'
@@ -863,10 +858,10 @@ test_configured_prune_type () { then new_cmdline="$new_cmdline refs/tags/*:refs/tags/*" fi - fi + fi && cmdline="$new_cmdline" - fi + fi && test_expect_success "$mode prune fetch.prune=$fetch_prune remote.origin.prune=$remote_origin_prune fetch.pruneTags=$fetch_prune_tags remote.origin.pruneTags=$remote_origin_prune_tags${cmdline:+ $cmdline}; branch:$expected_branch tag:$expected_tag" ' # make sure a newbranch is there in . and also in one
@@ -915,6 +910,13 @@ test_configured_prune_type () { esac ) ' + + return 0 +} + +test_configured_prune () { + test_configured_prune_type "$@" "name" && + test_configured_prune_type "$@" "link" } # $1 config: fetch.prune
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 2/10 fetch tests: use named, not positional parameters
In reply to Ævar Arnfjörð BjarmasonUse the named parameters we've already unpacked from "$@" rather than the positional paremeters. There's no functional changes here.
This changes code that dates back to the initial introduction of the function in 737c5a9cde7 (fetch: make --prune configurable, 2013-07-13), although e.g. e249ce0ccdb (fetch tests: add scaffolding for the new fetch.pruneTags, 2018-02-09) added many more parameters.
--- t/t5510-fetch.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5510-fetch.sh +1 -1
@@ -868,7 +868,7 @@ test_configured_prune_type () { cmdline="$new_cmdline" fi - test_expect_success "$mode prune fetch.prune=$1 remote.origin.prune=$2 fetch.pruneTags=$3 remote.origin.pruneTags=$4${7:+ $7}; branch:$5 tag:$6" ' + test_expect_success "$mode prune fetch.prune=$fetch_prune remote.origin.prune=$remote_origin_prune fetch.pruneTags=$fetch_prune_tags remote.origin.pruneTags=$remote_origin_prune_tags${cmdline:+ $cmdline}; branch:$expected_branch tag:$expected_tag" ' # make sure a newbranch is there in . and also in one git branch -f newbranch && git tag -f newtag &&
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 5/10 fetch tests: pass "mode" parameter first, pave way for "$@"
In reply to Ævar Arnfjörð BjarmasonChange the "$mode" parameter to be passed first, and setup a command-line parser we'll be able to use for getting rid of many of the boilerplate parameters.
This will allow us to unquote the command-line argument, and process fetch arguments as a list of "$@". For now we need to do more work to unpack these, but in a subsequent commit we'll be able to make the shell quote handling here much simpler.
--- t/t5510-fetch.sh | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-)
t/t5510-fetch.sh +33 -10
@@ -822,14 +822,29 @@ set_config_tristate () { } test_configured_prune_type () { + mode= && + while test $# != 0 + do + case "$1" in + --mode) + mode="$2" && + shift + ;; + *) + break + ;; + esac && + shift + done && + local fetch_prune="$1" && local remote_origin_prune="$2" && local fetch_prune_tags="$3" && local remote_origin_prune_tags="$4" && local expected_branch="$5" && local expected_tag="$6" && - local cmdline="$7" && - local mode="$8" && + shift 6 && + local cmdline="$@" && if test -z "$cmdline_setup" then
@@ -915,8 +930,8 @@ test_configured_prune_type () { } test_configured_prune () { - test_configured_prune_type "$@" "name" && - test_configured_prune_type "$@" "link" + test_configured_prune_type --mode name "$@" && + test_configured_prune_type --mode link "$@" } # $1 config: fetch.prune
@@ -1007,11 +1022,19 @@ test_configured_prune unset unset unset true pruned kept \ # +refs/heads/*:refs/remotes/origin/* refspec and supplying it on the # command-line negates --prune-tags, the branches will not be pruned. test_configured_prune_type_branch () { - test_configured_prune_type "$1" "$2" "$3" "$4" pruned "$6" "$7" "name" - test_configured_prune_type "$1" "$2" "$3" "$4" kept "$6" "$7" "link" + local cfg_fp="$1" && + local cfg_rnp="$2" && + local cfg_fpt="$3" && + local cfg_rnpt="$4" && + local arg_branch="$5" && + local arg_tag="$6" && + shift 6 && + + test_configured_prune_type --mode name "$cfg_fp" "$cfg_rnp" "$cfg_fpt" "$cfg_rnpt" pruned "$arg_tag" "$@" + test_configured_prune_type --mode link "$cfg_fp" "$cfg_rnp" "$cfg_fpt" "$cfg_rnpt" kept "$arg_tag" "$@" } -test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "name" -test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "link" +test_configured_prune_type --mode name unset unset unset unset kept kept "origin --prune-tags" +test_configured_prune_type --mode link unset unset unset unset kept kept "origin --prune-tags" test_configured_prune_type_branch unset unset unset unset - pruned "origin --prune --prune-tags" test_configured_prune_type_branch unset unset unset unset - pruned "--prune --prune-tags origin" test_configured_prune_type_branch unset unset true unset - pruned "--prune origin"
@@ -1029,8 +1052,8 @@ test_expect_success 'remove remote.origin.fetch "one"' ' git config --unset-all remote.origin.fetch ) ' -test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "name" -test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "link" +test_configured_prune_type --mode name unset unset unset unset kept pruned "origin --prune --prune-tags" +test_configured_prune_type --mode link unset unset unset unset kept pruned "origin --prune --prune-tags" test_expect_success 'all boundary commits are excluded' ' test_commit base &&
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 6/10 fetch tests: pass a list, not a string of arguments
In reply to Ævar Arnfjörð BjarmasonFollow-up on the preceding commit where we've started to intercept these arguments as "$@" and pass them as a list, not as a string, before we'd only have a "$@" equivalent to "$1".
We're still not doing anything really useful with these, but this is getting us towards fixing the quote handling in "test_configured_prune_type()".
--- t/t5510-fetch.sh | 116 +++++++++++++++++++++++------------------------ 1 file changed, 58 insertions(+), 58 deletions(-)
t/t5510-fetch.sh +58 -58
@@ -940,82 +940,82 @@ test_configured_prune () { # $4 config: remote.<name>.pruneTags # $5 expect: branch to be pruned? # $6 expect: tag to be pruned? -# $7 git-fetch $cmdline: +# $7... git-fetch $cmdline: # -# $1 $2 $3 $4 $5 $6 $7 -test_configured_prune unset unset unset unset kept kept "" -test_configured_prune unset unset unset unset kept kept "--no-prune" -test_configured_prune unset unset unset unset pruned kept "--prune" +# $1 $2 $3 $4 $5 $6 $7... +test_configured_prune unset unset unset unset kept kept +test_configured_prune unset unset unset unset kept kept --no-prune +test_configured_prune unset unset unset unset pruned kept --prune test_configured_prune unset unset unset unset kept pruned \ - "--prune origin refs/tags/*:refs/tags/*" + --prune origin "refs/tags/*:refs/tags/*" test_configured_prune unset unset unset unset pruned pruned \ - "--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*" + --prune origin "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" -test_configured_prune false unset unset unset kept kept "" -test_configured_prune false unset unset unset kept kept "--no-prune" -test_configured_prune false unset unset unset pruned kept "--prune" +test_configured_prune false unset unset unset kept kept +test_configured_prune false unset unset unset kept kept --no-prune +test_configured_prune false unset unset unset pruned kept --prune -test_configured_prune true unset unset unset pruned kept "" -test_configured_prune true unset unset unset pruned kept "--prune" -test_configured_prune true unset unset unset kept kept "--no-prune" +test_configured_prune true unset unset unset pruned kept +test_configured_prune true unset unset unset pruned kept --prune +test_configured_prune true unset unset unset kept kept --no-prune -test_configured_prune unset false unset unset kept kept "" -test_configured_prune unset false unset unset kept kept "--no-prune" -test_configured_prune unset false unset unset pruned kept "--prune" +test_configured_prune unset false unset unset kept kept +test_configured_prune unset false unset unset kept kept --no-prune +test_configured_prune unset false unset unset pruned kept --prune -test_configured_prune false false unset unset kept kept "" -test_configured_prune false false unset unset kept kept "--no-prune" -test_configured_prune false false unset unset pruned kept "--prune" +test_configured_prune false false unset unset kept kept +test_configured_prune false false unset unset kept kept --no-prune +test_configured_prune false false unset unset pruned kept --prune test_configured_prune false false unset unset kept pruned \ - "--prune origin refs/tags/*:refs/tags/*" + --prune origin "refs/tags/*:refs/tags/*" test_configured_prune false false unset unset pruned pruned \ - "--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*" + --prune origin "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" -test_configured_prune true false unset unset kept kept "" -test_configured_prune true false unset unset pruned kept "--prune" -test_configured_prune true false unset unset kept kept "--no-prune" +test_configured_prune true false unset unset kept kept +test_configured_prune true false unset unset pruned kept --prune +test_configured_prune true false unset unset kept kept --no-prune -test_configured_prune unset true unset unset pruned kept "" -test_configured_prune unset true unset unset kept kept "--no-prune" -test_configured_prune unset true unset unset pruned kept "--prune" +test_configured_prune unset true unset unset pruned kept +test_configured_prune unset true unset unset kept kept --no-prune +test_configured_prune unset true unset unset pruned kept --prune -test_configured_prune false true unset unset pruned kept "" -test_configured_prune false true unset unset kept kept "--no-prune" -test_configured_prune false true unset unset pruned kept "--prune" +test_configured_prune false true unset unset pruned kept +test_configured_prune false true unset unset kept kept --no-prune +test_configured_prune false true unset unset pruned kept --prune -test_configured_prune true true unset unset pruned kept "" -test_configured_prune true true unset unset pruned kept "--prune" -test_configured_prune true true unset unset kept kept "--no-prune" +test_configured_prune true true unset unset pruned kept +test_configured_prune true true unset unset pruned kept --prune +test_configured_prune true true unset unset kept kept --no-prune test_configured_prune true true unset unset kept pruned \ - "--prune origin refs/tags/*:refs/tags/*" + --prune origin "refs/tags/*:refs/tags/*" test_configured_prune true true unset unset pruned pruned \ - "--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*" + --prune origin "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" # --prune-tags on its own does nothing, needs --prune as well, same # for fetch.pruneTags without fetch.prune -test_configured_prune unset unset unset unset kept kept "--prune-tags" -test_configured_prune unset unset true unset kept kept "" -test_configured_prune unset unset unset true kept kept "" +test_configured_prune unset unset unset unset kept kept --prune-tags +test_configured_prune unset unset true unset kept kept +test_configured_prune unset unset unset true kept kept # These will prune the tags -test_configured_prune unset unset unset unset pruned pruned "--prune --prune-tags" -test_configured_prune true unset true unset pruned pruned "" -test_configured_prune unset true unset true pruned pruned "" +test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags +test_configured_prune true unset true unset pruned pruned +test_configured_prune unset true unset true pruned pruned # remote.<name>.pruneTags overrides fetch.pruneTags, just like # remote.<name>.prune overrides fetch.prune if set. -test_configured_prune true unset true unset pruned pruned "" -test_configured_prune false true false true pruned pruned "" -test_configured_prune true false true false kept kept "" +test_configured_prune true unset true unset pruned pruned +test_configured_prune false true false true pruned pruned +test_configured_prune true false true false kept kept # When --prune-tags is supplied it's ignored if an explicit refspec is # given, same for the configuration options. test_configured_prune unset unset unset unset pruned kept \ - "--prune --prune-tags origin +refs/heads/*:refs/remotes/origin/*" + --prune --prune-tags origin "+refs/heads/*:refs/remotes/origin/*" test_configured_prune unset unset true unset pruned kept \ - "--prune origin +refs/heads/*:refs/remotes/origin/*" + --prune origin "+refs/heads/*:refs/remotes/origin/*" test_configured_prune unset unset unset true pruned kept \ - "--prune origin +refs/heads/*:refs/remotes/origin/*" + --prune origin "+refs/heads/*:refs/remotes/origin/*" # Pruning that also takes place if a file:// url replaces a named # remote. However, because there's no implicit
@@ -1033,15 +1033,15 @@ test_configured_prune_type_branch () { test_configured_prune_type --mode name "$cfg_fp" "$cfg_rnp" "$cfg_fpt" "$cfg_rnpt" pruned "$arg_tag" "$@" test_configured_prune_type --mode link "$cfg_fp" "$cfg_rnp" "$cfg_fpt" "$cfg_rnpt" kept "$arg_tag" "$@" } -test_configured_prune_type --mode name unset unset unset unset kept kept "origin --prune-tags" -test_configured_prune_type --mode link unset unset unset unset kept kept "origin --prune-tags" -test_configured_prune_type_branch unset unset unset unset - pruned "origin --prune --prune-tags" -test_configured_prune_type_branch unset unset unset unset - pruned "--prune --prune-tags origin" -test_configured_prune_type_branch unset unset true unset - pruned "--prune origin" -test_configured_prune_type_branch unset unset unset true - pruned "--prune origin" -test_configured_prune_type_branch true unset true unset - pruned "origin" -test_configured_prune_type_branch unset true true unset - pruned "origin" -test_configured_prune_type_branch unset true unset true - pruned "origin" +test_configured_prune_type --mode name unset unset unset unset kept kept origin --prune-tags +test_configured_prune_type --mode link unset unset unset unset kept kept origin --prune-tags +test_configured_prune_type_branch unset unset unset unset - pruned origin --prune --prune-tags +test_configured_prune_type_branch unset unset unset unset - pruned --prune --prune-tags origin +test_configured_prune_type_branch unset unset true unset - pruned --prune origin +test_configured_prune_type_branch unset unset unset true - pruned --prune origin +test_configured_prune_type_branch true unset true unset - pruned origin +test_configured_prune_type_branch unset true true unset - pruned origin +test_configured_prune_type_branch unset true unset true - pruned origin # When all remote.origin.fetch settings are deleted a --prune # --prune-tags still implicitly supplies refs/tags/*:refs/tags/* so
@@ -1052,8 +1052,8 @@ test_expect_success 'remove remote.origin.fetch "one"' ' git config --unset-all remote.origin.fetch ) ' -test_configured_prune_type --mode name unset unset unset unset kept pruned "origin --prune --prune-tags" -test_configured_prune_type --mode link unset unset unset unset kept pruned "origin --prune --prune-tags" +test_configured_prune_type --mode name unset unset unset unset kept pruned origin --prune --prune-tags +test_configured_prune_type --mode link unset unset unset unset kept pruned origin --prune --prune-tags test_expect_success 'all boundary commits are excluded' ' test_commit base &&
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 7/10 fetch tests: remove lazy variable setup
In reply to Ævar Arnfjörð BjarmasonRemove the lazy test setup added in e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) to make it clear that these variables aren't changing across runs.
We can also do away with the shell invocations here, but let's do that in a subsequent commit, for now this shows that it's safe to do this more than once.
--- t/t5510-fetch.sh | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
t/t5510-fetch.sh +3 -8
@@ -846,14 +846,9 @@ test_configured_prune_type () { shift 6 && local cmdline="$@" && - if test -z "$cmdline_setup" - then - test_expect_success 'setup cmdline_setup variable for subsequent test' ' - remote_url="file://$(git -C one config remote.origin.url)" && - remote_fetch="$(git -C one config remote.origin.fetch)" && - cmdline_setup="\"$remote_url\" \"$remote_fetch\"" - ' - fi && + remote_url="file://$(git -C one config remote.origin.url)" && + remote_fetch="$(git -C one config remote.origin.fetch)" && + cmdline_setup="\"$remote_url\" \"$remote_fetch\"" && if test "$mode" = 'link' then
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 8/10 fetch tests: remove shelling out for previously "lazy" variables
In reply to Ævar Arnfjörð BjarmasonAmend the variable assignments added in e1790f9245f (fetch tests:
fetch <url> <spec> as well as fetch [<remote>], 2018-02-09). As of the last commit this test was slower as we'd do this on every run, but it was easier to see what was going on.
But let's instead set this on the basis of $TRASH_DIRECTORY, which is a lot more obvious than the roundabout way of getting the configuration from the repo that we created earlier with:
git clone . one
Subsequent commits will make use of the new "$refspec_heads" for other purposes, so let's declare it as a variable, and use it instead of "$remote_fetch".
--- t/t5510-fetch.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
t/t5510-fetch.sh +4 -3
@@ -837,6 +837,8 @@ test_configured_prune_type () { shift done && + local refspec_heads='+refs/heads/*:refs/remotes/origin/*' && + local fetch_prune="$1" && local remote_origin_prune="$2" && local fetch_prune_tags="$3" &&
@@ -846,9 +848,8 @@ test_configured_prune_type () { shift 6 && local cmdline="$@" && - remote_url="file://$(git -C one config remote.origin.url)" && - remote_fetch="$(git -C one config remote.origin.fetch)" && - cmdline_setup="\"$remote_url\" \"$remote_fetch\"" && + local remote_url="file://$TRASH_DIRECTORY/." && + local cmdline_setup="\"$remote_url\" \"$refspec_heads\"" if test "$mode" = 'link' then
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 9/10 fetch tests: stop implicitly adding refspecs
In reply to Ævar Arnfjörð BjarmasonRemove the implicit behavior added in 97716d217c1 (fetch: add a --prune-tags option and fetch.pruneTags config, 2018-02-09) of providing the heads/tags refspecs and the "$remote_url" for tests that needed the "--mode link".
In that case we need the URI on the command-line, and would then add these refspecs.
Note that this was added in other cases, but these are the only ones where it mattered, i.e. we redundantly modified the command-line before.
Yes, the "\"$remote_url\"" quoting here is buggy, but so is the pre-image in the same way, but removing this edge case will make it easier to eventually deal with that. So let's make that variable non-local for now and pass it in.
--- t/t5510-fetch.sh | 42 +++++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 17 deletions(-)
t/t5510-fetch.sh +25 -17
@@ -848,7 +848,7 @@ test_configured_prune_type () { shift 6 && local cmdline="$@" && - local remote_url="file://$TRASH_DIRECTORY/." && + remote_url="file://$TRASH_DIRECTORY/." && # NOT local yet! local cmdline_setup="\"$remote_url\" \"$refspec_heads\"" if test "$mode" = 'link'
@@ -862,15 +862,6 @@ test_configured_prune_type () { new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') fi && - if test "$fetch_prune_tags" = 'true' || - test "$remote_origin_prune_tags" = 'true' - then - if ! printf '%s' "$cmdline\n" | grep -q refs/remotes/origin/ - then - new_cmdline="$new_cmdline refs/tags/*:refs/tags/*" - fi - fi && - cmdline="$new_cmdline" fi &&
@@ -990,19 +981,36 @@ test_configured_prune true true unset unset pruned pruned \ # --prune-tags on its own does nothing, needs --prune as well, same # for fetch.pruneTags without fetch.prune test_configured_prune unset unset unset unset kept kept --prune-tags -test_configured_prune unset unset true unset kept kept -test_configured_prune unset unset unset true kept kept +test_configured_prune_type --mode name unset unset true unset kept kept +test_configured_prune_type --mode link unset unset true unset kept kept \ + origin "refs/tags/*:refs/tags/*" +test_configured_prune_type --mode name unset unset unset true kept kept +test_configured_prune_type --mode link unset unset unset true kept kept \ + origin "refs/tags/*:refs/tags/*" # These will prune the tags test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags -test_configured_prune true unset true unset pruned pruned -test_configured_prune unset true unset true pruned pruned + +test_configured_prune_type --mode name true unset true unset pruned pruned +test_configured_prune_type --mode link true unset true unset pruned pruned \ + "\"$remote_url\"" \ + "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" +test_configured_prune_type --mode name unset true unset true pruned pruned +test_configured_prune_type --mode link unset true unset true pruned pruned \ + "\"$remote_url\"" \ + "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" # remote.<name>.pruneTags overrides fetch.pruneTags, just like # remote.<name>.prune overrides fetch.prune if set. -test_configured_prune true unset true unset pruned pruned -test_configured_prune false true false true pruned pruned -test_configured_prune true false true false kept kept +test_configured_prune_type --mode name true unset true unset pruned pruned +test_configured_prune_type --mode link true unset true unset pruned pruned \ + "\"$remote_url\"" \ + "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" +test_configured_prune_type --mode name false true false true pruned pruned +test_configured_prune_type --mode link false true false true pruned pruned \ + "\"$remote_url\"" \ + "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" +test_configured_prune true false true false kept kept # When --prune-tags is supplied it's ignored if an explicit refspec is # given, same for the configuration options.
Signature · 1 line
2.36.1.1239.gfba91521d90
PATCH 10/10 fetch tests: fix needless and buggy re-quoting
In reply to Ævar Arnfjörð BjarmasonChange the test_configured_prune_type() function to take full advantage of its arguments being passed in as a list, rather than needing hacks to work around its quoting issues.
When the test_configured_prune() function was implemented in 737c5a9cde7 (fetch: make --prune configurable, 2013-07-13) it passed in its arguments to "git fetch" as one argument (although at the time only one argument was passed)..
Then in preparation for passing more arguments the first quoting hack was added in 82f34e03e91 (fetch tests: double quote a variable for interpolation, 2018-02-09), with e1790f9245f (fetch tests: fetch <url> <spec> as well as fetch [<remote>], 2018-02-09) following after that. This was all to implement the "git fetch --prune-tags" feature in 97716d217c1 (fetch: add a --prune-tags option and fetch.pruneTags config, 2018-02-09).
At the time the edge cases introduced by this quoting were a known issue, but the alternative was a larger refactoring of this test file. In preceding commits we've done that refactoring, so let's finally take advantage of it. As reported in [1] the existing workaround(s) weren't enough.
We can now drop the "perl" (or "sed" or whatever, see [2]), this will also handle other cases, such as those mentioned in [3].
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://lore.kernel.org/git/YrFwcL2dRS%[email protected]/
--- t/t5510-fetch.sh | 79 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 62 insertions(+), 17 deletions(-)
t/t5510-fetch.sh +62 -17
@@ -837,6 +837,7 @@ test_configured_prune_type () { shift done && + local refspec_tags='refs/tags/*:refs/tags/*' && local refspec_heads='+refs/heads/*:refs/remotes/origin/*' && local fetch_prune="$1" &&
@@ -846,23 +847,59 @@ test_configured_prune_type () { local expected_branch="$5" && local expected_tag="$6" && shift 6 && - local cmdline="$@" && + local cmdline="" && + local arg_fetch_prune="" && + local arg_fetch_no_prune="" && + local arg_fetch_prune_tags="" && + local arg_fetch_origin="" && + local arg_fetch_url="" && + local arg_fetch_refspec_tags="" && + local arg_fetch_refspec_heads="" && + while test $# != 0 + do + cmdline="${cmdline:+$cmdline }$1" && + case "$1" in + --prune) + arg_fetch_prune=t + ;; + --no-prune) + arg_fetch_no_prune=t + ;; + --prune-tags) + arg_fetch_prune_tags=t + ;; + origin) + arg_fetch_origin=t + ;; + REMOTE_URL) + arg_fetch_url=t + ;; + $refspec_tags) + arg_fetch_refspec_tags=t + ;; + $refspec_heads) + arg_fetch_refspec_heads=t + ;; + *) + BUG "unknown argument: '$1'" + ;; + esac && + shift + done && - remote_url="file://$TRASH_DIRECTORY/." && # NOT local yet! - local cmdline_setup="\"$remote_url\" \"$refspec_heads\"" + local remote_url="file://$TRASH_DIRECTORY/." && if test "$mode" = 'link' then - new_cmdline="" && - if test -z "$cmdline" then - new_cmdline=$cmdline_setup - else - new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g') - fi && - - cmdline="$new_cmdline" + arg_fetch_refspec_heads=t + arg_fetch_url=t + elif test -n "$arg_fetch_origin" + then + arg_fetch_origin= + arg_fetch_url=t + fi fi && test_expect_success "$mode prune fetch.prune=$fetch_prune remote.origin.prune=$remote_origin_prune fetch.pruneTags=$fetch_prune_tags remote.origin.pruneTags=$remote_origin_prune_tags${cmdline:+ $cmdline}; branch:$expected_branch tag:$expected_tag" '
@@ -871,7 +908,7 @@ test_configured_prune_type () { git tag -f newtag && ( cd one && - git fetch '"$cmdline_setup"' && + git fetch "$remote_url" "$refspec_heads" && git rev-parse --verify refs/remotes/origin/newbranch && git rev-parse --verify refs/tags/newtag ) &&
@@ -893,7 +930,15 @@ test_configured_prune_type () { then git_fetch_c="" fi && - git$git_fetch_c fetch '"$cmdline"' && + git$git_fetch_c fetch \ + ${arg_fetch_prune:+--prune} \ + ${arg_fetch_no_prune:+--no-prune} \ + ${arg_fetch_prune_tags:+--prune-tags} \ + ${arg_fetch_origin:+origin} \ + ${arg_fetch_url:+"$remote_url"} \ + ${arg_fetch_refspec_tags:+"refs/tags/*:refs/tags/*"} \ + ${arg_fetch_refspec_heads:+"+refs/heads/*:refs/remotes/origin/*"} && + case "$expected_branch" in pruned) test_must_fail git rev-parse --verify refs/remotes/origin/newbranch
@@ -993,22 +1038,22 @@ test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags test_configured_prune_type --mode name true unset true unset pruned pruned test_configured_prune_type --mode link true unset true unset pruned pruned \ - "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" test_configured_prune_type --mode name unset true unset true pruned pruned test_configured_prune_type --mode link unset true unset true pruned pruned \ - "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" # remote.<name>.pruneTags overrides fetch.pruneTags, just like # remote.<name>.prune overrides fetch.prune if set. test_configured_prune_type --mode name true unset true unset pruned pruned test_configured_prune_type --mode link true unset true unset pruned pruned \ - "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" test_configured_prune_type --mode name false true false true pruned pruned test_configured_prune_type --mode link false true false true pruned pruned \ - "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*" test_configured_prune true false true false kept kept
Signature · 1 line
2.36.1.1239.gfba91521d90
Re: [PATCH 01/10] fetch tests: remove redundant test_unconfig()
In reply to Ævar Arnfjörð BjarmasonQuoted 4 lines · Ævar Arnfjörð Bjarmason
The
test_unconfig()calls here were added as boilerplate in737c5a9cde7(fetch: make --prune configurable, 2013-07-13), and then faithfully reproduced ine249ce0ccdb(fetch tests: add scaffolding for the new fetch.pruneTags, 2018-02-09). But they were never necessary,
unnecessary because ...?
is it because nothing has happened in this directory before?
is it because these will all be overriden by the command line options?
is it because of something else?
Quoted 27 lines
so let's remove them.
This actually improves our test coverage, as we'll now be asserting that whatever configuration we leave here (in the "one" block below) won't affect this particular "git fetch" command.
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- t/t5510-fetch.sh | 4 ---- 1 file changed, 4 deletions(-)t/t5510-fetch.sh +0
-4@@ -874,10 +874,6 @@ test_configured_prune_type () { git tag -f newtag && ( cd one && - test_unconfig fetch.prune && - test_unconfig fetch.pruneTags && - test_unconfig remote.origin.prune && - test_unconfig remote.origin.pruneTags && git fetch '"$cmdline_setup"' && git rev-parse --verify refs/remotes/origin/newbranch && git rev-parse --verify refs/tags/newtag
Re: [PATCH 05/10] fetch tests: pass "mode" parameter first, pave way for "$@"
In reply to Ævar Arnfjörð Bjarmasont/t5510-fetch.sh set_config_tristate () { · line 847
Quoted 1 line · Ævar Arnfjörð Bjarmason
t/t5510-fetch.sh +1
-0+ local cmdline="$@" &&
Unless you are using the magic trait of "$@" that lets you pass each positional parameter without getting split at internal $IFS, it is less confusing to write "$*", which will make it obvious that a later use of $cmdline will lose the distinction between inter-parameter spaces and a whitespace that was embedded inside an individual parameter.
Quoted 14 lines
t/t5510-fetch.sh +2
-2@@ -915,8 +930,8 @@ test_configured_prune_type () { } test_configured_prune () { - test_configured_prune_type "$@" "name" && - test_configured_prune_type "$@" "link" + test_configured_prune_type --mode name "$@" && + test_configured_prune_type --mode link "$@" } # $1 config: fetch.prune@@ -1007,11 +1022,19 @@ test_configured_prune unset unset unset true pruned kept \ # +refs/heads/*:refs/remotes/origin/* refspec and supplying it on the # command-line negates --prune-tags, the branches will not be pruned.
t/t5510-fetch.sh test_configured_prune unset unset unset true pruned kept \ · line 1031
Quoted 10 lines
t/t5510-fetch.sh +7
-2test_configured_prune_type_branch () { - test_configured_prune_type "$1" "$2" "$3" "$4" pruned "$6" "$7" "name" - test_configured_prune_type "$1" "$2" "$3" "$4" kept "$6" "$7" "link" + local cfg_fp="$1" && + local cfg_rnp="$2" && + local cfg_fpt="$3" && + local cfg_rnpt="$4" && + local arg_branch="$5" && + local arg_tag="$6" && + shift 6 &&
Unless we plan to never allow new parameters to be added (or existing one retired) from this helper, it would probably be easier to maintain if you wrote
local cfg_fp="$1" && shift && local cfg_rnp="$1" && shift && ...
Re: [PATCH 10/10] fetch tests: fix needless and buggy re-quoting
In reply to Ævar Arnfjörð Bjarmasont/t5510-fetch.sh test_configured_prune_type () { · line 852
Quoted 2 lines · Ævar Arnfjörð Bjarmason
t/t5510-fetch.sh +0
-2- remote_url="file://$TRASH_DIRECTORY/." && # NOT local yet! - local cmdline_setup="\"$remote_url\" \"$refspec_heads\""
Good riddance ;-)
Quoted 5 lines
t/t5510-fetch.sh +2
-1+ local remote_url="file://$TRASH_DIRECTORY/." && - git fetch '"$cmdline_setup"' && + git fetch "$remote_url" "$refspec_heads" &&
t/t5510-fetch.sh test_configured_prune_type () { · line 941
Quoted 10 lines
t/t5510-fetch.sh +9
-0+ git$git_fetch_c fetch \ + ${arg_fetch_prune:+--prune} \ + ${arg_fetch_no_prune:+--no-prune} \ + ${arg_fetch_prune_tags:+--prune-tags} \ + ${arg_fetch_origin:+origin} \ + ${arg_fetch_url:+"$remote_url"} \ + ${arg_fetch_refspec_tags:+"refs/tags/*:refs/tags/*"} \ + ${arg_fetch_refspec_heads:+"+refs/heads/*:refs/remotes/origin/*"} && +
This makes it a lot clearer, with no perl, no sed, no eval. It does become louder, but should be easier to follow in general ...
t/t5510-fetch.sh test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags · line 1042
Quoted 4 lines
test_configured_prune_type--mode link true unset true unset pruned pruned \t/t5510-fetch.sh +1
-1- "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*"
... except for a magic like this one.
We may remember the REMOTE_URL -> $remote_url trick used here this week, but I am not sure if we find it sensible in 3 months.
But overall I think this makes it simpler. I am not 100% sold on the necessity of lengthy earlier steps, though.
Thanks.
Re: [PATCH 10/10] fetch tests: fix needless and buggy re-quoting
Quoted 12 lines · Junio C Hamano
Ævar Arnfjörð Bjarmason <[email protected]> writes:
This makes it a lot clearer, with no perl, no sed, no eval. It does become louder, but should be easier to follow in general ...
t/t5510-fetch.sh test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags · line 1042
Quoted 4 lines
test_configured_prune_type--mode link true unset true unset pruned pruned \t/t5510-fetch.sh +1
-1- "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*"... except for a magic like this one.
Quoted 5 lines
We may remember the
REMOTE_URL-> $remote_url trick used here this week, but I am not sure if we find it sensible in 3 months.But overall I think this makes it simpler. I am not 100% sold on the necessity of lengthy earlier steps, though.
When I saw that this was a series with 10 patches (without reading anything else) I expected that you had created a test-lib.sh helper that allowed replacing a word in a string with another string, and then the remaining patches were fixing the other tests that have similar breaks when using "@" in the path.
(Heck, I'd even take a "test-tool replace-word <string> <word> <replacement>" implementation to avoid all of these issues we have due to using scripting languages that rely on special characters to define the match and replacement operation.)
It seems like this isn't the last time we are going to have a problem with string replacement like this, and having a well-defined helper would go far.
The rest of the changes to the test script seem more complicated than necessary for what _should_ be a simple problem.
Thanks,
-Stolee
Re: [PATCH 10/10] fetch tests: fix needless and buggy re-quoting
Quoted 16 lines · Derrick Stolee
Quoted 15 lines · Junio C Hamano
Ævar Arnfjörð Bjarmason <[email protected]> writes:
This makes it a lot clearer, with no perl, no sed, no eval. It does become louder, but should be easier to follow in general ...
t/t5510-fetch.sh test_configured_prune unset unset unset unset pruned pruned --prune --prune-tags · line 1042
Quoted 4 lines
test_configured_prune_type--mode link true unset true unset pruned pruned \t/t5510-fetch.sh +1
-1- "\"$remote_url\"" \ + REMOTE_URL \ "refs/tags/*:refs/tags/*" "+refs/heads/*:refs/remotes/origin/*"... except for a magic like this one.
We may remember the
REMOTE_URL-> $remote_url trick used here this week, but I am not sure if we find it sensible in 3 months.
Quoted 8 lines
Quoted 2 lines
But overall I think this makes it simpler. I am not 100% sold on the necessity of lengthy earlier steps, though.
When I saw that this was a series with 10 patches (without reading anything else) I expected that you had created a
test-lib.shhelper that allowed replacing a word in a string with another string, and then the remaining patches were fixing the other tests that have similar breaks when using "@" in the path.
I guess we could have such a helper, but I can't imagine a use-case for it where the answer wouldn't be the same as what this series suggests:
Just stop passing a quoted argument list as one giant string.
Quoted 2 lines
(Heck, I'd even take a "test-tool replace-word <string> <word> <replacement>" implementation [...]
If we need to replace a string we can use sed or perl, but much better is to not need to do so in the first place. E.g. the "origin" match we had before is now just "origin" in a case statement. I'm assuming that you mean a test-tool to do something similar to strvec_split() (or whatever the quoted variant was?).
Quoted 3 lines
to avoid all of these issues we have
due to using scripting languages that rely on special characters to define the match and replacement operation.)
We've got plenty of issues inherent in shellscript as a language with no little in the way of complex types (e.g. no hashes).
But in this case the language gives us the right type (list), we just weren't using it. No?
Quoted 3 lines
It seems like this isn't the last time we are going to have a problem with string replacement like this, and having a well-defined helper would go far.
...I guess I'm not seeing the use-case for it, in this case it was "just pass a list then", wouldn't that be the case in other similar scenarios.
Quoted 2 lines
The rest of the changes to the test script seem more complicated than necessary for what _should_ be a simple problem.
I tried to optimize it for ease of review as opposed to diff size or patch numbers, so e.g. doing mechanical replacements in their own commits.
Do you have any specific things in mind that you think are too complicated?
Yes, it should ideally not be such a pain, but as we made wide use of a string instead of a list digging ourselves out of that hole took some doing.
But I really think fixing the underlying issue is worth it, as opposed to just plastering over it.
Re: [PATCH 10/10] fetch tests: fix needless and buggy re-quoting
In reply to Derrick StoleeQuoted 3 lines · Derrick Stolee
It seems like this isn't the last time we are going to have a problem with string replacement like this, and having a well-defined helper would go far.
I think the idea of [10/10] is to use shell itself as a well-defined helper, with "string replacement" being "$variable_interpolation".
Which isn't a bad approach, I would say.
Quoted 2 lines
The rest of the changes to the test script seem more complicated than necessary for what _should_ be a simple problem.
True. I am not sure which parts are unnecessary "churn while at it, burying the most interesting and beneficial one at the end as a hostage", and which ones are absolutely needed to reach [10/10]. Perhaps all of them are the latter? I dunno.