Skip to contentSkip to conversation

ci: upgrade to the latest Azure Pipelines agent pools

PATCH Superseded Johannes Schindelin via GitGitGadget 4 participants 16 messages Feb 26, 2020 – Feb 28, 2020

lore
Share & tools
lore

Johannes Schindelin via GitGitGadget

author
#0 lore raw

cover ci: upgrade to the latest Azure Pipelines agent pools

At least the Windows agents we use will go away on March 23rd, 2020
[https://devblogs.microsoft.com/devops/removing-older-images-in-azure-pipelines-hosted-pools/]
, in favor of newer, faster, better ones. We might just as well use the latter already long before the cut-off date.

While at it, let's also update to a newer macOS pool. This necessitates two distinct fixes, though:

* Our ci/ scripts were ill-prepared to deal with System Integrity Protection [https://support.apple.com/en-us/HT204899]. This resulted in the p4d -V call seemingly "hanging" (Narrator's voice: it did not "hang", it actually opened a GUI dialog to ask the user whether it was okay to execute this quarantined executable).

* Apparently nobody ever bothered running the Apache2-based tests (t5616, t5702 and t5703, probably more) on macOS. I say that because they don't work, over-relying on GNU-specific behavior of sed. I rewrote those sed calls to use Perl instead.

Johannes Schindelin (3):
t/lib-httpd: avoid using BSD's sed
ci: prevent perforce from being quarantined Azure Pipeline: switch to the latest agent pools

 azure-pipelines.yml                | 37 +++++++++++++++++++----------
 ci/install-dependencies.sh         |  4 ++--
 t/lib-httpd.sh                     |  2 +-
 t/lib-httpd/apache.conf            |  6 ++---
 t/lib-httpd/apply-one-time-perl.sh | 27 +++++++++++++++++++++
 t/lib-httpd/apply-one-time-sed.sh  | 24 -------------------
 t/t5537-fetch-shallow.sh           | 10 ++++----
 t/t5616-partial-clone.sh           | 38 +++++++++++++++++-------------
 t/t5702-protocol-v2.sh             | 12 +++++-----
 t/t5703-upload-pack-ref-in-want.sh |  6 ++---
 10 files changed, 93 insertions(+), 73 deletions(-)
 create mode 100644 t/lib-httpd/apply-one-time-perl.sh
 delete mode 100644 t/lib-httpd/apply-one-time-sed.sh

base-commit: 2d2118b814c11f509e1aa76cb07110f7231668dc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-714%2Fdscho%2Fazure-pipelines-latest-pools-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-714/dscho/azure-pipelines-latest-pools-v1
Pull-Request: https://github.com/git/git/pull/714

Signature · 1 line
gitgitgadget
author
#11 lore raw

PATCH 3/3 Azure Pipeline: switch to the latest agent pools

From: Johannes Schindelin <[email protected]>

It would seem that at least the vs2015-win2012r2 pool (which we use via its old name, Hosted) is about to be phased out. Let's switch before that.

While at it, use the newer pool names as suggested at
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent

Signed-off-by Johannes Schindelin <[email protected]>
---
 azure-pipelines.yml | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)
azure-pipelines.yml +25 -12
index af2a5ea4845..675c3a43c9c 100644
@@ -5,7 +5,8 @@ jobs:
 - job: windows_build
   displayName: Windows Build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   steps:
   - powershell: |

@@ -61,7 +62,8 @@ jobs:
   displayName: Windows Test
   dependsOn: windows_build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   strategy:
     parallel: 10

@@ -133,7 +135,8 @@ jobs:
 - job: vs_build
   displayName: Visual Studio Build
   condition: succeeded()
-  pool: Hosted VS2017
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   steps:
   - powershell: |

@@ -181,6 +184,7 @@ jobs:
       platform: x64
       configuration: Release
       maximumCpuCount: 4
+      msbuildArguments: /p:PlatformToolset=v142
   - powershell: |
       & compat\vcbuild\vcpkg_copy_dlls.bat release
       if (!$?) { exit(1) }

@@ -224,7 +228,8 @@ jobs:
   displayName: Visual Studio Test
   dependsOn: vs_build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   strategy:
     parallel: 10

@@ -292,7 +297,8 @@ jobs:
 - job: linux_clang
   displayName: linux-clang
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -330,7 +336,8 @@ jobs:
 - job: linux_gcc
   displayName: linux-gcc
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -367,7 +374,8 @@ jobs:
 - job: osx_clang
   displayName: osx-clang
   condition: succeeded()
-  pool: Hosted macOS
+  pool:
+    vmImage: macOS-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -402,7 +410,8 @@ jobs:
 - job: osx_gcc
   displayName: osx-gcc
   condition: succeeded()
-  pool: Hosted macOS
+  pool:
+    vmImage: macOS-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -435,7 +444,8 @@ jobs:
 - job: gettext_poison
   displayName: GETTEXT_POISON
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -472,7 +482,8 @@ jobs:
 - job: linux32
   displayName: Linux32
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -506,7 +517,8 @@ jobs:
 - job: static_analysis
   displayName: StaticAnalysis
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -526,7 +538,8 @@ jobs:
 - job: documentation
   displayName: Documentation
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
Signature · 1 line
gitgitgadget
author
#1 lore raw

PATCH 1/3 t/lib-httpd: avoid using BSD's sed

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Among other differences relative to GNU sed, BSD sed always ends its output with a trailing newline, even if the input did not have such a trailing newline.

Surprisingly, this makes three httpd-based tests fail on macOS: t5616, t5702 and t5703. ("Surprisingly" because those tests have been around for some time, but apparently nobody runs them on macOS with a working Apache2 setup.)

The reason is that we use sed in those tests to filter the response of the web server. Apart from the fact that we use GNU constructs (such as using a space after the c command instead of a backslash and a newline), we have another problem: BSD sed LF-only newlines while webservers are supposed to use CR/LF ones.

Even worse, t5616 uses sed to replace a binary part of the response with a new binary part (kind of hoping that the replaced binary part does not contain a 0x0a byte which would be interpreted as a newline).

To that end, it calls on Perl to read the binary pack file and hex-encode it, then calls on sed to prefix every hex digit pair with a \x in order to construct the text that the c statement of the sed invocation is supposed to insert. So we call Perl and sed to construct a sed statement. The final nail in the coffin is that BSD sed does not even interpret those \x<hex> constructs.

Let's just replace all of that by Perl snippets. With Perl, at least, we do not have to deal with GNU vs BSD semantics, we do not have to worry about unwanted trailing newlines, and we do not have to spawn commands to construct arguments for other commands to be spawned (i.e. we can avoid a whole lot of shell scripting complexity).

The upshot is that this fixes t5616, t5702 and t5703 on macOS with Apache2.

Signed-off-by Johannes Schindelin <[email protected]>
---
 t/lib-httpd.sh                     |  2 +-
 t/lib-httpd/apache.conf            |  6 ++---
 t/lib-httpd/apply-one-time-perl.sh | 27 +++++++++++++++++++++
 t/lib-httpd/apply-one-time-sed.sh  | 24 -------------------
 t/t5537-fetch-shallow.sh           | 10 ++++----
 t/t5616-partial-clone.sh           | 38 +++++++++++++++++-------------
 t/t5702-protocol-v2.sh             | 12 +++++-----
 t/t5703-upload-pack-ref-in-want.sh |  6 ++---
 8 files changed, 66 insertions(+), 59 deletions(-)
 create mode 100644 t/lib-httpd/apply-one-time-perl.sh
 delete mode 100644 t/lib-httpd/apply-one-time-sed.sh
t/lib-httpd.sh +1 -1
index 656997b4d66..1449ee95e9e 100644
@@ -132,7 +132,7 @@ prepare_httpd() {
 	install_script broken-smart-http.sh
 	install_script error-smart-http.sh
 	install_script error.sh
-	install_script apply-one-time-sed.sh
+	install_script apply-one-time-perl.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +3 -3
index 5c1c86c193a..994e5290d63 100644
@@ -113,7 +113,7 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
 	SetEnv GIT_HTTP_EXPORT_ALL
 </LocationMatch>
-<LocationMatch /one_time_sed/>
+<LocationMatch /one_time_perl/>
 	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
 	SetEnv GIT_HTTP_EXPORT_ALL
 </LocationMatch>

@@ -122,7 +122,7 @@ ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1
 ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
-ScriptAliasMatch /one_time_sed/(.*) apply-one-time-sed.sh/$1
+ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>

@@ -135,7 +135,7 @@ ScriptAliasMatch /one_time_sed/(.*) apply-one-time-sed.sh/$1
 <Files error.sh>
   Options ExecCGI
 </Files>
-<Files apply-one-time-sed.sh>
+<Files apply-one-time-perl.sh>
 	Options ExecCGI
 </Files>
 <Files ${GIT_EXEC_PATH}/git-http-backend>
t/lib-httpd/apply-one-time-perl.sh new file +27 -0
new file mode 100644
index 00000000000..09a0abdff7c
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# If "one-time-perl" exists in $HTTPD_ROOT_PATH, run perl on the HTTP response,
+# using the contents of "one-time-perl" as the perl command to be run. If the
+# response was modified as a result, delete "one-time-perl" so that subsequent
+# HTTP responses are no longer modified.
+#
+# This can be used to simulate the effects of the repository changing in
+# between HTTP request-response pairs.
+if test -f one-time-perl
+then
+	LC_ALL=C
+	export LC_ALL
+
+	"$GIT_EXEC_PATH/git-http-backend" >out
+	perl -pe "$(cat one-time-perl)" out >out_modified
+
+	if cmp -s out out_modified
+	then
+		cat out
+	else
+		cat out_modified
+		rm one-time-perl
+	fi
+else
+	"$GIT_EXEC_PATH/git-http-backend"
+fi
t/lib-httpd/apply-one-time-sed.sh deleted +0 -24
deleted file mode 100644
index bf7689d0202..00000000000
@@ -1,24 +0,0 @@
-#!/bin/sh
-
-# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response,
-# using the contents of "one-time-sed" as the sed command to be run. If the
-# response was modified as a result, delete "one-time-sed" so that subsequent
-# HTTP responses are no longer modified.
-#
-# This can be used to simulate the effects of the repository changing in
-# between HTTP request-response pairs.
-if test -f one-time-sed
-then
-	"$GIT_EXEC_PATH/git-http-backend" >out
-	sed "$(cat one-time-sed)" out >out_modified
-
-	if cmp -s out out_modified
-	then
-		cat out
-	else
-		cat out_modified
-		rm one-time-sed
-	fi
-else
-	"$GIT_EXEC_PATH/git-http-backend"
-fi
t/t5537-fetch-shallow.sh +5 -5
index 9e16512fe31..4f10057e9f1 100755
@@ -237,7 +237,7 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	git -C "$REPO" config protocol.version 2 &&
 	git -C client config protocol.version 2 &&
 
-	git -C client fetch --depth=2 "$HTTPD_URL/one_time_sed/repo" master:a_branch &&
+	git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" master:a_branch &&
 
 	# Craft a situation in which the server sends back an unshallow request
 	# with an empty packfile. This is done by refetching with a shorter

@@ -246,13 +246,13 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	printf "$(test_oid sed)" \
 	       "$(git -C "$REPO" rev-parse HEAD)" \
 	       "$(git -C "$REPO" rev-parse HEAD^)" \
-	       >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	       >"$HTTPD_ROOT_PATH/one-time-perl" &&
 	test_must_fail env GIT_TEST_SIDEBAND_ALL=0 git -C client \
-		fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \
+		fetch --depth=1 "$HTTPD_URL/one_time_perl/repo" \
 		master:a_branch &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed" &&
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	# Ensure that the resulting repo is consistent, despite our failure to
 	# fetch.
t/t5616-partial-clone.sh +21 -17
index 9a9178fd281..0eb5b1c47b1 100755
@@ -398,14 +398,18 @@ intersperse () {
 	sed 's/\(..\)/'$1'\1/g'
 }
 
-# Create a one-time-sed command to replace the existing packfile with $1.
+# Create a one-time-perl command to replace the existing packfile with $1.
 replace_packfile () {
 	# The protocol requires that the packfile be sent in sideband 1, hence
 	# the extra \x01 byte at the beginning.
-	printf "1,/packfile/!c %04x\\\\x01%s0000" \
-		"$(($(wc -c <$1) + 5))" \
-		"$(hex_unpack <$1 | intersperse '\\x')" \
-		>"$HTTPD_ROOT_PATH/one-time-sed"
+	cp $1 "$HTTPD_ROOT_PATH/one-time-pack" &&
+	echo 'if (/packfile/) {
+		print;
+		my $length = -s "one-time-pack";
+		printf "%04x\x01", $length + 5;
+		print `cat one-time-pack` . "0000";
+		last
+	}' >"$HTTPD_ROOT_PATH/one-time-perl"
 }
 
 test_expect_success 'upon cloning, check that all refs point to objects' '

@@ -429,16 +433,16 @@ test_expect_success 'upon cloning, check that all refs point to objects' '
 	# \x01 byte at the beginning.
 	replace_packfile incomplete.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 	test_must_fail git -c protocol.version=2 clone \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo 2>err &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo 2>err &&
 
 	test_i18ngrep "did not send all necessary objects" err &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 test_expect_success 'when partial cloning, tolerate server not sending target of tag' '

@@ -469,17 +473,17 @@ test_expect_success 'when partial cloning, tolerate server not sending target of
 	# \x01 byte at the beginning.
 	replace_packfile incomplete.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 
 	# Exercise to make sure it works.
 	git -c protocol.version=2 clone \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo 2> err &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo 2> err &&
 	! grep "missing object referenced by" err &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 test_expect_success 'tolerate server sending REF_DELTA against missing promisor objects' '

@@ -502,7 +506,7 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 
 	# Clone. The client has deltabase_have but not deltabase_missing.
 	git -c protocol.version=2 clone --no-checkout \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo &&
 	git -C repo hash-object -w -- "$SERVER/have.txt" &&
 
 	# Sanity check to ensure that the client does not have

@@ -543,7 +547,7 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 
 	replace_packfile thin.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 

@@ -556,8 +560,8 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 	grep "want $(cat deltabase_missing)" trace &&
 	! grep "want $(cat deltabase_have)" trace &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 # DO NOT add non-httpd-specific tests here, because the last part of this
t/t5702-protocol-v2.sh +6 -6
index 7fd7102c874..5039e66dc47 100755
@@ -712,11 +712,11 @@ test_expect_success 'when server sends "ready", expect DELIM' '
 
 	# After "ready" in the acknowledgments section, pretend that a FLUSH
 	# (0000) was sent instead of a DELIM (0001).
-	printf "/ready/,$ s/0001/0000/" \
-		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+	printf "\$ready = 1 if /ready/; \$ready && s/0001/0000/" \
+		>"$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	test_must_fail git -C http_child -c protocol.version=2 \
-		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+		fetch "$HTTPD_URL/one_time_perl/http_parent" 2> err &&
 	test_i18ngrep "expected packfile to be sent after .ready." err
 '
 

@@ -737,12 +737,12 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
 
 	# After the acknowledgments section, pretend that a DELIM
 	# (0001) was sent instead of a FLUSH (0000).
-	printf "/acknowledgments/,$ s/0000/0001/" \
-		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+	printf "\$ack = 1 if /acknowledgments/; \$ack && s/0000/0001/" \
+		>"$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \
 		-c protocol.version=2 \
-		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+		fetch "$HTTPD_URL/one_time_perl/http_parent" 2> err &&
 	grep "fetch< .*acknowledgments" log &&
 	! grep "fetch< .*ready" log &&
 	test_i18ngrep "expected no other sections to be sent after no .ready." err
t/t5703-upload-pack-ref-in-want.sh +3 -3
index 8aeeaac5091..7fba3063bf9 100755
@@ -314,7 +314,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		test_commit m3 &&
 		git tag -d m2 m3
 	) &&
-	git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_sed/repo" &&
+	git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_perl/repo" &&
 	git -C "$LOCAL_PRISTINE" config protocol.version 2
 '
 

@@ -327,7 +327,7 @@ inconsistency () {
 	# RPCs during a single negotiation.
 	oid1=$(git -C "$REPO" rev-parse $1) &&
 	oid2=$(git -C "$REPO" rev-parse $2) &&
-	echo "s/$oid1/$oid2/" >"$HTTPD_ROOT_PATH/one-time-sed"
+	echo "s/$oid1/$oid2/" >"$HTTPD_ROOT_PATH/one-time-perl"
 }
 
 test_expect_success 'server is initially ahead - no ref in want' '

@@ -379,7 +379,7 @@ test_expect_success 'server loses a ref - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	echo "s/master/raster/" >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	echo "s/master/raster/" >"$HTTPD_ROOT_PATH/one-time-perl" &&
 	test_must_fail git -C local fetch 2>err &&
 
 	test_i18ngrep "fatal: remote error: unknown ref refs/heads/raster" err
Signature · 1 line
gitgitgadget
author
#10 lore raw

PATCH 2/3 ci: prevent `perforce` from being quarantined

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The most recent Azure Pipelines macOS agents enable what Apple calls "System Integrity Protection". This makes p4d -V hang: there is some sort of GUI dialog waiting for the user to acknowledge that the copied binaries are legit and may be executed, but on build agents, there is no user who could acknowledge that.

Let's ask Homebrew specifically to _not_ quarantine the Perforce binaries.

Helped-by Aleksandr Chebotov
Signed-off-by Johannes Schindelin <[email protected]>
---
 ci/install-dependencies.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +2 -2
index cd59855d73d..497fd32ca83 100755
@@ -40,11 +40,11 @@ osx-clang|osx-gcc)
 	test -z "$BREW_INSTALL_PACKAGES" ||
 	brew install $BREW_INSTALL_PACKAGES
 	brew link --force gettext
-	brew cask install perforce || {
+	brew cask install --no-quarantine perforce || {
 		# Update the definitions and try again
 		cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask &&
 		git -C "$cask_repo" pull --no-stat &&
-		brew cask install perforce
+		brew cask install --no-quarantine perforce
 	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
Signature · 1 line
gitgitgadget
#2 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

In reply to Johannes Schindelin via GitGitGadget
Quoted 11 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Among other differences relative to GNU sed, BSD sed always ends its output with a trailing newline, even if the input did not have such a trailing newline.

Surprisingly, this makes three httpd-based tests fail on macOS: t5616, t5702 and t5703. ("Surprisingly" because those tests have been around for some time, but apparently nobody runs them on macOS with a working Apache2 setup.)

Hmm, this is interesting - all tests (that are executed) are passing on FreeBSD, in CI.

I tried on FreeBSD and do not see a trailing newline added; I'm not sure how sed behaves on other BSDs. However, you probably want to refer to macOS sed rather than BSD sed in the commit.

#7 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

In reply to Johannes Schindelin via GitGitGadget

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 14 lines · writes:
t/t5537-fetch-shallow.sh +1 -1
index 9e16512fe31..4f10057e9f1 100755
@@ -237,7 +237,7 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	git -C "$REPO" config protocol.version 2 &&
 	git -C client config protocol.version 2 &&
 
-	git -C client fetch --depth=2 "$HTTPD_URL/one_time_sed/repo" master:a_branch &&
+	git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" master:a_branch &&
 
 	# Craft a situation in which the server sends back an unshallow request
 	# with an empty packfile. This is done by refetching with a shorter

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 246

Quoted 1 line
t/t5537-fetch-shallow.sh +0 -0
@@ -246,13 +246,13 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	printf "$(test_oid sed)" \

Hmm, shouldn't the test-oid token "sed" whose value is set up in the setup section of this test script also be renamed to "perl"? Or, if we are actively taking advantage of the fact that the syntax of the replacement operator is the same between the languages, perhaps "sed" is better renamed to something more language agnostic and reflects the purpose/reason why we extend the packet header by two bytes with the one-time munging process?

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 249

Quoted 4 lines
t/t5537-fetch-shallow.sh +1 -1
 	       "$(git -C "$REPO" rev-parse HEAD)" \
 	       "$(git -C "$REPO" rev-parse HEAD^)" \
-	       >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	       >"$HTTPD_ROOT_PATH/one-time-perl" &&

Other than that, this step looked quite sensible. Thanks.

#8 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

Quoted 27 lines · Junio C Hamano

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 14 lines · writes:
t/t5537-fetch-shallow.sh +1 -1
index 9e16512fe31..4f10057e9f1 100755
@@ -237,7 +237,7 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	git -C "$REPO" config protocol.version 2 &&
 	git -C client config protocol.version 2 &&
 
-	git -C client fetch --depth=2 "$HTTPD_URL/one_time_sed/repo" master:a_branch &&
+	git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" master:a_branch &&
 
 	# Craft a situation in which the server sends back an unshallow request
 	# with an empty packfile. This is done by refetching with a shorter

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 246

Quoted 1 line
t/t5537-fetch-shallow.sh +0 -0
@@ -246,13 +246,13 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	printf "$(test_oid sed)" \

Hmm, shouldn't the test-oid token "sed" whose value is set up in the setup section of this test script also be renamed to "perl"? Or, if we are actively taking advantage of the fact that the syntax of the replacement operator is the same between the languages, perhaps "sed" is better renamed to something more language agnostic and reflects the purpose/reason why we extend the packet header by two bytes with the one-time munging process?

Quoted 6 lines

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 249

Quoted 4 lines
t/t5537-fetch-shallow.sh +1 -1
 	       "$(git -C "$REPO" rev-parse HEAD)" \
 	       "$(git -C "$REPO" rev-parse HEAD^)" \
-	       >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	       >"$HTTPD_ROOT_PATH/one-time-perl" &&

Other than that, this step looked quite sensible. Thanks.

Hmm, is it because you wanted to backport this down to 'maint' (otherwise, your tests will start failing in a month) that you left the "test_oid sed" thing untouched? If so, that makes sort-of sense.

I expect that the series will be rerolled, if only for s/BSD/macOS/ mentioned elsewhere in the thread, but in the meantime, I'll rebase them on 'maint' "as a practice" while queuing.

author
#12 lore raw

v2 cover ci: upgrade to the latest Azure Pipelines agent pools

In reply to Johannes Schindelin via GitGitGadget

At least the Windows agents we use will go away on March 23rd, 2020
[https://devblogs.microsoft.com/devops/removing-older-images-in-azure-pipelines-hosted-pools/]
, in favor of newer, faster, better ones. We might just as well use the latter already long before the cut-off date.

While at it, let's also update to a newer macOS pool. This necessitates two distinct fixes, though:

* Our ci/ scripts were ill-prepared to deal with System Integrity Protection [https://support.apple.com/en-us/HT204899]. This resulted in the p4d -V call seemingly "hanging" (Narrator's voice: it did not "hang", it actually opened a GUI dialog to ask the user whether it was okay to execute this quarantined executable).

* Apparently nobody ever bothered running the Apache2-based tests (t5616, t5702 and t5703, probably more) on macOS. I say that because they don't work, over-relying on GNU-specific behavior of sed. I rewrote those sed calls to use Perl instead.

Changes since v1:

* Avoid blaming BSD for what is (at least nowadays) squarely a macOS issue.
* Targeting maint

Johannes Schindelin (3):
t/lib-httpd: avoid using macOS' sed
ci: prevent perforce from being quarantined Azure Pipeline: switch to the latest agent pools

 azure-pipelines.yml                | 37 +++++++++++++++++++----------
 ci/install-dependencies.sh         |  4 ++--
 t/lib-httpd.sh                     |  2 +-
 t/lib-httpd/apache.conf            |  6 ++---
 t/lib-httpd/apply-one-time-perl.sh | 27 +++++++++++++++++++++
 t/lib-httpd/apply-one-time-sed.sh  | 24 -------------------
 t/t5537-fetch-shallow.sh           | 10 ++++----
 t/t5616-partial-clone.sh           | 38 +++++++++++++++++-------------
 t/t5702-protocol-v2.sh             | 12 +++++-----
 t/t5703-upload-pack-ref-in-want.sh |  6 ++---
 10 files changed, 93 insertions(+), 73 deletions(-)
 create mode 100644 t/lib-httpd/apply-one-time-perl.sh
 delete mode 100644 t/lib-httpd/apply-one-time-sed.sh

base-commit: c522f061d551c9bb8684a7c3859b2ece4499b56b
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-714%2Fdscho%2Fazure-pipelines-latest-pools-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-714/dscho/azure-pipelines-latest-pools-v2
Pull-Request: https://github.com/git/git/pull/714

Range-diff vs v1:

1: 9900cacbfef ! 1: af28d115ded t/lib-httpd: avoid using BSD's sed
@@ -1,8 +1,8 @@
Author: Johannes Schindelin <[email protected]>

- t/lib-httpd: avoid using BSD's sed
+ t/lib-httpd: avoid using macOS' sed

- Among other differences relative to GNU sed, BSD sed always ends its
+ Among other differences relative to GNU sed, macOS' sed always ends its
output with a trailing newline, even if the input did not have such a trailing newline.

@@ -14,7 +14,7 @@
The reason is that we use sed in those tests to filter the response of the web server. Apart from the fact that we use GNU constructs (such as using a space after the c command instead of a backslash and a
- newline), we have another problem: BSD sed LF-only newlines while
+ newline), we have another problem: macOS' sed LF-only newlines while
webservers are supposed to use CR/LF ones.

Even worse, t5616 uses sed to replace a binary part of the response
@@ -25,11 +25,11 @@
hex-encode it, then calls on sed to prefix every hex digit pair with a \x in order to construct the text that the c statement of the sed invocation is supposed to insert. So we call Perl and sed to construct a
- sed statement. The final nail in the coffin is that BSD sed does not
+ sed statement. The final nail in the coffin is that macOS' sed does not
even interpret those \x<hex> constructs.

Let's just replace all of that by Perl snippets. With Perl, at least, we
- do not have to deal with GNU vs BSD semantics, we do not have to worry
+ do not have to deal with GNU vs macOS semantics, we do not have to worry
about unwanted trailing newlines, and we do not have to spawn commands to construct arguments for other commands to be spawned (i.e. we can avoid a whole lot of shell scripting complexity).
@@ -159,7 +159,7 @@
# Craft a situation in which the server sends back an unshallow request # with an empty packfile. This is done by refetching with a shorter
@@
- printf "$(test_oid sed)" \
+ printf "s/0034shallow %s/0036unshallow %s/" \
"$(git -C "$REPO" rev-parse HEAD)" \
"$(git -C "$REPO" rev-parse HEAD^)" \
- >"$HTTPD_ROOT_PATH/one-time-sed" &&

 2:  8739f8bac0f = 2:  5e5fcf3b76b ci: prevent `perforce` from being quarantined
 3:  f141c295fd8 = 3:  6d2f8e7e70e Azure Pipeline: switch to the latest agent pools
Signature · 1 line
gitgitgadget
author
#14 lore raw

PATCH v2 2/3 ci: prevent `perforce` from being quarantined

From: Johannes Schindelin <[email protected]>

The most recent Azure Pipelines macOS agents enable what Apple calls "System Integrity Protection". This makes p4d -V hang: there is some sort of GUI dialog waiting for the user to acknowledge that the copied binaries are legit and may be executed, but on build agents, there is no user who could acknowledge that.

Let's ask Homebrew specifically to _not_ quarantine the Perforce binaries.

Helped-by Aleksandr Chebotov
Signed-off-by Johannes Schindelin <[email protected]>
---
 ci/install-dependencies.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +2 -2
index cd59855d73d..497fd32ca83 100755
@@ -40,11 +40,11 @@ osx-clang|osx-gcc)
 	test -z "$BREW_INSTALL_PACKAGES" ||
 	brew install $BREW_INSTALL_PACKAGES
 	brew link --force gettext
-	brew cask install perforce || {
+	brew cask install --no-quarantine perforce || {
 		# Update the definitions and try again
 		cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask &&
 		git -C "$cask_repo" pull --no-stat &&
-		brew cask install perforce
+		brew cask install --no-quarantine perforce
 	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
Signature · 1 line
gitgitgadget
author
#15 lore raw

PATCH v2 3/3 Azure Pipeline: switch to the latest agent pools

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

It would seem that at least the vs2015-win2012r2 pool (which we use via its old name, Hosted) is about to be phased out. Let's switch before that.

While at it, use the newer pool names as suggested at
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent

Signed-off-by Johannes Schindelin <[email protected]>
---
 azure-pipelines.yml | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)
azure-pipelines.yml +25 -12
index af2a5ea4845..675c3a43c9c 100644
@@ -5,7 +5,8 @@ jobs:
 - job: windows_build
   displayName: Windows Build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   steps:
   - powershell: |

@@ -61,7 +62,8 @@ jobs:
   displayName: Windows Test
   dependsOn: windows_build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   strategy:
     parallel: 10

@@ -133,7 +135,8 @@ jobs:
 - job: vs_build
   displayName: Visual Studio Build
   condition: succeeded()
-  pool: Hosted VS2017
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   steps:
   - powershell: |

@@ -181,6 +184,7 @@ jobs:
       platform: x64
       configuration: Release
       maximumCpuCount: 4
+      msbuildArguments: /p:PlatformToolset=v142
   - powershell: |
       & compat\vcbuild\vcpkg_copy_dlls.bat release
       if (!$?) { exit(1) }

@@ -224,7 +228,8 @@ jobs:
   displayName: Visual Studio Test
   dependsOn: vs_build
   condition: succeeded()
-  pool: Hosted
+  pool:
+    vmImage: windows-latest
   timeoutInMinutes: 240
   strategy:
     parallel: 10

@@ -292,7 +297,8 @@ jobs:
 - job: linux_clang
   displayName: linux-clang
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -330,7 +336,8 @@ jobs:
 - job: linux_gcc
   displayName: linux-gcc
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -367,7 +374,8 @@ jobs:
 - job: osx_clang
   displayName: osx-clang
   condition: succeeded()
-  pool: Hosted macOS
+  pool:
+    vmImage: macOS-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -402,7 +410,8 @@ jobs:
 - job: osx_gcc
   displayName: osx-gcc
   condition: succeeded()
-  pool: Hosted macOS
+  pool:
+    vmImage: macOS-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -435,7 +444,8 @@ jobs:
 - job: gettext_poison
   displayName: GETTEXT_POISON
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -472,7 +482,8 @@ jobs:
 - job: linux32
   displayName: Linux32
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -506,7 +517,8 @@ jobs:
 - job: static_analysis
   displayName: StaticAnalysis
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1

@@ -526,7 +538,8 @@ jobs:
 - job: documentation
   displayName: Documentation
   condition: succeeded()
-  pool: Hosted Ubuntu 1604
+  pool:
+    vmImage: ubuntu-latest
   steps:
   - bash: |
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
Signature · 1 line
gitgitgadget
author
#13 lore raw

PATCH v2 1/3 t/lib-httpd: avoid using macOS' sed

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Among other differences relative to GNU sed, macOS' sed always ends its output with a trailing newline, even if the input did not have such a trailing newline.

Surprisingly, this makes three httpd-based tests fail on macOS: t5616, t5702 and t5703. ("Surprisingly" because those tests have been around for some time, but apparently nobody runs them on macOS with a working Apache2 setup.)

The reason is that we use sed in those tests to filter the response of the web server. Apart from the fact that we use GNU constructs (such as using a space after the c command instead of a backslash and a newline), we have another problem: macOS' sed LF-only newlines while webservers are supposed to use CR/LF ones.

Even worse, t5616 uses sed to replace a binary part of the response with a new binary part (kind of hoping that the replaced binary part does not contain a 0x0a byte which would be interpreted as a newline).

To that end, it calls on Perl to read the binary pack file and hex-encode it, then calls on sed to prefix every hex digit pair with a \x in order to construct the text that the c statement of the sed invocation is supposed to insert. So we call Perl and sed to construct a sed statement. The final nail in the coffin is that macOS' sed does not even interpret those \x<hex> constructs.

Let's just replace all of that by Perl snippets. With Perl, at least, we do not have to deal with GNU vs macOS semantics, we do not have to worry about unwanted trailing newlines, and we do not have to spawn commands to construct arguments for other commands to be spawned (i.e. we can avoid a whole lot of shell scripting complexity).

The upshot is that this fixes t5616, t5702 and t5703 on macOS with Apache2.

Signed-off-by Johannes Schindelin <[email protected]>
---
 t/lib-httpd.sh                     |  2 +-
 t/lib-httpd/apache.conf            |  6 ++---
 t/lib-httpd/apply-one-time-perl.sh | 27 +++++++++++++++++++++
 t/lib-httpd/apply-one-time-sed.sh  | 24 -------------------
 t/t5537-fetch-shallow.sh           | 10 ++++----
 t/t5616-partial-clone.sh           | 38 +++++++++++++++++-------------
 t/t5702-protocol-v2.sh             | 12 +++++-----
 t/t5703-upload-pack-ref-in-want.sh |  6 ++---
 8 files changed, 66 insertions(+), 59 deletions(-)
 create mode 100644 t/lib-httpd/apply-one-time-perl.sh
 delete mode 100644 t/lib-httpd/apply-one-time-sed.sh
t/lib-httpd.sh +1 -1
index 656997b4d66..1449ee95e9e 100644
@@ -132,7 +132,7 @@ prepare_httpd() {
 	install_script broken-smart-http.sh
 	install_script error-smart-http.sh
 	install_script error.sh
-	install_script apply-one-time-sed.sh
+	install_script apply-one-time-perl.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +3 -3
index 5c1c86c193a..994e5290d63 100644
@@ -113,7 +113,7 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
 	SetEnv GIT_HTTP_EXPORT_ALL
 </LocationMatch>
-<LocationMatch /one_time_sed/>
+<LocationMatch /one_time_perl/>
 	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
 	SetEnv GIT_HTTP_EXPORT_ALL
 </LocationMatch>

@@ -122,7 +122,7 @@ ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1
 ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
-ScriptAliasMatch /one_time_sed/(.*) apply-one-time-sed.sh/$1
+ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>

@@ -135,7 +135,7 @@ ScriptAliasMatch /one_time_sed/(.*) apply-one-time-sed.sh/$1
 <Files error.sh>
   Options ExecCGI
 </Files>
-<Files apply-one-time-sed.sh>
+<Files apply-one-time-perl.sh>
 	Options ExecCGI
 </Files>
 <Files ${GIT_EXEC_PATH}/git-http-backend>
t/lib-httpd/apply-one-time-perl.sh new file +27 -0
new file mode 100644
index 00000000000..09a0abdff7c
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# If "one-time-perl" exists in $HTTPD_ROOT_PATH, run perl on the HTTP response,
+# using the contents of "one-time-perl" as the perl command to be run. If the
+# response was modified as a result, delete "one-time-perl" so that subsequent
+# HTTP responses are no longer modified.
+#
+# This can be used to simulate the effects of the repository changing in
+# between HTTP request-response pairs.
+if test -f one-time-perl
+then
+	LC_ALL=C
+	export LC_ALL
+
+	"$GIT_EXEC_PATH/git-http-backend" >out
+	perl -pe "$(cat one-time-perl)" out >out_modified
+
+	if cmp -s out out_modified
+	then
+		cat out
+	else
+		cat out_modified
+		rm one-time-perl
+	fi
+else
+	"$GIT_EXEC_PATH/git-http-backend"
+fi
t/lib-httpd/apply-one-time-sed.sh deleted +0 -24
deleted file mode 100644
index bf7689d0202..00000000000
@@ -1,24 +0,0 @@
-#!/bin/sh
-
-# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response,
-# using the contents of "one-time-sed" as the sed command to be run. If the
-# response was modified as a result, delete "one-time-sed" so that subsequent
-# HTTP responses are no longer modified.
-#
-# This can be used to simulate the effects of the repository changing in
-# between HTTP request-response pairs.
-if test -f one-time-sed
-then
-	"$GIT_EXEC_PATH/git-http-backend" >out
-	sed "$(cat one-time-sed)" out >out_modified
-
-	if cmp -s out out_modified
-	then
-		cat out
-	else
-		cat out_modified
-		rm one-time-sed
-	fi
-else
-	"$GIT_EXEC_PATH/git-http-backend"
-fi
t/t5537-fetch-shallow.sh +5 -5
index 97a67728ca9..dd9cd4b51a2 100755
@@ -233,7 +233,7 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	git -C "$REPO" config protocol.version 2 &&
 	git -C client config protocol.version 2 &&
 
-	git -C client fetch --depth=2 "$HTTPD_URL/one_time_sed/repo" master:a_branch &&
+	git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" master:a_branch &&
 
 	# Craft a situation in which the server sends back an unshallow request
 	# with an empty packfile. This is done by refetching with a shorter

@@ -242,13 +242,13 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	printf "s/0034shallow %s/0036unshallow %s/" \
 	       "$(git -C "$REPO" rev-parse HEAD)" \
 	       "$(git -C "$REPO" rev-parse HEAD^)" \
-	       >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	       >"$HTTPD_ROOT_PATH/one-time-perl" &&
 	test_must_fail env GIT_TEST_SIDEBAND_ALL=0 git -C client \
-		fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \
+		fetch --depth=1 "$HTTPD_URL/one_time_perl/repo" \
 		master:a_branch &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed" &&
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	# Ensure that the resulting repo is consistent, despite our failure to
 	# fetch.
t/t5616-partial-clone.sh +21 -17
index 9a9178fd281..0eb5b1c47b1 100755
@@ -398,14 +398,18 @@ intersperse () {
 	sed 's/\(..\)/'$1'\1/g'
 }
 
-# Create a one-time-sed command to replace the existing packfile with $1.
+# Create a one-time-perl command to replace the existing packfile with $1.
 replace_packfile () {
 	# The protocol requires that the packfile be sent in sideband 1, hence
 	# the extra \x01 byte at the beginning.
-	printf "1,/packfile/!c %04x\\\\x01%s0000" \
-		"$(($(wc -c <$1) + 5))" \
-		"$(hex_unpack <$1 | intersperse '\\x')" \
-		>"$HTTPD_ROOT_PATH/one-time-sed"
+	cp $1 "$HTTPD_ROOT_PATH/one-time-pack" &&
+	echo 'if (/packfile/) {
+		print;
+		my $length = -s "one-time-pack";
+		printf "%04x\x01", $length + 5;
+		print `cat one-time-pack` . "0000";
+		last
+	}' >"$HTTPD_ROOT_PATH/one-time-perl"
 }
 
 test_expect_success 'upon cloning, check that all refs point to objects' '

@@ -429,16 +433,16 @@ test_expect_success 'upon cloning, check that all refs point to objects' '
 	# \x01 byte at the beginning.
 	replace_packfile incomplete.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 	test_must_fail git -c protocol.version=2 clone \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo 2>err &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo 2>err &&
 
 	test_i18ngrep "did not send all necessary objects" err &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 test_expect_success 'when partial cloning, tolerate server not sending target of tag' '

@@ -469,17 +473,17 @@ test_expect_success 'when partial cloning, tolerate server not sending target of
 	# \x01 byte at the beginning.
 	replace_packfile incomplete.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 
 	# Exercise to make sure it works.
 	git -c protocol.version=2 clone \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo 2> err &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo 2> err &&
 	! grep "missing object referenced by" err &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 test_expect_success 'tolerate server sending REF_DELTA against missing promisor objects' '

@@ -502,7 +506,7 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 
 	# Clone. The client has deltabase_have but not deltabase_missing.
 	git -c protocol.version=2 clone --no-checkout \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo &&
+		--filter=blob:none $HTTPD_URL/one_time_perl/server repo &&
 	git -C repo hash-object -w -- "$SERVER/have.txt" &&
 
 	# Sanity check to ensure that the client does not have

@@ -543,7 +547,7 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 
 	replace_packfile thin.pack &&
 
-	# Use protocol v2 because the sed command looks for the "packfile"
+	# Use protocol v2 because the perl command looks for the "packfile"
 	# section header.
 	test_config -C "$SERVER" protocol.version 2 &&
 

@@ -556,8 +560,8 @@ test_expect_success 'tolerate server sending REF_DELTA against missing promisor
 	grep "want $(cat deltabase_missing)" trace &&
 	! grep "want $(cat deltabase_have)" trace &&
 
-	# Ensure that the one-time-sed script was used.
-	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+	# Ensure that the one-time-perl script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-perl"
 '
 
 # DO NOT add non-httpd-specific tests here, because the last part of this
t/t5702-protocol-v2.sh +6 -6
index 7fd7102c874..5039e66dc47 100755
@@ -712,11 +712,11 @@ test_expect_success 'when server sends "ready", expect DELIM' '
 
 	# After "ready" in the acknowledgments section, pretend that a FLUSH
 	# (0000) was sent instead of a DELIM (0001).
-	printf "/ready/,$ s/0001/0000/" \
-		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+	printf "\$ready = 1 if /ready/; \$ready && s/0001/0000/" \
+		>"$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	test_must_fail git -C http_child -c protocol.version=2 \
-		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+		fetch "$HTTPD_URL/one_time_perl/http_parent" 2> err &&
 	test_i18ngrep "expected packfile to be sent after .ready." err
 '
 

@@ -737,12 +737,12 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
 
 	# After the acknowledgments section, pretend that a DELIM
 	# (0001) was sent instead of a FLUSH (0000).
-	printf "/acknowledgments/,$ s/0000/0001/" \
-		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+	printf "\$ack = 1 if /acknowledgments/; \$ack && s/0000/0001/" \
+		>"$HTTPD_ROOT_PATH/one-time-perl" &&
 
 	test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \
 		-c protocol.version=2 \
-		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+		fetch "$HTTPD_URL/one_time_perl/http_parent" 2> err &&
 	grep "fetch< .*acknowledgments" log &&
 	! grep "fetch< .*ready" log &&
 	test_i18ngrep "expected no other sections to be sent after no .ready." err
t/t5703-upload-pack-ref-in-want.sh +3 -3
index 1424fabd4aa..e1eec7d7bb7 100755
@@ -313,7 +313,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		test_commit m3 &&
 		git tag -d m2 m3
 	) &&
-	git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_sed/repo" &&
+	git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_perl/repo" &&
 	git -C "$LOCAL_PRISTINE" config protocol.version 2
 '
 

@@ -326,7 +326,7 @@ inconsistency () {
 	# RPCs during a single negotiation.
 	oid1=$(git -C "$REPO" rev-parse $1) &&
 	oid2=$(git -C "$REPO" rev-parse $2) &&
-	echo "s/$oid1/$oid2/" >"$HTTPD_ROOT_PATH/one-time-sed"
+	echo "s/$oid1/$oid2/" >"$HTTPD_ROOT_PATH/one-time-perl"
 }
 
 test_expect_success 'server is initially ahead - no ref in want' '

@@ -378,7 +378,7 @@ test_expect_success 'server loses a ref - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	echo "s/master/raster/" >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	echo "s/master/raster/" >"$HTTPD_ROOT_PATH/one-time-perl" &&
 	test_must_fail git -C local fetch 2>err &&
 
 	test_i18ngrep "fatal: remote error: unknown ref refs/heads/raster" err
Signature · 1 line
gitgitgadget
#3 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

In reply to Ed Maste

Hi Ed,

Quoted 14 lines · Ed Maste
Quoted 11 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Among other differences relative to GNU sed, BSD sed always ends its output with a trailing newline, even if the input did not have such a trailing newline.

Surprisingly, this makes three httpd-based tests fail on macOS: t5616, t5702 and t5703. ("Surprisingly" because those tests have been around for some time, but apparently nobody runs them on macOS with a working Apache2 setup.)

Quoted 6 lines

Hmm, this is interesting - all tests (that are executed) are passing on FreeBSD, in CI.

I tried on FreeBSD and do not see a trailing newline added; I'm not sure how sed behaves on other BSDs. However, you probably want to refer to macOS sed rather than BSD sed in the commit.

My bad. I looked at StackOverflow and there the claim was that all BSD seds behave that way.

Of course, SO always lags behind by a couple years (although even such old threads are often very useful), so it is possible that _old_ BSD sed behaved that way.

In any case, I adjusted the commit message.

Related, I saw that Cirrus CI offers FreeBSD builds, maybe you'd be interested in supporting that out of the box in
https://github.com/git/git?

Ciao,
Dscho

#9 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

In reply to Junio C Hamano

Hi Junio,

Quoted 16 lines · Junio C Hamano
Quoted 14 lines · Junio C Hamano

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 11 lines · writes:
t/t5537-fetch-shallow.sh +1 -1
index 9e16512fe31..4f10057e9f1 100755
@@ -237,7 +237,7 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	git -C "$REPO" config protocol.version 2 &&
 	git -C client config protocol.version 2 &&
 
-	git -C client fetch --depth=2 "$HTTPD_URL/one_time_sed/repo" master:a_branch &&
+	git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" master:a_branch &&
Quoted 7 lines
Quoted 7 lines

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 246

Quoted 4 lines
t/t5537-fetch-shallow.sh +0 -0
 	# Craft a situation in which the server sends back an unshallow request
 	# with an empty packfile. This is done by refetching with a shorter

@@ -246,13 +246,13 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f
 	printf "$(test_oid sed)" \

Hmm, shouldn't the test-oid token "sed" whose value is set up in the setup section of this test script also be renamed to "perl"?

Ooops...

Quoted 13 lines
Quoted 6 lines

Or, if we are actively taking advantage of the fact that the syntax of the replacement operator is the same between the languages, perhaps "sed" is better renamed to something more language agnostic and reflects the purpose/reason why we extend the packet header by two bytes with the one-time munging process?

Quoted 6 lines

t/t5537-fetch-shallow.sh test_expect_success 'shallow fetches check connectivity before writing shallow f · line 249

Quoted 4 lines
t/t5537-fetch-shallow.sh +1 -1
 	       "$(git -C "$REPO" rev-parse HEAD)" \
 	       "$(git -C "$REPO" rev-parse HEAD^)" \
-	       >"$HTTPD_ROOT_PATH/one-time-sed" &&
+	       >"$HTTPD_ROOT_PATH/one-time-perl" &&

Other than that, this step looked quite sensible. Thanks.

Quoted 4 lines

Hmm, is it because you wanted to backport this down to 'maint' (otherwise, your tests will start failing in a month) that you left the "test_oid sed" thing untouched? If so, that makes sort-of sense.

That's a good point. I target maint in v2, and offered an add-on patch meant to be applied on top of the merge into master (or next, or pu).

Quoted 3 lines

I expect that the series will be rerolled, if only for s/BSD/macOS/ mentioned elsewhere in the thread, but in the meantime, I'll rebase them on 'maint' "as a practice" while queuing.

Thanks ;-)

Ciao,
Dscho

#4 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

In reply to Johannes Schindelin
Quoted 7 lines · Johannes Schindelin

My bad. I looked at StackOverflow and there the claim was that all BSD seds behave that way.

Of course, SO always lags behind by a couple years (although even such old threads are often very useful), so it is possible that _old_ BSD sed behaved that way.

Yeah, I wondered about the different behaviour, and asked on Twitter about it. You're right, it is historical BSD behaviour and NetBSD at least still appends the newline. FreeBSD changed this in 2014 -
http://bugs.freebsd.org/160745.

Quoted 3 lines

Related, I saw that Cirrus CI offers FreeBSD builds, maybe you'd be interested in supporting that out of the box in
https://github.com/git/git?

Indeed - there is a .cirrus.yml in git now which builds and runs tests (on FreeBSD 12.1). I'll look into working with the GitHub organization owners for git and gitgitgadget to see about allowing Cirrus to access the repositories.

#5 lore raw

Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

Hi Ed,

Quoted 6 lines · Ed Maste
Quoted 4 lines · Johannes Schindelin

My bad. I looked at StackOverflow and there the claim was that all BSD seds behave that way.

Quoted 8 lines
Quoted 3 lines

Of course, SO always lags behind by a couple years (although even such old threads are often very useful), so it is possible that _old_ BSD sed behaved that way.

Yeah, I wondered about the different behaviour, and asked on Twitter about it. You're right, it is historical BSD behaviour and NetBSD at least still appends the newline. FreeBSD changed this in 2014 -
http://bugs.freebsd.org/160745.

Thank you for digging into this!

Quoted 8 lines
Quoted 3 lines

Related, I saw that Cirrus CI offers FreeBSD builds, maybe you'd be interested in supporting that out of the box in
https://github.com/git/git?

Indeed - there is a .cirrus.yml in git now which builds and runs tests (on FreeBSD 12.1). I'll look into working with the GitHub organization owners for git and gitgitgadget to see about allowing Cirrus to access the repositories.

Oy, I had forgotten that you worked on this. I enabled this in the git and in the gitgitgadget orgs. The next pushes/PRs should benefit from this.

Thanks,
Dscho

#6 lore raw

CI/PR builds on FreeBSD, was Re: [PATCH 1/3] t/lib-httpd: avoid using BSD's sed

Hi Ed,

Quoted 9 lines · Johannes Schindelin
Quoted 7 lines · Ed Maste
Quoted 3 lines · Johannes Schindelin

Related, I saw that Cirrus CI offers FreeBSD builds, maybe you'd be interested in supporting that out of the box in
https://github.com/git/git?

Quoted 8 lines
Quoted 4 lines

Indeed - there is a .cirrus.yml in git now which builds and runs tests (on FreeBSD 12.1). I'll look into working with the GitHub organization owners for git and gitgitgadget to see about allowing Cirrus to access the repositories.

Oy, I had forgotten that you worked on this. I enabled this in the git and in the gitgitgadget orgs. The next pushes/PRs should benefit from this.

This works now. When you click on the green checkmarks (and the one red X) at https://github.com/git/git/branches/active, you will see the freebsd_12 build.

Ciao,
Dscho