ci: some speedups for slow tests

Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

- Installing the Git for Windows SDK speeds up from 3 minutes to 10 seconds by using tar(1) instead of Extract-Archive.

- t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding the overhead of spawning redundant processes.

- t5551 and t5559 both speed up by a factor of 10 on macOS, going from 20 minutes each to 2 minutes each by using reftables in a repo with 100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18 minutes end-to-end.

- "fedora-breaking-changes-meson" speeds up by a factor of 2, going from 35 minutes to 16 minutes. Turns out our SHA256 implementation is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
gitlab-ci: speed up installation of Git for Windows SDK
gitlab-ci: exclude intermediate outputs from Windows artifacts
ci: build optimized binaries with Meson
t0027: optimize how we reset repository state
t0027: cache expected output files
t1092: set up repositories from templates
t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           | 13 ++++++++++--
 ci/install-sdk.ps1                       | 26 ++++++++++++++++++------
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 33 +++++++++++++++++--------------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 74 insertions(+), 39 deletions(-)

---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via Expand-Archive, which is expensive.

Improve this by caching the SDK and by using tar(1) instead of Expand-Archive. Note that caching actually only gives us a win of a couple seconds. The bigger win here is using tar(1), which speeds up extracting the archive from 3 minutes to about 10 seconds. Regardless of that though, caching the SDK feels like a sensible thing to do as it removes one step that may spuriously fail.

---
 .gitlab-ci.yml     |  9 ++++++++-
 ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
 2 files changed, 28 insertions(+), 7 deletions(-)

index 9b7698a0037..15419a954b3 100644
@@ -127,9 +127,16 @@ build:mingw64:
     - saas-windows-medium-amd64
   variables:
     NO_PERL: 1
+  cache:
+    key:
+      files:
+        - ci/install-sdk.ps1
+        - ci/install-dependencies.ps1
+    paths:
+      - .dependencies
   before_script:
     - *windows_before_script
-    - ./ci/install-sdk.ps1 -directory "git-sdk"
+    - ./ci/install-sdk.ps1 -Directory "git-sdk"
     - ./ci/install-dependencies.ps1 -Mingw
   script:
     # The minimal SDK's profile resets PATH.

index 66f24838a40..1dbcef04a18 100755
@@ -1,12 +1,26 @@
 param(
-    [string]$directory='git-sdk',
-    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
+    [string]$Directory='git-sdk',
+    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
+    [string]$DownloadDirectory='.dependencies'
 )
 
-Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
-Remove-Item -Path git-sdk.zip
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+New-Item -Path $DownloadDirectory -ItemType Directory -Force
+
+$archive = Join-Path $DownloadDirectory 'git-sdk.zip'
+if (-not (Test-Path $archive)) {
+    Invoke-WebRequest "$URL" -OutFile $archive -TimeoutSec 300
+}
+
+New-Item -Path $Directory -ItemType Directory -Force
+tar -xf $archive -C $Directory
+if ($LASTEXITCODE -ne 0) {
+    throw "Failed to extract $archive with exit code $LASTEXITCODE"
+}
 
 New-Item -Path .git/info -ItemType Directory -Force
 New-Item -Path .git/info/exclude -ItemType File -Force
-Add-Content -Path .git/info/exclude -Value "/$directory"
+Add-Content -Path .git/info/exclude -Value "/$Directory"
+Add-Content -Path .git/info/exclude -Value "/$DownloadDirectory"

2.56.0.500.g5d06dc02cd.dirty

Wow, is a tool to move the bits from one part of a rusted iron surface to another part really so much more expensive compared to another tool to move the same bits, to make a difference that is worth doing this?

One thing that may make readers curious is if we need to worry about this cached copy going stale.

I was surprised by this, as well.

Yes, it may go stale, but I'm not even sure whether that really matters. I decided to just do this as-is and Cc Johannes to maybe facilitate his input on this part. But arguably, we can just skip the cache altogether as it didn't really buy much anyway.

Well, with the new DownloadDirectory parameter everything else would have been inconsistent, so I made those changes while at it. I'll dropt he cache in v2 though, and with that we can drop these changes, too.

Patrick

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

index 15419a954b3..036d2f3c7f2 100644
@@ -194,6 +194,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

2.56.0.500.g5d06dc02cd.dirty

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:

- The job uses Meson to compile Git, and Meson by default uses the "debug" build type that translates into "-O0". Our Makefile in contrast to that uses "-g -O2" by default.

- The job enables breaking changes and thus runs with SHA256 by default with the "block" backend, which benefits heavily from compiler optimizations. For example, hashing a 500MB blob with SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5 seconds with "-O2". The SHA1 code path only slows down from 0.9 seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

index 036d2f3c7f2..df82cd3a5df 100644
@@ -189,7 +189,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:

index 68b91e31439..fd8136af342 100755
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

2.56.0.500.g5d06dc02cd.dirty

That is an impressive improvement.

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is checkout_files (), which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

---
 t/t0027-auto-crlf.sh | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

index 49dbf09da77..eefb346d11f 100755
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
-	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
-	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
-	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- \
+			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

2.56.0.500.g5d06dc02cd.dirty

"Wow, so it can go" - a well working, but slow code is speed up so much.

I used the following (before reading the suggestion from Junio)
and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.

    test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
        files="\
           crlf_false_attr__LF.txt \
           crlf_false_attr__CRLF.txt \
           crlf_false_attr__LF_mix_CR.txt \
           crlf_false_attr__CRLF_mix_LF.txt \
           crlf_false_attr__LF_nul.txt" &&
        rm -fv $files &&
        git ${ceol:+-c core.eol=$ceol} checkout -- $files
    '

Hmm. I think I like Junio's alternative a bit better, but both are definitely an improvement over my take. Will add to the next version, thanks.

Patrick

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is compare_files () and compare_ws_file (), which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

index eefb346d11f..25ed6b33704 100755
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

2.56.0.500.g5d06dc02cd.dirty

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via init_repos (), which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

index 05b54062b3b..5d81ec54c23 100755
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

2.56.0.500.g5d06dc02cd.dirty

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

index 130053c57aa..34945059531 100755
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000

@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

2.56.0.500.g5d06dc02cd.dirty

Thanks for these patches. I didn't leave comments on later parts of the series, but all of them were interesting read and fully made sense to me.

Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

- Installing the Git for Windows SDK speeds up from 3 minutes to 10 seconds by using tar(1) instead of Extract-Archive.

- t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding the overhead of spawning redundant processes.

- t5551 and t5559 both speed up by a factor of 10 on macOS, going from 20 minutes each to 2 minutes each by using reftables in a repo with 100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18 minutes end-to-end.

- "fedora-breaking-changes-meson" speeds up by a factor of 2, going from 35 minutes to 16 minutes. Turns out our SHA256 implementation is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].

Changes in v2:
- Drop the cache for the GfW SDK again.
- Drop repetitive paths in t0027.
- Link to v1: https://patch.msgid.link/[email protected]

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
gitlab-ci: speed up installation of Git for Windows SDK
gitlab-ci: exclude intermediate outputs from Windows artifacts
ci: build optimized binaries with Meson
t0027: optimize how we reset repository state
t0027: cache expected output files
t1092: set up repositories from templates
t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 26 +++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 47 insertions(+), 30 deletions(-)

Range-diff versus v1:

1:  e25adb1b8c8 < -:  ----------- gitlab-ci: speed up installation of Git for Windows SDK
-:  ----------- > 1:  582a9ab6468 gitlab-ci: speed up installation of Git for Windows SDK
2:  116c8b8a716 = 2:  bff0a624127 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  a035b8a902c = 3:  713e69bfb59 ci: build optimized binaries with Meson
4:  1a1238f4be7 < -:  ----------- t0027: optimize how we reset repository state
-:  ----------- > 4:  e421eb2a275 t0027: optimize how we reset repository state
5:  c652f5b380a = 5:  16905b5be62 t0027: cache expected output files
6:  9a90272c620 = 6:  43f355fbb02 t1092: set up repositories from templates
7:  187128e7d5d = 7:  960716da178 t5551: use reftables for repositories with many refs

---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via Expand-Archive, which is expensive.

Improve this by using tar(1) instead of Expand-Archive. While at it, make the script error on failures and suppress progress output.

---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

index 66f24838a40..bb332c1c10f 100755
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force

2.56.0.611.g1030f0b940.dirty

Wow. You learn new things every day.

This is in ci/install-sdk.ps1 file. Right now that script is referred to only from .gitlab-ci.yml so as long as GitLab CI environment has tar that works this way we would be fine, but I thought that ci/* was designed to be portable across CI and other environments. At least we would want a comment next to it that says we assume BSD derived tar that can grok zip archives here, probably?

Yeah, it's fun. Windows uses bsdtar, which is in turn based on libarchive, and hence it also supports ZIP and 7zip files [1].

Well, bsdtar is nowadays distributed with Windows by default, so this should be portable to other CI environments, too. So I don't think it really warrants a comment as it should just work everwhere. That's certainly something to point out in the commit message though.

In fact I wonder whether we should maybe adapt GitHub Workflows to use that script, too. That was my intent back when introducing this script, and for a brief time we in fact did that. But we eventually figured that it was too slow in contrast to the git-for-windows/setup-git-for-windows-sdk step, so we reverted that change on GitHub again in 55d62306eea (GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again, 2024-12-17).

But now that we have found a way to remove the slowness we could unify both systems again. I intentionally left that out of this patch series though, as we can still do that as a subsequent step if we want to. In any case, I want Johannes' opinion on this topic before doing such a change.

Patrick

[1]: https://learn.microsoft.com/en-us/windows/tar/

Yeah, this installer is about Windows, so any current or future CI that uses this script would end up running BSD tar, which means we can assume readers know what the script is doing and why it works.

Yes, indeed. Thanks for these patches.

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

index 9b7698a0037..3631393a8cc 100644
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

2.56.0.611.g1030f0b940.dirty

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:

- The job uses Meson to compile Git, and Meson by default uses the "debug" build type that translates into "-O0". Our Makefile in contrast to that uses "-g -O2" by default.

- The job enables breaking changes and thus runs with SHA256 by default with the "block" backend, which benefits heavily from compiler optimizations. For example, hashing a 500MB blob with SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5 seconds with "-O2". The SHA1 code path only slows down from 0.9 seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

index 3631393a8cc..449fed1b392 100644
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:

index 68b91e31439..fd8136af342 100755
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

2.56.0.611.g1030f0b940.dirty

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is checkout_files (), which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

---
 t/t0027-auto-crlf.sh | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

index 49dbf09da77..504a7ea54bb 100755
@@ -305,11 +305,12 @@ checkout_files () {
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

2.56.0.611.g1030f0b940.dirty

Yeah, makes sense.

Patrick

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is compare_files () and compare_ws_file (), which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

index 504a7ea54bb..b021f0855a8 100755
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

2.56.0.611.g1030f0b940.dirty

Thanks! I benchmarked this and it doesn't seem to make much of a difference. So I'd propose to stick with the simple change for now, if that's alright with you.

Patrick

Good to hear that the benchmark was done.
OK to stick your patch "as is"

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via init_repos (), which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

index 05b54062b3b..5d81ec54c23 100755
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

2.56.0.611.g1030f0b940.dirty

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

index 130053c57aa..34945059531 100755
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000

@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

2.56.0.611.g1030f0b940.dirty

Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

- Installing the Git for Windows SDK speeds up from 3 minutes to 10 seconds by using tar(1) instead of Extract-Archive.

- t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding the overhead of spawning redundant processes.

- t5551 and t5559 both speed up by a factor of 10 on macOS, going from 20 minutes each to 2 minutes each by using reftables in a repo with 100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18 minutes end-to-end.

- "fedora-breaking-changes-meson" speeds up by a factor of 2, going from 35 minutes to 16 minutes. Turns out our SHA256 implementation is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].

Changes in v3:
- Document why Windows' tar version can extract ZIP archives.
- Clear paths variable.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Drop the cache for the GfW SDK again.
- Drop repetitive paths in t0027.
- Link to v1: https://patch.msgid.link/[email protected]

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
gitlab-ci: speed up installation of Git for Windows SDK
gitlab-ci: exclude intermediate outputs from Windows artifacts
ci: build optimized binaries with Meson
t0027: optimize how we reset repository state
t0027: cache expected output files
t1092: set up repositories from templates
t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 27 ++++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 48 insertions(+), 30 deletions(-)

Range-diff versus v2:

1: 41c3a0e8d7d ! 1: 8fabb336df3 gitlab-ci: speed up installation of Git for Windows SDK
@@ Commit message
Windows SDK on every run and then extracts it via Expand-Archive, which is expensive.

- Improve this by using tar(1) instead of Expand-Archive. While at it,
- make the script error on failures and suppress progress output.
+ Improve this by using tar(1) instead of Expand-Archive. Note that
+ we're using it to extract a ZIP archive, which may seem puzzling at
+ first. But Windows ships bsdtar by default [1], which also supports ZIP
+ and 7zip files.
+
+ While at it, make the script error on failures and suppress progress
+ output.
+
+ [1]: https://learn.microsoft.com/en-us/windows/tar/

2:  439c12cb15d = 2:  4a4d1e6f2a8 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  070bef6fecf = 3:  c23b7a7c58d ci: build optimized binaries with Meson
4:  598a143fc24 ! 4:  942d70be750 t0027: optimize how we reset repository state
    @@ Commit message
     
      ## t/t0027-auto-crlf.sh ##
     @@ t/t0027-auto-crlf.sh: checkout_files () {
    + 		git config core.autocrlf $crlf
    + 	'
      	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
    ++	paths=
      	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
      	do
     -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
5:  a325efbaddb = 5:  460cd0c88fd t0027: cache expected output files
6:  a2d33441e50 = 6:  44c77717749 t1092: set up repositories from templates
7:  bfc93fdfb4e = 7:  d64a61efec2 t5551: use reftables for repositories with many refs

---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via Expand-Archive, which is expensive.

Improve this by using tar(1) instead of Expand-Archive. Note that we're using it to extract a ZIP archive, which may seem puzzling at first. But Windows ships bsdtar by default [1], which also supports ZIP and 7zip files.

While at it, make the script error on failures and suppress progress output.

[1]: https://learn.microsoft.com/en-us/windows/tar/

---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

index 66f24838a40..bb332c1c10f 100755
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force

2.56.0.611.g1030f0b940.dirty

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

index 9b7698a0037..3631393a8cc 100644
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

2.56.0.611.g1030f0b940.dirty

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:

- The job uses Meson to compile Git, and Meson by default uses the "debug" build type that translates into "-O0". Our Makefile in contrast to that uses "-g -O2" by default.

- The job enables breaking changes and thus runs with SHA256 by default with the "block" backend, which benefits heavily from compiler optimizations. For example, hashing a 500MB blob with SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5 seconds with "-O2". The SHA1 code path only slows down from 0.9 seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

index 3631393a8cc..449fed1b392 100644
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:

index 68b91e31439..fd8136af342 100755
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

2.56.0.611.g1030f0b940.dirty

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is checkout_files (), which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

---
 t/t0027-auto-crlf.sh | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

index 49dbf09da77..a3904b729b3 100755
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
+	paths=
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

2.56.0.611.g1030f0b940.dirty

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is compare_files () and compare_ws_file (), which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

index a3904b729b3..38ce06cbc1e 100755
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

2.56.0.611.g1030f0b940.dirty

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via init_repos (), which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

index 05b54062b3b..5d81ec54c23 100755
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

2.56.0.611.g1030f0b940.dirty

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

index 130053c57aa..34945059531 100755
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000

@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

2.56.0.611.g1030f0b940.dirty