ci: avoid pounding on the poor ci-artifacts container

PATCH Superseded Johannes Schindelin via GitGitGadget 4 participants 9 messages May 12, 2020 – May 15, 2020

lore

Share & tools

lore

Johannes Schindelin via GitGitGadget

author

2020-05-12 20:47 UTC #0 lore raw

PATCH 1/1 ci: avoid pounding on the poor ci-artifacts container

From: Johannes Schindelin <[email protected]>

When this developer tested how the git-sdk-64-minimal artifact could be served to all the GitHub workflow runs that need it, Azure Blobs looked like a pretty good choice: it is reliable, fast and we already use it in Git for Windows to serve components like OpenSSL, cURL, etc

It came as an unpleasant surprise just _how many_ times this artifact was downloaded. It exploded the bandwidth to a point where the free tier would no longer be enough, threatening to block other, essential Git for Windows services.

Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.

To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action actions/upload-artifact in the windows-build job and the GitHub Action actions/download-artifact in the windows-test and vs-test jobs (the latter now depends on windows-build for that reason, too).

Signed-off-by Johannes Schindelin <[email protected]>

---
ci: avoid pounding on the poor ci-artifacts container

I did not think through how I designed the Windows job initializing the minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc):
it currently accesses Azure Blobs and uses way too much bandwidth, blowing right through my quota.

So for now, all the Windows/Visual Studio builds will be failing to download that SDK subset.

With this patch, we use the (slower) method of downloading the git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which fixes that particular problem.

To be sure, it is a shame that we now spend around a whole minute downloading that SDK subset instead of those sweet, sweet seven seconds. But what does not work does not work, and that's that.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/632

 .github/workflows/main.yml | 65 +++++++++++++++++++++++++++++++-------
 1 file changed, 54 insertions(+), 11 deletions(-)

.github/workflows/main.yml +54 ~~-11~~

index fd4df939b50..e2e1611aa22 100644
@@ -12,7 +12,24 @@ jobs:
     - uses: actions/checkout@v1
     - name: download git-sdk-64-minimal
       shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Add `json_pp` to the search path
+        PATH=$PATH:/usr/bin/core_perl
+
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          json_pp |
+          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          json_pp |
+          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+          -o artifacts.zip "$download_url"
+
+        ## Unzip and remove the artifact
+        unzip artifacts.zip
+        rm artifacts.zip
     - name: build
       shell: powershell
       env:

@@ -30,6 +47,11 @@ jobs:
       with:
         name: windows-artifacts
         path: artifacts
+    - name: upload git-sdk-64-minimal
+      uses: actions/upload-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: git-sdk-64-minimal
   windows-test:
     runs-on: windows-latest
     needs: [windows-build]

@@ -38,9 +60,6 @@ jobs:
         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
     steps:
     - uses: actions/checkout@v1
-    - name: download git-sdk-64-minimal
-      shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
     - name: download build artifacts
       uses: actions/download-artifact@v1
       with:

@@ -49,6 +68,11 @@ jobs:
     - name: extract build artifacts
       shell: bash
       run: tar xf artifacts.tar.gz
+    - name: download git-sdk-64-minimal
+      uses: actions/download-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: ${{github.workspace}}/git-sdk-64-minimal/
     - name: test
       shell: powershell
       run: |

@@ -79,7 +103,24 @@ jobs:
     - uses: actions/checkout@v1
     - name: download git-sdk-64-minimal
       shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Add `json_pp` to the search path
+        PATH=$PATH:/usr/bin/core_perl
+
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          json_pp |
+          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          json_pp |
+          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+          -o artifacts.zip "$download_url"
+
+        ## Unzip and remove the artifact
+        unzip artifacts.zip
+        rm artifacts.zip
     - name: generate Visual Studio solution
       shell: powershell
       run: |

@@ -119,15 +160,17 @@ jobs:
         path: artifacts
   vs-test:
     runs-on: windows-latest
-    needs: [vs-build]
+    needs: [vs-build, windows-build]
     strategy:
       matrix:
         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
     steps:
     - uses: actions/checkout@v1
-    - name: download git-64-portable
-      shell: bash
-      run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+    - name: download git-sdk-64-minimal
+      uses: actions/download-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: ${{github.workspace}}/git-sdk-64-minimal/
     - name: download build artifacts
       uses: actions/download-artifact@v1
       with:

@@ -143,9 +186,9 @@ jobs:
         NO_SVN_TESTS: 1
         GIT_TEST_SKIP_REBASE_P: 1
       run: |
-        & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @"
+        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
           # Let Git ignore the SDK and the test-cache
-          printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude
+          printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude
 
           cd t &&
           PATH=\"`$PWD/helper:`$PATH\" &&

base-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c

Signature · 1 line

gitgitgadget

Junio C Hamano

2020-05-12 23:39 UTC #1 lore raw

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 8 lines · writes:

Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.
To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action actions/upload-artifact in the windows-build job and the GitHub Action actions/download-artifact in the windows-test and vs-test jobs (the latter now depends on windows-build for that reason, too).

I guess this answers a question I sent earlier to the list (our mails almost crossed, I guess, as two of us were looking at the same problem at around the same time?).

Hopefully when cmake-for-windows-build topic lands, this can go away altogether, but that is probably at least 8 weeks away (3 weeks remaining before the next cycle opens, plus a half of 10 week per cycle for a typical major release).

Today's final integration (these days I'm pushing out twice or three times a day) contains this one, and it seems to have passed ;-)

Thanks.

Johannes Schindelin

2020-05-15 13:59 UTC #2 lore raw

Hi Junio,

Quoted 12 lines · Junio C Hamano

"Johannes Schindelin via GitGitGadget" <[email protected]>
Quoted 8 lines · writes:
Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.
To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action actions/upload-artifact in the windows-build job and the GitHub Action actions/download-artifact in the windows-test and vs-test jobs (the latter now depends on windows-build for that reason, too).

Quoted 3 lines

I guess this answers a question I sent earlier to the list (our mails almost crossed, I guess, as two of us were looking at the same problem at around the same time?).

I am terribly sorry, but I did not get to read the Git mailing list at all this week (or for that matter, my private mail). So I would not even have seen your message... :-(

Quoted 4 lines

Hopefully when cmake-for-windows-build topic lands, this can go away altogether, but that is probably at least 8 weeks away (3 weeks remaining before the next cycle opens, plus a half of 10 week per cycle for a typical major release).

The cmake-for-windows-build would address only the build part for Visual Studio. The regular Windows build, as well as the parallelized tests _still_ need the git-sdk-64-minimal artifact. With or without CMake. That's because neither CMake nor Visual Studio can accommodate the fact that our test suite is implemented in shell script _and_ requires a working Perl interpreter.

Quoted 2 lines

Today's final integration (these days I'm pushing out twice or three times a day) contains this one, and it seems to have passed ;-)

Excellent!

Thanks,
Dscho

Đoàn Trần Công Danh

2020-05-13 12:44 UTC #3 lore raw

In reply to Johannes Schindelin via GitGitGadget

Review

Quoted 13 lines · Johannes Schindelin via GitGitGadget

(quoted diff) +12 -1

-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Add `json_pp` to the search path
+        PATH=$PATH:/usr/bin/core_perl
+
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          json_pp |
+          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          json_pp |
+          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"

Hi Dscho,

I wonder if it's acceptable to introduce jq (already installed in GitHub Actions and Travis) into our codebase (only in GitHub Actions).

If yes, I think this will be easier to follow than depending on static number of space and sed branching. ---------------------8<-----------------

.github/workflows/main.yml +4 ~~-14~~

index e2e1611aa2..482df46651 100644
@@ -13,17 +13,12 @@ jobs:
     - name: download git-sdk-64-minimal
       shell: bash
       run: |
-        ## Add `json_pp` to the search path
-        PATH=$PATH:/usr/bin/core_perl
-
         ## Get artifact
         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
         id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          json_pp |
-          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+          jq -r ".value[] | .id")
         download_url="$(curl "$urlbase/$id/artifacts" |
-          json_pp |
-          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
           -o artifacts.zip "$download_url"

@@ -104,17 +99,12 @@ jobs:
     - name: download git-sdk-64-minimal
       shell: bash
       run: |
-        ## Add `json_pp` to the search path
-        PATH=$PATH:/usr/bin/core_perl
-
         ## Get artifact
         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
         id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          json_pp |
-          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+          jq -r ".value[] | .id")
         download_url="$(curl "$urlbase/$id/artifacts" |
-          json_pp |
-          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
           -o artifacts.zip "$download_url"

----------->8------------

Signature · 1 line

Danh

Johannes Schindelin

2020-05-15 14:01 UTC #4 lore raw

Hi Danh,

Quoted 58 lines · Đoàn Trần Công Danh

Review

Quoted 13 lines · Johannes Schindelin via GitGitGadget

(quoted diff) +12 -1

-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Add `json_pp` to the search path
+        PATH=$PATH:/usr/bin/core_perl
+
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          json_pp |
+          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          json_pp |
+          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"

Hi Dscho,

I wonder if it's acceptable to introduce jq (already installed in GitHub Actions and Travis) into our codebase (only in GitHub Actions).

If yes, I think this will be easier to follow than depending on static number of space and sed branching. ---------------------8<-----------------

.github/workflows/main.yml +2 ~~-11~~

index e2e1611aa2..482df46651 100644
@@ -13,17 +13,12 @@ jobs:
     - name: download git-sdk-64-minimal
       shell: bash
       run: |
-        ## Add `json_pp` to the search path
-        PATH=$PATH:/usr/bin/core_perl
-
         ## Get artifact
         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
         id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          json_pp |
-          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+          jq -r ".value[] | .id")
         download_url="$(curl "$urlbase/$id/artifacts" |
-          json_pp |
-          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
           -o artifacts.zip "$download_url"

@@ -104,17 +99,12 @@ jobs:
     - name: download git-sdk-64-minimal
       shell: bash
       run: |
-        ## Add `json_pp` to the search path
-        PATH=$PATH:/usr/bin/core_perl
-
         ## Get artifact
         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
         id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          json_pp |

.github/workflows/main.yml jobs:

Quoted 10 lines

.github/workflows/main.yml +2 -4

-          sed -n 's/^         "id" : \([1-9][0-9]*\).*/\1/p')
+          jq -r ".value[] | .id")
         download_url="$(curl "$urlbase/$id/artifacts" |
-          json_pp |
-          sed -n '/^      {/{:1;N;/\n      }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
           -o artifacts.zip "$download_url"
 
----------->8------------

Thank you for that. Indeed, I should have checked whether jq is available on the Windows build agents (and then I should have learned how to hold that tool right).

I sent out a v2 with this change, and I already merged that version into Git for Windows' master branch in preparation for v2.27.0-rc0.

Thanks,
Dscho

Johannes Schindelin via GitGitGadget

author

2020-05-15 07:55 UTC #5 lore raw

PATCH v2 1/1 ci: avoid pounding on the poor ci-artifacts container

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.

Helped-by Đoàn Trần Công Danh <[email protected]>

Signed-off-by Johannes Schindelin <[email protected]>

---
ci: avoid pounding on the poor ci-artifacts container

So for now, all the Windows/Visual Studio builds will be failing to download that SDK subset.

With this patch, we use the (slower) method of downloading the git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which fixes that particular problem.

To be sure, it is a shame that we now spend around a whole minute downloading that SDK subset instead of those sweet, sweet seven seconds. But what does not work does not work, and that's that.

Change since v1:

* Simplifying the scripted code by using jq (thanks, Danh!)

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/632

Range-diff vs v1:

1: 274ae816be6 ! 1: 14606cc2ed7 ci: avoid pounding on the poor ci-artifacts container
@@ Commit message
and vs-test jobs (the latter now depends on windows-build for that reason, too).

+ Helped-by: Đoàn Trần Công Danh <[email protected]>

Signed-off-by Johannes Schindelin <[email protected]>

## .github/workflows/main.yml ##
@@ .github/workflows/main.yml: jobs:
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
-+ ## Add json_pp to the search path
-+ PATH=$PATH:/usr/bin/core_perl
-+
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-+ json_pp |
-+ sed -n 's/^ "id" : $[1-9][0-9]*$.*/\1/p')
++ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
-+ json_pp |
-+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "$[^\"]*$.*/\1/p}}')"
++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+
@@ .github/workflows/main.yml: jobs:
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
-+ ## Add json_pp to the search path
-+ PATH=$PATH:/usr/bin/core_perl
-+
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-+ json_pp |
-+ sed -n 's/^ "id" : $[1-9][0-9]*$.*/\1/p')
++ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
-+ json_pp |
-+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "$[^\"]*$.*/\1/p}}')"
++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+

 .github/workflows/main.yml | 55 ++++++++++++++++++++++++++++++--------
 1 file changed, 44 insertions(+), 11 deletions(-)

.github/workflows/main.yml +44 ~~-11~~

index fd4df939b50..482df466510 100644
@@ -12,7 +12,19 @@ jobs:
     - uses: actions/checkout@v1
     - name: download git-sdk-64-minimal
       shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          jq -r ".value[] | .id")
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+          -o artifacts.zip "$download_url"
+
+        ## Unzip and remove the artifact
+        unzip artifacts.zip
+        rm artifacts.zip
     - name: build
       shell: powershell
       env:

@@ -30,6 +42,11 @@ jobs:
       with:
         name: windows-artifacts
         path: artifacts
+    - name: upload git-sdk-64-minimal
+      uses: actions/upload-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: git-sdk-64-minimal
   windows-test:
     runs-on: windows-latest
     needs: [windows-build]

@@ -38,9 +55,6 @@ jobs:
         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
     steps:
     - uses: actions/checkout@v1
-    - name: download git-sdk-64-minimal
-      shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
     - name: download build artifacts
       uses: actions/download-artifact@v1
       with:

@@ -49,6 +63,11 @@ jobs:
     - name: extract build artifacts
       shell: bash
       run: tar xf artifacts.tar.gz
+    - name: download git-sdk-64-minimal
+      uses: actions/download-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: ${{github.workspace}}/git-sdk-64-minimal/
     - name: test
       shell: powershell
       run: |

@@ -79,7 +98,19 @@ jobs:
     - uses: actions/checkout@v1
     - name: download git-sdk-64-minimal
       shell: bash
-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+      run: |
+        ## Get artifact
+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+          jq -r ".value[] | .id")
+        download_url="$(curl "$urlbase/$id/artifacts" |
+          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+          -o artifacts.zip "$download_url"
+
+        ## Unzip and remove the artifact
+        unzip artifacts.zip
+        rm artifacts.zip
     - name: generate Visual Studio solution
       shell: powershell
       run: |

@@ -119,15 +150,17 @@ jobs:
         path: artifacts
   vs-test:
     runs-on: windows-latest
-    needs: [vs-build]
+    needs: [vs-build, windows-build]
     strategy:
       matrix:
         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
     steps:
     - uses: actions/checkout@v1
-    - name: download git-64-portable
-      shell: bash
-      run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+    - name: download git-sdk-64-minimal
+      uses: actions/download-artifact@v1
+      with:
+        name: git-sdk-64-minimal
+        path: ${{github.workspace}}/git-sdk-64-minimal/
     - name: download build artifacts
       uses: actions/download-artifact@v1
       with:

@@ -143,9 +176,9 @@ jobs:
         NO_SVN_TESTS: 1
         GIT_TEST_SKIP_REBASE_P: 1
       run: |
-        & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @"
+        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
           # Let Git ignore the SDK and the test-cache
-          printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude
+          printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude
 
           cd t &&
           PATH=\"`$PWD/helper:`$PATH\" &&

base-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c

Signature · 1 line

gitgitgadget

Junio C Hamano

2020-05-15 15:11 UTC #6 lore raw

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 21 lines · writes:

From: Johannes Schindelin <[email protected]>
When this developer tested how the git-sdk-64-minimal artifact could be served to all the GitHub workflow runs that need it, Azure Blobs looked like a pretty good choice: it is reliable, fast and we already use it in Git for Windows to serve components like OpenSSL, cURL, etc
It came as an unpleasant surprise just _how many_ times this artifact was downloaded. It exploded the bandwidth to a point where the free tier would no longer be enough, threatening to block other, essential Git for Windows services.
Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.
To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action actions/upload-artifact in the windows-build job and the GitHub Action actions/download-artifact in the windows-test and vs-test jobs (the latter now depends on windows-build for that reason, too).

Quoted 7 lines

Helped-by Đoàn Trần Công Danh <[email protected]>
Signed-off-by Johannes Schindelin <[email protected]>
---
Change since v1:
* Simplifying the scripted code by using jq (thanks, Danh!)

Thanks, both. Will replace and let's merge it down before -rc1.

Johannes Schindelin

2020-05-15 21:35 UTC #7 lore raw

Hi Junio,

Quoted 28 lines · Junio C Hamano

"Johannes Schindelin via GitGitGadget" <[email protected]>
Quoted 25 lines · writes:
From: Johannes Schindelin <[email protected]>
When this developer tested how the git-sdk-64-minimal artifact could be served to all the GitHub workflow runs that need it, Azure Blobs looked like a pretty good choice: it is reliable, fast and we already use it in Git for Windows to serve components like OpenSSL, cURL, etc
It came as an unpleasant surprise just _how many_ times this artifact was downloaded. It exploded the bandwidth to a point where the free tier would no longer be enough, threatening to block other, essential Git for Windows services.
Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being.
To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action actions/upload-artifact in the windows-build job and the GitHub Action actions/download-artifact in the windows-test and vs-test jobs (the latter now depends on windows-build for that reason, too).
Helped-by Đoàn Trần Công Danh <[email protected]>
Signed-off-by Johannes Schindelin <[email protected]>
---

Quoted 5 lines

Quoted 3 lines
Change since v1:
* Simplifying the scripted code by using jq (thanks, Danh!)
Thanks, both. Will replace and let's merge it down before -rc1.

Thank you!
Dscho

Junio C Hamano

2020-05-15 21:38 UTC #8 lore raw

Quoted 5 lines · Johannes Schindelin

Hi Junio,
Quoted 1 line · Junio C Hamano
Thanks, both. Will replace and let's merge it down before -rc1.

'next' passes with this

https://github.com/git/git/runs/679473788
https://github.com/git/git/runs/679473866

etc.

Thanks.