ci: only run win+VS build & tests in Git for Windows' fork

From: Johannes Schindelin <[email protected]>

It has been a frequent matter of contention that the win+VS jobs not only take a long time to run, but are also more easily broken than the other jobs (because they do not use the same Makefile-based builds as all other jobs), and to make matters worse, these breakages are also much harder to diagnose and fix than other jobs', especially for contributors who are happy to stay away from Windows.

The purpose of these win+VS jobs is to maintain the CMake-based build of Git, with the target audience being Visual Studio users on Windows who are typically quite unfamiliar with make and POSIX shell scripting, but the benefit of whose expertise we want for the Git project nevertheless.

The CMake support was introduced for that specific purpose, and already early on concerns were raised that it would put an undue burden on contributors to ensure that these jobs pass in CI, when they do not have access to Windows machines (nor want to have that).

This developer's initial hope was that it would be enough to fix win+VS failures and provide the changes to be squashed into contributors' patches, and that it would be worth the benefit of attracting Windows-based developers' contributions.

Neither of these hopes have panned out.

To lower the frustration, and incidentally benefit from using way less build minutes, let's just not run the win+VS jobs by default, which appears to be the consensus of the mail thread leading up to
https://lore.kernel.org/git/[email protected]/

Since the Git for Windows project still needs to at least try to attract more of said Windows-based developers, let's keep the jobs, but disable them everywhere except in Git for Windows' fork. This will help because Git for Windows' branch thicket is "continuously rebased" via automation to the shears/maint, shears/main, shears/next and shears/seen branches at https://github.com/git-for-windows/git. That way, the Git for Windows project will still be notified early on about potential breakages, but the Git project won't be burdened with fixing them anymore, which seems to be the best compromise we can get on this issue.

---
Explicitly make CMake-based CI failures the responsibility of "Windows folks"

Ævar and I have brain-stormed off-list what would be the best way to resolve the mounting frustration with CI failures that are caused by needing to mirror Makefile changes into the CMake-based build, a burden that the Git project never wanted to bear.

While he still wants to improve the CMake support (which will benefit Git for Windows), the main driver of trying to extend CMake support to Linux (which does not need it because make works very well there, indeed) was said frustration with CI failures.

A much quicker method to reduce that friction to close to nil is to simply disable the win+VS jobs, which is what this proposal is about. (Almost, at least, we still want to keep those job definitions and run them in Git for Windows' fork to ensure that CMake-based builds still work.)

A very welcome side effect is to reduce the CI build time again, which became alarmingly long as of recent, causing friction on its own.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1445%2Fdscho%2Fonly-run-win%2BVS-in-the-git-for-windows-fork-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1445/dscho/only-run-win+VS-in-the-git-for-windows-fork-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1445

 .github/workflows/main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index e67847a682c..8af3c67f605 100644
@@ -132,7 +132,7 @@ jobs:
   vs-build:
     name: win+VS build
     needs: ci-config
-    if: needs.ci-config.outputs.enabled == 'yes'
+    if: github.event.repository.owner.login == 'git-for-windows' && needs.ci-config.outputs.enabled == 'yes'
     env:
       NO_PERL: 1
       GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"

base-commit: c48035d29b4e524aed3a32f0403676f0d9128863

gitgitgadget

Hi Dscho

I'm sad that those hopes have not panned out, but I think this patch is probably the best way forward. It is certainly the lowest effort for non-windows developers and the improvement it CI minutes is very welcome.

Best Wishes

Phillip

I thought the initial purpose of it was to test compiling & testing with MSVC rather than GCC?

It's only after 4c2c38e800f (ci: modification of main.yml to use cmake for vs-build job, 2020-06-26) that it got co-opted to test cmake too.

Whatever we do about the job at this point not discussing 4c2c38e800f in the commit seems like a big omission, why not revert & adjust it if the cmake part of it is a perceived hassle?

Now, I don't think that it's worth spending time on running two sets of Windows tests all the time just to tease out GCC v.s. MSVC differences, but I think we should still run the build on both. That takes around ~5m, and seems worth it to avoid and flag any portability issues.

Before this we test on gcc/clang/msvc, after this we'll have dropped that last one. Whatever we do about the tests and/or cmake I think we really should avoid compiler monoculture, and fix portability issues early.

I think this is a good goal, but what does the "as of recent" refer to here? When the ASAN job was introduced?

"Johannes Schindelin via GitGitGadget" <[email protected]>

Thanks, both.

;-)

Hi Ævar,

I really do not see any value in that.

If you _really_ want to run those tests, you can easily add a commit on top of your branch.

It's not like you always want to run the win+VS tests on a certain subset of your branches.

But. A big part of the reason for this patch is to codify that it is _not_ the responsibility of core Git contributors to take care of these tests. Trying to do this via the utterly convoluted and under-documented ci-config would only water down this quite clear statement.

I fear that the tb/ci-concurrency topic is a good example of "death by committee". Let's put in a collective effort to avoid that here.

That might have been the motivation, but there have been preciously few patches coming in from that side.

So few, in fact, that the question "is it worth spending all of that energy to run these builds and tests all the time" most likely has to be answered with a resounding "No".

I can think of just two bugs that were identified in the MSVC builds, one where we had to change some code to explicitly use a stable sort (which MSVC would otherwise not have used), and one where we now avoid an unsigned/signed comparison where MSVC cast the signed value to a now-insanely-large unsigned, i.e. in the wrong direction.

That's two bugs identified in how many years?

We still can catch those bugs in git-for-windows/git. And avoid some of the long build times.

It's not only linux-asan. The overall build time of every single job has gone up. I picked randomly two runs that were roughly 6 months apart,
https://github.com/git/git/actions/runs/2537915353 and
https://github.com/git/git/actions/runs/3728047162. As an indicator, the osx-gcc job took 35m half a year ago, now it takes 40m. The overall time went up from 7h40 (which is already _quite_ a cost!) to 8h01.

I don't think that we, collectively, are judicious enough about the balance between cost and benefit here.

Having said that, there is a silver lining: while the linux-asan job is new and takes a whopping 44m to complete, the difference between the total run time 6 months ago and today is less than that, so we must have saved on _something_, _somewhere_. Or GitHub bought faster hardware.

Ciao,
Johannes