ci: update caskroom/cask/perforce to new location

Running CI on Mac OS X in Azure Pipelines is currently broken due to a moved homebrew package.

Thanks, -Stolee

Johannes Schindelin (1):
ci(osx): use new location of the perforce cask

 ci/install-dependencies.sh | 1 +
 1 file changed, 1 insertion(+)

base-commit: 108b97dc372828f0e72e56bbb40cae8e1e83ece6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-400%2Fderrickstolee%2Fci-caskroom-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-400/derrickstolee/ci-caskroom-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/400
--
gitgitgadget

From: Johannes Schindelin <[email protected]>

The CI builds are failing for Mac OS X due to a change in the

location of the perforce cask. The command outputs the following error:

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.

So let's try to call brew cask install perforce first (which is what that error message suggests, in a most round-about way).

The "caskroom" way was added in 672f51cb (travis-ci:
fix Perforce install on macOS, 2017-01-22) and the justification is that the call "brew cask install perforce" can fail due to a checksum
mismatch: the recipe simply downloads the official Perforce distro, and whenever that is updated, the recipe needs to be updated, too.

CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if brew cask install perforce fails, by specifically pulling the latest master of the homebrew-cask repository.

This will still fail, of course, when homebrew-cask falls behind Perforce's release schedule. But once it is updated, we can now simply re-run the failed jobs and they will pick up that update.

As for updating homebrew-cask: the beginnings of automating this in
https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary
will be finished once the next Perforce upgrade comes around.

Signed-off-by Johannes Schindelin <[email protected]>

Signed-off-by Derrick Stolee <[email protected]>

---
 ci/install-dependencies.sh | 5 +++++
 1 file changed, 5 insertions(+)

ci/install-dependencies.sh +5 -0

index 85a9d6b15c..ce149ed39c 100755
@@ -40,6 +40,11 @@ osx-clang|osx-gcc)
 	test -z "$BREW_INSTALL_PACKAGES" ||
 	brew install $BREW_INSTALL_PACKAGES
 	brew link --force gettext
+	brew cask install perforce || {
+		# Update the definitions and try again
+		git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull &&
+		brew cask install perforce
+	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)

Signature · 1 line

gitgitgadget

Quoted 3 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CI builds are failing for Mac OS X due to a change in the

s/CI/Azure Pipelines/

Our Travis CI builds are fine.

Quoted 6 lines

location of the perforce cask. The command outputs the following error:

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.

Quoted 8 lines

So let's try to call brew cask install perforce first (which is what that error message suggests, in a most round-about way).

The "caskroom" way was added in 672f51cb (travis-ci:
fix Perforce install on macOS, 2017-01-22) and the justification is that the call "brew cask install perforce" can fail due to a checksum
mismatch: the recipe simply downloads the official Perforce distro, and whenever that is updated, the recipe needs to be updated, too.

This paragraph is wrong, it mixes up things too much.

Prior to 672f51cb we used to install the 'perforce' _package_ with 'brew install perforce' (note: no 'cask' in there). The justification for 672f51cb was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". It was _their_ justification for this move that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of:

==> No checksum defined for Cask perforce, skipping verification

It's unclear to me why 672f51cb used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then.

Anyway, as the error message at the top of the log message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that.

Note that on Travis CI we explicitly specify which macOS image to use, and nowadays we don't run 'brew update' during the build process [1], so both commands work in our builds there.
...
Now, let's take a step back.

All 'brew cask install perforce' really does is ...
... in fact, that's what we have been doing in some of our Linux jobs since the very beginning, so basically only the download URL has to be adjusted.

Quoted 53 lines · SZEDER Gábor

Quoted 3 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CI builds are failing for Mac OS X due to a change in the

s/CI/Azure Pipelines/

Our Travis CI builds are fine.

Quoted 6 lines

location of the perforce cask. The command outputs the following error:

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.

Quoted 8 lines

So let's try to call brew cask install perforce first (which is what that error message suggests, in a most round-about way).

The "caskroom" way was added in 672f51cb (travis-ci:
fix Perforce install on macOS, 2017-01-22) and the justification is that the call "brew cask install perforce" can fail due to a checksum
mismatch: the recipe simply downloads the official Perforce distro, and whenever that is updated, the recipe needs to be updated, too.

This paragraph is wrong, it mixes up things too much.

Prior to 672f51cb we used to install the 'perforce' _package_ with 'brew install perforce' (note: no 'cask' in there). The justification for 672f51cb was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". It was _their_ justification for this move that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of:

==> No checksum defined for Cask perforce, skipping verification

It's unclear to me why 672f51cb used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then.

Anyway, as the error message at the top of the log message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that.

Note that on Travis CI we explicitly specify which macOS image to use, and nowadays we don't run 'brew update' during the build process [1], so both commands work in our builds there.
...
Now, let's take a step back.

Quoted 4 lines

All 'brew cask install perforce' really does is ...
... in fact, that's what we have been doing in some of our Linux jobs since the very beginning, so basically only the download URL has to be adjusted.

This is already in 'next' X-<; reverting a merge is cheap but I prefer to do so when we already have a replacement.

Thanks for taking a closer look.

Quoted 2 lines

This is already in 'next' X-<; reverting a merge is cheap but I prefer to do so when we already have a replacement.

I force-pushed (see https://github.com/gitgitgadget/git/pull/400), and once Stolee approves, he will submit v3. This will only change the commit message, though, as I disagree that hard-coding the URL would be an improvement: the nice thing about a package management system is that the user does not need to know the details (or need to know if the details change, like, ever).

Quoted 3 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CI builds are failing for Mac OS X due to a change in the

s/CI/Azure Pipelines/

Our Travis CI builds are fine.

Quoted 6 lines

location of the perforce cask. The command outputs the following error:

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.

Quoted 8 lines

So let's try to call brew cask install perforce first (which is what that error message suggests, in a most round-about way).

The "caskroom" way was added in 672f51cb (travis-ci:
fix Perforce install on macOS, 2017-01-22) and the justification is that the call "brew cask install perforce" can fail due to a checksum
mismatch: the recipe simply downloads the official Perforce distro, and whenever that is updated, the recipe needs to be updated, too.

This paragraph is wrong, it mixes up things too much.

Prior to 672f51cb we used to install the 'perforce' _package_ with 'brew install perforce' (note: no 'cask' in there). The justification for 672f51cb was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". It was _their_ justification for this move that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of:

==> No checksum defined for Cask perforce, skipping verification

It's unclear to me why 672f51cb used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then.

Anyway, as the error message at the top of the log message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that.

Note that on Travis CI we explicitly specify which macOS image to use, and nowadays we don't run 'brew update' during the build process [1], so both commands work in our builds there.

[1] f2f4715033 (ci: don't update Homebrew, 2019-07-03)

Quoted 4 lines

CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if brew cask install perforce fails, by specifically pulling the latest master of the homebrew-cask repository.

Homebrew didn't record a checksum for Perforce versions r17.1, r17.2 and r18.1, so installing those still works fine. Our Travis CI images install r18.1.

However, when Homebrew updated to Perforce r19.1, they included the checksum again for some reason (intentional or accidental; I didn't look why). This worked fine for a while, until a couple of days ago Perforce updated the r19.1 binaries in place, breaking those checksums.

If we were to still run 'brew update', then it would shortly fix the checksum mismatch. But we don't run it, and we do not want to run it because it takes ages. Falling back to pull from the 'homebrew-cask' repository could be a reasonable and quick workaround.

Quoted 3 lines

This will still fail, of course, when homebrew-cask falls behind Perforce's release schedule. But once it is updated, we can now simply re-run the failed jobs and they will pick up that update.

In our CI builds we don't at all care what the checksums of the Perforce binaries are, so I would really like to tell 'brew' to ignore any checksum mismatch when installing 'perforce'. Alas, it appears that 'brew' has no public options to turn of or to ignore checksum verification.

Now, let's take a step back.

All 'brew cask install perforce' really does is run 'curl' to download a tar.gz from the Perforce servers, verify its checksum, unpack it, and put the executables somewhere on $PATH. That's not rocket science, we could easily do that ourselves; we don't even have to deal with a tar.gz, the 'p4' and 'p4d' binaries for mac are readily available for download at:

http://filehost.perforce.com/perforce/r19.1/bin.macosx1010x86_64/

And, in fact, that's what we have been doing in some of our Linux jobs since the very beginning, so basically only the download URL has to be adjusted.

Quoted 3 lines

As for updating homebrew-cask: the beginnings of automating this in
https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary
will be finished once the next Perforce upgrade comes around.

Quoted 25 lines

Signed-off-by Johannes Schindelin <[email protected]>

Signed-off-by Derrick Stolee <[email protected]>

---
 ci/install-dependencies.sh | 5 +++++
 1 file changed, 5 insertions(+)

ci/install-dependencies.sh +5 -1

index 85a9d6b15c..ce149ed39c 100755
@@ -40,6 +40,11 @@ osx-clang|osx-gcc)
 	test -z "$BREW_INSTALL_PACKAGES" ||
 	brew install $BREW_INSTALL_PACKAGES
 	brew link --force gettext
+	brew cask install perforce || {
+		# Update the definitions and try again
+		git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull &&
+		brew cask install perforce
+	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)
--

gitgitgadget

Quoted 3 lines · SZEDER Gábor

Quoted 2 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Quoted 5 lines

Quoted 1 line

The CI builds are failing for Mac OS X due to a change in the

s/CI/Azure Pipelines/

Our Travis CI builds are fine.

For the moment ;-)

If you don't mind, I am going to copy/edit these three paragraphs into the commit message,

Quoted 5 lines

In our CI builds we don't at all care what the checksums of the Perforce binaries are, so I would really like to tell 'brew' to ignore any checksum mismatch when installing 'perforce'. Alas, it appears that 'brew' has no public options to turn of or to ignore checksum verification.

Sad, yet true, that we indeed have no command-line option to say "you know what, your checksum possibly mismatches, but we really don't care".

Quoted 9 lines

Now, let's take a step back.

All 'brew cask install perforce' really does is run 'curl' to download a tar.gz from the Perforce servers, verify its checksum, unpack it, and put the executables somewhere on $PATH. That's not rocket science, we could easily do that ourselves; we don't even have to deal with a tar.gz, the 'p4' and 'p4d' binaries for mac are readily available for download at:

Quoted 5 lines

http://filehost.perforce.com/perforce/r19.1/bin.macosx1010x86_64/

And, in fact, that's what we have been doing in some of our Linux jobs since the very beginning, so basically only the download URL has to be adjusted.

I'd rather not.

Just because there is no better way on Linux, and just because the current perforce cask recipe happens to just download and unpack that file does not mean that this won't change.

From: Johannes Schindelin <[email protected]>

The Azure Pipelines builds are failing for macOS due to a change in the location of the perforce cask. The command outputs the following error:

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.
...
In any case, as the error message at the top of this commit message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that.

"Johannes Schindelin via GitGitGadget" <[email protected]>

Quoted 5 lines · writes:

From: Johannes Schindelin <[email protected]>

The Azure Pipelines builds are failing for macOS due to a change in the location of the perforce cask. The command outputs the following error:

Quoted 7 lines

+ brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.
...
In any case, as the error message at the top of this commit message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that.

It appears that OSX jobs at Travis are getting hit by this issue. Here is what our failed build ends with, for example:

+brew install caskroom/cask/perforce
Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.

cf. https://travis-ci.org/git/git/jobs/601697815

Today's 'pu' has this topic queued, and it seems to help even the builds at Travis ('pu' seems to fail the test for totally different reason, though):

+brew link gcc@8
Error: No such keg: /usr/local/Cellar/gcc@8

cf. https://travis-ci.org/git/git/jobs/601697903

CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if brew cask install perforce fails, by specifically pulling the latest master of the homebrew-cask repository.

ci/install-dependencies.sh +0 -0

index 85a9d6b15c..ce149ed39c 100755

ci/install-dependencies.sh +2 -0

+		brew cask install perforce
+	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)

Signature · 1 line

gitgitgadget

Quoted 10 lines · Johannes Schindelin via GitGitGadget

CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if brew cask install perforce fails, by specifically pulling the latest master of the homebrew-cask repository.

ci/install-dependencies.sh +0 -0

index 85a9d6b15c..ce149ed39c 100755

ci/install-dependencies.sh osx-clang|osx-gcc) · line 45

Quoted 6 lines

ci/install-dependencies.sh +3 -0

@@ -40,6 +40,11 @@ osx-clang|osx-gcc)
 	test -z "$BREW_INSTALL_PACKAGES" ||
 	brew install $BREW_INSTALL_PACKAGES
 	brew link --force gettext
+	brew cask install perforce || {
+		# Update the definitions and try again
+		git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull &&

In the build of v2.24.0 this 'git pull' printed just short of 600 lines worth of diffstat. Two weeks went by since then, and in today's 'pu' build that diffstat is already over 1000 lines long.

Perhaps we could use --quiet here, though that would suppress the transfer progress as well.

Quoted 7 lines

ci/install-dependencies.sh +2 -1

+		brew cask install perforce
+	} ||
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)
--

gitgitgadget

Quoted 8 lines

Review

Quoted 3 lines

(quoted diff) +3 -0

+	brew cask install perforce || {
+		# Update the definitions and try again
+		git -C "$(brew --repository)"/Library/Taps/homebrew/homebrew-cask pull &&

In the build of v2.24.0 this 'git pull' printed just short of 600 lines worth of diffstat. Two weeks went by since then, and in today's 'pu' build that diffstat is already over 1000 lines long.

Quoted 2 lines

Perhaps we could use --quiet here, though that would suppress the transfer progress as well.

Isn't there an option to suppress the diffstat specifically?

-- >8 --
Subject: ci(osx): update homebrew-cask repository with less noise

The OSX CI build procedure updates the homebrew-cask repository before attempting to install perforce again, after seeing an installation failure. This involves a "git pull" that by default computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world.

Suppress the diffstat to both save cycles to generate it, and strain on the eyeballs to skip it.

Reported-by SZEDER Gábor <[email protected]>

Helped-by Johannes Schindelin <[email protected]>

Signed-off-by Junio C Hamano <[email protected]>

---
 ci/install-dependencies.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

ci/install-dependencies.sh +0 -0

index 4e64a19112..b6b4f4707f 100755

ci/install-dependencies.sh +0 -0

 		brew cask install perforce
 	} ||
 	brew install caskroom/cask/perforce

Quoted 10 lines

Subject: ci(osx): update homebrew-cask repository with less noise

The OSX CI build procedure updates the homebrew-cask repository before attempting to install perforce again, after seeing an installation failure. This involves a "git pull" that by default computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world.

Quoted 6 lines

Suppress the diffstat to both save cycles to generate it, and strain on the eyeballs to skip it.

Reported-by SZEDER Gábor <[email protected]>

Helped-by Johannes Schindelin <[email protected]>

Signed-off-by Junio C Hamano <[email protected]>

Subject: ci(osx): update homebrew-cask repository with less noise

The OSX CI build procedure updates the homebrew-cask repository before attempting to install perforce again, after seeing an installation failure. This involves a "git pull" that by default computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world.

Suppress the diffstat to both save cycles to generate it, and strain on the eyeballs to skip it.

Reported-by SZEDER Gábor <[email protected]>

Helped-by Johannes Schindelin <[email protected]>

Signed-off-by Junio C Hamano <[email protected]>

---
 ci/install-dependencies.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

ci/install-dependencies.sh +0 -0

index 4e64a19112..b6b4f4707f 100755

ci/install-dependencies.sh +0 -0

 		brew cask install perforce
 	} ||
 	brew install caskroom/cask/perforce

Quoted 5 lines · Junio C Hamano

...
computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world.

Minor clarification: I, too, thought that the CI build images are cast in stone, but the Travis CI build images are definitely not (don't know about Azure Pipelines), but receive smaller updates from time to time, including updates to their Homebrew database. Such a recent update necessitated commit 7d4733c501 (ci: fix GCC install in the Travis CI GCC OSX job, 2019-10-24). I contacted Travis CI support about it, and they confirmed.

Quoted 9 lines · Junio C Hamano

Subject: ci(osx): update homebrew-cask repository with less noise

The OSX CI build procedure updates the homebrew-cask repository before attempting to install perforce again, after seeing an installation failure. This involves a "git pull" that by default computes and outputs diffstat, which would only grow as the time goes by and the repository cast in stone in the CI build image becomes more and more stale relative to the upstream repository in the outside world.

Minor clarification: I, too, thought that the CI build images are cast in stone, but the Travis CI build images are definitely not (don't know about Azure Pipelines),