ci: make perforce installation optional in macOS
PATCH 1/1 ci: make perforce installation optional in macOS
Using brew to install perforce has several documented[1,2,3,4] edge cases that make it fail periodically, so if problems were found while installing it, just continue without it.
This means that until the problem is solved all perforce tests will be skipped in macOS, but they are still most likely covered by the other unaffected runs and will be covered again once the issue solves itself.
1 0eb3671ed96 (ci(osx): use new location of the perforce cask, 2019-10-23) 2 5ed9fc3fc86 (ci: prevent perforce from being quarantined, 2020-02-27) 3 3831132ace6 (ci/install-depends: attempt to fix "brew cask" stuff, 2021-01-14) 4 https://lore.kernel.org/git/[email protected]/
---
This is based on master and can merge upwards except with seen where it conflicts with the recently added ab/ci-osx-loosen-package-requirement which it is meant to replace.
A successful sample run when merged with master available in :
https://github.com/carenas/git/actions/runs/2204519374
ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
wi
ci/install-dependencies.sh +1 -1
@@ -43,7 +43,7 @@ macos-latest) git -C "$cask_repo" pull --no-stat --ff-only && brew install --cask --no-quarantine perforce } || - brew install homebrew/cask/perforce + brew install homebrew/cask/perforce || true if test -n "$CC_PACKAGE" then
Signature · 1 line
2.36.0.266.g59f845bde02
Quoted 8 lines · Carlo Marcelo Arenas Belón
Using brew to install perforce has several documented[1,2,3,4] edge cases that make it fail periodically, so if problems were found while installing it, just continue without it.
This means that until the problem is solved all perforce tests will be skipped in macOS, but they are still most likely covered by the other unaffected runs and will be covered again once the issue solves itself.
Quoted 5 lines
1
0eb3671ed96(ci(osx): use new location of theperforcecask, 2019-10-23) 25ed9fc3fc86(ci: preventperforcefrom being quarantined, 2020-02-27) 33831132ace6(ci/install-depends: attempt to fix "brew cask" stuff, 2021-01-14) 4 https://lore.kernel.org/git/[email protected]/
---
Missing sign-off.
Quoted 23 lines · Carlo Marcelo Arenas Belón
Using brew to install perforce has several documented[1,2,3,4] edge cases that make it fail periodically, so if problems were found while installing it, just continue without it.
This means that until the problem is solved all perforce tests will be skipped in macOS, but they are still most likely covered by the other unaffected runs and will be covered again once the issue solves itself.
1
0eb3671ed96(ci(osx): use new location of theperforcecask, 2019-10-23) 25ed9fc3fc86(ci: preventperforcefrom being quarantined, 2020-02-27) 33831132ace6(ci/install-depends: attempt to fix "brew cask" stuff, 2021-01-14) 4 https://lore.kernel.org/git/[email protected]/
---
This is based on master and can merge upwards except with seen where it conflicts with the recently added ab/ci-osx-loosen-package-requirement which it is meant to replace.A successful sample run when merged with master available in :
ci/install-dependencies.sh macos-latest) · line 43
Quoted 2 lines
ci/install-dependencies.sh +1
-1@@ -43,7 +43,7 @@ macos-latest) - brew install homebrew/cask/perforce + brew install homebrew/cask/perforce || true
You do get a vaguely alarming error message[1] with the solution implemented by this patch:
ci/install-dependencies.sh: line 81: type: p4d: not found
However, that's an unrelated and existing minor mistake (outside of the scope of this fix, probably). It should be using command -v rather than type in the couple conditionals at the bottom of the script.
[1]: https://github.com/carenas/git/runs/6119943232?check_suite_focus=true#step:3:116
v2 cover ci: avoid perforce/brew issues affecting macOS
In reply to Carlo Marcelo Arenas BelónThis series is a simpler alternative to ab/ci-osx-loosen-package-requirement which while it doesn't address the underlying issue, avoids the CI failures in macOS as well.
For an example of a successful run with it merged with next see:
https://github.com/carenas/git/actions/runs/2205005763
Carlo Marcelo Arenas Belón (2):
ci: make failure to find perforce more user friendly
ci: make perforce installation optional in macOS
ci/install-dependencies.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
Signature · 1 line
2.36.0.266.g59f845bde02
PATCH v2 1/2 ci: make failure to find perforce more user friendly
In preparation for a future change that will make perforce installation optional in macOS, make sure that the check for it is done without triggering scary looking errors and add a user friendly message instead.
Only one invocation of type is changed as that is what is only needed for the expected future use case, and because type is recommended in the CodingGuidelines, so changing that recommendation or a more complex change has been specifically punted.
--- ci/install-dependencies.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
ci/install-dependencies.sh +3 -1
@@ -78,12 +78,14 @@ linux-gcc-default) ;; esac -if type p4d >/dev/null && type p4 >/dev/null +if command -v p4d >/dev/null && type p4 >/dev/null then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else + echo "WARNING: perforce wasn't installed, see above for clues why" fi if type git-lfs >/dev/null then
Signature · 1 line
2.36.0.266.g59f845bde02
Re: [PATCH v2 1/2] ci: make failure to find perforce more user friendly
Quoted 8 lines · Carlo Marcelo Arenas Belón
In preparation for a future change that will make perforce installation optional in macOS, make sure that the check for it is done without triggering scary looking errors and add a user friendly message instead.
Only one invocation of type is changed as that is what is only needed for the expected future use case, and because
typeis recommended in the CodingGuidelines, so changing that recommendation or a more complex change has been specifically punted.
This may be in the "POSIX may say this but the real world may not work that way" territory. As far as I can tell, "command -v" [*1*] and "type" [*2*] both ought to give diagnostic messages to their standard error stream, and they both should signal an error with non-zero exit status. It may be that the shell implementation you have tried had "command -v" that is less noisy than "type" when given a command that is not installed, but I wonder if the next shell implementation you find has "command -v" that is as noisy and scary as "type", in which case this patch amounts to a no-op.
I wonder if "type p4d >/dev/null 2>/dev/null" (or "command -v" with the same) is a more futureproof fix.
[References]
*1* https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html
*2* https://pubs.opengroup.org/onlinepubs/9699919799/utilities/type.html
Quoted 26 lines
Helped-by Eric Sunshine <[email protected]>Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>--- ci/install-dependencies.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)ci/install-dependencies.sh +3
-1@@ -78,12 +78,14 @@ linux-gcc-default) ;; esac -if type p4d >/dev/null && type p4 >/dev/null +if command -v p4d >/dev/null && type p4 >/dev/null then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else + echo "WARNING: perforce wasn't installed, see above for clues why" fi if type git-lfs >/dev/null then
Re: [PATCH v2 1/2] ci: make failure to find perforce more user friendly
Quoted 10 lines · Junio C Hamano
This may be in the "POSIX may say this but the real world may not work that way" territory. As far as I can tell, "command -v" [*1*] and "type" [*2*] both ought to give diagnostic messages to their standard error stream, and they both should signal an error with non-zero exit status. It may be that the shell implementation you have tried had "command -v" that is less noisy than "type" when given a command that is not installed, but I wonder if the next shell implementation you find has "command -v" that is as noisy and scary as "type", in which case this patch amounts to a no-op.
Quoted 2 lines
I wonder if "type p4d >/dev/null 2>/dev/null" (or "command -v" with the same) is a more futureproof fix.
So, how about replacing it with something like this?
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
From: Carlo Marcelo Arenas Belón <[email protected]>
In preparation for a future change that will make perforce installation optional in macOS, make sure that the check for it is done without triggering scary looking errors and add a user friendly message instead.
All other existing uses of 'type <cmd>' in our shell scripts that check the availability of a command <cmd> send both standard output and error stream to /dev/null to squelch "<cmd> not found" diagnostic output, but this script left the standard error stream shown.
Redirect it just like everybody else to squelch this error message that we fully expect to see.
--- ci/install-dependencies.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +4 -2
@@ -78,14 +78,16 @@ linux-gcc-default) ;; esac -if type p4d >/dev/null && type p4 >/dev/null +if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1 then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else + echo "WARNING: perforce wasn't installed, see above for clues why" fi -if type git-lfs >/dev/null +if type git-lfs >/dev/null 2>&1 then echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" git-lfs version
Signature · 1 line
2.36.0-184-gca3de164ba
Re: [PATCH v2 1/2] ci: make failure to find perforce more user friendly
Quoted 2 lines · Junio C Hamano
So, how about replacing it with something like this?
Agree 100%.
It would theoretically make the issue raised by Ævar of not knowing when perforce was skipped slightly worse
but getting that fixed would seem like something that could be done in a follow up.
I have to also admit, with all the on the fly changes to these same files, it might be wiser to wait until later anyway.
Carlo
Re: [PATCH v2 1/2] ci: make failure to find perforce more user friendly
Quoted 6 lines · Carlo Arenas
Quoted 2 lines · Junio C Hamano
So, how about replacing it with something like this?
Agree 100%.
Quoted 7 lines
It would theoretically make the issue raised by Ævar of not knowing when perforce was skipped slightly worse
but getting that fixed would seem like something that could be done in a follow up.I have to also admit, with all the on the fly changes to these same files, it might be wiser to wait until later anyway.
Actually, I was thinking about taking these two (possibly with Ævar's "download with https://, we are in 21st century" change to make them 3-patch series) and fast-tracking.
The homebrew specific "packagers can take a bit of time to catch up and we may see hash mismatch" and other tweaks Ævar has can come on top once we see the basic "any parts of tests can fail and the rest can still be tested, why does it have to be world-stopping if we fail to install p4?" working.
Thanks.
Re: [PATCH v2 1/2] ci: make failure to find perforce more user friendly
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Carlo Arenas
I have to also admit, with all the on the fly changes to these same files, it might be wiser to wait until later anyway.
Actually, I was thinking about taking these two (possibly with Ævar's "download with https://, we are in 21st century" change to make them 3-patch series) and fast-tracking.
Sounds like a good plan to me, and unlikely to introduce much pain to all other on the fly changes, but I think it is worth mentioning that some kind soul did fix the mismatch in homebrew already, so there is not really an urgent need for fast-tracking right now either.
Carlo
PATCH v2 2/2 ci: make perforce installation optional in macOS
In reply to Carlo Marcelo Arenas BelónUsing brew to install perforce has several documented[1,2,3,4] edge cases that make it fail periodically, so if problems were found while installing it, just continue without it.
This means that until the problem is solved all perforce tests will be skipped in macOS, but they are still most likely covered by the other unaffected runs and will be covered again once the issue solves itself.
1 0eb3671ed96 (ci(osx): use new location of the perforce cask, 2019-10-23) 2 5ed9fc3fc86 (ci: prevent perforce from being quarantined, 2020-02-27) 3 3831132ace6 (ci/install-depends: attempt to fix "brew cask" stuff, 2021-01-14) 4 https://lore.kernel.org/git/[email protected]/
--- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ci/install-dependencies.sh +1 -1
@@ -43,7 +43,7 @@ macos-latest) git -C "$cask_repo" pull --no-stat --ff-only && brew install --cask --no-quarantine perforce } || - brew install homebrew/cask/perforce + brew install homebrew/cask/perforce || true if test -n "$CC_PACKAGE" then
Signature · 1 line
2.36.0.266.g59f845bde02
v3 cover ci: avoid perforce/brew issues affecting macOS
In reply to Carlo Marcelo Arenas BelónThis series is an alternative to ab/ci-osx-loosen-package-requirement and includes the parts proposed on it in v2 that didn't conflict.
The first patch is just a slightly modified version from the one proposed by Junio and that adds also a message if git-lfs wasn't found just like we do now for perforce while making those messages go to stderr.
The second patch replaces the original proposal with a more complicated one that removes the use of brew as suggested by Ævar, it is also a little bigger than it really needs to be just to make sure it won't conflict with on the fly changes, and for the same reason uses some hardcoded values that would be worth fixing after.
The third patch is only needed to avoid regressions with Azure Pipelines which might be deprecated as proposed in other on the fly changes, so it might not be needed in seen, but is still needed if this progresses further first. It was made independent so it could be easy to revert if Azure support is going away just like TravisCI did.
The last patch was taken from Ævar's v2 and is the only one that will require changes in other core on the fly but the conflict resolution is trivial, and more importantly doesn't conflict when merged to seen in the current spot this branch uses.
For an example of a "successful" run with it merged on top of seen (with the previous iteration reverted) see:
https://github.com/carenas/git/runs/6140236148
Note that perforce no longer fails because someone fixed the issue with their Cask but still most of the new code is excercised correctly.
This introduces a problem identified by Ævar of making the CI runs less consistent, as they might be skipping some of the perforce integrations if installing that fails again, but the likelyhood of that failing has been reduced by removing the inherent brew flakyness, and so improving the UI so that it will be more visible when that happens is probably something that can be addressed later, once all other CI related changes are settled, or we could reintroduce the hard dependency on installing perforce for macOS (just like it is done in Linux)
Carlo Marcelo Arenas Belón (3):
ci: make failure to find perforce more user friendly
ci: avoid brew for installing perforce
ci: reintroduce prevention from perforce being quarantined in macOS
Ævar Arnfjörð Bjarmason (1):
CI: use https, not http to download binaries from perforce.com
ci/install-dependencies.sh | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
Signature · 1 line
2.36.0.266.g59f845bde02
PATCH v3 1/4 ci: make failure to find perforce more user friendly
In preparation for a future change that will make perforce installation optional in macOS, make sure that the check for it is done without triggering scary looking errors and add a user friendly message instead.
All other existing uses of 'type <cmd>' in our shell scripts that check the availability of a command <cmd> send both standard output and error stream to /dev/null to squelch "<cmd> not found" diagnostic output, but this script left the standard error stream shown.
Redirect it just like everybody else to squelch this error message that we fully expect to see.
--- ci/install-dependencies.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +6 -2
@@ -78,15 +78,19 @@ linux-gcc-default) ;; esac -if type p4d >/dev/null && type p4 >/dev/null +if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1 then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else + echo "WARNING: perforce wasn't installed, see above for clues why" >2 fi -if type git-lfs >/dev/null +if type git-lfs >/dev/null 2>&1 then echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" git-lfs version +else + echo "WARNING: git-lfs wasn't installed, see above for clues why" >2 fi
Signature · 1 line
2.36.0.266.g59f845bde02
Re: [PATCH v3 1/4] ci: make failure to find perforce more user friendly
Quoted 8 lines · Carlo Marcelo Arenas Belón
ci/install-dependencies.sh +2
-1-if type p4d >/dev/null && type p4 >/dev/null +if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1 then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else
ci/install-dependencies.sh linux-gcc-default) · line 96
Quoted 10 lines
ci/install-dependencies.sh +4
-1+ echo "WARNING: perforce wasn't installed, see above for clues why" >2 fi -if type git-lfs >/dev/null +if type git-lfs >/dev/null 2>&1 then echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" git-lfs version +else + echo "WARNING: git-lfs wasn't installed, see above for clues why" >2 fi
NO! Why do we want to create a file whose name is "2" here?
Good that I caught them before I merged them to 'next'.
Will locally amend and see if it fixes CI failure in 'seen'
Thanks.
PATCH v3 2/4 ci: avoid brew for installing perforce
In reply to Carlo Marcelo Arenas BelónPerfoce's cask in brew is meant[1] to be used only by humans, so replace its use from the CI with a scripted binary download which is less likely to fail, as it is done in Linux.
Kept the logic together so it will be less likely to break when moved around as on the fly code changes in this area are settled, at which point it will also feasable to ammend it to avoid some of the hardcoded values by using similar variables to the ones Linux does.
In that same line, a POSIX sh syntax is used instead of the similar one used in Linux in preparation for an unrelated future change that might change the shell currently configured for it.
This change reintroduces the risk that the installed binaries might not work because of being quarantined that was fixed with 5ed9fc3fc86 (ci:
prevent perforce from being quarantined, 2020-02-27) but fixing that now was also punted for simplicity and since the affected cloud provider is scheduled to be retired with an on the fly change, but should be addressed if that other change is not integrated further.
The discussion on the need to keep 2 radically different versions of the binaries to be tested with Linux vs macOS or how to upgrade to newer versions now that brew won't do that automatically for us has been punted for now as well. On that line the now obsolete comment about it in lib.sh was originally being updated by this change but created conflicts as it is moved around by other on the fly changes, so will be addressed independently as well.
[1] https://github.com/Homebrew/homebrew-cask/pull/122347#discussion_r856026584
--- ci/install-dependencies.sh | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
ci/install-dependencies.sh +8 -7
@@ -37,13 +37,14 @@ macos-latest) test -z "$BREW_INSTALL_PACKAGES" || brew install $BREW_INSTALL_PACKAGES brew link --force gettext - brew install --cask --no-quarantine perforce || { - # Update the definitions and try again - cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask && - git -C "$cask_repo" pull --no-stat --ff-only && - brew install --cask --no-quarantine perforce - } || - brew install homebrew/cask/perforce + mkdir -p $HOME/bin + ( + cd $HOME/bin + wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" && + tar -xf helix-core-server.tgz + ) + PATH="$PATH:${HOME}/bin" + export PATH if test -n "$CC_PACKAGE" then
Signature · 1 line
2.36.0.266.g59f845bde02
Re: [PATCH v3 2/4] ci: avoid brew for installing perforce
Quoted 3 lines · Carlo Marcelo Arenas Belón
Perfoce's cask in brew is meant[1] to be used only by humans, so replace its use from the CI with a scripted binary download which is less likely to fail, as it is done in Linux.
s/Perfoce/Perforce/
Quoted 4 lines
Kept the logic together so it will be less likely to break when moved around as on the fly code changes in this area are settled, at which point it will also feasable to ammend it to avoid some of the hardcoded values by using similar variables to the ones Linux does.
s/Kept/Keep/
s/also/also be/
s/feasable/feasible/
s/ammend/amend/
Quoted 22 lines
In that same line, a POSIX sh syntax is used instead of the similar one used in Linux in preparation for an unrelated future change that might change the shell currently configured for it.
This change reintroduces the risk that the installed binaries might not work because of being quarantined that was fixed with
5ed9fc3fc86(ci:
preventperforcefrom being quarantined, 2020-02-27) but fixing that now was also punted for simplicity and since the affected cloud provider is scheduled to be retired with an on the fly change, but should be addressed if that other change is not integrated further.The discussion on the need to keep 2 radically different versions of the binaries to be tested with Linux vs macOS or how to upgrade to newer versions now that brew won't do that automatically for us has been punted for now as well. On that line the now obsolete comment about it in
lib.shwas originally being updated by this change but created conflicts as it is moved around by other on the fly changes, so will be addressed independently as well.[1] https://github.com/Homebrew/homebrew-cask/pull/122347#discussion_r856026584
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>
Re: [PATCH v3 2/4] ci: avoid brew for installing perforce
In reply to Carlo Marcelo Arenas BelónHi Carlo,
Quoted 9 lines · Carlo Marcelo Arenas Belón
ci/install-dependencies.sh +0
-1@@ -37,13 +37,14 @@ macos-latest) test -z "$BREW_INSTALL_PACKAGES" || brew install $BREW_INSTALL_PACKAGES brew link --force gettext - brew install --cask --no-quarantine perforce || {
ci/install-dependencies.sh macos-latest) · line 43
Quoted 10 lines
ci/install-dependencies.sh +4
-6- # Update the definitions and try again - cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask && - git -C "$cask_repo" pull --no-stat --ff-only && - brew install --cask --no-quarantine perforce - } || - brew install homebrew/cask/perforce + mkdir -p $HOME/bin + ( + cd $HOME/bin + wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" &&
I vaguely recall that Gábor Szeder attempted something similar, but I _think_ that ultimately there were too many moving parts in that URL that we did not want to hardcode.
For example, when opening a PR against maint or an even older topic branch, we are stuck with the exact code in that revision that obtains the perforce package. But since that server is outside our control, it can very well evolve to a state that is not amenable to hard-coding such a URL.
While I don't think that we can solve this fully, I would prefer to keep the existing brew install calls but fall back to downloading from a hard-coded URL. That still would have the shortcoming of hard-coded r21.2 and 1015, but it should fail less often than the proposed patch.
For the record, the recent problems stem from the fact that the package was cached on GitHub's build agents (I guess to avoid many identical downloads), and the cached package did not match what was recorded in the updated package definition. Which means that those brew errors are only transient, until a new VM image is built that caches the then-current perforce package.
Ciao,
Dscho
Quoted 11 lines
ci/install-dependencies.sh +4
-1+ tar -xf helix-core-server.tgz + ) + PATH="$PATH:${HOME}/bin" + export PATH if test -n "$CC_PACKAGE" then --2.36.0.266.g59f845bde02
Re: [PATCH v3 2/4] ci: avoid brew for installing perforce
Quoted 18 lines · Johannes Schindelin
Quoted 15 lines · Carlo Marcelo Arenas Belón
ci/install-dependencies.sh +0
-7@@ -37,13 +37,14 @@ macos-latest) test -z "$BREW_INSTALL_PACKAGES" || brew install $BREW_INSTALL_PACKAGES brew link --force gettext - brew install --cask --no-quarantine perforce || { - # Update the definitions and try again - cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask && - git -C "$cask_repo" pull --no-stat --ff-only && - brew install --cask --no-quarantine perforce - } || - brew install homebrew/cask/perforce
Quoted 8 lines
ci/install-dependencies.sh macos-latest) · line 43
Quoted 4 lines
ci/install-dependencies.sh +4
-0+ mkdir -p $HOME/bin + ( + cd $HOME/bin + wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" &&I vaguely recall that Gábor Szeder attempted something similar, but I _think_ that ultimately there were too many moving parts in that URL that we did not want to hardcode.
I can see that, and indeed that is also why those series still keep perforce installation as optional so it wouldn't block CI when it does.
BTW most of the reasons why those values are hardcoded here is just to avoid conflicts with on the fly changes, and making it smarter and avoiding the hardcoding would be done as a prerequisite to making it mandatory again (if that is what is preferred)
Quoted 3 lines
While I don't think that we can solve this fully, I would prefer to keep the existing
brew installcalls but fall back to downloading from a hard-coded URL.
Considering that the expectation from brew maintainers (as documented[1]) is that those mismatches would happen and should be solved manually and that perforce releases just do inplace binary pushes I still stand by my suggestion to avoid brew, but it might be useful as a fallback of the wget approach if that helps with your concerns above.
Quoted 6 lines
For the record, the recent problems stem from the fact that the package was cached on GitHub's build agents (I guess to avoid many identical downloads), and the cached package did not match what was recorded in the updated package definition. Which means that those
brewerrors are only transient, until a new VM image is built that caches the then-current perforce package.
FWIW as I updated the brew cask[2], I realized that livecheck pointed to an even newer release, so if we are adding brew back it might be worth adding yet another
workaround to delete the cached file and try without a SHA as Ævar suggested.
Carlo
[1] https://docs.brew.sh/Common-Issues#cask---checksum-does-not-match
[2] https://github.com/Homebrew/homebrew-cask/pull/122347
PATCH v3 3/4 ci: reintroduce prevention from perforce being quarantined in macOS
In reply to Carlo Marcelo Arenas Belón5ed9fc3fc86 (ci: prevent perforce from being quarantined, 2020-02-27) introduces this prevention for brew, but brew has been removed in a previous commit, so reintroduce an equivalent option to avoid a possible regression.
This doesn't affect github actions (as configure now) and is therefore done silently to avoid any possible scary irrelevant messages.
--- ci/install-dependencies.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
ci/install-dependencies.sh +2 -1
@@ -41,7 +41,8 @@ macos-latest) ( cd $HOME/bin wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" && - tar -xf helix-core-server.tgz + tar -xf helix-core-server.tgz && + sudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true ) PATH="$PATH:${HOME}/bin" export PATH
Signature · 1 line
2.36.0.266.g59f845bde02
Re: [PATCH v3 3/4] ci: reintroduce prevention from perforce being quarantined in macOS
Quoted 7 lines · Carlo Marcelo Arenas Belón
5ed9fc3fc86(ci: preventperforcefrom being quarantined, 2020-02-27) introduces this prevention for brew, but brew has been removed in a previous commit, so reintroduce an equivalent option to avoid a possible regression.This doesn't affect github actions (as configure now) and is therefore done silently to avoid any possible scary irrelevant messages.
s/configure/configured/
ci/install-dependencies.sh macos-latest) · line 48
Quoted 13 lines
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
ci/install-dependencies.sh +2
-1@@ -41,7 +41,8 @@ macos-latest) ( cd $HOME/bin wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" && - tar -xf helix-core-server.tgz + tar -xf helix-core-server.tgz && + sudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true ) PATH="$PATH:${HOME}/bin" export PATH
I don't see the point of the || true since the code doesn't care about the eventual outcome of the &&-chain. (The &&-chain itself makes sense, though, to avoid executing commands pointlessly if an earlier step failed.) May not be worth a reroll, though.
PATCH v3 4/4 CI: use https, not http to download binaries from perforce.com
In reply to Carlo Marcelo Arenas BelónFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Since 522354d70f4 (Add Travis CI support, 2015-11-27) the CI has used
http://filehost.perforce.com/perforce/ to download binaries from filehost.perforce.com, they were then moved to this script in 657343a602e (travis-ci: move Travis CI code into dedicated scripts, 2017-09-10).
Let's use https instead for good measure. I don't think we need to worry about the DNS or network between the GitHub CI and perforce.com being MitM'd, but using https gives us extra validation of the payload at least, and is one less thing to worry about when checking where else we rely on non-TLS'd http connections.
--- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ci/install-dependencies.sh +1 -1
@@ -5,7 +5,7 @@ . ${0%/*}/lib.sh -P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION +P4WHENCE=https://filehost.perforce.com/perforce/r$LINUX_P4_VERSION LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl
Signature · 1 line
2.36.0.266.g59f845bde02
v4 cover ci: avoid perforce/brew issues affecting macOS
In reply to Carlo Marcelo Arenas BelónWe see macOS CI job failures from time to time when the version of p4 brew expects is slightly older than what is offered at Perforce's download site. Instead of failing to run the tests altogether, just skip "git-p4" tests, like we do for tests that depends on any other optional packages, when this happens.
This is essentially unchanged from the v3 posted by Carlo on April 23rd, with a few fix-ups that have been cooking on top squashed in.
The changes from the previous round (i.e. the "fix-ups" that have been floating on top, which got squashed in) are:
* The first step has a fix to a couple of wrong redirections that created a junk file whose name is "2", instead of sending the messages to the standard error.
* The last step makes two places that we download Perforce packages to use https://cdist2.perforce.com/ (earlier one of the places used that, the other https://filehost.perforce.com/).
I am sending this out as a final "complain now, or this will go to 'next' soonish" warning. The "What's cooking" report is getting crowded with too many topics marked as "Expecting a reroll", and I'm trying to do easier ones myself to see how much reduction we can make.
Carlo Marcelo Arenas Belón (3):
ci: make failure to find perforce more user friendly
ci: avoid brew for installing perforce
ci: reintroduce prevention from perforce being quarantined in macOS
Ævar Arnfjörð Bjarmason (1):
ci: use https, not http to download binaries from perforce.com
ci/install-dependencies.sh | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
1: 5730a749bc ! 1: 024459486c ci: make failure to find perforce more user friendly
@@ ci/install-dependencies.sh: linux-gcc-default)
echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)"
p4 -V | grep Rev.
+else
-+ echo "WARNING: perforce wasn't installed, see above for clues why" >2
++ echo >&2 "WARNING: perforce wasn't installed, see above for clues why"
fi
-if type git-lfs >/dev/null
+if type git-lfs >/dev/null 2>&1
@@ ci/install-dependencies.sh: linux-gcc-default)
echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)"
git-lfs version
+else
-+ echo "WARNING: git-lfs wasn't installed, see above for clues why" >2
++ echo >&2 "WARNING: git-lfs wasn't installed, see above for clues why"
fi
2: 8cd96645ae = 2: 8717dbe8d9 ci: avoid brew for installing perforce 3: 1f06d0ba06 = 3: 6a4f085d63 ci: reintroduce prevention from perforce being quarantined in macOS 4: 6bf267b995 ! 4: 5be72d9150 ci: use https, not http to download binaries from perforce.com
@@ Commit message
at least, and is one less thing to worry about when checking where else we rely on non-TLS'd http connections.
+ Also, use the same download site at perforce.com for Linux and macOS
+ tarballs for consistency.
+
+ Signed-off-by: Carlo Marcelo Arenas Belón <[email protected]>
## ci/install-dependencies.sh ##
@@ ci/install-dependencies.sh
. ${0%/*}/lib.sh
-P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION
-+P4WHENCE=https://filehost.perforce.com/perforce/r$LINUX_P4_VERSION ++P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION
LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev
tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl
Signature · 1 line
2.36.1-338-g1c7f76a54c
PATCH v4 1/4 ci: make failure to find perforce more user friendly
From: Carlo Marcelo Arenas Belón <[email protected]>
In preparation for a future change that will make perforce installation optional in macOS, make sure that the check for it is done without triggering scary looking errors and add a user friendly message instead.
All other existing uses of 'type <cmd>' in our shell scripts that check the availability of a command <cmd> send both standard output and error stream to /dev/null to squelch "<cmd> not found" diagnostic output, but this script left the standard error stream shown.
Redirect it just like everybody else to squelch this error message that we fully expect to see.
--- ci/install-dependencies.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +6 -2
@@ -78,15 +78,19 @@ linux-gcc-default) ;; esac -if type p4d >/dev/null && type p4 >/dev/null +if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1 then echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" p4d -V | grep Rev. echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V | grep Rev. +else + echo >&2 "WARNING: perforce wasn't installed, see above for clues why" fi -if type git-lfs >/dev/null +if type git-lfs >/dev/null 2>&1 then echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" git-lfs version +else + echo >&2 "WARNING: git-lfs wasn't installed, see above for clues why" fi
Signature · 1 line
2.36.1-338-g1c7f76a54c
PATCH v4 2/4 ci: avoid brew for installing perforce
In reply to Junio C HamanoFrom: Carlo Marcelo Arenas Belón <[email protected]>
Perfoce's cask in brew is meant[1] to be used only by humans, so replace its use from the CI with a scripted binary download which is less likely to fail, as it is done in Linux.
Kept the logic together so it will be less likely to break when moved around as on the fly code changes in this area are settled, at which point it will also feasable to ammend it to avoid some of the hardcoded values by using similar variables to the ones Linux does.
In that same line, a POSIX sh syntax is used instead of the similar one used in Linux in preparation for an unrelated future change that might change the shell currently configured for it.
This change reintroduces the risk that the installed binaries might not work because of being quarantined that was fixed with 5ed9fc3fc86 (ci:
prevent perforce from being quarantined, 2020-02-27) but fixing that now was also punted for simplicity and since the affected cloud provider is scheduled to be retired with an on the fly change, but should be addressed if that other change is not integrated further.
The discussion on the need to keep 2 radically different versions of the binaries to be tested with Linux vs macOS or how to upgrade to newer versions now that brew won't do that automatically for us has been punted for now as well. On that line the now obsolete comment about it in lib.sh was originally being updated by this change but created conflicts as it is moved around by other on the fly changes, so will be addressed independently as well.
[1] https://github.com/Homebrew/homebrew-cask/pull/122347#discussion_r856026584
--- ci/install-dependencies.sh | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
ci/install-dependencies.sh +8 -7
@@ -37,13 +37,14 @@ macos-latest) test -z "$BREW_INSTALL_PACKAGES" || brew install $BREW_INSTALL_PACKAGES brew link --force gettext - brew install --cask --no-quarantine perforce || { - # Update the definitions and try again - cask_repo="$(brew --repository)"/Library/Taps/homebrew/homebrew-cask && - git -C "$cask_repo" pull --no-stat --ff-only && - brew install --cask --no-quarantine perforce - } || - brew install homebrew/cask/perforce + mkdir -p $HOME/bin + ( + cd $HOME/bin + wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" && + tar -xf helix-core-server.tgz + ) + PATH="$PATH:${HOME}/bin" + export PATH if test -n "$CC_PACKAGE" then
Signature · 1 line
2.36.1-338-g1c7f76a54c
PATCH v4 3/4 ci: reintroduce prevention from perforce being quarantined in macOS
In reply to Junio C HamanoFrom: Carlo Marcelo Arenas Belón <[email protected]>
5ed9fc3fc86 (ci: prevent perforce from being quarantined, 2020-02-27) introduces this prevention for brew, but brew has been removed in a previous commit, so reintroduce an equivalent option to avoid a possible regression.
This doesn't affect github actions (as configure now) and is therefore done silently to avoid any possible scary irrelevant messages.
--- ci/install-dependencies.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
ci/install-dependencies.sh +2 -1
@@ -41,7 +41,8 @@ macos-latest) ( cd $HOME/bin wget -q "https://cdist2.perforce.com/perforce/r21.2/bin.macosx1015x86_64/helix-core-server.tgz" && - tar -xf helix-core-server.tgz + tar -xf helix-core-server.tgz && + sudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true ) PATH="$PATH:${HOME}/bin" export PATH
Signature · 1 line
2.36.1-338-g1c7f76a54c
PATCH v4 4/4 ci: use https, not http to download binaries from perforce.com
In reply to Junio C HamanoFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Since 522354d70f4 (Add Travis CI support, 2015-11-27) the CI has used
http://filehost.perforce.com/perforce/ to download binaries from filehost.perforce.com, they were then moved to this script in 657343a602e (travis-ci: move Travis CI code into dedicated scripts, 2017-09-10).
Let's use https instead for good measure. I don't think we need to worry about the DNS or network between the GitHub CI and perforce.com being MitM'd, but using https gives us extra validation of the payload at least, and is one less thing to worry about when checking where else we rely on non-TLS'd http connections.
Also, use the same download site at perforce.com for Linux and macOS tarballs for consistency.
--- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ci/install-dependencies.sh +1 -1
@@ -5,7 +5,7 @@ . ${0%/*}/lib.sh -P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION +P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl
Signature · 1 line
2.36.1-338-g1c7f76a54c