repo.eclipse.org outage breaking all our linux CI jobs
As https://www.eclipsestatus.io/ reports there is no ETA for recovery, and due to the failure of downloading JGit material in an early stage of our CI jobs, our linux CI jobs are all failing, I am very tempted to apply the following to 'maint' immediately and propagate it all the way up to 'master', 'next', and 'seen'.
I would very very much appreciate additional thoughts and advices by anybody more involved in JGit community and more clueful than I am on the situation.
Anyway, what is somewhat funny is that at the end of this script, there is an attempt to notice and report the lack of jgit (as well as p4 and lfs) but still continuing:
...
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi
end_group "Install dependencies"
but because ci/lib.sh does "set -e", we fail way before we hit this code. I am tempted to suggest we remove that "set -e" as a long term maintainability improvement measure, but that is a separate topic.
ci/install-dependencies.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +3 -2
@@ -74,8 +74,9 @@ ubuntu-*|i386/ubuntu-*|debian-*) -C "$CUSTOM_PATH" --strip-components=1 "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" - wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" - chmod a+x "$CUSTOM_PATH/jgit" + wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" && + chmod a+x "$CUSTOM_PATH/jgit" || + echo >&2 "JGit download failed, but we do not care and keep going" ;; esac ;;
PATCH 1/1 ci: skip unavailable external software
The ci/install-dependencies.sh script used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" in ci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.
Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi
will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".
--- ci/install-dependencies.sh | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)
ci/install-dependencies.sh +22 -9
@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*) mkdir --parents "$CUSTOM_PATH" wget --quiet --directory-prefix="$CUSTOM_PATH" \ - "$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4" - chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" - - wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" + "$P4WHENCE/bin.linux26x86_64/p4d" \ + "$P4WHENCE/bin.linux26x86_64/p4" && + chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || { + rm -f "$CUSTOM_PATH/p4" + rm -f "$CUSTOM_PATH/p4d" + echo >&2 "P4 download (optional) failed" + } + + wget --quiet \ + "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" && tar -xzf "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" \ - -C "$CUSTOM_PATH" --strip-components=1 "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" - rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" - - wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" - chmod a+x "$CUSTOM_PATH/jgit" + -C "$CUSTOM_PATH" --strip-components=1 \ + "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" && + rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" || { + rm -f "$CUSTOM_PATH/git-lfs" + echo >&2 "LFS download (optional) failed" + } + + wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" && + chmod a+x "$CUSTOM_PATH/jgit" || { + rm -f "$CUSTOM_PATH/jgit" + echo >&2 "JGit download (optional) failed" + } ;; esac ;;
Signature · 1 line
2.49.0-555-g089a0e42c4
Re: [PATCH] ci: skip unavailable external software
Quoted 20 lines · Junio C Hamano
The
ci/install-dependencies.shscript used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" inci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi
Quoted 2 lines
will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".
Yeah, I think this is a sensible direction to go. It is unfortunate that this may lead to silent breakage of these dependencies unless somebody explicitly looks for those warnings. But that feels like the lesser evil compared to failing the whole pipeline.
Quoted 14 lines
Signed-off-by Junio C Hamano <[email protected]>--- ci/install-dependencies.sh | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)ci/install-dependencies.sh +0
-1@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*) mkdir --parents "$CUSTOM_PATH" wget --quiet --directory-prefix="$CUSTOM_PATH" \ - "$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4"
ci/install-dependencies.sh ubuntu-*|i386/ubuntu-*|debian-*) · line 75
Quoted 10 lines
ci/install-dependencies.sh +7
-3- chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" - - wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" + "$P4WHENCE/bin.linux26x86_64/p4d" \ + "$P4WHENCE/bin.linux26x86_64/p4" && + chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || { + rm -f "$CUSTOM_PATH/p4" + rm -f "$CUSTOM_PATH/p4d" + echo >&2 "P4 download (optional) failed" + }
I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.
Patrick
Re: [PATCH] ci: skip unavailable external software
Quoted 4 lines · Patrick Steinhardt
I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.
You may think so (as I thought so too before hitting a snag or two) ;-).
The thing is, failing "wget --output-document" leaves an empty file, and an empty file does not still cause "if type jgit; then" to take the "else" clause, and the invocation of "jgit version" would fail, taking the whole thing out, due to the "set -e" thing.
Re: [PATCH] ci: skip unavailable external software
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Patrick Steinhardt
I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.
Quoted 6 lines
You may think so (as I thought so too before hitting a snag or two) ;-).
The thing is, failing "wget --output-document" leaves an empty file, and an empty file does not still cause "if type jgit; then" to take the "else" clause, and the invocation of "jgit version" would fail, taking the whole thing out, due to the "set -e" thing.
But shouldn't the failing wget cause an error, too? So the || { } cleanup branch would execute in that case and we can prune the empty file there. So in other words, shouldn't the following work alright?
if wget --output-document=...
then
massage output
else
rm output
fiOr am I still missing the obvious?
Patrick
Re: [PATCH] ci: skip unavailable external software
In reply to Patrick SteinhardtHi Patrick,
Quoted 50 lines · Patrick Steinhardt
Quoted 20 lines · Junio C Hamano
The
ci/install-dependencies.shscript used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" inci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fiQuoted 2 lines
will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".
Yeah, I think this is a sensible direction to go. It is unfortunate that this may lead to silent breakage of these dependencies unless somebody explicitly looks for those warnings. But that feels like the lesser evil compared to failing the whole pipeline.
Quoted 21 lines
Signed-off-by Junio C Hamano <[email protected]>--- ci/install-dependencies.sh | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)ci/install-dependencies.sh +4
-4@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*) mkdir --parents "$CUSTOM_PATH" wget --quiet --directory-prefix="$CUSTOM_PATH" \ - "$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4" - chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" - - wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" + "$P4WHENCE/bin.linux26x86_64/p4d" \ + "$P4WHENCE/bin.linux26x86_64/p4" && + chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || { + rm -f "$CUSTOM_PATH/p4"
Quoted 8 lines
ci/install-dependencies.sh ubuntu-*|i386/ubuntu-*|debian-*) · line 75
Quoted 3 lines
ci/install-dependencies.sh +3
-0+ rm -f "$CUSTOM_PATH/p4d" + echo >&2 "P4 download (optional) failed" + }I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.
That is true. It would probably also make sense to mark the message as a ::warning:: on GitHub (and the equivalent on GitLab), so that it is shown a bit more prominently in the CI summary.
Further, as per Matthias Sohn's (i.e. the JGit maintainer's) recommendation at
https://discord.com/channels/1042895022950994071/1364872237710184520/1364886912044765216,
the JGit download link in particular should probably be changed to
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
(which would work around the CI failures as well and could take the pressure off of working on more graceful dependency management in Git's CI, but then, we don't need more time to discuss the patch in this here thread because it already was fast-tracked to master).
Ciao,
Johannes
Re: [PATCH] ci: skip unavailable external software
In reply to Patrick SteinhardtQuoted 4 lines · Patrick Steinhardt
But shouldn't the failing wget cause an error, too? So the
|| { }cleanup branch would execute in that case and we can prune the empty file there. So in other words, shouldn't the following work alright?
Quoted 8 lines
if wget --output-document=... then massage output else rm output fiOr am I still missing the obvious?
While the above "works", what is "obvious" is that it is way too verbose and the merit of going verbose is dubious, especially given that the reason that trigger the argument to favor the above construct over the more concise
wget && massage || rm
is "massage part should never fail".
We do want to notice a failure in something, if that something is what should never fail, don't we? Not necessarily so!
Our CI jobs are not in the business of checking and ensuring wget or chmod keeps working. If either of them fail, the more important part is its practical impact to the rest of the CI job---resulting "jgit" file is harmful to be left on disk and needs to be removed.
Another to think about this is to imagine if we are having this conversation, had "wget" had (just like its --output-document argument) a "--chmod" argument. (wget && massage) as a unit is conceptually a single "download the jgit binary" in this case, and if either of them fail, we failed to download it.
So, yes, either would "work", but I do not think longhand is warranted in this case.
Re: [PATCH] ci: skip unavailable external software
In reply to Johannes SchindelinQuoted 3 lines · Johannes Schindelin
That is true. It would probably also make sense to mark the message as a
::warning::on GitHub (and the equivalent on GitLab), so that it is shown a bit more prominently in the CI summary.
::warning:: may be a good idea, but I actually think we should remove the message when we say "ah wget && chmod failed, so let's remove". At the end of the same script, there already is "ah jgit is not there, so let's warn" code exists (and it would falsely say "jgit file exists but cannot be executed" and take the whole thing down if we do not remove after a failed download). The ::warning::
should belong there.
Thanks.
Quoted 23 lines · Junio C Hamano
As https://www.eclipsestatus.io/ reports there is no ETA for recovery, and due to the failure of downloading JGit material in an early stage of our CI jobs, our linux CI jobs are all failing, I am very tempted to apply the following to 'maint' immediately and propagate it all the way up to 'master', 'next', and 'seen'.
I would very very much appreciate additional thoughts and advices by anybody more involved in JGit community and more clueful than I am on the situation.
Anyway, what is somewhat funny is that at the end of this script, there is an attempt to notice and report the lack of jgit (as well as p4 and lfs) but still continuing:
...
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi
Quoted 7 lines
end_group"Install dependencies"but because
ci/lib.shdoes "set -e", we fail way before we hit this code. I am tempted to suggest we remove that "set -e" as a long term maintainability improvement measure, but that is a separate topic.
I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.
And if we have the caches, we could somehow avoid problems by hitting the cache when third-party services were down. However, I do not dive into the CI before, so there may be something wrong about my statement.
Thanks,
Jialuo
Quoted 3 lines · shejialuo
I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.
It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).
cover ci: update unavailable external software handling
In reply to Junio C HamanoSo here is a small update based on Dscho's suggestions to the change we fast-tracked down to 'maint' in order to work around CI failures due to eclipse.org's outage.
Junio C Hamano (2):
ci: update the message for unavailble third-party software
ci: download JGit from maven, not eclipse.org
ci/install-dependencies.sh | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)
Signature · 1 line
2.49.0-564-g9a5a794ec8
PATCH 1/2 ci: update the message for unavailble third-party software
An earlier fix added an extra message immediately after failing to download a third-party package. But near the end of the script, their availability is checked again and given a message.
Remove the new ones added with a recent fix, as they are redundant. If we were to add more places to download these software (e.g. for other platforms we currently do not download them on), the existing warnning near the end of the script will also trigger.
While at it, as Dscho suggests, rewrite the WARNING: label on the warning message to ::warning::, which presumably should be shown a bit more prominently in the CI summary.
--- ci/install-dependencies.sh | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
ci/install-dependencies.sh +7 -12
@@ -71,7 +71,6 @@ ubuntu-*|i386/ubuntu-*|debian-*) chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || { rm -f "$CUSTOM_PATH/p4" rm -f "$CUSTOM_PATH/p4d" - echo >&2 "P4 download (optional) failed" } wget --quiet \
@@ -79,16 +78,12 @@ ubuntu-*|i386/ubuntu-*|debian-*) tar -xzf "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" \ -C "$CUSTOM_PATH" --strip-components=1 \ "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" && - rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" || { - rm -f "$CUSTOM_PATH/git-lfs" - echo >&2 "LFS download (optional) failed" - } + rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" || + rm -f "$CUSTOM_PATH/git-lfs" wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" && - chmod a+x "$CUSTOM_PATH/jgit" || { - rm -f "$CUSTOM_PATH/jgit" - echo >&2 "JGit download (optional) failed" - } + chmod a+x "$CUSTOM_PATH/jgit" || + rm -f "$CUSTOM_PATH/jgit" ;; esac ;;
@@ -151,7 +146,7 @@ then echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" p4 -V else - echo >&2 "WARNING: perforce wasn't installed, see above for clues why" + echo >&2 "::warning:: perforce wasn't installed, see above for clues why" fi if type git-lfs >/dev/null 2>&1
@@ -159,7 +154,7 @@ then echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" git-lfs version else - echo >&2 "WARNING: git-lfs wasn't installed, see above for clues why" + echo >&2 "::warning:: git-lfs wasn't installed, see above for clues why" fi if type jgit >/dev/null 2>&1
@@ -167,7 +162,7 @@ then echo "$(tput setaf 6)JGit Version$(tput sgr0)" jgit version else - echo >&2 "WARNING: JGit wasn't installed, see above for clues why" + echo >&2 "::warning:: JGit wasn't installed, see above for clues why" fi end_group "Install dependencies"
Signature · 1 line
2.49.0-564-g9a5a794ec8
PATCH 2/2 ci: download JGit from maven, not eclipse.org
In reply to Junio C HamanoAs Matthias Sohn, JGit maintainer, recommends, update the JGit download link from repo.eclipse.org to a one in maven.org
--- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ci/install-dependencies.sh +1 -1
@@ -9,7 +9,7 @@ begin_group "Install dependencies" P4WHENCE=https://cdist2.perforce.com/perforce/r23.2 LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION -JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh +JGITWHENCE=https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh # Make sudo a no-op and execute the command directly when running as root. # While using sudo would be fine on most platforms when we are root already,
Signature · 1 line
2.49.0-564-g9a5a794ec8
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · shejialuo
I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.
It also feels wasteful to me that the CI jobs need to do a full
install-dependencies.shover and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).
That's right. I'll investigate how to implement this.
Quoted 7 lines · shejialuo
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · shejialuo
I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.
Quoted 6 lines
Quoted 4 lines
It also feels wasteful to me that the CI jobs need to do a full
install-dependencies.shover and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).That's right. I'll investigate how to implement this.
It would be nice if we could adapt the Linux-based jobs to use pre-seeded Docker images. The idea would be that those images are only built once and then used by later steps of the pipeline. In theory, this could even be extended so that we only rebuild images as-needed when something changes so that the images are reused for multiple pipelines.
Another big benefit would be that this results in a fully-reproducible environment for developers that can in theory be uploaded to a container registry. So if you see that something fails only with a specific job image, you can now trivially fetch that image and try to reproduce the issue in the exact same image as CI used.
Patrick
Quoted 13 lines · Patrick Steinhardt
Quoted 12 lines · shejialuo
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · shejialuo
I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.
It also feels wasteful to me that the CI jobs need to do a full
install-dependencies.shover and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).
Quoted 8 lines
Quoted 1 line
That's right. I'll investigate how to implement this.
It would be nice if we could adapt the Linux-based jobs to use pre-seeded Docker images. The idea would be that those images are only built once and then used by later steps of the pipeline. In theory, this could even be extended so that we only rebuild images as-needed when something changes so that the images are reused for multiple pipelines.
I agree that it would be better if we provide pre-built container image. But there is only one problem, which platform we should upload to? I somehow know that github container registry is free for public repository.
So, we may just push the image into github container registry. However, I am not a member of the Git organization. So, I need a help from ones who have the access.
Quoted 6 lines
Another big benefit would be that this results in a fully-reproducible environment for developers that can in theory be uploaded to a container registry. So if you see that something fails only with a specific job image, you can now trivially fetch that image and try to reproduce the issue in the exact same image as CI used.
That's right, we could easily set up the development environment to replicate the problem without setting up the environment in our own machine step by step.
Quoted 1 line
Patrick
Thanks,
Jialuo