Skip to contentSkip to conversation

repo.eclipse.org outage breaking all our linux CI jobs

PATCH In review Junio C Hamano 4 participants 16 messages Apr 24, 2025 – Apr 28, 2025

lore
Share & tools
lore
#0 lore raw

As https://www.eclipsestatus.io/ reports there is no ETA for recovery, and due to the failure of downloading JGit material in an early stage of our CI jobs, our linux CI jobs are all failing, I am very tempted to apply the following to 'maint' immediately and propagate it all the way up to 'master', 'next', and 'seen'.

I would very very much appreciate additional thoughts and advices by anybody more involved in JGit community and more clueful than I am on the situation.

Anyway, what is somewhat funny is that at the end of this script, there is an attempt to notice and report the lack of jgit (as well as p4 and lfs) but still continuing:

...
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi

end_group "Install dependencies"

but because ci/lib.sh does "set -e", we fail way before we hit this code. I am tempted to suggest we remove that "set -e" as a long term maintainability improvement measure, but that is a separate topic.

 ci/install-dependencies.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
ci/install-dependencies.sh +3 -2
index be9ba5e30a..4dda5db7e5 100755
@@ -74,8 +74,9 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 			-C "$CUSTOM_PATH" --strip-components=1 "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs"
 		rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
 
-		wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit"
-		chmod a+x "$CUSTOM_PATH/jgit"
+		wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" &&
+		chmod a+x "$CUSTOM_PATH/jgit" ||
+		echo >&2 "JGit download failed, but we do not care and keep going"
 		;;
 	esac
 	;;
#1 lore raw

PATCH 1/1 ci: skip unavailable external software

The ci/install-dependencies.sh script used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" in ci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.

Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:

if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi

will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".

---
 ci/install-dependencies.sh | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)
ci/install-dependencies.sh +22 -9
index 0df74610d0..e51304c3b0 100755
@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 		mkdir --parents "$CUSTOM_PATH"
 
 		wget --quiet --directory-prefix="$CUSTOM_PATH" \
-			"$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4"
-		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4"
-
-		wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
+			"$P4WHENCE/bin.linux26x86_64/p4d" \
+			"$P4WHENCE/bin.linux26x86_64/p4" &&
+		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || {
+			rm -f "$CUSTOM_PATH/p4"
+			rm -f "$CUSTOM_PATH/p4d"
+			echo >&2 "P4 download (optional) failed"
+		}
+
+		wget --quiet \
+		     "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" &&
 		tar -xzf "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" \
-			-C "$CUSTOM_PATH" --strip-components=1 "git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs"
-		rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
-
-		wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit"
-		chmod a+x "$CUSTOM_PATH/jgit"
+			-C "$CUSTOM_PATH" --strip-components=1 \
+			"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" &&
+		rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" || {
+			rm -f "$CUSTOM_PATH/git-lfs"
+			echo >&2 "LFS download (optional) failed"
+		}
+
+		wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" &&
+		chmod a+x "$CUSTOM_PATH/jgit" || {
+			rm -f "$CUSTOM_PATH/jgit"
+			echo >&2 "JGit download (optional) failed"
+		}
 		;;
 	esac
 	;;
Signature · 1 line
2.49.0-555-g089a0e42c4
#2 lore raw

Re: [PATCH] ci: skip unavailable external software

Quoted 20 lines · Junio C Hamano

The ci/install-dependencies.sh script used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" in ci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.

Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:

if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi

Quoted 2 lines

will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".

Yeah, I think this is a sensible direction to go. It is unfortunate that this may lead to silent breakage of these dependencies unless somebody explicitly looks for those warnings. But that feels like the lesser evil compared to failing the whole pipeline.

Quoted 14 lines
---
 ci/install-dependencies.sh | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)
ci/install-dependencies.sh +0 -1
index 0df74610d0..e51304c3b0 100755
@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 		mkdir --parents "$CUSTOM_PATH"
 
 		wget --quiet --directory-prefix="$CUSTOM_PATH" \
-			"$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4"

ci/install-dependencies.sh ubuntu-*|i386/ubuntu-*|debian-*) · line 75

Quoted 10 lines
ci/install-dependencies.sh +7 -3
-		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4"
-
-		wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
+			"$P4WHENCE/bin.linux26x86_64/p4d" \
+			"$P4WHENCE/bin.linux26x86_64/p4" &&
+		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || {
+			rm -f "$CUSTOM_PATH/p4"
+			rm -f "$CUSTOM_PATH/p4d"
+			echo >&2 "P4 download (optional) failed"
+		}

I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.

Patrick

#3 lore raw

Re: [PATCH] ci: skip unavailable external software

Quoted 4 lines · Patrick Steinhardt

I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.

You may think so (as I thought so too before hitting a snag or two) ;-).

The thing is, failing "wget --output-document" leaves an empty file, and an empty file does not still cause "if type jgit; then" to take the "else" clause, and the invocation of "jgit version" would fail, taking the whole thing out, due to the "set -e" thing.

#4 lore raw

Re: [PATCH] ci: skip unavailable external software

Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Patrick Steinhardt

I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.

Quoted 6 lines

You may think so (as I thought so too before hitting a snag or two) ;-).

The thing is, failing "wget --output-document" leaves an empty file, and an empty file does not still cause "if type jgit; then" to take the "else" clause, and the invocation of "jgit version" would fail, taking the whole thing out, due to the "set -e" thing.

But shouldn't the failing wget cause an error, too? So the || { } cleanup branch would execute in that case and we can prune the empty file there. So in other words, shouldn't the following work alright?

    if wget --output-document=...
    then
        massage output
    else
        rm output
    fi

Or am I still missing the obvious?

Patrick

#5 lore raw

Re: [PATCH] ci: skip unavailable external software

Quoted 4 lines · Patrick Steinhardt

But shouldn't the failing wget cause an error, too? So the || { } cleanup branch would execute in that case and we can prune the empty file there. So in other words, shouldn't the following work alright?

Quoted 8 lines
    if wget --output-document=...
    then
        massage output
    else
        rm output
    fi

Or am I still missing the obvious?

While the above "works", what is "obvious" is that it is way too verbose and the merit of going verbose is dubious, especially given that the reason that trigger the argument to favor the above construct over the more concise

	wget && massage || rm

is "massage part should never fail".

We do want to notice a failure in something, if that something is what should never fail, don't we? Not necessarily so!

Our CI jobs are not in the business of checking and ensuring wget or chmod keeps working. If either of them fail, the more important part is its practical impact to the rest of the CI job---resulting "jgit" file is harmful to be left on disk and needs to be removed.

Another to think about this is to imagine if we are having this conversation, had "wget" had (just like its --output-document argument) a "--chmod" argument. (wget && massage) as a unit is conceptually a single "download the jgit binary" in this case, and if either of them fail, we failed to download it.

So, yes, either would "work", but I do not think longhand is warranted in this case.

#6 lore raw

Re: [PATCH] ci: skip unavailable external software

In reply to Patrick Steinhardt

Hi Patrick,

Quoted 50 lines · Patrick Steinhardt
Quoted 20 lines · Junio C Hamano

The ci/install-dependencies.sh script used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" in ci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.

Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:

if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi

Quoted 2 lines

will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".

Yeah, I think this is a sensible direction to go. It is unfortunate that this may lead to silent breakage of these dependencies unless somebody explicitly looks for those warnings. But that feels like the lesser evil compared to failing the whole pipeline.

Quoted 21 lines
---
 ci/install-dependencies.sh | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)
ci/install-dependencies.sh +4 -4
index 0df74610d0..e51304c3b0 100755
@@ -66,16 +66,29 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 		mkdir --parents "$CUSTOM_PATH"
 
 		wget --quiet --directory-prefix="$CUSTOM_PATH" \
-			"$P4WHENCE/bin.linux26x86_64/p4d" "$P4WHENCE/bin.linux26x86_64/p4"
-		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4"
-
-		wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
+			"$P4WHENCE/bin.linux26x86_64/p4d" \
+			"$P4WHENCE/bin.linux26x86_64/p4" &&
+		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || {
+			rm -f "$CUSTOM_PATH/p4"
Quoted 8 lines

ci/install-dependencies.sh ubuntu-*|i386/ubuntu-*|debian-*) · line 75

Quoted 3 lines
ci/install-dependencies.sh +3 -0
+			rm -f "$CUSTOM_PATH/p4d"
+			echo >&2 "P4 download (optional) failed"
+		}

I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.

That is true. It would probably also make sense to mark the message as a ::warning:: on GitHub (and the equivalent on GitLab), so that it is shown a bit more prominently in the CI summary.

Further, as per Matthias Sohn's (i.e. the JGit maintainer's) recommendation at
https://discord.com/channels/1042895022950994071/1364872237710184520/1364886912044765216,
the JGit download link in particular should probably be changed to
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
(which would work around the CI failures as well and could take the pressure off of working on more graceful dependency management in Git's CI, but then, we don't need more time to discuss the patch in this here thread because it already was fast-tracked to master).

Ciao,
Johannes

#7 lore raw

Re: [PATCH] ci: skip unavailable external software

Quoted 3 lines · Johannes Schindelin

That is true. It would probably also make sense to mark the message as a ::warning:: on GitHub (and the equivalent on GitLab), so that it is shown a bit more prominently in the CI summary.

::warning:: may be a good idea, but I actually think we should remove the message when we say "ah wget && chmod failed, so let's remove". At the end of the same script, there already is "ah jgit is not there, so let's warn" code exists (and it would falsely say "jgit file exists but cannot be executed" and take the whole thing down if we do not remove after a failed download). The ::warning::
should belong there.

Thanks.

#8 lore raw

cover ci: update unavailable external software handling

So here is a small update based on Dscho's suggestions to the change we fast-tracked down to 'maint' in order to work around CI failures due to eclipse.org's outage.

Junio C Hamano (2):
ci: update the message for unavailble third-party software
ci: download JGit from maven, not eclipse.org

 ci/install-dependencies.sh | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)
Signature · 1 line
2.49.0-564-g9a5a794ec8
#9 lore raw

PATCH 1/2 ci: update the message for unavailble third-party software

An earlier fix added an extra message immediately after failing to download a third-party package. But near the end of the script, their availability is checked again and given a message.

Remove the new ones added with a recent fix, as they are redundant. If we were to add more places to download these software (e.g. for other platforms we currently do not download them on), the existing warnning near the end of the script will also trigger.

While at it, as Dscho suggests, rewrite the WARNING: label on the warning message to ::warning::, which presumably should be shown a bit more prominently in the CI summary.

---
 ci/install-dependencies.sh | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
ci/install-dependencies.sh +7 -12
index e51304c3b0..be20271d3c 100755
@@ -71,7 +71,6 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 		chmod a+x "$CUSTOM_PATH/p4d" "$CUSTOM_PATH/p4" || {
 			rm -f "$CUSTOM_PATH/p4"
 			rm -f "$CUSTOM_PATH/p4d"
-			echo >&2 "P4 download (optional) failed"
 		}
 
 		wget --quiet \
@@ -79,16 +78,12 @@ ubuntu-*|i386/ubuntu-*|debian-*)
 		tar -xzf "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" \
 			-C "$CUSTOM_PATH" --strip-components=1 \
 			"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs" &&
-		rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" || {
-			rm -f "$CUSTOM_PATH/git-lfs"
-			echo >&2 "LFS download (optional) failed"
-		}
+		rm "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" ||
+		rm -f "$CUSTOM_PATH/git-lfs"
 
 		wget --quiet "$JGITWHENCE" --output-document="$CUSTOM_PATH/jgit" &&
-		chmod a+x "$CUSTOM_PATH/jgit" || {
-			rm -f "$CUSTOM_PATH/jgit"
-			echo >&2 "JGit download (optional) failed"
-		}
+		chmod a+x "$CUSTOM_PATH/jgit" ||
+		rm -f "$CUSTOM_PATH/jgit"
 		;;
 	esac
 	;;
@@ -151,7 +146,7 @@ then
 	echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)"
 	p4 -V
 else
-	echo >&2 "WARNING: perforce wasn't installed, see above for clues why"
+	echo >&2 "::warning:: perforce wasn't installed, see above for clues why"
 fi
 
 if type git-lfs >/dev/null 2>&1
@@ -159,7 +154,7 @@ then
 	echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)"
 	git-lfs version
 else
-	echo >&2 "WARNING: git-lfs wasn't installed, see above for clues why"
+	echo >&2 "::warning:: git-lfs wasn't installed, see above for clues why"
 fi
 
 if type jgit >/dev/null 2>&1
@@ -167,7 +162,7 @@ then
 	echo "$(tput setaf 6)JGit Version$(tput sgr0)"
 	jgit version
 else
-	echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
+	echo >&2 "::warning:: JGit wasn't installed, see above for clues why"
 fi
 
 end_group "Install dependencies"
Signature · 1 line
2.49.0-564-g9a5a794ec8
#10 lore raw

PATCH 2/2 ci: download JGit from maven, not eclipse.org

In reply to Junio C Hamano

As Matthias Sohn, JGit maintainer, recommends, update the JGit download link from repo.eclipse.org to a one in maven.org

---
 ci/install-dependencies.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ci/install-dependencies.sh +1 -1
index be20271d3c..d9004ab24f 100755
@@ -9,7 +9,7 @@ begin_group "Install dependencies"
 
 P4WHENCE=https://cdist2.perforce.com/perforce/r23.2
 LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
-JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
+JGITWHENCE=https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
 
 # Make sudo a no-op and execute the command directly when running as root.
 # While using sudo would be fine on most platforms when we are root already,
Signature · 1 line
2.49.0-564-g9a5a794ec8
#11 lore raw
In reply to Junio C Hamano
Quoted 23 lines · Junio C Hamano

As https://www.eclipsestatus.io/ reports there is no ETA for recovery, and due to the failure of downloading JGit material in an early stage of our CI jobs, our linux CI jobs are all failing, I am very tempted to apply the following to 'maint' immediately and propagate it all the way up to 'master', 'next', and 'seen'.

I would very very much appreciate additional thoughts and advices by anybody more involved in JGit community and more clueful than I am on the situation.

Anyway, what is somewhat funny is that at the end of this script, there is an attempt to notice and report the lack of jgit (as well as p4 and lfs) but still continuing:

...
if type jgit >/dev/null 2>&1
then
echo "$(tput setaf 6)JGit Version$(tput sgr0)"
jgit version
else
echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
fi

Quoted 7 lines

end_group "Install dependencies"

but because ci/lib.sh does "set -e", we fail way before we hit this code. I am tempted to suggest we remove that "set -e" as a long term maintainability improvement measure, but that is a separate topic.

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

And if we have the caches, we could somehow avoid problems by hitting the cache when third-party services were down. However, I do not dive into the CI before, so there may be something wrong about my statement.

Thanks,
Jialuo

#12 lore raw
Quoted 3 lines · shejialuo

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).

#13 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · shejialuo

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).

That's right. I'll investigate how to implement this.

#14 lore raw
Quoted 7 lines · shejialuo
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · shejialuo

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

Quoted 6 lines
Quoted 4 lines

It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).

That's right. I'll investigate how to implement this.

It would be nice if we could adapt the Linux-based jobs to use pre-seeded Docker images. The idea would be that those images are only built once and then used by later steps of the pipeline. In theory, this could even be extended so that we only rebuild images as-needed when something changes so that the images are reused for multiple pipelines.

Another big benefit would be that this results in a fully-reproducible environment for developers that can in theory be uploaded to a container registry. So if you see that something fails only with a specific job image, you can now trivially fetch that image and try to reproduce the issue in the exact same image as CI used.

Patrick

#15 lore raw
Quoted 13 lines · Patrick Steinhardt
Quoted 12 lines · shejialuo
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · shejialuo

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).

Quoted 8 lines
Quoted 1 line

That's right. I'll investigate how to implement this.

It would be nice if we could adapt the Linux-based jobs to use pre-seeded Docker images. The idea would be that those images are only built once and then used by later steps of the pipeline. In theory, this could even be extended so that we only rebuild images as-needed when something changes so that the images are reused for multiple pipelines.

I agree that it would be better if we provide pre-built container image. But there is only one problem, which platform we should upload to? I somehow know that github container registry is free for public repository.

So, we may just push the image into github container registry. However, I am not a member of the Git organization. So, I need a help from ones who have the access.

Quoted 6 lines

Another big benefit would be that this results in a fully-reproducible environment for developers that can in theory be uploaded to a container registry. So if you see that something fails only with a specific job image, you can now trivially fetch that image and try to reproduce the issue in the exact same image as CI used.

That's right, we could easily set up the development environment to replicate the problem without setting up the environment in our own machine step by step.

Quoted 1 line

Patrick

Thanks,
Jialuo