travis-ci: no longer use containers
PATCH 1/1 travis-ci: no longer use containers
Travis CI will soon deprecate the container-based infrastructure enabled by sudo: false in ce59dffb34190e780be2fa9f449f842cadee9753.
More info:
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructures
--- .travis.yml | 2 -- 1 file changed, 2 deletions(-)
.travis.yml +0 -2
@@ -1,7 +1,5 @@ language: c -sudo: false - cache: directories: - $HOME/travis-cache
--
2.19.1
Quoted 5 lines · Sebastian Staudt
Travis CI will soon deprecate the container-based infrastructure enabled by
sudo: falseince59dffb34190e780be2fa9f449f842cadee9753.More info:
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructures
Thanks for posting a patch that would serve as a good discussion starter. This is not a criticism on your patch, but more is a RFD to those who helped our use of Travis by contributing to .travis.yml and ci/.
Don't we need to do some other things so that we can run in vm environment, rather than in container environment, before doing this change? IOW, aren't we doing in .travis.yml something we can do only in container but not in vm (if there is any), and if so, shouldn't we be rewriting that something so that we can run in vm?
I know ce59dffb ("travis-ci: explicity use container-based infrastructure", 2016-01-26) only added "sudo: false" without doing anything else (e.g. adding things that are only available to those who run in container), but if we added stuff that are not usable in vm environment after that commit since then, we need to adjust them so that we can migrate to the container-based environment, no?
To me, removing that "sudo: false" line seems like the least thing we need to worry about. After all, they say that whether we have "sudo: false" or not, the CI jobs will start running in vm environment and not in container. So if the rest of .travis.yml is ready to run in vm environment, we do not have to do anything ;-).
In short, my question to Lars and SZEDER is, are we already prepared to be thrown into a vm environment?
If the answer is "yes", then I think removing "sudo: false" is probably still a good thing to do for documentation purposes (i.e. showing that we knew we are ready to go through their migration).
Thanks.
Quoted 19 lines
Signed-off-by Sebastian Staudt <[email protected]>--- .travis.yml | 2 -- 1 file changed, 2 deletions(-).travis.yml +0
-3@@ -1,7 +1,5 @@ language: c -sudo: false - cache: directories: - $HOME/travis-cache --2.19.1
Quoted 13 lines · Junio C Hamano
Quoted 5 lines · Sebastian Staudt
Travis CI will soon deprecate the container-based infrastructure enabled by
sudo: falseince59dffb34190e780be2fa9f449f842cadee9753.More info:
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructuresThanks for posting a patch that would serve as a good discussion starter. This is not a criticism on your patch, but more is a RFD to those who helped our use of Travis by contributing to .travis.yml and ci/.
Quoted 5 lines
Don't we need to do some other things so that we can run in vm environment, rather than in container environment, before doing this change? IOW, aren't we doing in .travis.yml something we can do only in container but not in vm (if there is any), and if so, shouldn't we be rewriting that something so that we can run in vm?
As far as I understand, the container-based infrastructure has only one benefit over the VMs, the shorter startup time.
OTOH, in VMs we can use sudo, which is not available in the container-based intra. This has the benefit that after switching to VMs, we'll be able to install packages by running 'sudo apt-get install ...'. Currently the necessary packages are listed in '.travis.yml' for Travis CI, while for Azure the whole install command is embedded in '.azure....yml'. After the switch we could consolidate installing packages by 'sudo apt-get...' in 'ci/install-dependencies.sh' for both.
Quoted 7 lines
I know
ce59dffb("travis-ci: explicity use container-based infrastructure", 2016-01-26) only added "sudo: false" without doing anything else (e.g. adding things that are only available to those who run in container), but if we added stuff that are not usable in vm environment after that commit since then, we need to adjust them so that we can migrate to the container-based environment, no?
Quoted 8 lines
To me, removing that "sudo: false" line seems like the least thing we need to worry about. After all, they say that whether we have "sudo: false" or not, the CI jobs will start running in vm environment and not in container. So if the rest of .travis.yml is ready to run in vm environment, we do not have to do anything ;-).
In short, my question to Lars and SZEDER is, are we already prepared to be thrown into a vm environment?
I think we are. I've run only two builds with this patch, and they run smoothly and finished successfully. After you update 'pu' I'll run more.
Quoted 4 lines
If the answer is "yes", then I think removing "sudo: false" is probably still a good thing to do for documentation purposes (i.e. showing that we knew we are ready to go through their migration).
I agree.
Quoted 19 lines
Quoted 19 lines
Signed-off-by Sebastian Staudt <[email protected]>--- .travis.yml | 2 -- 1 file changed, 2 deletions(-).travis.yml +0
-3@@ -1,7 +1,5 @@ language: c -sudo: false - cache: directories: - $HOME/travis-cache --2.19.1
PATCH 1/1 travis-ci: install packages in 'ci/install-dependencies.sh'
Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon. While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves. With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well.
Let's make use of this and install necessary packages in 'ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file. Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well. Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using 'ci/install-dependencies.sh' to install packages, and neither of these two build jobs depend on and thus install those.
This change will presumably be beneficial for the upcoming Azure Pipelines integration [1]: preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running 'ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'.
---
This patch should go on top of 'ss/travis-ci-force-vm-mode'.
I'm not sure about the last paragraph, because:
- It talks about presumed benefits for a currently still work-in-progress patch series of an other contributor, and I'm not really sure that that's a good thing. Perhaps I should have rather put it below the '---'.
- I'm confused about the name of this Azure thing. The cover letter mentions "Azure Pipelines", the file is called 'azure-pipelines.yml', but the relevant patch I link to talks about "Azure DevOps" in the commit message.
Anyway, keep that last paragraph or drop it as you see fit.
.travis.yml | 21 --------------------- ci/install-dependencies.sh | 35 +++++++++++++++++++++++++++++------ 2 files changed, 29 insertions(+), 27 deletions(-)
.travis.yml +0 -21
@@ -12,16 +12,6 @@ compiler: - clang - gcc -addons: - apt: - sources: - - ubuntu-toolchain-r-test - packages: - - language-pack-is - - git-svn - - apache2 - - gcc-8 - matrix: include: - env: jobname=GETTEXT_POISON
@@ -50,22 +40,11 @@ matrix: - env: jobname=StaticAnalysis os: linux compiler: - addons: - apt: - packages: - - coccinelle - before_install: script: ci/run-static-analysis.sh after_failure: - env: jobname=Documentation os: linux compiler: - addons: - apt: - packages: - - asciidoc - - xmlto - before_install: script: ci/test-documentation.sh after_failure:
ci/install-dependencies.sh +29 -6
@@ -10,6 +10,15 @@ LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VE case "$jobname" in linux-clang|linux-gcc) + sudo apt-add-repository -y "ppa:ubuntu-toolchain-r/test" + sudo apt-get -q update + sudo apt-get -q -y install language-pack-is git-svn apache2 + case "$jobname" in + linux-gcc) + sudo apt-get -q -y install gcc-8 + ;; + esac + mkdir --parents "$P4_PATH" pushd "$P4_PATH" wget --quiet "$P4WHENCE/bin.linux26x86_64/p4d"
@@ -32,11 +41,25 @@ osx-clang|osx-gcc) brew link --force gettext brew install caskroom/cask/perforce ;; +StaticAnalysis) + sudo apt-get -q update + sudo apt-get -q -y install coccinelle + ;; +Documentation) + sudo apt-get -q update + sudo apt-get -q -y install asciidoc xmlto + ;; esac -echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" -p4d -V | grep Rev. -echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" -p4 -V | grep Rev. -echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" -git-lfs version +if type p4d >/dev/null && type p4 >/dev/null +then + echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" + p4d -V | grep Rev. + echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" + p4 -V | grep Rev. +fi +if type git-lfs >/dev/null +then + echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" + git-lfs version +fi
Signature · 1 line
2.19.1.838.g101e68a972
Re: [PATCH] travis-ci: install packages in 'ci/install-dependencies.sh'
Quoted 8 lines · SZEDER Gábor
Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon. While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves. With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well.
OK, so far we learned that this is now _doable_; but not enough to decide if this is a good thing to do or not. Let's read on to find out.
Quoted 4 lines
Let's make use of this and install necessary packages in '
ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file.
So we used to have two waysto prepare the test environment; non packaged software were done via install-dependencies.sh, but packaged ones weren't. Unifying them so that the script installs both would be a good change to simplify the procedure.
Is that how this sentence argues for this change?
Quoted 2 lines
Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well.
Is this "unneeded gcc-8 was installed" something we can fix only because we now stopped doing the installation via apt addon? Or we could have fixed it while we were on apt addon but we didn't bother, and this patch fixes it "while at it"---primarily because the shell script is far more flexible to work with than travis.yml matrix and this kind of customization is far easier to do?
Quoted 11 lines
Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using '
ci/install-dependencies.sh' to install packages, and neither of these two build jobs depend on and thus install those.This change will presumably be beneficial for the upcoming Azure Pipelines integration [1]: preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running '
ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'.
So the main point of this change is to have less knowledge to prepare the target configuration in the .travis.yml file and keep them all in ci/install-dependencies.sh, which hopefully is more reusable than .travis.yml in a non Travis environment?
If that is the case, it makes sense to me.
Quoted 9 lines
This patch should go on top of 'ss/travis-ci-force-vm-mode'.
I'm not sure about the last paragraph, because:
- It talks about presumed benefits for a currently still work-in-progress patch series of an other contributor, and I'm not really sure that that's a good thing. Perhaps I should have rather put it below the '---'.
Quoted 6 lines
- I'm confused about the name of this Azure thing. The cover letter mentions "Azure Pipelines", the file is called 'azure-pipelines.yml', but the relevant patch I link to talks about "Azure DevOps" in the commit message.
Anyway, keep that last paragraph or drop it as you see fit.
I hope we'll hear from Dscho in one or two revolutions of the Earth ;-)
Quoted 52 lines
ci/install-dependencies.sh +29
-6@@ -10,6 +10,15 @@ LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VE case "$jobname" in linux-clang|linux-gcc) + sudo apt-add-repository -y "ppa:ubuntu-toolchain-r/test" + sudo apt-get -q update + sudo apt-get -q -y install language-pack-is git-svn apache2 + case "$jobname" in + linux-gcc) + sudo apt-get -q -y install gcc-8 + ;; + esac + mkdir --parents "$P4_PATH" pushd "$P4_PATH" wget --quiet "$P4WHENCE/bin.linux26x86_64/p4d"@@ -32,11 +41,25 @@ osx-clang|osx-gcc) brew link --force gettext brew install caskroom/cask/perforce ;; +StaticAnalysis) + sudo apt-get -q update + sudo apt-get -q -y install coccinelle + ;; +Documentation) + sudo apt-get -q update + sudo apt-get -q -y install asciidoc xmlto + ;; esac -echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" -p4d -V | grep Rev. -echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" -p4 -V | grep Rev. -echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" -git-lfs version +if type p4d >/dev/null && type p4 >/dev/null +then + echo "$(tput setaf 6)Perforce Server Version$(tput sgr0)" + p4d -V | grep Rev. + echo "$(tput setaf 6)Perforce Client Version$(tput sgr0)" + p4 -V | grep Rev. +fi +if type git-lfs >/dev/null +then + echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)" + git-lfs version +fi
Re: [PATCH] travis-ci: install packages in 'ci/install-dependencies.sh'
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · SZEDER Gábor
Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon. While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves. With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well.
Quoted 3 lines
OK, so far we learned that this is now _doable_; but not enough to decide if this is a good thing to do or not. Let's read on to find out.
Yeah, this paragraph is just a bit of background about how the current situation came to be and what recent change made the switch possible.
Quoted 11 lines
Quoted 4 lines
Let's make use of this and install necessary packages in '
ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file.So we used to have two waysto prepare the test environment; non packaged software were done via
install-dependencies.sh, but packaged ones weren't. Unifying them so that the script installs both would be a good change to simplify the procedure.Is that how this sentence argues for this change?
Yes.
Quoted 5 lines
Quoted 2 lines
Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well.
Is this "unneeded gcc-8 was installed" something we can fix only because we now stopped doing the installation via apt addon?
Now that you mention it: no. It would have been possible to install gcc-8 only in the 'linux-gcc' build job even via the apt addon, namely by removing the two Linux build jobs from the implicit build matrix and adding them as two independent build jobs in the 'matrix.include' section of '.travis.yml'. The drawback is that all the extra packages used in both build jobs would have to be duplicated.
Quoted 5 lines
Or we
could have fixed it while we were on apt addon but we didn't bother, and this patch fixes it "while at it"---primarily because the shell script is far more flexible to work with than travis.yml matrix and this kind of customization is far easier to do?
Basically yes (though I think it's not about not bothering; I don't know about others, but it just occured to me that it would have been doable, however, even if it occured to me earlier, because of the duplicated list of common packages I wouldn't have done it).
Doing it in good old shell is indeed easier and the common packages are then only listed once.
Quoted 12 lines
Quoted 11 lines
Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using '
ci/install-dependencies.sh' to install packages, and neither of these two build jobs depend on and thus install those.This change will presumably be beneficial for the upcoming Azure Pipelines integration [1]: preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running '
ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'.
Quoted 4 lines
So the main point of this change is to have less knowledge to prepare the target configuration in the .travis.yml file and keep them all in
ci/install-dependencies.sh, which hopefully is more reusable than .travis.yml in a non Travis environment?
Oh, "more reusable" indeed, that's a more eloquent way to put it.
Quoted 16 lines
If that is the case, it makes sense to me.
Quoted 14 lines
This patch should go on top of 'ss/travis-ci-force-vm-mode'.
I'm not sure about the last paragraph, because:
- It talks about presumed benefits for a currently still work-in-progress patch series of an other contributor, and I'm not really sure that that's a good thing. Perhaps I should have rather put it below the '---'.
- I'm confused about the name of this Azure thing. The cover letter mentions "Azure Pipelines", the file is called 'azure-pipelines.yml', but the relevant patch I link to talks about "Azure DevOps" in the commit message.
Quoted 4 lines
Quoted 1 line
Anyway, keep that last paragraph or drop it as you see fit.
I hope we'll hear from Dscho in one or two revolutions of the Earth ;-)
... revolutions around what? :)
Re: [PATCH] travis-ci: install packages in 'ci/install-dependencies.sh'
Quoted 12 lines · SZEDER Gábor
Quoted 12 lines
Quoted 12 lines
I'm not sure about the last paragraph, because:
- It talks about presumed benefits for a currently still work-in-progress patch series of an other contributor, and I'm not really sure that that's a good thing. Perhaps I should have rather put it below the '---'.
- I'm confused about the name of this Azure thing. The cover letter mentions "Azure Pipelines", the file is called 'azure-pipelines.yml', but the relevant patch I link to talks about "Azure DevOps" in the commit message.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Anyway, keep that last paragraph or drop it as you see fit.
I hope we'll hear from Dscho in one or two revolutions of the Earth ;-)
... revolutions around what? :)
Originally I meant its own axis, but perhaps the moon.
Re: [PATCH] travis-ci: install packages in 'ci/install-dependencies.sh'
Hi,
Quoted 14 lines · Junio C Hamano
Quoted 12 lines · SZEDER Gábor
Quoted 12 lines
Quoted 12 lines
I'm not sure about the last paragraph, because:
- It talks about presumed benefits for a currently still work-in-progress patch series of an other contributor, and I'm not really sure that that's a good thing. Perhaps I should have rather put it below the '---'.
- I'm confused about the name of this Azure thing. The cover letter mentions "Azure Pipelines", the file is called 'azure-pipelines.yml', but the relevant patch I link to talks about "Azure DevOps" in the commit message.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Anyway, keep that last paragraph or drop it as you see fit.
I hope we'll hear from Dscho in one or two revolutions of the Earth ;-)
... revolutions around what? :)
Originally I meant its own axis, but perhaps the moon.
I see, you had fun talking about a revolution [*1*]...
I am much in favor of this patch, as it indeed will simplify the integration into Azure Pipelines.
As to the DevOps vs Pipelines thing: DevOps is the umbrella, it consists of much more than just the automated builds (Pipelines), it also has Boards, Packages, etc, but I don't think we will ever use that in the Git project, as our workflow is based out of a mailing list.
Ciao,
Dscho
Footnote *1*: https://www.youtube.com/watch?v=Xv8FBjo1Y8I
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Sebastian Staudt
Travis CI will soon deprecate the container-based infrastructure enabled by
sudo: falseince59dffb34190e780be2fa9f449f842cadee9753.
Quoted 7 lines
Quoted 2 lines
More info:
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructuresThanks for posting a patch that would serve as a good discussion starter. This is not a criticism on your patch, but more is a RFD to those who helped our use of Travis by contributing to .travis.yml and ci/.
In fact this was the intention while creating this patch. Although I see I could have made this a bit clearer in the initial message.
Having a patch that may cause a broken build or other CI problems seems more appropriate than waiting for Travis CI to flip the switch and searching for the problem afterwards.
Quoted 48 lines
Don't we need to do some other things so that we can run in vm environment, rather than in container environment, before doing this change? IOW, aren't we doing in .travis.yml something we can do only in container but not in vm (if there is any), and if so, shouldn't we be rewriting that something so that we can run in vm?
I know
ce59dffb("travis-ci: explicity use container-based infrastructure", 2016-01-26) only added "sudo: false" without doing anything else (e.g. adding things that are only available to those who run in container), but if we added stuff that are not usable in vm environment after that commit since then, we need to adjust them so that we can migrate to the container-based environment, no?To me, removing that "sudo: false" line seems like the least thing we need to worry about. After all, they say that whether we have "sudo: false" or not, the CI jobs will start running in vm environment and not in container. So if the rest of .travis.yml is ready to run in vm environment, we do not have to do anything ;-).
In short, my question to Lars and SZEDER is, are we already prepared to be thrown into a vm environment?
If the answer is "yes", then I think removing "sudo: false" is probably still a good thing to do for documentation purposes (i.e. showing that we knew we are ready to go through their migration).
Thanks.
Quoted 19 lines
Signed-off-by Sebastian Staudt <[email protected]>--- .travis.yml | 2 -- 1 file changed, 2 deletions(-).travis.yml +0
-3@@ -1,7 +1,5 @@ language: c -sudo: false - cache: directories: - $HOME/travis-cache --2.19.1