travis: Use caching when possible
cover travis: Use caching when possible
This series tries to reduce Travis CI failures due to network issues, and tries to speed builds a bit, reducing failures due to job timeouting.
Travis feature: https://docs.travis-ci.com/user/caching/
To see improvement over time, I recommend to apply this series on top of different merge commit on master, to see changes in cache use.
Few patches come from:
https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.html
Regards,
Phil.
Philippe Mathieu-Daudé (10):
.travis.yml: Enable multiple caching features .travis.yml: Increase cache timeout from 3min to 20min .travis.yml: Cache Python PIP packages
.travis.yml: Cache Avocado cache
.travis.yml: Improve ccache use
.travis.yml: Enable ccache on OSX
.travis.yml: Document how the build matrix use caches .travis.yml: Cache Linux/GCC 'debug profile' jobs together .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together .travis.yml: Cache Linux/Clang jobs together
.travis.yml | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
PATCH 1/10 .travis.yml: Enable multiple caching features
From: Philippe Mathieu-Daudé <[email protected]>
Using the 'multiple caching features' means explode the YAML array, thus it eases the git workflow (it is easier to move patches around).
See https://docs.travis-ci.com/user/caching#enabling-multiple-caching-features
Message-Id: <[email protected]>
--- .travis.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
.travis.yml +2 -1
@@ -5,7 +5,8 @@ dist: xenial language: c compiler: - gcc -cache: ccache +cache: + ccache: true addons:
Signature · 1 line
2.20.1
PATCH 2/10 .travis.yml: Increase cache timeout from 3min to 20min
In reply to Philippe Mathieu-DaudéWe are going to cache few gigabytes, increase the cache timeout to avoid build failures when uploading our cache.
See https://docs.travis-ci.com/user/caching/#setting-the-timeout
--- .travis.yml | 1 + 1 file changed, 1 insertion(+)
.travis.yml +1 -0
@@ -6,6 +6,7 @@ language: c compiler: - gcc cache: + timeout: 1200 ccache: true
Signature · 1 line
2.20.1
PATCH 3/10 .travis.yml: Cache Python PIP packages
In reply to Philippe Mathieu-DaudéWe always install the same packages ever and ever, cache them.
See https://docs.travis-ci.com/user/caching/#pip-cache
--- .travis.yml | 1 + 1 file changed, 1 insertion(+)
.travis.yml +1 -0
@@ -8,6 +8,7 @@ compiler: cache: timeout: 1200 ccache: true + pip: true addons:
Signature · 1 line
2.20.1
PATCH 4/10 .travis.yml: Cache Avocado cache
In reply to Philippe Mathieu-DaudéAvocado tests download artifacts from various sources. These sources sometime have network issues resulting in build failures. Cache Avocado cache to reduce build failure.
See https://docs.travis-ci.com/user/caching/#arbitrary-directories
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+)
.travis.yml +2 -0
@@ -9,6 +9,8 @@ cache: timeout: 1200 ccache: true pip: true + directories: + - $HOME/avocado/data/cache addons:
Signature · 1 line
2.20.1
PATCH 5/10 .travis.yml: Improve ccache use
In reply to Philippe Mathieu-DaudéFrom: Philippe Mathieu-Daudé <[email protected]>
Per https://ccache.dev/manual/latest.html:
By default, ccache tries to give as few false cache hits as possible. However, in certain situations it’s possible that you know things that ccache can’t take for granted.
[The CCACHE_SLOPINESS environment variable] makes it possible to tell ccache to relax some checks in order to increase the hit rate.
We can relax the ctime/mtime header checks:
- include_file_ctime
By default, ccache also will not cache a file if it includes a header whose ctime is too new. This option disables that check.
- include_file_mtime
By default, ccache will not cache a file if it includes a header whose mtime is too new. This option disables that check.
We also add a call to clear the cache statistics before running the build, and display them when the build finishes.
See https://docs.travis-ci.com/user/caching/#ccache-cache
--- .travis.yml | 6 ++++++ 1 file changed, 6 insertions(+)
.travis.yml +6 -0
@@ -70,6 +70,9 @@ env: - TEST_CMD="make check -j3 V=1" # This is broadly a list of "mainline" softmmu targets which have support across the major distros - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu" + - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime" + - CCACHE_MAXSIZE=1G + git: # we want to do this ourselves
@@ -77,10 +80,13 @@ git: before_script: + - command -v ccache && ccache --zero-stats - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR} - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; } script: - make -j3 && travis_retry ${TEST_CMD} +after_script: + - command -v ccache && ccache --show-stats matrix:
Signature · 1 line
2.20.1
PATCH 6/10 .travis.yml: Enable ccache on OSX
In reply to Philippe Mathieu-DaudéBy default, ccache is not installed on macOS environments.
See https://docs.travis-ci.com/user/caching/#ccache-on-macos
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+)
.travis.yml +2 -0
@@ -45,6 +45,7 @@ addons: - gcovr homebrew: packages: + - ccache - glib - pixman - gnu-sed
@@ -80,6 +81,7 @@ git: before_script: + - if [ "$TRAVIS_OS_NAME" == "osx" ] ; then export PATH="/usr/local/opt/ccache/libexec:$PATH" ; fi - command -v ccache && ccache --zero-stats - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR} - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
Signature · 1 line
2.20.1
PATCH 7/10 .travis.yml: Document how the build matrix use caches
In reply to Philippe Mathieu-DaudéWe will set the CACHE_NAME variable to improve the caching of various jobs using the same characteristics. Document it first.
See https://docs.travis-ci.com/user/caching/#caches-and-build-matrices
--- .travis.yml | 6 ++++++ 1 file changed, 6 insertions(+)
.travis.yml +6 -0
@@ -6,6 +6,12 @@ language: c compiler: - gcc cache: + # There is one cache per branch and compiler version. + # characteristics of each job are used to identify the cache: + # - OS name (currently, linux, osx, or windows) + # - OS distribution (for Linux, xenial, trusty, or precise) + # - macOS image name (e.g., xcode7.2) + # - Names and values of visible environment variables set in .travis.yml or Settings panel timeout: 1200 ccache: true pip: true
Signature · 1 line
2.20.1
PATCH 8/10 .travis.yml: Cache Linux/GCC 'debug profile' jobs together
In reply to Philippe Mathieu-DaudéThese jobs build different components but use the same host features. Put them in the same cache bucket.
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+)
.travis.yml +2 -0
@@ -120,11 +120,13 @@ matrix: - env: - CONFIG="--enable-debug --enable-debug-tcg --disable-user" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug" # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions - env: - CONFIG="--enable-debug-tcg --disable-system" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug" - env:
Signature · 1 line
2.20.1
CACHE_NAME in .travis.yml (was Re: [Qemu-devel] [PATCH 08/10] .travis.yml: Cache Linux/GCC 'debug profile' jobs together)
I just stumbled upon this when editing .travis.yml, and I'm trying to understand how the $CACHE_NAME trick is supposed to work.
If I understood the Travis documentation correctly, all environment variables will affect the cache id (including $CONFIG, in the case below). This would mean the two entries would still use a different cache even if $CACHE_NAME is the same.
See, for example:
https://travis-ci.org/github/qemu/qemu/jobs/735738551#L965
https://travis-ci.org/github/qemu/qemu/jobs/735738553#L967
Both build tasks have the same value for $CACHE_NAME, but they are fetching two different cache URLs:
master/cache--linux-focal-50ba7438064455bdb04bb41dcde2c91c34bb797265fe435b84cee4bae7d9b61d--compiler-gcc.tgz master/cache--linux-focal-8991cfc9b7f3f268e7e50aff4218daa4c206597ae796c60643a84ed58bba0cfb--compiler-gcc.tgz
Quoted 30 lines · Philippe Mathieu-Daudé
These jobs build different components but use the same host features. Put them in the same cache bucket.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+).travis.yml +2
-0@@ -120,11 +120,13 @@ matrix: - env: - CONFIG="--enable-debug --enable-debug-tcg --disable-user" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug" # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions - env: - CONFIG="--enable-debug-tcg --disable-system" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug" - env:Signature · 1 line
2.20.1
Signature · 1 line
Eduardo
PATCH 9/10 .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together
In reply to Philippe Mathieu-DaudéThese jobs build different components but use the same host features. Put them in the same cache bucket.
--- .travis.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+)
.travis.yml +13 -0
@@ -101,15 +101,18 @@ matrix: include: - env: - CONFIG="--disable-system --static" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" # we split the system builds as it takes a while to build them all - env: - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" - env: - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" # Just build tools and run minimal unit and softfloat checks
@@ -117,6 +120,8 @@ matrix: - BASE_CONFIG="--enable-tools" - CONFIG="--disable-user --disable-system" - TEST_CMD="make check-unit check-softfloat -j3" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" + - env: - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
@@ -136,6 +141,7 @@ matrix: # Module builds are mostly of interest to major distros - env: - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" # Alternate coroutines implementations are only really of interest to KVM users
@@ -155,6 +161,7 @@ matrix: - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.." - BASE_CONFIG="--enable-tools --enable-docs" - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" addons: apt: packages:
@@ -196,6 +203,7 @@ matrix: # We manually include builds which we disable "make check" for - env: - CONFIG="--without-default-devices --disable-user" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" - TEST_CMD=""
@@ -239,6 +247,7 @@ matrix: # Python builds - env: - CONFIG="--target-list=x86_64-softmmu" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" language: python python: - "3.4"
@@ -246,6 +255,7 @@ matrix: - env: - CONFIG="--target-list=x86_64-softmmu" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" language: python python: - "3.6"
@@ -312,8 +322,11 @@ matrix: - env: - CONFIG="--disable-system" - TEST_CMD="make -j3 check-tcg V=1" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" + # Run check-tcg against softmmu targets - env: - CONFIG="--target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu" - TEST_CMD="make -j3 check-tcg V=1" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
Signature · 1 line
2.20.1
PATCH 10/10 .travis.yml: Cache Linux/Clang jobs together
In reply to Philippe Mathieu-DaudéThese jobs build different components but use the same host features. Put them in the same cache bucket.
--- .travis.yml | 3 +++ 1 file changed, 3 insertions(+)
.travis.yml +3 -0
@@ -173,11 +173,13 @@ matrix: # Test with Clang for compile portability (Travis uses clang-5.0) - env: - CONFIG="--disable-system" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default" compiler: clang - env: - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default" compiler: clang
@@ -190,6 +192,7 @@ matrix: - env: - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}" + - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default" compiler: clang
Signature · 1 line
2.20.1
Quoted 3 lines · Philippe Mathieu-Daudé
This series tries to reduce Travis CI failures due to network issues, and tries to speed builds a bit, reducing failures due to job timeouting.
Looking good.
Queued to testing/next, thanks.
Quoted 26 lines
Travis feature: https://docs.travis-ci.com/user/caching/
To see improvement over time, I recommend to apply this series on top of different merge commit on master, to see changes in cache use.
Few patches come from:
https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.htmlRegards,
Phil.
Philippe Mathieu-Daudé (10):
.travis.yml: Enable multiple caching features .travis.yml: Increase cache timeout from 3min to 20min .travis.yml: Cache Python PIP packages
.travis.yml: Cache Avocado cache
.travis.yml: Improve ccache use
.travis.yml: Enable ccache on OSX
.travis.yml: Document how the build matrix use caches .travis.yml: Cache Linux/GCC 'debug profile' jobs together .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together .travis.yml: Cache Linux/Clang jobs together
Quoted 2 lines
.travis.yml | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
--
Alex Bennée