Skip to contentSkip to conversation

travis: Use caching when possible

PATCH In review Philippe Mathieu-Daudé 3 participants 13 messages Sep 3, 2019 – Oct 14, 2020

lore
Share & tools
lore
#0 lore raw

cover travis: Use caching when possible

This series tries to reduce Travis CI failures due to network issues, and tries to speed builds a bit, reducing failures due to job timeouting.

Travis feature: https://docs.travis-ci.com/user/caching/

To see improvement over time, I recommend to apply this series on top of different merge commit on master, to see changes in cache use.

Few patches come from:
https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.html

Regards,

Phil.

Philippe Mathieu-Daudé (10):
.travis.yml: Enable multiple caching features .travis.yml: Increase cache timeout from 3min to 20min .travis.yml: Cache Python PIP packages
.travis.yml: Cache Avocado cache
.travis.yml: Improve ccache use
.travis.yml: Enable ccache on OSX
.travis.yml: Document how the build matrix use caches .travis.yml: Cache Linux/GCC 'debug profile' jobs together .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together .travis.yml: Cache Linux/Clang jobs together

 .travis.yml | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/10 .travis.yml: Enable multiple caching features

From: Philippe Mathieu-Daudé <[email protected]>

Using the 'multiple caching features' means explode the YAML array, thus it eases the git workflow (it is easier to move patches around).

See https://docs.travis-ci.com/user/caching#enabling-multiple-caching-features

Message-Id: <[email protected]>

---
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
.travis.yml +2 -1
index caf0a1f8fa..7478d758e1 100644
@@ -5,7 +5,8 @@ dist: xenial
 language: c
 compiler:
   - gcc
-cache: ccache
+cache:
+  ccache: true
 
 
 addons:
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/10 .travis.yml: Increase cache timeout from 3min to 20min

In reply to Philippe Mathieu-Daudé

We are going to cache few gigabytes, increase the cache timeout to avoid build failures when uploading our cache.

See https://docs.travis-ci.com/user/caching/#setting-the-timeout

---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)
.travis.yml +1 -0
index 7478d758e1..a65d13a592 100644
@@ -6,6 +6,7 @@ language: c
 compiler:
   - gcc
 cache:
+  timeout: 1200
   ccache: true
 
 
Signature · 1 line
2.20.1
#3 lore raw

PATCH 3/10 .travis.yml: Cache Python PIP packages

In reply to Philippe Mathieu-Daudé

We always install the same packages ever and ever, cache them.

See https://docs.travis-ci.com/user/caching/#pip-cache

---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)
.travis.yml +1 -0
index a65d13a592..3211667adb 100644
@@ -8,6 +8,7 @@ compiler:
 cache:
   timeout: 1200
   ccache: true
+  pip: true
 
 
 addons:
Signature · 1 line
2.20.1
#4 lore raw

PATCH 4/10 .travis.yml: Cache Avocado cache

In reply to Philippe Mathieu-Daudé

Avocado tests download artifacts from various sources. These sources sometime have network issues resulting in build failures. Cache Avocado cache to reduce build failure.

See https://docs.travis-ci.com/user/caching/#arbitrary-directories

---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index 3211667adb..a3db5b3f63 100644
@@ -9,6 +9,8 @@ cache:
   timeout: 1200
   ccache: true
   pip: true
+  directories:
+  - $HOME/avocado/data/cache
 
 
 addons:
Signature · 1 line
2.20.1
#5 lore raw

PATCH 5/10 .travis.yml: Improve ccache use

In reply to Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

Per https://ccache.dev/manual/latest.html:

By default, ccache tries to give as few false cache hits as possible. However, in certain situations it’s possible that you know things that ccache can’t take for granted.

[The CCACHE_SLOPINESS environment variable] makes it possible to tell ccache to relax some checks in order to increase the hit rate.

We can relax the ctime/mtime header checks:

- include_file_ctime

By default, ccache also will not cache a file if it includes a header whose ctime is too new. This option disables that check.

- include_file_mtime

By default, ccache will not cache a file if it includes a header whose mtime is too new. This option disables that check.

We also add a call to clear the cache statistics before running the build, and display them when the build finishes.

See https://docs.travis-ci.com/user/caching/#ccache-cache

---
 .travis.yml | 6 ++++++
 1 file changed, 6 insertions(+)
.travis.yml +6 -0
index a3db5b3f63..fccc1e091d 100644
@@ -70,6 +70,9 @@ env:
     - TEST_CMD="make check -j3 V=1"
     # This is broadly a list of "mainline" softmmu targets which have support across the major distros
     - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+    - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
+    - CCACHE_MAXSIZE=1G
+
 
 git:
   # we want to do this ourselves
@@ -77,10 +80,13 @@ git:
 
 
 before_script:
+  - command -v ccache && ccache --zero-stats
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
 script:
   - make -j3 && travis_retry ${TEST_CMD}
+after_script:
+  - command -v ccache && ccache --show-stats
 
 
 matrix:
Signature · 1 line
2.20.1
#6 lore raw

PATCH 6/10 .travis.yml: Enable ccache on OSX

In reply to Philippe Mathieu-Daudé

By default, ccache is not installed on macOS environments.

See https://docs.travis-ci.com/user/caching/#ccache-on-macos

---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index fccc1e091d..640d84f201 100644
@@ -45,6 +45,7 @@ addons:
       - gcovr
   homebrew:
     packages:
+      - ccache
       - glib
       - pixman
       - gnu-sed
@@ -80,6 +81,7 @@ git:
 
 
 before_script:
+  - if [ "$TRAVIS_OS_NAME" == "osx" ] ; then export PATH="/usr/local/opt/ccache/libexec:$PATH" ; fi
   - command -v ccache && ccache --zero-stats
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
Signature · 1 line
2.20.1
#7 lore raw

PATCH 7/10 .travis.yml: Document how the build matrix use caches

In reply to Philippe Mathieu-Daudé

We will set the CACHE_NAME variable to improve the caching of various jobs using the same characteristics. Document it first.

See https://docs.travis-ci.com/user/caching/#caches-and-build-matrices

---
 .travis.yml | 6 ++++++
 1 file changed, 6 insertions(+)
.travis.yml +6 -0
index 640d84f201..c64a551829 100644
@@ -6,6 +6,12 @@ language: c
 compiler:
   - gcc
 cache:
+  # There is one cache per branch and compiler version.
+  # characteristics of each job are used to identify the cache:
+  # - OS name (currently, linux, osx, or windows)
+  # - OS distribution (for Linux, xenial, trusty, or precise)
+  # - macOS image name (e.g., xcode7.2)
+  # - Names and values of visible environment variables set in .travis.yml or Settings panel
   timeout: 1200
   ccache: true
   pip: true
Signature · 1 line
2.20.1
#8 lore raw

PATCH 8/10 .travis.yml: Cache Linux/GCC 'debug profile' jobs together

In reply to Philippe Mathieu-Daudé

These jobs build different components but use the same host features. Put them in the same cache bucket.

---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index c64a551829..e362c3fe02 100644
@@ -120,11 +120,13 @@ matrix:
 
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions
     - env:
         - CONFIG="--enable-debug-tcg --disable-system"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     - env:
Signature · 1 line
2.20.1
#9 lore raw

CACHE_NAME in .travis.yml (was Re: [Qemu-devel] [PATCH 08/10] .travis.yml: Cache Linux/GCC 'debug profile' jobs together)

I just stumbled upon this when editing .travis.yml, and I'm trying to understand how the $CACHE_NAME trick is supposed to work.

If I understood the Travis documentation correctly, all environment variables will affect the cache id (including $CONFIG, in the case below). This would mean the two entries would still use a different cache even if $CACHE_NAME is the same.

See, for example:
https://travis-ci.org/github/qemu/qemu/jobs/735738551#L965
https://travis-ci.org/github/qemu/qemu/jobs/735738553#L967

Both build tasks have the same value for $CACHE_NAME, but they are fetching two different cache URLs:

master/cache--linux-focal-50ba7438064455bdb04bb41dcde2c91c34bb797265fe435b84cee4bae7d9b61d--compiler-gcc.tgz master/cache--linux-focal-8991cfc9b7f3f268e7e50aff4218daa4c206597ae796c60643a84ed58bba0cfb--compiler-gcc.tgz

Quoted 30 lines · Philippe Mathieu-Daudé

These jobs build different components but use the same host features. Put them in the same cache bucket.

---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
.travis.yml +2 -0
index c64a551829..e362c3fe02 100644
@@ -120,11 +120,13 @@ matrix:
 
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions
     - env:
         - CONFIG="--enable-debug-tcg --disable-system"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     - env:
Signature · 1 line
2.20.1
Signature · 1 line
Eduardo
#10 lore raw

PATCH 9/10 .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together

In reply to Philippe Mathieu-Daudé

These jobs build different components but use the same host features. Put them in the same cache bucket.

---
 .travis.yml | 13 +++++++++++++
 1 file changed, 13 insertions(+)
.travis.yml +13 -0
index e362c3fe02..7132cf8fba 100644
@@ -101,15 +101,18 @@ matrix:
   include:
     - env:
         - CONFIG="--disable-system --static"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # we split the system builds as it takes a while to build them all
     - env:
         - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     - env:
         - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # Just build tools and run minimal unit and softfloat checks
@@ -117,6 +120,8 @@ matrix:
         - BASE_CONFIG="--enable-tools"
         - CONFIG="--disable-user --disable-system"
         - TEST_CMD="make check-unit check-softfloat -j3"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
+
 
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
@@ -136,6 +141,7 @@ matrix:
     # Module builds are mostly of interest to major distros
     - env:
         - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # Alternate coroutines implementations are only really of interest to KVM users
@@ -155,6 +161,7 @@ matrix:
         - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
         - BASE_CONFIG="--enable-tools --enable-docs"
         - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       addons:
         apt:
           packages:
@@ -196,6 +203,7 @@ matrix:
     # We manually include builds which we disable "make check" for
     - env:
         - CONFIG="--without-default-devices --disable-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
         - TEST_CMD=""
 
 
@@ -239,6 +247,7 @@ matrix:
     # Python builds
     - env:
         - CONFIG="--target-list=x86_64-softmmu"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       language: python
       python:
         - "3.4"
@@ -246,6 +255,7 @@ matrix:
 
     - env:
         - CONFIG="--target-list=x86_64-softmmu"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       language: python
       python:
         - "3.6"
@@ -312,8 +322,11 @@ matrix:
     - env:
         - CONFIG="--disable-system"
         - TEST_CMD="make -j3 check-tcg V=1"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
+
 
     # Run check-tcg against softmmu targets
     - env:
         - CONFIG="--target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
         - TEST_CMD="make -j3 check-tcg V=1"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
Signature · 1 line
2.20.1
#11 lore raw

PATCH 10/10 .travis.yml: Cache Linux/Clang jobs together

In reply to Philippe Mathieu-Daudé

These jobs build different components but use the same host features. Put them in the same cache bucket.

---
 .travis.yml | 3 +++
 1 file changed, 3 insertions(+)
.travis.yml +3 -0
index 7132cf8fba..bd891fe284 100644
@@ -173,11 +173,13 @@ matrix:
     # Test with Clang for compile portability (Travis uses clang-5.0)
     - env:
         - CONFIG="--disable-system"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
     - env:
         - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
@@ -190,6 +192,7 @@ matrix:
 
     - env:
         - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
Signature · 1 line
2.20.1
#12 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 3 lines · Philippe Mathieu-Daudé

This series tries to reduce Travis CI failures due to network issues, and tries to speed builds a bit, reducing failures due to job timeouting.

Looking good.

Queued to testing/next, thanks.

Quoted 26 lines

Travis feature: https://docs.travis-ci.com/user/caching/

To see improvement over time, I recommend to apply this series on top of different merge commit on master, to see changes in cache use.

Few patches come from:
https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.html

Regards,

Phil.

Philippe Mathieu-Daudé (10):
.travis.yml: Enable multiple caching features .travis.yml: Increase cache timeout from 3min to 20min .travis.yml: Cache Python PIP packages
.travis.yml: Cache Avocado cache
.travis.yml: Improve ccache use
.travis.yml: Enable ccache on OSX
.travis.yml: Document how the build matrix use caches .travis.yml: Cache Linux/GCC 'debug profile' jobs together .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together .travis.yml: Cache Linux/Clang jobs together

Quoted 2 lines
 .travis.yml | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

--
Alex Bennée