gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
RFC PATCH 1/1 gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
To fix this problem, use "meson test --no-rebuild" instead of "make check" to avoid rebuilding all binaries every time. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
---
Marked as "RFC" since I'm not quite sure whether "meson test" has the same test coverage as "make check"... Paolo?
.gitlab-ci.yml | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-)
.gitlab-ci.yml +22 -19
@@ -34,6 +34,19 @@ include: make -j"$JOBS" $MAKE_CHECK_ARGS ; fi +.native_meson_test_job: + stage: test + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + script: + - cd build + - touch * + - make git-submodule-update + - if [ -x ../meson/meson.py ]; then + ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ; + else + meson test --no-rebuild -t 5 $MESON_TEST_ARGS ; + fi + .native_test_job_template: &native_test_job_definition stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
@@ -83,17 +96,15 @@ build-system-alpine: artifacts: expire_in: 2 days paths: - - .git-submodule-status - build check-system-alpine: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-alpine artifacts: true variables: IMAGE: alpine - MAKE_CHECK_ARGS: check acceptance-system-alpine: <<: *native_test_job_definition
@@ -118,13 +129,12 @@ build-system-ubuntu: - build check-system-ubuntu: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 - MAKE_CHECK_ARGS: check acceptance-system-ubuntu: <<: *native_test_job_definition
@@ -149,13 +159,12 @@ build-system-debian: - build check-system-debian: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-debian artifacts: true variables: IMAGE: debian-amd64 - MAKE_CHECK_ARGS: check # No targets are built here, just tools, docs, and unit tests. This # also feeds into the eventual documentation deployment steps later
@@ -194,13 +203,12 @@ build-system-fedora: - build check-system-fedora: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-fedora artifacts: true variables: IMAGE: fedora - MAKE_CHECK_ARGS: check acceptance-system-fedora: <<: *native_test_job_definition
@@ -226,13 +234,12 @@ build-system-centos: - build check-system-centos: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-centos artifacts: true variables: IMAGE: centos8 - MAKE_CHECK_ARGS: check acceptance-system-centos: <<: *native_test_job_definition
@@ -256,13 +263,12 @@ build-system-opensuse: - build check-system-opensuse: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-opensuse artifacts: true variables: IMAGE: opensuse-leap - MAKE_CHECK_ARGS: check acceptance-system-opensuse: <<: *native_test_job_definition
@@ -525,13 +531,12 @@ build-crypto-old-nettle: - build check-crypto-old-nettle: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-old-nettle artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check build-crypto-old-gcrypt:
@@ -546,13 +551,12 @@ build-crypto-old-gcrypt: - build check-crypto-old-gcrypt: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-old-gcrypt artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check build-crypto-only-gnutls:
@@ -567,13 +571,12 @@ build-crypto-only-gnutls: - build check-crypto-only-gnutls: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-only-gnutls artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check # We don't need to exercise every backend with every front-end build-trace-multi-user:
Signature · 1 line
2.27.0
check-block is not run by "meson test".
Paolo
Il dom 24 gen 2021, 08:58 Thomas Huth <[email protected]> ha scritto:
Quoted 187 lines
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
To fix this problem, use "meson test --no-rebuild" instead of "make check" to avoid rebuilding all binaries every time. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
Signed-off-by Thomas Huth <[email protected]>---
Marked as "RFC" since I'm not quite sure whether "meson test" has the same test coverage as "make check"... Paolo?.gitlab-ci.yml | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-).gitlab-ci.yml +22
-20@@ -34,6 +34,19 @@ include: make -j"$JOBS" $MAKE_CHECK_ARGS ; fi +.native_meson_test_job: + stage: test + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + script: + - cd build + - touch * + - make git-submodule-update + - if [ -x ../meson/meson.py ]; then + ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ; + else + meson test --no-rebuild -t 5 $MESON_TEST_ARGS ; + fi + .native_test_job_template: &native_test_job_definition stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest@@ -83,17 +96,15 @@ build-system-alpine: artifacts: expire_in: 2 days paths: - - .git-submodule-status - build check-system-alpine: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-alpine artifacts: true variables: IMAGE: alpine - MAKE_CHECK_ARGS: check acceptance-system-alpine: <<: *native_test_job_definition@@ -118,13 +129,12 @@ build-system-ubuntu: - build check-system-ubuntu: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 - MAKE_CHECK_ARGS: check acceptance-system-ubuntu: <<: *native_test_job_definition@@ -149,13 +159,12 @@ build-system-debian: - build check-system-debian: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-debian artifacts: true variables: IMAGE: debian-amd64 - MAKE_CHECK_ARGS: check # No targets are built here, just tools, docs, and unit tests. This # also feeds into the eventual documentation deployment steps later@@ -194,13 +203,12 @@ build-system-fedora: - build check-system-fedora: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-fedora artifacts: true variables: IMAGE: fedora - MAKE_CHECK_ARGS: check acceptance-system-fedora: <<: *native_test_job_definition@@ -226,13 +234,12 @@ build-system-centos: - build check-system-centos: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-centos artifacts: true variables: IMAGE: centos8 - MAKE_CHECK_ARGS: check acceptance-system-centos: <<: *native_test_job_definition@@ -256,13 +263,12 @@ build-system-opensuse: - build check-system-opensuse: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-system-opensuse artifacts: true variables: IMAGE: opensuse-leap - MAKE_CHECK_ARGS: check acceptance-system-opensuse: <<: *native_test_job_definition@@ -525,13 +531,12 @@ build-crypto-old-nettle: - build check-crypto-old-nettle: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-old-nettle artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check build-crypto-old-gcrypt:@@ -546,13 +551,12 @@ build-crypto-old-gcrypt: - build check-crypto-old-gcrypt: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-old-gcrypt artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check build-crypto-only-gnutls:@@ -567,13 +571,12 @@ build-crypto-only-gnutls: - build check-crypto-only-gnutls: - <<: *native_test_job_definition + extends: .native_meson_test_job needs: - job: build-crypto-only-gnutls artifacts: true variables: IMAGE: centos7 - MAKE_CHECK_ARGS: check # We don't need to exercise every backend with every front-end build-trace-multi-user: --2.27.0
Quoted 1 line · Paolo Bonzini
check-block is not run by "meson test".
Thanks! I've just send a v1 now which also runs the iotests again.
Thomas
Quoted 189 lines
Il dom 24 gen 2021, 08:58 Thomas Huth <[email protected] <mailto:[email protected]>> ha scritto:
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
To fix this problem, use "meson test --no-rebuild" instead of "make check" to avoid rebuilding all binaries every time. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
Signed-off-by Thomas Huth <[email protected] <mailto:[email protected]>>---
Marked as "RFC" since I'm not quite sure whether "meson test" has the same test coverage as "make check"... Paolo?.gitlab-ci.yml | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-)
diff --git
a/.gitlab-ci.ymlb/.gitlab-ci.yml
indexde3a3d25b5..c9fb11c325100644
---a/.gitlab-ci.yml
+++b/.gitlab-ci.yml
@@ -34,6 +34,19 @@ include:
make -j"$JOBS" $MAKE_CHECK_ARGS ;
fi+.native_meson_test_job:
+ stage: test
+ image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+ script:
+ - cd build
+ - touch *
+ - make git-submodule-update
+ - if [ -x../meson/meson.py]; then
+../meson/meson.pytest --no-rebuild -t 5 $MESON_TEST_ARGS ; + else
+ meson test --no-rebuild -t 5 $MESON_TEST_ARGS ; + fi
+
.native_test_job_template:&native_test_job_definition
stage: test
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
@@ -83,17 +96,15 @@ build-system-alpine:
artifacts:
expire_in: 2 days
paths:
- - .git-submodule-status
- buildcheck-system-alpine:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-alpine
artifacts: true
variables:
IMAGE: alpine
-MAKE_CHECK_ARGS: checkacceptance-system-alpine:
<<:*native_test_job_definition
@@ -118,13 +129,12 @@ build-system-ubuntu:
- buildcheck-system-ubuntu:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-ubuntu
artifacts: true
variables:
IMAGE: ubuntu2004
-MAKE_CHECK_ARGS: checkacceptance-system-ubuntu:
<<:*native_test_job_definition
@@ -149,13 +159,12 @@ build-system-debian:
- buildcheck-system-debian:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-debian
artifacts: true
variables:
IMAGE: debian-amd64
-MAKE_CHECK_ARGS: check# No targets are built here, just tools, docs, and unit tests. This # also feeds into the eventual documentation deployment steps later
@@ -194,13 +203,12 @@ build-system-fedora:
- buildcheck-system-fedora:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-fedora
artifacts: true
variables:
IMAGE: fedora
-MAKE_CHECK_ARGS: checkacceptance-system-fedora:
<<:*native_test_job_definition
@@ -226,13 +234,12 @@ build-system-centos:
- buildcheck-system-centos:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-centos
artifacts: true
variables:
IMAGE: centos8
-MAKE_CHECK_ARGS: checkacceptance-system-centos:
<<:*native_test_job_definition
@@ -256,13 +263,12 @@ build-system-opensuse:
- buildcheck-system-opensuse:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-opensuse
artifacts: true
variables:
IMAGE: opensuse-leap
-MAKE_CHECK_ARGS: checkacceptance-system-opensuse:
<<:*native_test_job_definition
@@ -525,13 +531,12 @@ build-crypto-old-nettle:
- buildcheck-crypto-old-nettle:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-old-nettle
artifacts: true
variables:
IMAGE: centos7
-MAKE_CHECK_ARGS: checkbuild-crypto-old-gcrypt:
@@ -546,13 +551,12 @@ build-crypto-old-gcrypt:
- buildcheck-crypto-old-gcrypt:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-old-gcrypt
artifacts: true
variables:
IMAGE: centos7
-MAKE_CHECK_ARGS: checkbuild-crypto-only-gnutls:
@@ -567,13 +571,12 @@ build-crypto-only-gnutls:
- buildcheck-crypto-only-gnutls:
- <<:*native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-only-gnutls
artifacts: true
variables:
IMAGE: centos7
-MAKE_CHECK_ARGS: check# We don't need to exercise every backend with every front-end build-trace-multi-user:
--
2.27.0