gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
PATCH 1/1 gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
So instead of trying to marry a freshly checked out source tree with the pre-built binaries in these jobs, let's simply pass the whole source including the submodules and the build tree as artifact to the test jobs. That way timestamps get preserved and there is no rebuild of the sources anymore. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
---
This is how a job looked like before my patch, running for 42 minutes:
https://gitlab.com/huth/qemu/-/jobs/978432757
And this is how it looks like afterwards - it just took 18 minutes:
https://gitlab.com/huth/qemu/-/jobs/979500316
.gitlab-ci.d/containers.yml | 1 + .gitlab-ci.yml | 40 +++++++++++++++++++++++++------------ 2 files changed, 28 insertions(+), 13 deletions(-)
.gitlab-ci.yml +27 -13
@@ -39,7 +39,6 @@ include: image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: - cd build - - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition
@@ -83,8 +82,7 @@ build-system-alpine: artifacts: expire_in: 2 days paths: - - .git-submodule-status - - build + - "*" check-system-alpine: <<: *native_test_job_definition
@@ -92,6 +90,7 @@ check-system-alpine: - job: build-system-alpine artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: alpine MAKE_CHECK_ARGS: check
@@ -101,6 +100,7 @@ acceptance-system-alpine: - job: build-system-alpine artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: alpine MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -116,7 +116,7 @@ build-system-ubuntu: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-ubuntu: <<: *native_test_job_definition
@@ -124,6 +124,7 @@ check-system-ubuntu: - job: build-system-ubuntu artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check
@@ -133,6 +134,7 @@ acceptance-system-ubuntu: - job: build-system-ubuntu artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -148,7 +150,7 @@ build-system-debian: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-debian: <<: *native_test_job_definition
@@ -156,6 +158,7 @@ check-system-debian: - job: build-system-debian artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-amd64 MAKE_CHECK_ARGS: check
@@ -170,7 +173,7 @@ build-tools-and-docs-debian: artifacts: expire_in: 2 days paths: - - build + - "*" acceptance-system-debian: <<: *native_test_job_definition
@@ -178,6 +181,7 @@ acceptance-system-debian: - job: build-system-debian artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -194,7 +198,7 @@ build-system-fedora: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-fedora: <<: *native_test_job_definition
@@ -202,6 +206,7 @@ check-system-fedora: - job: build-system-fedora artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: fedora MAKE_CHECK_ARGS: check
@@ -211,6 +216,7 @@ acceptance-system-fedora: - job: build-system-fedora artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -226,7 +232,7 @@ build-system-centos: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-centos: <<: *native_test_job_definition
@@ -234,6 +240,7 @@ check-system-centos: - job: build-system-centos artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos8 MAKE_CHECK_ARGS: check
@@ -243,6 +250,7 @@ acceptance-system-centos: - job: build-system-centos artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos8 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -257,7 +265,7 @@ build-system-opensuse: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-opensuse: <<: *native_test_job_definition
@@ -265,6 +273,7 @@ check-system-opensuse: - job: build-system-opensuse artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: opensuse-leap MAKE_CHECK_ARGS: check
@@ -274,6 +283,7 @@ acceptance-system-opensuse: - job: build-system-opensuse artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: opensuse-leap MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition
@@ -444,7 +454,7 @@ build-deprecated: artifacts: expire_in: 2 days paths: - - build + - "*" # We split the check-tcg step as test failures are expected but we still # want to catch the build breaking.
@@ -454,6 +464,7 @@ check-deprecated: - job: build-deprecated artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-all-test-cross MAKE_CHECK_ARGS: check-tcg allow_failure: true
@@ -526,7 +537,7 @@ build-crypto-old-nettle: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-old-nettle: <<: *native_test_job_definition
@@ -534,6 +545,7 @@ check-crypto-old-nettle: - job: build-crypto-old-nettle artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: check
@@ -547,7 +559,7 @@ build-crypto-old-gcrypt: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-old-gcrypt: <<: *native_test_job_definition
@@ -555,6 +567,7 @@ check-crypto-old-gcrypt: - job: build-crypto-old-gcrypt artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: check
@@ -568,7 +581,7 @@ build-crypto-only-gnutls: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-only-gnutls: <<: *native_test_job_definition
@@ -576,6 +589,7 @@ check-crypto-only-gnutls: - job: build-crypto-only-gnutls artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: check
Signature · 1 line
2.27.0
Quoted 7 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
This used to work in the past didn't it ? Did something change in meson to break this, or have we just not noticed before.
Quoted 6 lines
So instead of trying to marry a freshly checked out source tree with the pre-built binaries in these jobs, let's simply pass the whole source including the submodules and the build tree as artifact to the test jobs. That way timestamps get preserved and there is no rebuild of the sources anymore. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
I'm a little worried we might end up hitting the artifact size limit which is supposedly 1GB on gitlab.com. Im guessing this must be measuring the compressed size though, as a src checkout with build dir and .git dir is already way over 1GB.
Quoted 257 lines
Signed-off-by Thomas Huth <[email protected]>---
This is how a job looked like before my patch, running for 42 minutes:
https://gitlab.com/huth/qemu/-/jobs/978432757And this is how it looks like afterwards - it just took 18 minutes:
https://gitlab.com/huth/qemu/-/jobs/979500316.gitlab-ci.d/containers.yml | 1 + .gitlab-ci.yml | 40 +++++++++++++++++++++++++------------ 2 files changed, 28 insertions(+), 13 deletions(-).gitlab-ci.yml +27
-13@@ -39,7 +39,6 @@ include: image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: - cd build - - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition@@ -83,8 +82,7 @@ build-system-alpine: artifacts: expire_in: 2 days paths: - - .git-submodule-status - - build + - "*" check-system-alpine: <<: *native_test_job_definition@@ -92,6 +90,7 @@ check-system-alpine: - job: build-system-alpine artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: alpine MAKE_CHECK_ARGS: check@@ -101,6 +100,7 @@ acceptance-system-alpine: - job: build-system-alpine artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: alpine MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -116,7 +116,7 @@ build-system-ubuntu: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-ubuntu: <<: *native_test_job_definition@@ -124,6 +124,7 @@ check-system-ubuntu: - job: build-system-ubuntu artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check@@ -133,6 +134,7 @@ acceptance-system-ubuntu: - job: build-system-ubuntu artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -148,7 +150,7 @@ build-system-debian: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-debian: <<: *native_test_job_definition@@ -156,6 +158,7 @@ check-system-debian: - job: build-system-debian artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-amd64 MAKE_CHECK_ARGS: check@@ -170,7 +173,7 @@ build-tools-and-docs-debian: artifacts: expire_in: 2 days paths: - - build + - "*" acceptance-system-debian: <<: *native_test_job_definition@@ -178,6 +181,7 @@ acceptance-system-debian: - job: build-system-debian artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -194,7 +198,7 @@ build-system-fedora: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-fedora: <<: *native_test_job_definition@@ -202,6 +206,7 @@ check-system-fedora: - job: build-system-fedora artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: fedora MAKE_CHECK_ARGS: check@@ -211,6 +216,7 @@ acceptance-system-fedora: - job: build-system-fedora artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -226,7 +232,7 @@ build-system-centos: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-centos: <<: *native_test_job_definition@@ -234,6 +240,7 @@ check-system-centos: - job: build-system-centos artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos8 MAKE_CHECK_ARGS: check@@ -243,6 +250,7 @@ acceptance-system-centos: - job: build-system-centos artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos8 MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -257,7 +265,7 @@ build-system-opensuse: artifacts: expire_in: 2 days paths: - - build + - "*" check-system-opensuse: <<: *native_test_job_definition@@ -265,6 +273,7 @@ check-system-opensuse: - job: build-system-opensuse artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: opensuse-leap MAKE_CHECK_ARGS: check@@ -274,6 +283,7 @@ acceptance-system-opensuse: - job: build-system-opensuse artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: opensuse-leap MAKE_CHECK_ARGS: check-acceptance <<: *acceptance_definition@@ -444,7 +454,7 @@ build-deprecated: artifacts: expire_in: 2 days paths: - - build + - "*" # We split the check-tcg step as test failures are expected but we still # want to catch the build breaking.@@ -454,6 +464,7 @@ check-deprecated: - job: build-deprecated artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: debian-all-test-cross MAKE_CHECK_ARGS: check-tcg allow_failure: true@@ -526,7 +537,7 @@ build-crypto-old-nettle: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-old-nettle: <<: *native_test_job_definition@@ -534,6 +545,7 @@ check-crypto-old-nettle: - job: build-crypto-old-nettle artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: check@@ -547,7 +559,7 @@ build-crypto-old-gcrypt: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-old-gcrypt: <<: *native_test_job_definition@@ -555,6 +567,7 @@ check-crypto-old-gcrypt: - job: build-crypto-old-gcrypt artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: check@@ -568,7 +581,7 @@ build-crypto-only-gnutls: MAKE_CHECK_ARGS: check-build artifacts: paths: - - build + - "*" check-crypto-only-gnutls: <<: *native_test_job_definition@@ -576,6 +589,7 @@ check-crypto-only-gnutls: - job: build-crypto-only-gnutls artifacts: true variables: + GIT_CHECKOUT: "false" IMAGE: centos7 MAKE_CHECK_ARGS: checkSignature · 1 line
2.27.0
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 11 lines · Daniel P. Berrangé
Quoted 7 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
This used to work in the past didn't it ? Did something change in meson to break this, or have we just not noticed before.
For to ask, could we address it by using 'meson test --no-rebuild' perhaps ?
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 11 lines · Daniel P. Berrangé
Quoted 7 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
This used to work in the past didn't it ? Did something change in meson to break this, or have we just not noticed before.
Likely https://github.com/mesonbuild/meson/pull/7900/
Kludge:
https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg05491.html
Quoted 11 lines · Daniel P. Berrangé
Quoted 7 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work anymore (with the current version of meson). Due to some changed time stamps, meson is always trying to rebuild the whole tree.
This used to work in the past didn't it ? Did something change in meson to break this, or have we just not noticed before.
As mentioned by Philippe already, there seems to be a problem in current meson which triggers a rebuild of the whole tree in some cases. See this thread here:
https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg05409.html
Quoted 11 lines
Quoted 6 lines
So instead of trying to marry a freshly checked out source tree with the pre-built binaries in these jobs, let's simply pass the whole source including the submodules and the build tree as artifact to the test jobs. That way timestamps get preserved and there is no rebuild of the sources anymore. This saves ca. 15 - 20 minutes of precious CI cycles in each run.
I'm a little worried we might end up hitting the artifact size limit which is supposedly 1GB on gitlab.com. Im guessing this must be measuring the compressed size though, as a src checkout with build dir and .git dir is already way over 1GB.
I've added a "du -sh" in one of the jobs and the size of the directory was indeed 2 GB already. But I did not see any problems due to that size. So either the limit must be bigger, or it's about the compressed size. Either way, it's working at least right now.
We could maybe also simply exclude the .git directory of the source checkout, I think that's the biggest chunk there and likely not needed for the tests.
Quoted 2 lines
For to ask, could we address it by using 'meson test --no-rebuild' perhaps ?
Interesting idea! I'll give it a try, and if it works, I'll send a patch for that instead.
Thomas