gitlab-ci.yml: Avoid recompiling the sources in the test jobs
PATCH 1/1 gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.
In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.
--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +3 -1
@@ -38,9 +38,11 @@ include: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: + - scripts/git-submodule.sh update + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') - cd build - find . -type f -exec touch {} + - - make $MAKE_CHECK_ARGS + - make NINJA=":" $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition cache:
Signature · 1 line
2.27.0
Quoted 28 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.
In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-).gitlab-ci.yml +0
-0
.gitlab-ci.yml include: · line 45
Quoted 9 lines
.gitlab-ci.yml +3
-1@@ -38,9 +38,11 @@ include: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: + - scripts/git-submodule.sh update + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') - cd build - find . -type f -exec touch {} + - - make $MAKE_CHECK_ARGS + - make NINJA=":" $MAKE_CHECK_ARGS
This ninja trick deserves a comment in the YAML file.
Otherwise:
Quoted 4 lines
.gitlab-ci.yml +0
-0.acceptance_template: &acceptance_definition cache:
Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 23 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.
In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 5 lines
.gitlab-ci.yml +0
-0.gitlab-ci.yml include: · line 45
Quoted 9 lines
.gitlab-ci.yml +3
-1@@ -38,9 +38,11 @@ include: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: + - scripts/git-submodule.sh update + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') - cd build - find . -type f -exec touch {} + - - make $MAKE_CHECK_ARGS + - make NINJA=":" $MAKE_CHECK_ARGSThis ninja trick deserves a comment in the YAML file.
I'll add:
# Avoid recompiling by hiding ninja with NINJA=":"
Ok?
Thomas
Quoted 43 lines · Thomas Huth
Quoted 33 lines · Philippe Mathieu-Daudé
Quoted 32 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.
In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Thomas Huth <[email protected]>---
.gitlab-ci.yml | 4 +++-1 file changed, 3 insertions(+), 1 deletion(-).gitlab-ci.yml +0
-0@@ -38,9 +38,11 @@ include:stage: test
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
script:
+ -scripts/git-submodule.shupdateQuoted 8 lines
Quoted 6 lines
+ $(grep
GIT_SUBMODULESbuild/config-host.mak | sed 's/GIT_SUBMODULES=//')
- cd build
- find . -type f -exec touch {} +
- - make $MAKE_CHECK_ARGS
+ - make NINJA=":" $MAKE_CHECK_ARGSThis ninja trick deserves a comment in the YAML file.
Quoted 5 lines
I'll add:
# Avoid recompiling by hiding ninja with NINJA=":"
Ok?
Perfect, thanks!
Phil.
Quoted 8 lines · Thomas Huth
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.
Quoted 8 lines
In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.
Thanks for working on this!
Quoted 6 lines
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 18 lines
.gitlab-ci.yml +3
-1@@ -38,9 +38,11 @@ include: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: + - scripts/git-submodule.sh update + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') - cd build - find . -type f -exec touch {} + - - make $MAKE_CHECK_ARGS + - make NINJA=":" $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition cache: