Skip to contentSkip to conversation

gitlab-ci.yml: Avoid recompiling the sources in the test jobs

PATCH In review Thomas Huth 3 participants 5 messages Jan 26, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gitlab-ci.yml: Avoid recompiling the sources in the test jobs

Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +3 -1
index de3a3d25b5..16fea25ba9 100644
@@ -38,9 +38,11 @@ include:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
     - cd build
     - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
   cache:
Signature · 1 line
2.27.0
Reviewed-by Tested-by on v1 1/1
#1 lore raw
Quoted 28 lines · Thomas Huth

Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +0 -0
index de3a3d25b5..16fea25ba9 100644

.gitlab-ci.yml include: · line 45

Quoted 9 lines
.gitlab-ci.yml +3 -1
@@ -38,9 +38,11 @@ include:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
     - cd build
     - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS

This ninja trick deserves a comment in the YAML file.

Otherwise:

Quoted 4 lines
.gitlab-ci.yml +0 -0
 .acceptance_template: &acceptance_definition
   cache:
 
#2 lore raw
Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 23 lines · Thomas Huth

Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

---
  .gitlab-ci.yml | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 5 lines
.gitlab-ci.yml +0 -0
index de3a3d25b5..16fea25ba9 100644

.gitlab-ci.yml include: · line 45

Quoted 9 lines
.gitlab-ci.yml +3 -1
@@ -38,9 +38,11 @@ include:
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
      - cd build
      - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS

This ninja trick deserves a comment in the YAML file.

I'll add:

# Avoid recompiling by hiding ninja with NINJA=":"

Ok?

Thomas

#3 lore raw
Quoted 43 lines · Thomas Huth
Quoted 33 lines · Philippe Mathieu-Daudé
Quoted 32 lines · Thomas Huth

Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

---
  .gitlab-ci.yml | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
.gitlab-ci.yml +0 -0
index de3a3d25b5..16fea25ba9 100644
@@ -38,9 +38,11 @@ include:

    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
+    - scripts/git-submodule.sh update

Quoted 8 lines
Quoted 6 lines

+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
      - cd build
      - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS

This ninja trick deserves a comment in the YAML file.

Quoted 5 lines

I'll add:

    # Avoid recompiling by hiding ninja with NINJA=":"

Ok?

Perfect, thanks!

Phil.

Reviewed-by on v1 1/1
#4 lore raw
In reply to Thomas Huth
Quoted 8 lines · Thomas Huth

Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree.

Quoted 8 lines

In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

Thanks for working on this!

Quoted 6 lines
---
  .gitlab-ci.yml | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 18 lines
.gitlab-ci.yml +3 -1
index de3a3d25b5..16fea25ba9 100644
@@ -38,9 +38,11 @@ include:
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
      - cd build
      - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS
 
  .acceptance_template: &acceptance_definition
    cache: