gitlab: expose more info about CI runner environment as artifacts
v2 cover gitlab: expose more info about CI runner environment as artifacts
To repeat the commit message from patch 6..
This captures the mount list, CPU model, memory information, device node lists, kernel info, user identity and all environment variables. For more details see patch 6.
The first 5 patches are all refactoring of the job templates to reduce redundancy, so we have more consistency across jobs with less code.
Sample pipeline showing the new 'ci-runner-env' artifacts in jobs:
https://gitlab.com/berrange/qemu/-/pipelines/2626072516
Daniel P. Berrangé (6):
gitlab: remove unused .cross_test_artifacts template
gitlab: move .meson_job_template into base.yaml
gitlab: use .base_meson_job_template from crossbuild jobs
gitlab: pull ccache setup into .base_meson_ccache_job_template
gitlab: pull before_script logic into .base_meson_job_template
gitlab: greatly expand captured info about CI runner environment
.gitlab-ci.d/base.yml | 48 ++++++++++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 58 ++++----------------------- .gitlab-ci.d/crossbuild-template.yml | 59 +++------------------------- 3 files changed, 62 insertions(+), 103 deletions(-)
Signature · 1 line
2.54.0
PATCH v2 1/6 gitlab: remove unused .cross_test_artifacts template
This is unused since
commit 0d1137550bbf3af7ebe34801a9bff5a4839cc34e
Author: Richard Henderson <[email protected]>
Date: Thu Dec 18 09:58:33 2025 +1100
gitlab: Remove 32-bit host testing
--- .gitlab-ci.d/crossbuild-template.yml | 12 ------------ 1 file changed, 12 deletions(-)
.gitlab-ci.d/crossbuild-template.yml +0 -12
@@ -119,15 +119,3 @@ $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi - section_end test - -# We can still run some tests on some of our cross build jobs. They can add this -# template to their extends to save the build logs and test results -.cross_test_artifacts: - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" - when: always - expire_in: 7 days - paths: - - build/meson-logs - reports: - junit: build/meson-logs/*.junit.xml
Signature · 1 line
2.54.0
Re: [PATCH v2 1/6] gitlab: remove unused .cross_test_artifacts template
Quoted 6 lines · Daniel P. Berrangé
This is unused since
commit
0d1137550bbf3af7ebe34801a9bff5a4839cc34e
Author: Richard Henderson <[email protected]>
Date: Thu Dec 18 09:58:33 2025 +1100
Quoted 6 lines
gitlab: Remove 32-bit host testing
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/crossbuild-template.yml | 12 ------------ 1 file changed, 12 deletions(-)
Re: [PATCH v2 1/6] gitlab: remove unused .cross_test_artifacts template
In reply to Daniel P. BerrangéQuoted 6 lines · Daniel P. Berrangé
This is unused since
commit
0d1137550bbf3af7ebe34801a9bff5a4839cc34e
Author: Richard Henderson <[email protected]>
Date: Thu Dec 18 09:58:33 2025 +1100
Quoted 7 lines
gitlab: Remove 32-bit host testing
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/crossbuild-template.yml | 12 ------------ 1 file changed, 12 deletions(-)
PATCH v2 2/6 gitlab: move .meson_job_template into base.yaml
In reply to Daniel P. BerrangéRename .meson_job_template to .base_meson_job_template and move it into base.yml, in prep for wider reuse.
--- .gitlab-ci.d/base.yml | 14 ++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 20 +++----------------- 2 files changed, 17 insertions(+), 17 deletions(-)
.gitlab-ci.d/base.yml +14 -0
@@ -129,3 +129,17 @@ variables: QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG - when: on_success + +# Any job running meson should capture meson logs +# by default. Some jobs might override the artifacts +# to capture further files +.base_meson_job_template: + extends: .base_job_template + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" + when: always + expire_in: 7 days + paths: + - build/meson-logs + reports: + junit: build/meson-logs/*.junit.xml
.gitlab-ci.d/buildtest-template.yml +3 -17
@@ -1,20 +1,6 @@ -# Any job running meson should capture meson logs -# by default. Some jobs might override the artifacts -# to capture further files -.meson_job_template: - extends: .base_job_template - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" - when: always - expire_in: 7 days - paths: - - build/meson-logs - reports: - junit: build/meson-logs/*.junit.xml - .native_build_job_template: - extends: .meson_job_template + extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG cache:
@@ -75,7 +61,7 @@ - build/**/*.c.o.d .common_test_job_template: - extends: .meson_job_template + extends: .base_meson_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG script:
@@ -133,7 +119,7 @@ QEMU_JOB_FUNCTIONAL: 1 .wasm_build_job_template: - extends: .meson_job_template + extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG before_script:
Signature · 1 line
2.54.0
Re: [PATCH v2 2/6] gitlab: move .meson_job_template into base.yaml
Quoted 9 lines · Daniel P. Berrangé
Rename .meson_job_template to .base_meson_job_template and move it into base.yml, in prep for wider reuse.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/base.yml | 14 ++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 20 +++----------------- 2 files changed, 17 insertions(+), 17 deletions(-)
PATCH v2 3/6 gitlab: use .base_meson_job_template from crossbuild jobs
In reply to Daniel P. BerrangéThis ensures that crossbuild jobs capture the build/meson-logs content as artifacts to aid in debugging.
--- .gitlab-ci.d/crossbuild-template.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
.gitlab-ci.d/crossbuild-template.yml +3 -3
@@ -1,5 +1,5 @@ .cross_system_build_job: - extends: .base_job_template + extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG cache:
@@ -54,7 +54,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $EXTRA_CONFIGURE_OPTS variable. .cross_accel_build_job: - extends: .base_job_template + extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 60m
@@ -87,7 +87,7 @@ - section_end test .cross_user_build_job: - extends: .base_job_template + extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG cache:
Signature · 1 line
2.54.0
Re: [PATCH v2 3/6] gitlab: use .base_meson_job_template from crossbuild jobs
Quoted 8 lines · Daniel P. Berrangé
This ensures that crossbuild jobs capture the build/meson-logs content as artifacts to aid in debugging.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/crossbuild-template.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
PATCH v2 4/6 gitlab: pull ccache setup into .base_meson_ccache_job_template
In reply to Daniel P. BerrangéThis provides a consistent ccache setup across all build jobs.
--- .gitlab-ci.d/base.yml | 16 +++++++++++++ .gitlab-ci.d/buildtest-template.yml | 14 ++--------- .gitlab-ci.d/crossbuild-template.yml | 35 +++++----------------------- 3 files changed, 24 insertions(+), 41 deletions(-)
.gitlab-ci.d/base.yml +16 -0
@@ -143,3 +143,19 @@ variables: - build/meson-logs reports: junit: build/meson-logs/*.junit.xml + +.base_meson_ccache_job_template: + extends: .base_meson_job_template + cache: + paths: + - ccache + key: "$CI_JOB_NAME" + when: always + script: + - export CCACHE_BASEDIR="$(pwd)" + - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" + - export CCACHE_MAXSIZE="500M" + - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - ccache --zero-stats + after_script: + - ccache --show-stats
.gitlab-ci.d/buildtest-template.yml +2 -12
@@ -1,13 +1,8 @@ .native_build_job_template: - extends: .base_meson_job_template + extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - cache: - paths: - - ccache - key: "$CI_JOB_NAME" - when: always before_script: - source scripts/ci/gitlab-ci-section - section_start setup "Pre-script setup"
@@ -15,14 +10,10 @@ - cat /packages.txt - section_end setup script: - - export CCACHE_BASEDIR="$(pwd)" - - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - - export CCACHE_MAXSIZE="500M" - - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - !reference [.base_meson_ccache_job_template, script] - du -sh .git - mkdir build - cd build - - ccache --zero-stats - section_start configure "Running configure" - ../configure --enable-werror --disable-docs --enable-fdt=system --disable-debug-info
@@ -43,7 +34,6 @@ $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ; fi - section_end test - - ccache --show-stats # We jump some hoops in common_test_job_template to avoid # rebuilding all the object files we skip in the artifacts
.gitlab-ci.d/crossbuild-template.yml +6 -29
@@ -1,12 +1,7 @@ .cross_system_build_job: - extends: .base_meson_job_template + extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - cache: - paths: - - ccache - key: "$CI_JOB_NAME" - when: always timeout: 80m before_script: - source scripts/ci/gitlab-ci-section
@@ -15,13 +10,9 @@ - cat /packages.txt - section_end setup script: - - export CCACHE_BASEDIR="$(pwd)" - - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - - export CCACHE_MAXSIZE="500M" - - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - !reference [.base_meson_ccache_job_template, script] - mkdir build - cd build - - ccache --zero-stats - section_start configure "Running configure" - ../configure --enable-werror --disable-docs --enable-fdt=system --disable-user $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS
@@ -46,7 +37,6 @@ mv -v qemu-setup*.exe qemu-setup-${version}.exe; fi - section_end installer - - ccache --show-stats # Job to cross-build specific accelerators. #
@@ -54,22 +44,15 @@ # KVM), and set extra options (such disabling other accelerators) via the # $EXTRA_CONFIGURE_OPTS variable. .cross_accel_build_job: - extends: .base_meson_job_template + extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 60m - cache: - paths: - - ccache/ - key: "$CI_JOB_NAME" before_script: - source scripts/ci/gitlab-ci-section - JOBS=$(expr $(nproc) + 1) script: - - export CCACHE_BASEDIR="$(pwd)" - - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - - export CCACHE_MAXSIZE="500M" - - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - !reference [.base_meson_ccache_job_template, script] - mkdir build - cd build - section_start configure "Running configure"
@@ -87,20 +70,14 @@ - section_end test .cross_user_build_job: - extends: .base_meson_job_template + extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - cache: - paths: - - ccache/ - key: "$CI_JOB_NAME" before_script: - source scripts/ci/gitlab-ci-section - JOBS=$(expr $(nproc) + 1) script: - - export CCACHE_BASEDIR="$(pwd)" - - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - - export CCACHE_MAXSIZE="500M" + - !reference [.base_meson_ccache_job_template, script] - mkdir build - cd build - section_start configure "Running configure"
Signature · 1 line
2.54.0
Re: [PATCH v2 4/6] gitlab: pull ccache setup into .base_meson_ccache_job_template
Quoted 9 lines · Daniel P. Berrangé
This provides a consistent ccache setup across all build jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/base.yml | 16 +++++++++++++ .gitlab-ci.d/buildtest-template.yml | 14 ++--------- .gitlab-ci.d/crossbuild-template.yml | 35 +++++----------------------- 3 files changed, 24 insertions(+), 41 deletions(-)
PATCH v2 5/6 gitlab: pull before_script logic into .base_meson_job_template
In reply to Daniel P. BerrangéThis ensures that all jobs have the $JOBS env set and capture the packages.txt content (where available) in their logs, and all use the job section headers.
--- .gitlab-ci.d/base.yml | 9 +++++++++ .gitlab-ci.d/buildtest-template.yml | 24 ++---------------------- .gitlab-ci.d/crossbuild-template.yml | 12 ------------ 3 files changed, 11 insertions(+), 34 deletions(-)
.gitlab-ci.d/base.yml +9 -0
@@ -143,6 +143,15 @@ variables: - build/meson-logs reports: junit: build/meson-logs/*.junit.xml + before_script: + - source scripts/ci/gitlab-ci-section + - section_start setup "Pre-script setup" + - JOBS=$(expr $(nproc) + 1) + # Prevent logs (if any) from prior build job artifacts + # from being duplicated in the new job artifacts + - rm -f build/meson-logs/* + - test -f /packages.txt && cat /packages.txt + - section_end setup .base_meson_ccache_job_template: extends: .base_meson_job_template
.gitlab-ci.d/buildtest-template.yml +2 -22
@@ -3,12 +3,6 @@ extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - before_script: - - source scripts/ci/gitlab-ci-section - - section_start setup "Pre-script setup" - - JOBS=$(expr $(nproc) + 1) - - cat /packages.txt - - section_end setup script: - !reference [.base_meson_ccache_job_template, script] - du -sh .git
@@ -74,11 +68,6 @@ .native_test_job_template: extends: .common_test_job_template - before_script: - # Prevent logs from the build job that run earlier - # from being duplicated in the test job artifacts - - rm -f build/meson-logs/* - .functional_test_job_template: extends: .common_test_job_template
@@ -96,27 +85,18 @@ - build/tests/functional/*/*/*.log reports: junit: build/meson-logs/*.junit.xml - before_script: - - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1 - - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache - # Prevent logs from the build job that run earlier - # from being duplicated in the test job artifacts - - rm -f build/meson-logs/* after_script: - cd build - du -chs ${CI_PROJECT_DIR}/*-cache variables: QEMU_JOB_FUNCTIONAL: 1 + QEMU_TEST_ALLOW_UNTRUSTED_CODE: 1 + QEMU_TEST_CACHE_DIR: ${CI_PROJECT_DIR}/functional-cache .wasm_build_job_template: extends: .base_meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - before_script: - - source scripts/ci/gitlab-ci-section - - section_start setup "Pre-script setup" - - JOBS=$(expr $(nproc) + 1) - - section_end setup script: - du -sh .git - mkdir build
.gitlab-ci.d/crossbuild-template.yml +0 -12
@@ -3,12 +3,6 @@ stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 80m - before_script: - - source scripts/ci/gitlab-ci-section - - section_start setup "Pre-script setup" - - JOBS=$(expr $(nproc) + 1) - - cat /packages.txt - - section_end setup script: - !reference [.base_meson_ccache_job_template, script] - mkdir build
@@ -48,9 +42,6 @@ stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 60m - before_script: - - source scripts/ci/gitlab-ci-section - - JOBS=$(expr $(nproc) + 1) script: - !reference [.base_meson_ccache_job_template, script] - mkdir build
@@ -73,9 +64,6 @@ extends: .base_meson_ccache_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG - before_script: - - source scripts/ci/gitlab-ci-section - - JOBS=$(expr $(nproc) + 1) script: - !reference [.base_meson_ccache_job_template, script] - mkdir build
Signature · 1 line
2.54.0
Re: [PATCH v2 5/6] gitlab: pull before_script logic into .base_meson_job_template
Quoted 11 lines · Daniel P. Berrangé
This ensures that all jobs have the $JOBS env set and capture the packages.txt content (where available) in their logs, and all use the job section headers.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/base.yml | 9 +++++++++ .gitlab-ci.d/buildtest-template.yml | 24 ++---------------------- .gitlab-ci.d/crossbuild-template.yml | 12 ------------ 3 files changed, 11 insertions(+), 34 deletions(-)
PATCH v2 6/6 gitlab: greatly expand captured info about CI runner environment
In reply to Daniel P. BerrangéTo aid in debugging wierd CI failures we need greater information about the CI runner environment. It is usually container based and can have some unexpected characteristics that significantly differ from a developer's local environment.
This captures the mount list, CPU model, memory information, device node lists, kernel info, user identity and all environment variables.
Since this information can get quite large it is not emitted directly in the logs, rather it is exposed in job artifacts under the "ci-runner-env" directory.
--- .gitlab-ci.d/base.yml | 11 ++++++++++- .gitlab-ci.d/buildtest-template.yml | 2 ++ 2 files changed, 12 insertions(+), 1 deletion(-)
.gitlab-ci.d/base.yml +10 -1
@@ -141,6 +141,7 @@ variables: expire_in: 7 days paths: - build/meson-logs + - ci-runner-env reports: junit: build/meson-logs/*.junit.xml before_script:
@@ -150,7 +151,15 @@ variables: # Prevent logs (if any) from prior build job artifacts # from being duplicated in the new job artifacts - rm -f build/meson-logs/* - - test -f /packages.txt && cat /packages.txt + - mkdir -p ci-runner-env + - test -f /packages.txt && cp /packages.txt ci-runner-env/packages.txt + - cp /proc/mounts ci-runner-env/mounts.txt + - cp /proc/cpuinfo ci-runner-env/cpuinfo.txt + - cp /proc/meminfo ci-runner-env/meminfo.txt + - ls -l /dev > ci-runner-env/devices.txt + - uname -a > ci-runner-env/kernel.txt + - id -a > ci-runner-env/user.txt + - printenv | sort > ci-runner-env/environ.txt - section_end setup .base_meson_ccache_job_template:
.gitlab-ci.d/buildtest-template.yml +2 -0
@@ -38,6 +38,7 @@ paths: - build - .git-submodule-status + - ci-runner-env exclude: - build/**/*.p - build/**/*.a.p
@@ -83,6 +84,7 @@ paths: - build/meson-logs - build/tests/functional/*/*/*.log + - ci-runner-env reports: junit: build/meson-logs/*.junit.xml after_script:
Signature · 1 line
2.54.0
Re: [PATCH v2 6/6] gitlab: greatly expand captured info about CI runner environment
Quoted 13 lines · Daniel P. Berrangé
To aid in debugging wierd CI failures we need greater information about the CI runner environment. It is usually container based and can have some unexpected characteristics that significantly differ from a developer's local environment.
This captures the mount list, CPU model, memory information, device node lists, kernel info, user identity and all environment variables.
Since this information can get quite large it is not emitted directly in the logs, rather it is exposed in job artifacts under the "ci-runner-env" directory.
Quoted 6 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/base.yml | 11 ++++++++++- .gitlab-ci.d/buildtest-template.yml | 2 ++ 2 files changed, 12 insertions(+), 1 deletion(-)
Quoted 22 lines · Daniel P. Berrangé
To repeat the commit message from patch 6..
This captures the mount list, CPU model, memory information, device node lists, kernel info, user identity and all environment variables. For more details see patch 6.
The first 5 patches are all refactoring of the job templates to reduce redundancy, so we have more consistency across jobs with less code.
Sample pipeline showing the new 'ci-runner-env' artifacts in jobs:
https://gitlab.com/berrange/qemu/-/pipelines/2626072516
Daniel P. Berrangé (6):
gitlab: remove unused .cross_test_artifacts template
gitlab: move .meson_job_template intobase.yaml
gitlab: use .base_meson_job_template from crossbuild jobs
gitlab: pull ccache setup into .base_meson_ccache_job_template
gitlab: pullbefore_scriptlogic into .base_meson_job_template
gitlab: greatly expand captured info about CI runner environment
Quoted 5 lines
.gitlab-ci.d/base.yml | 48 ++++++++++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 58 ++++----------------------- .gitlab-ci.d/crossbuild-template.yml | 59 +++------------------------- 3 files changed, 62 insertions(+), 103 deletions(-)
This was merged into master (8f1d3b586f1265023f75ea9c227c35d463321aef). Thank you for your contribution!
Regards,
Pierrick