Skip to contentSkip to conversation

gitlab: expose installed package info in build logs

PATCH Applied Daniel P. Berrangé 5 participants 10 messages Jul 24, 2024 – Jul 25, 2024

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 gitlab: record installed packages in /packages.txt in containers

In reply to Daniel P. Berrangé

The lcitool created containers save the full distro package list details into /packages.txt. The idea is that build jobs will 'cat' this file, so that the build log has a record of what packages were used. This is important info, because when it comes to debug failures, the original container is often lost.

This extends the manually written dockerfiles to also create the /packages.txt file.

---
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
 6 files changed, 12 insertions(+), 6 deletions(-)
tests/docker/dockerfiles/debian-all-test-cross.docker +2 -1
index 6cc38a3633..8ab244e018 100644
@@ -62,7 +62,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         gcc-s390x-linux-gnu \
         libc6-dev-s390x-cross \
         gcc-sparc64-linux-gnu \
-        libc6-dev-sparc64-cross
+        libc6-dev-sparc64-cross && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
tests/docker/dockerfiles/debian-hexagon-cross.docker +2 -1
index f2d40f2dee..23152b4918 100644
@@ -33,7 +33,8 @@ RUN apt-get update && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
-    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
tests/docker/dockerfiles/debian-legacy-test-cross.docker +2 -1
index d75e0b85e2..5a6616b7d3 100644
@@ -36,7 +36,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         python3-pip \
         python3-setuptools \
         python3-venv \
-        python3-wheel
+        python3-wheel && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
tests/docker/dockerfiles/debian-loongarch-cross.docker +2 -1
index 6a9197528b..79eab5621e 100644
@@ -32,7 +32,8 @@ RUN apt-get update && \
         python3-pip \
         python3-setuptools \
         python3-venv \
-        python3-wheel
+        python3-wheel && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
tests/docker/dockerfiles/debian-tricore-cross.docker +2 -1
index 16276aa21d..479b4d6eba 100644
@@ -34,7 +34,8 @@ RUN apt update && \
        python3-pip \
        python3-setuptools \
        python3-wheel \
-       python3-venv
+       python3-venv && \
+       dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
tests/docker/dockerfiles/debian-xtensa-cross.docker +2 -1
index 413881899b..d011eee2ad 100644
@@ -16,7 +16,8 @@ RUN apt-get update && \
         curl \
         gettext \
         git \
-        python3-minimal
+        python3-minimal && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 ENV CPU_LIST dc232b dc233c de233_fpu dsp3400
 ENV TOOLCHAIN_RELEASE 2020.07
Signature · 1 line
2.45.2
#2 lore raw

PATCH 2/2 gitlab: display /packages.txt in build jobs

In reply to Daniel P. Berrangé

The lcitool created containers save the full distro package list details into /packages.txt. The idea is that build jobs will 'cat' this file, so that the build log has a record of what packages were used. This is important info, because when it comes to debug failures, the original container is often lost.

---
 .gitlab-ci.d/buildtest-template.yml  | 1 +
 .gitlab-ci.d/crossbuild-template.yml | 2 ++
 2 files changed, 3 insertions(+)
.gitlab-ci.d/buildtest-template.yml +1 -0
index 8f7ebfaed8..844c26623d 100644
@@ -9,6 +9,7 @@
     when: always
   before_script:
     - JOBS=$(expr $(nproc) + 1)
+    - cat /packages.txt
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
.gitlab-ci.d/crossbuild-template.yml +2 -0
index d9f81b7061..53051ec793 100644
@@ -8,6 +8,8 @@
     key: "$CI_JOB_NAME"
     when: always
   timeout: 80m
+  before_script:
+    - cat /packages.txt
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
Signature · 1 line
2.45.2
#0 lore raw

cover gitlab: expose installed package info in build logs

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

The lcitool generated containers create a /packages.txt file whose content is intended to be output in the build phase, so record the packages associated with the build.

This adds packages.txt to the manually written containers, and modifies the build jobs to display this content. This will improve our future debuggability of CI problems.

Daniel P. Berrangé (2):
gitlab: record installed packages in /packages.txt in containers
gitlab: display /packages.txt in build jobs

 .gitlab-ci.d/buildtest-template.yml                      | 1 +
 .gitlab-ci.d/crossbuild-template.yml                     | 2 ++
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
 8 files changed, 15 insertions(+), 6 deletions(-)
Signature · 1 line
2.45.2
Applied
#3 lore raw
Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

Quoted 7 lines

The lcitool generated containers create a /packages.txt file whose content is intended to be output in the build phase, so record the packages associated with the build.

This adds packages.txt to the manually written containers, and modifies the build jobs to display this content. This will improve our future debuggability of CI problems.

Queued to maintainer/for-9.1, thanks.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#5 lore raw
In reply to Daniel P. Berrangé

Hello Daniel,

Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic?

Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few months and inform of any updates so that we can manually bump them.

Manos

Quoted 26 lines

The lcitool generated containers create a /packages.txt file whose content is intended to be output in the build phase, so record the packages associated with the build.

This adds packages.txt to the manually written containers, and modifies the build jobs to display this content. This will improve our future debuggability of CI problems.

Daniel P. Berrangé (2):
gitlab: record installed packages in /packages.txt in containers
gitlab: display /packages.txt in build jobs

.gitlab-ci.d/buildtest-template.yml                      | 1 +
.gitlab-ci.d/crossbuild-template.yml                     | 2 ++
tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
8 files changed, 15 insertions(+), 6 deletions(-)
Signature · 1 line
2.45.2
#6 lore raw
Quoted 10 lines · Manos Pitsidianakis

Hello Daniel,

Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic?

Even if tests would be more deterministic that way, I also don't think that's what we want: We would test with backlevel packages that users are not using (since they'll normally use the latest version of a package from a certain version of a distro). So in our CI, we would not be able to reproduce the issues that the users are seeing on their systems.

Thomas

#8 lore raw
In reply to Manos Pitsidianakis
Quoted 11 lines · Manos Pitsidianakis

Hello Daniel,

Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic?

We want to be testing against latest packages because that reflects what our users are likely to see on their own systems. IOW, having failures after newly updated packages is a good thing - provided we can easily identify what caused the breakage, to enable us to fix it promptly - by promptly I mean same-day.

Quoted 2 lines

Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few months and inform of any updates so that we can manually bump them.

Only seeing breakage from new packages as long as a few months after it happens is a bad thing. We benefit from fast detection and fast fixing. In cases where the new package is broken, rather than QEMU, it is also better to be able to report that back to the distro within days of them rolling out the problem as it'll be fresh in their minds to fix.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw
In reply to Manos Pitsidianakis
Quoted 11 lines · Manos Pitsidianakis

Hello Daniel,

Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic?

Generally we want track the latest LTS version and usually the distros are pretty good about being cautious about updates. I think the recent case was because BSD's have a different policy about updating python.

IOW this is a nice to have for debugging but I don't think we want to freeze the acceptable package set for any given install.

Quoted 32 lines

Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few months and inform of any updates so that we can manually bump them.

Manos

Quoted 25 lines

The lcitool generated containers create a /packages.txt file whose content is intended to be output in the build phase, so record the packages associated with the build.

This adds packages.txt to the manually written containers, and modifies the build jobs to display this content. This will improve our future debuggability of CI problems.

Daniel P. Berrangé (2):
gitlab: record installed packages in /packages.txt in containers
gitlab: display /packages.txt in build jobs

.gitlab-ci.d/buildtest-template.yml                      | 1 +
.gitlab-ci.d/crossbuild-template.yml                     | 2 ++
tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
8 files changed, 15 insertions(+), 6 deletions(-)

-- 2.45.2

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#7 lore raw
In reply to Thomas Huth
Quoted 10 lines · Thomas Huth
Quoted 9 lines · Manos Pitsidianakis

Hello Daniel,

Quoted 5 lines · Daniel P. Berrangé

Many times we see a build job start failing, we wonder if the installed packages have changed since the last passing build. We can rarely diagnose this, however, since we only have the new container image, not the old one.

Quoted 4 lines
Quoted 1 line

APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic?

Even if tests would be more deterministic that way, I also don't think that's what we want: We would test with backlevel packages that users are not using (since they'll normally use the latest version of a package from a certain version of a distro). So in our CI, we would not be able to reproduce the issues that the users are seeing on their systems.

Definitely, but the current testing makes it harder to discern the lines between QEMU regression and QEMU not working with latest updates. If we had e.g. a pinned job and a "latest" job disagreeing we'd know the cause. Of course that means extra CI minutes, maintainer upkeep etc etc, I know, but this was my reasoning, I don't dismiss testing for what users experience. :)

Manos