Skip to contentSkip to conversation

CI: part 3 (slimline, and PPC/RISCV fixes)

PATCH In review Andrew Cooper 4 participants 72 messages Jul 11, 2024 – Jul 15, 2024 RISC-V

lore
Share & tools
lore
#0 lore raw

v2 cover CI: part 3 (slimline, and PPC/RISCV fixes)

Patch 1 fixes a bug in the containerize script

Patches 2-5 remove useless/obsolete testing

Patch 6 fixes a bug with the archlinux testing configuration

Patches 7-9 adjust the PPC64 testing

Patches 10-11 adjust the RISCV64 testing

Patch 12 refreshes the custom GCC-IBT container

All changes here follow best-guidance for dockerfiles (non-root, heredocs for improved legibility), and naming consistency improvements discussed previously on the Committers call. Breifly that's:

$DISTRO-$VERSION(numeric)-$ARCH-*

which sort more nicely and don't require e.g. people to remember which order buster/bullseye/bookworm are in terms of debain release.

For x86, this involves inserting an $ARCH of x86_64. Right now, x86_64 is implied by the absence, and all other variations have to state one.

Still to come:

* OpenSUSE container fixes (resolves the final rolling distro vs allowed_failure problem)
* Ubuntu/debian rename/trim
* Add new jobs for latest releases (Ubuntu, Alpine, Fedora)

Andrew Cooper (12):
CI: Fix CONTAINER_UID0=1 scripts/containerize

CI: Remove useless/misleading randconfig jobs
CI: Drop Debian Jessie dockerfiles
CI: Drop Debian Stretch testing
CI: Drop Ubuntu Trusty testing

CI: Mark Archlinux/x86 as allowing failures

CI: Introduce a debian:12-ppc64le container
CI: Use debian:12-ppc64le for both build and test
CI: Refresh bullseye-ppc64le as debian:11-ppc64le

CI: Introduce debian:11/12-riscv64 containers
CI: Swap to debian for riscv64 build and test

CI: Refresh and upgrade the GCC-IBT container

 .../archlinux/current-riscv64.dockerfile      |  22 --
 automation/build/debian/11-ppc64le.dockerfile |  33 +++
 automation/build/debian/11-riscv64.dockerfile |  33 +++
 automation/build/debian/12-ppc64le.dockerfile |  36 +++
 automation/build/debian/12-riscv64.dockerfile |  36 +++
 ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} |  81 +++---
 .../build/debian/bullseye-ppc64le.dockerfile  |  32 ---
 .../build/debian/jessie-i386.dockerfile       |  55 -----
 automation/build/debian/jessie.dockerfile     |  52 ----
 .../build/debian/stretch-i386.dockerfile      |  57 -----
 automation/build/debian/stretch.dockerfile    |  60 -----
 automation/build/ubuntu/trusty.dockerfile     |  51 ----
 automation/gitlab-ci/build.yaml               | 233 +++++++-----------
 automation/gitlab-ci/test.yaml                |  13 +-
 automation/scripts/containerize               |  15 +-
 automation/scripts/qemu-smoke-ppc64le.sh      |   4 +-
 .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  |  38 ---
 17 files changed, 282 insertions(+), 569 deletions(-)
 delete mode 100644 automation/build/archlinux/current-riscv64.dockerfile
 create mode 100644 automation/build/debian/11-ppc64le.dockerfile
 create mode 100644 automation/build/debian/11-riscv64.dockerfile
 create mode 100644 automation/build/debian/12-ppc64le.dockerfile
 create mode 100644 automation/build/debian/12-riscv64.dockerfile
 rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12-x86_64-gcc-ibt.dockerfile} (50%)
 delete mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile
 delete mode 100644 automation/build/debian/jessie-i386.dockerfile
 delete mode 100644 automation/build/debian/jessie.dockerfile
 delete mode 100644 automation/build/debian/stretch-i386.dockerfile
 delete mode 100644 automation/build/debian/stretch.dockerfile
 delete mode 100644 automation/build/ubuntu/trusty.dockerfile
 delete mode 100644 automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile
Signature · 1 line
2.39.2
#1 lore raw

PATCH v2 1/12 CI: Fix CONTAINER_UID0=1 scripts/containerize

Right now, most build containers use root. Archlinux, Fedora and Yocto set up a regular user called user.

For those containers, trying to containerize as root fails, because CONTAINER_UID0=1 does nothing, whereas CONTAINER_UID0=0 forces the user away from root.

To make CONTAINER_UID0=1 work reliably, force to root if requested.

Fixes 17fbe6504dfd ("automation: introduce a new variable to control container user")

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

v2:
* Add fixes tag

---
 automation/scripts/containerize | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
automation/scripts/containerize +1 -1
index acdef1b54813..03bc4837350c 100755
@@ -51,7 +51,7 @@ esac
 
 # Use this variable to control whether root should be used
 case "_${CONTAINER_UID0}" in
-    _1)   userarg= ;;
+    _1)   userarg="-u 0" ;;
     _0|_) userarg="-u $(id -u) $userns_podman" ;;
 esac
 
Signature · 1 line
2.39.2
Reviewed-by on v2 1/12
#2 lore raw

Re: [PATCH 01/12] CI: Fix CONTAINER_UID0=1 scripts/containerize

Quoted 11 lines · Andrew Cooper

Right now, most build containers use root. Archlinux, Fedora and Yocto set up a regular user called user.

For those containers, trying to containerize as root fails, because CONTAINER_UID0=1 does nothing, whereas CONTAINER_UID0=0 forces the user away from root.

To make CONTAINER_UID0=1 work reliably, force to root if requested.

Fixes 17fbe6504dfd ("automation: introduce a new variable to control container user")

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#3 lore raw

PATCH v2 2/12 CI: Remove useless/misleading randconfig jobs

In reply to Andrew Cooper

Randconfig builds pick CONFIG_DEBUG with 50% probability. Therefore $foo{,-debug}-randconfig are two identical jobs with misleading names.

Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm build environments. Both use GCC 12.2, so we have 4 identical jobs. Delete 3 of them.

This leaves us with 4 primary randconfig jobs:

gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:

as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special job with a custom compiler.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

Here is an example debug-randconfig with a release builds:
https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7312848876

If we want to run multiple identical randconfig jobs, that's spelt

parallel: 5

in the configuration, and here is an example of what such a run looks like:
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041

Notice the randconfig jobs have a 5 in place of a retry button, and show a submenu when clicked on.

---
 automation/gitlab-ci/build.yaml | 39 ---------------------------------
 1 file changed, 39 deletions(-)
automation/gitlab-ci/build.yaml +0 -39
index ff5c9055d1f0..da7b8885aa30 100644
@@ -385,13 +385,6 @@ debian-bookworm-gcc-arm32-randconfig:
     HYPERVISOR_ONLY: y
     RANDCONFIG: y
 
-debian-bookworm-gcc-arm32-debug-randconfig:
-  extends: .gcc-arm32-cross-build-debug
-  variables:
-    CONTAINER: debian:bookworm-arm64v8-arm32-gcc
-    HYPERVISOR_ONLY: y
-    RANDCONFIG: y
-
 debian-bookworm-gcc-arm32-debug-staticmem:
   extends: .gcc-arm32-cross-build-debug
   variables:
@@ -423,18 +416,6 @@ debian-bookworm-gcc-debug-arm64:
   variables:
     CONTAINER: debian:bookworm-arm64v8
 
-debian-bookworm-gcc-arm64-randconfig:
-  extends: .gcc-arm64-build
-  variables:
-    CONTAINER: debian:bookworm-arm64v8
-    RANDCONFIG: y
-
-debian-bookworm-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
-  variables:
-    CONTAINER: debian:bookworm-arm64v8
-    RANDCONFIG: y
-
 alpine-3.18-gcc-arm64:
   extends: .gcc-arm64-build
   variables:
@@ -451,12 +432,6 @@ alpine-3.18-gcc-arm64-randconfig:
     CONTAINER: alpine:3.18-arm64v8
     RANDCONFIG: y
 
-alpine-3.18-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
-  variables:
-    CONTAINER: alpine:3.18-arm64v8
-    RANDCONFIG: y
-
 alpine-3.18-gcc-debug-arm64-staticmem:
   extends: .gcc-arm64-build-debug
   variables:
@@ -525,14 +500,6 @@ archlinux-current-gcc-riscv64-randconfig:
     RANDCONFIG: y
     <<: *riscv-fixed-randconfig
 
-archlinux-current-gcc-riscv64-debug-randconfig:
-  extends: .gcc-riscv64-cross-build-debug
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    RANDCONFIG: y
-    <<: *riscv-fixed-randconfig
-
 # Power cross-build
 debian-bullseye-gcc-ppc64le:
   extends: .gcc-ppc64le-cross-build
@@ -676,12 +643,6 @@ debian-bookworm-gcc-randconfig:
     CONTAINER: debian:bookworm
     RANDCONFIG: y
 
-debian-bookworm-gcc-debug-randconfig:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:bookworm
-    RANDCONFIG: y
-
 debian-bookworm-32-clang-debug:
   extends: .clang-x86-32-build-debug
   variables:
Signature · 1 line
2.39.2
Reviewed-by on v2 2/12
#4 lore raw

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

Quoted 14 lines · Andrew Cooper

Randconfig builds pick CONFIG_DEBUG with 50% probability. Therefore $foo{,-debug}-randconfig are two identical jobs with misleading names.

Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm build environments. Both use GCC 12.2, so we have 4 identical jobs. Delete 3 of them.

This leaves us with 4 primary randconfig jobs:

gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:

Quoted 4 lines

as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special job with a custom compiler.

Thanks,

Quoted 9 lines

If we want to run multiple identical randconfig jobs, that's spelt

parallel: 5

in the configuration, and here is an example of what such a run looks like:
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041

Notice the randconfig jobs have a 5 in place of a retry button, and show a submenu when clicked on.

We already have "parallel: 2" on the whole pipeline, give or take, but those are quick jobs so it might be worth it to run them a few more time on a pipeline.

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#5 lore raw

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

Quoted 15 lines · Anthony PERARD
Quoted 14 lines · Andrew Cooper

Randconfig builds pick CONFIG_DEBUG with 50% probability. Therefore $foo{,-debug}-randconfig are two identical jobs with misleading names.

Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm build environments. Both use GCC 12.2, so we have 4 identical jobs. Delete 3 of them.

This leaves us with 4 primary randconfig jobs:

gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:

Quoted 5 lines
Quoted 4 lines

as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special job with a custom compiler.

Thanks.

Quoted 7 lines

Thanks,

Quoted 4 lines

If we want to run multiple identical randconfig jobs, that's spelt

parallel: 5

Quoted 8 lines
Quoted 5 lines

in the configuration, and here is an example of what such a run looks like:
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041

Notice the randconfig jobs have a 5 in place of a retry button, and show a submenu when clicked on.

We already have "parallel: 2" on the whole pipeline, give or take, but those are quick jobs so it might be worth it to run them a few more time on a pipeline.

I'm afraid I don't follow.

The runners themselves are set up to run concurrent jobs.  I could well believe that's spelt parallel: in the runner config files.

Here, its really jobs:parallel which is the way of saying "I want 5 copies of this job to run in the pipeline".

Obviously, 5 is arbitrary.  I was thinking of having a scheduled (once per day, at 3AM or whenever) running randconfig only with a parallel:20 or so, so we can make good use of otherwise-idle time.

~Andrew

#6 lore raw

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

Quoted 6 lines · Andrew Cooper
Quoted 3 lines · Anthony PERARD

We already have "parallel: 2" on the whole pipeline, give or take, but those are quick jobs so it might be worth it to run them a few more time on a pipeline.

I'm afraid I don't follow.

Sorry, I just meant that the whole test suite is run twice, once in
https://gitlab.com/xen-project/hardware/xen/-/pipelines
and once in
https://gitlab.com/xen-project/xen/-/pipelines
(with just a few tests that are differents)

Quoted 3 lines

Obviously, 5 is arbitrary.  I was thinking of having a scheduled (once per day, at 3AM or whenever) running randconfig only with a parallel:20 or so, so we can make good use of otherwise-idle time.

That sounds like a good idea.

Cheers,

--

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#7 lore raw

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

Quoted 11 lines · Anthony PERARD
Quoted 5 lines · Andrew Cooper
Quoted 3 lines · Anthony PERARD

We already have "parallel: 2" on the whole pipeline, give or take, but those are quick jobs so it might be worth it to run them a few more time on a pipeline.

I'm afraid I don't follow.

Sorry, I just meant that the whole test suite is run twice, once in
https://gitlab.com/xen-project/hardware/xen/-/pipelines
and once in
https://gitlab.com/xen-project/xen/-/pipelines
(with just a few tests that are differents)

And the rest.  Every time I re-sync my tree, and probably the same for the other maintainers.

However, these have nothing on the overhead from patchew, which would be far more palatable if we didn't essentially throw away the result by not sending it anywhere useful.

~Andrew

#8 lore raw

PATCH v2 3/12 CI: Drop Debian Jessie dockerfiles

In reply to Andrew Cooper

These were removed from testing in Xen 4.18.

Fixes 3817e3c1b4b8 ("automation: Remove testing on Debian Jessie")

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 .../build/debian/jessie-i386.dockerfile       | 55 -------------------
 automation/build/debian/jessie.dockerfile     | 52 ------------------
 automation/scripts/containerize               |  2 -
 3 files changed, 109 deletions(-)
 delete mode 100644 automation/build/debian/jessie-i386.dockerfile
 delete mode 100644 automation/build/debian/jessie.dockerfile
automation/build/debian/jessie-i386.dockerfile deleted +0 -55
deleted file mode 100644
index dfa6dae06cb8..000000000000
@@ -1,55 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/i386 debian/eol:jessie
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-ENTRYPOINT ["linux32"]
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ jessie main contrib non-free
-deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ jessie/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libc6-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
automation/build/debian/jessie.dockerfile deleted +0 -52

52 lines not on this page · raw message

automation/scripts/containerize +0 -2
index 03bc4837350c..57db42d10cb5 100755
@@ -31,8 +31,6 @@ case "_${CONTAINER}" in
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
-    _jessie) CONTAINER="${BASE}/debian:jessie" ;;
-    _jessie-i386) CONTAINER="${BASE}/debian:jessie-i386" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
     _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
     _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;;
Signature · 1 line
2.39.2
Reviewed-by on v2 3/12
#9 lore raw

Re: [PATCH 03/12] CI: Drop Debian Jessie dockerfiles

Quoted 4 lines · Andrew Cooper

These were removed from testing in Xen 4.18.

Fixes 3817e3c1b4b8 ("automation: Remove testing on Debian Jessie")

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#10 lore raw

PATCH v2 4/12 CI: Drop Debian Stretch testing

In reply to Andrew Cooper

Debian stretch is also End of Life. Update a couple of test steps to use bookworm instead.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 .../build/debian/stretch-i386.dockerfile      | 57 ------------------
 automation/build/debian/stretch.dockerfile    | 60 -------------------
 automation/gitlab-ci/build.yaml               | 34 +----------
 automation/gitlab-ci/test.yaml                |  4 +-
 automation/scripts/containerize               |  2 -
 5 files changed, 4 insertions(+), 153 deletions(-)
 delete mode 100644 automation/build/debian/stretch-i386.dockerfile
 delete mode 100644 automation/build/debian/stretch.dockerfile
automation/build/debian/stretch-i386.dockerfile deleted +0 -57
deleted file mode 100644
index 905096ab0ef1..000000000000
@@ -1,57 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/i386 debian:stretch
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-ENTRYPOINT ["linux32"]
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ stretch main contrib non-free
-deb http://archive.debian.org/debian/ stretch-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ stretch/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libc6-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        apt-transport-https \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
automation/build/debian/stretch.dockerfile deleted +0 -60

60 lines not on this page · raw message

automation/gitlab-ci/build.yaml +2 -32
index da7b8885aa30..2c6eb17bef50 100644
@@ -354,10 +354,10 @@ alpine-3.18-gcc-debug:
   variables:
     CONTAINER: alpine:3.18
 
-debian-stretch-gcc-debug:
+debian-bookworm-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: debian:stretch
+    CONTAINER: debian:bookworm
 
 debian-bookworm-clang-debug:
   extends: .clang-x86-64-build-debug
@@ -589,31 +589,6 @@ centos-7-gcc-debug:
   variables:
     CONTAINER: centos:7
 
-debian-stretch-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-32-clang-debug:
-  extends: .clang-x86-32-build-debug
-  variables:
-    CONTAINER: debian:stretch-i386
-
-debian-stretch-32-gcc-debug:
-  extends: .gcc-x86-32-build-debug
-  variables:
-    CONTAINER: debian:stretch-i386
-
 debian-buster-gcc-ibt:
   extends: .gcc-x86-64-build
   variables:
@@ -632,11 +607,6 @@ debian-bookworm-gcc:
   variables:
     CONTAINER: debian:bookworm
 
-debian-bookworm-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:bookworm
-
 debian-bookworm-gcc-randconfig:
   extends: .gcc-x86-64-build
   variables:
automation/gitlab-ci/test.yaml +2 -2
index 898d2adc8c5b..d50e522f9f79 100644
@@ -412,7 +412,7 @@ qemu-smoke-x86-64-gcc:
   script:
     - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
   needs:
-    - debian-stretch-gcc-debug
+    - debian-bookworm-gcc-debug
 
 qemu-smoke-x86-64-clang:
   extends: .qemu-x86-64
@@ -426,7 +426,7 @@ qemu-smoke-x86-64-gcc-pvh:
   script:
     - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
   needs:
-    - debian-stretch-gcc-debug
+    - debian-bookworm-gcc-debug
 
 qemu-smoke-x86-64-clang-pvh:
   extends: .qemu-x86-64
automation/scripts/containerize +0 -2
index 57db42d10cb5..5140765cee98 100755
@@ -32,8 +32,6 @@ case "_${CONTAINER}" in
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
-    _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
-    _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
Signature · 1 line
2.39.2
Reviewed-by on v2 4/12
#11 lore raw

Re: [PATCH 04/12] CI: Drop Debian Stretch testing

Quoted 4 lines · Andrew Cooper

Debian stretch is also End of Life. Update a couple of test steps to use bookworm instead.

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#12 lore raw

PATCH v2 5/12 CI: Drop Ubuntu Trusty testing

In reply to Andrew Cooper

This is also End of Life.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/build/ubuntu/trusty.dockerfile | 51 -----------------------
 automation/gitlab-ci/build.yaml           | 12 ------
 automation/scripts/containerize           |  1 -
 3 files changed, 64 deletions(-)
 delete mode 100644 automation/build/ubuntu/trusty.dockerfile
automation/build/ubuntu/trusty.dockerfile deleted +0 -51
deleted file mode 100644
index 1cda1bbd3fea..000000000000
@@ -1,51 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:14.04
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-
-# Remove expired certificate that Let's Encrypt certificates used to relie on.
-# (Not needed anymore)
-RUN sed -i 's#mozilla/DST_Root_CA_X3\.crt#!\0#' /etc/ca-certificates.conf && \
-    update-ca-certificates
automation/gitlab-ci/build.yaml +0 -12
index 2c6eb17bef50..3e299f34f461 100644
@@ -633,18 +633,6 @@ fedora-gcc-debug:
   variables:
     CONTAINER: fedora:29
 
-# Ubuntu Trusty's Clang is 3.4 while Xen requires 3.5
-
-ubuntu-trusty-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: ubuntu:trusty
-
-ubuntu-trusty-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:trusty
-
 ubuntu-xenial-clang:
   extends: .clang-x86-64-build
   variables:
automation/scripts/containerize +0 -1
index 5140765cee98..e9ad20aa0c80 100755
@@ -39,7 +39,6 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _trusty) CONTAINER="${BASE}/ubuntu:trusty" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
Signature · 1 line
2.39.2
#14 lore raw

PATCH v2 6/12 CI: Mark Archlinux/x86 as allowing failures

In reply to Andrew Cooper

Archlinux is a rolling distro, and is subject to arbitrary changes as the container is rebuilt.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/gitlab-ci/build.yaml | 2 ++
 1 file changed, 2 insertions(+)
automation/gitlab-ci/build.yaml +2 -0
index 3e299f34f461..1446bc1d0ea7 100644
@@ -573,11 +573,13 @@ archlinux-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: archlinux:current
+  allow_failure: true
 
 archlinux-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
     CONTAINER: archlinux:current
+  allow_failure: true
 
 centos-7-gcc:
   extends: .gcc-x86-64-build
Signature · 1 line
2.39.2
Reviewed-by on v2 6/12
#15 lore raw

Re: [PATCH 06/12] CI: Mark Archlinux/x86 as allowing failures

Quoted 2 lines · Andrew Cooper

Archlinux is a rolling distro, and is subject to arbitrary changes as the container is rebuilt.

I disagree about the changes been arbitrary, it is just new version of software. :-)

But I guess "arbirary changes" could be a description for package been removed from the main/official archive, and left for the users to build themselfs.

Quoted 1 line

Anyway:

(Next, automatically rebuild the container every month, I could probably revive my old patch series doing that)

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#16 lore raw

Re: [PATCH 06/12] CI: Mark Archlinux/x86 as allowing failures

Quoted 9 lines · Anthony PERARD
Quoted 2 lines · Andrew Cooper

Archlinux is a rolling distro, and is subject to arbitrary changes as the container is rebuilt.

I disagree about the changes been arbitrary, it is just new version of software. :-)

But I guess "arbirary changes" could be a description for package been removed from the main/official archive, and left for the users to build themselfs.

Specifically, rebuilding the container for staging has a side effect of jumping the toolchain version under all stable trees.

I'll see about tweaking the wording.

Quoted 4 lines
Quoted 2 lines

Anyway:

Thanks.

Quoted 2 lines

(Next, automatically rebuild the container every month, I could probably revive my old patch series doing that)

That's also on the list, but not for 4.19.  Qemu rebuild their containers as the first pipeline stage, and while I think that's overkill, I was going to see if I could turn it into a weekly(?) scheduled job.

~Andrew

#17 lore raw

PATCH v2 7/12 CI: Introduce a debian:12-ppc64le container

In reply to Andrew Cooper

Make it non-root by default, and set XEN_TARGET_ARCH=ppc64 too.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/build/debian/12-ppc64le.dockerfile | 36 +++++++++++++++++++
 automation/scripts/containerize               |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 automation/build/debian/12-ppc64le.dockerfile
automation/build/debian/12-ppc64le.dockerfile new file +36 -0
new file mode 100644
index 000000000000..3b311e675ef5
@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-ppc
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/scripts/containerize +1 -0
index e9ad20aa0c80..aa3433f0dc98 100755
@@ -32,6 +32,7 @@ case "_${CONTAINER}" in
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
+    _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
Signature · 1 line
2.39.2
Reviewed-by on v2 7/12
#18 lore raw

Re: [PATCH 07/12] CI: Introduce a debian:12-ppc64le container

Quoted 6 lines · Andrew Cooper
automation/build/debian/12-ppc64le.dockerfile new file +0 -0
new file mode 100644
index 000000000000..3b311e675ef5
@@ -0,0 +1,36 @@

...

automation/build/debian/12-ppc64le.dockerfile line 8

Quoted 2 lines
automation/build/debian/12-ppc64le.dockerfile new file +2 -0
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64

I don't really like these two envvars. The second one is already set in the env by gitlab, the first one could in theory also come from gitlab-ci.

But I guess that comment comes from the fact that I don't really know why we would want a second bookworm containers with only a few packages.

Quoted 14 lines
automation/build/debian/12-ppc64le.dockerfile new file +14 -0
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy

automation/build/debian/12-ppc64le.dockerfile line 32

Quoted 10 lines
automation/build/debian/12-ppc64le.dockerfile new file +10 -0
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-ppc
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*

Other containers do also apt-get clean is this not needed? Or maybe the original containers is setup to not save packages?

/me looking into /var/cache

Looks empty enough, so I guess apt-get clean is optional.

Changes looks good enough, so:

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#19 lore raw

Re: [PATCH 07/12] CI: Introduce a debian:12-ppc64le container

Quoted 8 lines · Anthony PERARD
Quoted 6 lines · Andrew Cooper
automation/build/debian/12-ppc64le.dockerfile new file +0 -0
new file mode 100644
index 000000000000..3b311e675ef5
@@ -0,0 +1,36 @@

...

Quoted 5 lines

automation/build/debian/12-ppc64le.dockerfile line 8

Quoted 2 lines
automation/build/debian/12-ppc64le.dockerfile new file +2 -0
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64

I don't really like these two envvars. The second one is already set in the env by gitlab, the first one could in theory also come from gitlab-ci.

The first one is pre-existing.  (It shows as new because enough of the file changed for it not to be deemed a rename).

The second is something I added, for improved usability.

Quoted 2 lines

But I guess that comment comes from the fact that I don't really know why we would want a second bookworm containers with only a few packages.

Right now (for better or worse) - we have single purpose containers for a single arch.

For PPC and RISCV, where we're only doing a cross-build of the hypervisor, the dependencies are almost identical.  But the moment we start doing userspace, it will diverge quickly.

Part of the work I'm doing is trimming the containers down.  Container size is a relevant factor for how long the CI tests take.  Also, with smaller containers, the "oops, out of disk space, delete everything" logic on various runners will trigger less frequently, so we get better cache utilisation.

We do not IMO want to have containers with a union of all deps of all architectures.  They'd be giant.

So while we do have single-purpose containers, these ENV vars are ~mandatory for using the container for it's intended purpose, and it's far more usable for a human when you can just enter the container and type make.

Quoted 27 lines
Quoted 14 lines
automation/build/debian/12-ppc64le.dockerfile new file +14 -0
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy

automation/build/debian/12-ppc64le.dockerfile line 32

Quoted 10 lines
automation/build/debian/12-ppc64le.dockerfile new file +10 -0
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-ppc
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*

Other containers do also apt-get clean is this not needed? Or maybe the original containers is setup to not save packages?

Quoted 3 lines

/me looking into /var/cache

Looks empty enough, so I guess apt-get clean is optional.

https://docs.docker.com/build/building/best-practices/ notes that the official Debian and Ubuntu images automatically clean themselves, so you don't need to do it manually.

https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105

Quoted 2 lines

Changes looks good enough, so:

Thanks.

~Andrew

#20 lore raw

PATCH v2 8/12 CI: Use debian:12-ppc64le for both build and test

In reply to Andrew Cooper

qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't understand the powernv9 machine.

However, bookworm's Qemu does, and this is preforable to maintaining a random build of Qemu ourselves.

Add bookworm builds, and retain bullseye to keep our lower-bound toolchain testing.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and reposition the jobs to optimise starting the smoke test.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/gitlab-ci/build.yaml               | 59 ++++++++++---------
 automation/gitlab-ci/test.yaml                |  5 +-
 automation/scripts/qemu-smoke-ppc64le.sh      |  4 +-
 .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  | 38 ------------
 4 files changed, 34 insertions(+), 72 deletions(-)
 delete mode 100644 automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile
automation/gitlab-ci/build.yaml +30 -29
index 1446bc1d0ea7..316dc4345f29 100644
@@ -302,20 +302,6 @@ qemu-system-aarch64-6.0.0-arm32-export:
   tags:
     - arm64
 
-# ppc64 test artifacts
-
-qemu-system-ppc64-8.1.0-ppc64-export:
-  extends: .test-jobs-artifact-common
-  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-ppc64:8.1.0-ppc64
-  script:
-    - mkdir binaries && cp /qemu-system-ppc64 /skiboot.lid binaries/
-  artifacts:
-    paths:
-      - binaries/qemu-system-ppc64
-      - binaries/skiboot.lid
-  tags:
-    - x86_64
-
 # x86_64 test artifacts
 
 alpine-3.18-rootfs-export:
@@ -364,6 +350,13 @@ debian-bookworm-clang-debug:
   variables:
     CONTAINER: debian:bookworm
 
+debian-12-ppc64le-gcc-debug:
+  extends: .gcc-ppc64le-cross-build-debug
+  variables:
+    CONTAINER: debian:12-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
 # Arm32 cross-build
 
 debian-bookworm-gcc-arm32:
@@ -500,21 +493,6 @@ archlinux-current-gcc-riscv64-randconfig:
     RANDCONFIG: y
     <<: *riscv-fixed-randconfig
 
-# Power cross-build
-debian-bullseye-gcc-ppc64le:
-  extends: .gcc-ppc64le-cross-build
-  variables:
-    CONTAINER: debian:bullseye-ppc64le
-    KBUILD_DEFCONFIG: ppc64_defconfig
-    HYPERVISOR_ONLY: y
-
-debian-bullseye-gcc-ppc64le-debug:
-  extends: .gcc-ppc64le-cross-build-debug
-  variables:
-    CONTAINER: debian:bullseye-ppc64le
-    KBUILD_DEFCONFIG: ppc64_defconfig
-    HYPERVISOR_ONLY: y
-
 # Yocto test jobs
 yocto-qemuarm64:
   extends: .yocto-test-arm64
@@ -738,3 +716,26 @@ opensuse-tumbleweed-gcc-debug:
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
+
+# PowerPC builds (x86 cross)
+debian-bullseye-gcc-ppc64le:
+  extends: .gcc-ppc64le-cross-build
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-bullseye-gcc-ppc64le-debug:
+  extends: .gcc-ppc64le-cross-build-debug
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-12-ppc64le-gcc:
+  extends: .gcc-ppc64le-cross-build
+  variables:
+    CONTAINER: debian:12-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
automation/gitlab-ci/test.yaml +2 -3
index d50e522f9f79..70c946f87799 100644
@@ -69,7 +69,7 @@
 .qemu-ppc64le:
   extends: .test-jobs-common
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:12-ppc64le
     LOGFILE: qemu-smoke-ppc64le.log
   artifacts:
     paths:
@@ -447,5 +447,4 @@ qemu-smoke-ppc64le-powernv9-gcc:
   script:
     - ./automation/scripts/qemu-smoke-ppc64le.sh powernv9 2>&1 | tee ${LOGFILE}
   needs:
-    - qemu-system-ppc64-8.1.0-ppc64-export
-    - debian-bullseye-gcc-ppc64le-debug
+    - debian-12-ppc64le-gcc-debug
automation/scripts/qemu-smoke-ppc64le.sh +2 -2
index 701d81107d18..9088881b731b 100755
@@ -12,8 +12,8 @@ rm -f ${serial_log}
 set +e
 
 timeout -k 1 20 \
-binaries/qemu-system-ppc64 \
-    -bios binaries/skiboot.lid \
+qemu-system-ppc64 \
+    -bios skiboot.lid \
     -M $machine \
     -m 2g \
     -smp 1 \
automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile deleted +0 -38
deleted file mode 100644
index e28d68649239..000000000000
@@ -1,38 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:bullseye-slim
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV QEMU_VERSION=8.1.0
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        curl \
-        python3 \
-        python3-pip \
-        python3-elementpath \
-        ninja-build \
-        pkg-config \
-        libglib2.0-dev \
-        libpixman-1-dev \
-        && \
-    \
-    curl -fsSLO https://download.qemu.org/qemu-"$QEMU_VERSION".tar.xz && \
-    tar xvJf qemu-"$QEMU_VERSION".tar.xz && \
-    cd qemu-"$QEMU_VERSION" && \
-    ./configure --target-list=ppc64-softmmu && \
-    make -j$(nproc) && \
-    cp ./build/qemu-system-ppc64 / && \
-    cp ./build/qemu-bundle/usr/local/share/qemu/skiboot.lid / && \
-    cd /build && \
-    rm -rf qemu-"$QEMU_VERSION"* && \
-    apt-get autoremove -y && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
Signature · 1 line
2.39.2
Reviewed-by on v2 8/12
#21 lore raw

Re: [PATCH 08/12] CI: Use debian:12-ppc64le for both build and test

Quoted 1 line · Andrew Cooper

qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't

QEMU is normally spelled "QEMU" unless we are in camel case context, or lower case context.

Quoted 3 lines

understand the powernv9 machine.

However, bookworm's Qemu does, and this is preforable to maintaining a random

"preforable", that's an intended mispell right? :-)

Quoted 6 lines

build of Qemu ourselves.

Add bookworm builds, and retain bullseye to keep our lower-bound toolchain testing.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and

There's no renamed jobs in this patch, but the new jobs does follow the new naming schema. Did you intend to rename both "debian-bullseye-gcc-ppc64le*" jobs?

Quoted 3 lines

reposition the jobs to optimise starting the smoke test.

Patch is mostly ok, so:

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#22 lore raw

Re: [PATCH 08/12] CI: Use debian:12-ppc64le for both build and test

Quoted 9 lines · Anthony PERARD
Quoted 1 line · Andrew Cooper

qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't

QEMU is normally spelled "QEMU" unless we are in camel case context, or lower case context.

Quoted 3 lines

understand the powernv9 machine.

However, bookworm's Qemu does, and this is preforable to maintaining a random

"preforable", that's an intended mispell right? :-)

Yes.  both fixed.

Quoted 9 lines
Quoted 7 lines

build of Qemu ourselves.

Add bookworm builds, and retain bullseye to keep our lower-bound toolchain testing.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and

There's no renamed jobs in this patch, but the new jobs does follow the new naming schema.

Oh - good point.  I'll tweak the wording.

Quoted 2 lines

Did you intend to rename both
"debian-bullseye-gcc-ppc64le*" jobs?

No - that's done in the following patch.

Quoted 6 lines
Quoted 4 lines

reposition the jobs to optimise starting the smoke test.

Patch is mostly ok, so:

Thanks.

~Andrew

#23 lore raw

PATCH v2 9/12 CI: Refresh bullseye-ppc64le as debian:11-ppc64le

In reply to Andrew Cooper

... in the style of bookworm-ppc64le

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/build/debian/11-ppc64le.dockerfile | 33 +++++++++++++++++++
 .../build/debian/bullseye-ppc64le.dockerfile  | 32 ------------------
 automation/gitlab-ci/build.yaml               |  8 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 38 insertions(+), 37 deletions(-)
 create mode 100644 automation/build/debian/11-ppc64le.dockerfile
 delete mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile
automation/build/debian/11-ppc64le.dockerfile new file +33 -0
new file mode 100644
index 000000000000..d846b670d02c
@@ -0,0 +1,33 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bullseye-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/debian/bullseye-ppc64le.dockerfile deleted +0 -32
deleted file mode 100644
index e166d205f38c..000000000000
@@ -1,32 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:bullseye-slim
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-# Add compiler path
-ENV CROSS_COMPILE powerpc64le-linux-gnu-
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        checkpolicy \
-        flex \
-        gawk \
-        gcc-powerpc64le-linux-gnu \
-        make \
-        python3-minimal \
-        # QEMU runtime dependencies for test phase
-        libglib2.0-0 \
-        libpixman-1-0 \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
automation/gitlab-ci/build.yaml +4 -4
index 316dc4345f29..e081664c4e95 100644
@@ -718,17 +718,17 @@ opensuse-tumbleweed-gcc-debug:
   allow_failure: true
 
 # PowerPC builds (x86 cross)
-debian-bullseye-gcc-ppc64le:
+debian-11-ppc64le-gcc:
   extends: .gcc-ppc64le-cross-build
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:11-ppc64le
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
-debian-bullseye-gcc-ppc64le-debug:
+debian-11-ppc64le-gcc-debug:
   extends: .gcc-ppc64le-cross-build-debug
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:11-ppc64le
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
automation/scripts/containerize +1 -1
index aa3433f0dc98..5c3f6782d9cd 100755
@@ -31,7 +31,7 @@ case "_${CONTAINER}" in
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
-    _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
+    _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
Signature · 1 line
2.39.2
#25 lore raw

PATCH v2 10/12 CI: Introduce debian:11/12-riscv64 containers

In reply to Andrew Cooper

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-riscv64         772MB
  registry.gitlab.com/xen-project/xen/debian      11-riscv64         422MB
  registry.gitlab.com/xen-project/xen/archlinux   current-riscv64    2.32GB

They also not rolling distros, so will be more predicatable testing for the stable trees in the future.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 automation/build/debian/11-riscv64.dockerfile | 33 +++++++++++++++++
 automation/build/debian/12-riscv64.dockerfile | 36 +++++++++++++++++++
 2 files changed, 69 insertions(+)
 create mode 100644 automation/build/debian/11-riscv64.dockerfile
 create mode 100644 automation/build/debian/12-riscv64.dockerfile
automation/build/debian/11-riscv64.dockerfile new file +33 -0
new file mode 100644
index 000000000000..1c99bc89ea1a
@@ -0,0 +1,33 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bullseye-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/debian/12-riscv64.dockerfile new file +36 -0
new file mode 100644
index 000000000000..03fd4b03059b
@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-riscv64
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
Signature · 1 line
2.39.2
Reviewed-by on v2 10/12
#26 lore raw

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

Quoted 11 lines · Andrew Cooper

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-riscv64         772MB
  registry.gitlab.com/xen-project/xen/debian      11-riscv64         422MB
  registry.gitlab.com/xen-project/xen/archlinux   current-riscv64    2.32GB

They also not rolling distros, so will be more predicatable testing for the stable trees in the future.

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#27 lore raw

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

In reply to Andrew Cooper
Quoted 7 lines · Andrew Cooper

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-riscv64        
772MB
  registry.gitlab.com/xen-project/xen/debian      11-riscv64        
422MB

Do we really need both 11-riscv64 and 12-riscv64?

Generally this patch LGTM: Reviewed-by: Oleksii Kurochko <[email protected]>

~ Oleksii

Quoted 108 lines

  registry.gitlab.com/xen-project/xen/archlinux   current-riscv64 2.32GB

They also not rolling distros, so will be more predicatable testing for the
stable trees in the future.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
---
 automation/build/debian/11-riscv64.dockerfile | 33 +++++++++++++++++ automation/build/debian/12-riscv64.dockerfile | 36
+++++++++++++++++++

 2 files changed, 69 insertions(+)

 create mode 100644 automation/build/debian/11-riscv64.dockerfile create mode 100644 automation/build/debian/12-riscv64.dockerfile

automation/build/debian/11-riscv64.dockerfile new file +33 -0
new file mode 100644
index 000000000000..1c99bc89ea1a
@@ -0,0 +1,33 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bullseye-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/debian/12-riscv64.dockerfile new file +36 -0
new file mode 100644
index 000000000000..03fd4b03059b
@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-riscv64
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
#28 lore raw

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

Quoted 9 lines · Oleksii
Quoted 7 lines · Andrew Cooper

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-riscv64        
772MB
  registry.gitlab.com/xen-project/xen/debian      11-riscv64        
422MB

Do we really need both 11-riscv64 and 12-riscv64?

Need? No, not strictly.

Want? Yes, absolutely.

You always want at least 2 different toolchains worth of testing, or what you will find happens is that you end up accidentally depending on a quirk of the single compiler your using, and that you discover this at some point in the future, rather than now(ish) when CI says no.

At the moment, the RISC-V builds are very trivial and quick, so it makes a lot of sense to have a second toolchain.  Before too much longer, you'll want to get Clang working too.

Quoted 2 lines

Generally this patch LGTM: Reviewed-by: Oleksii Kurochko <[email protected]>

Thanks.

~Andrew

#29 lore raw

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

Quoted 24 lines · Andrew Cooper
Quoted 11 lines · Oleksii
Quoted 9 lines · Andrew Cooper

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-
riscv64        
772MB
  registry.gitlab.com/xen-project/xen/debian      11-
riscv64        
422MB

Do we really need both 11-riscv64 and 12-riscv64?

Need? No, not strictly.

Want? Yes, absolutely.

You always want at least 2 different toolchains worth of testing, or what you will find happens is that you end up accidentally depending on
a quirk of the single compiler your using, and that you discover this at
some point in the future, rather than now(ish) when CI says no.

Quoted 4 lines

At the moment, the RISC-V builds are very trivial and quick, so it makes
a lot of sense to have a second toolchain.  Before too much longer, you'll want to get Clang working too.

Agree, I should consider to add Clang. I will do that in the one of my nearest patch series.

~ Oleksii

#30 lore raw

PATCH v2 11/12 CI: Swap to debian for riscv64 build and test

In reply to Andrew Cooper

The containers are both much smaller, with stable toolchains over time, and this at least means we're not doing all testing with a single compiler.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and reposition the jobs to optimise starting the smoke test.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 .../archlinux/current-riscv64.dockerfile      | 22 -----
 automation/gitlab-ci/build.yaml               | 83 +++++++++++--------
 automation/gitlab-ci/test.yaml                |  4 +-
 automation/scripts/containerize               |  3 +-
 4 files changed, 52 insertions(+), 60 deletions(-)
 delete mode 100644 automation/build/archlinux/current-riscv64.dockerfile
automation/build/archlinux/current-riscv64.dockerfile deleted +0 -22
deleted file mode 100644
index f7770bf82a78..000000000000
@@ -1,22 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 archlinux
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-# Packages needed for the build
-RUN pacman --noconfirm --needed -Syu \
-    base-devel \
-    git \
-    inetutils \
-    riscv64-linux-gnu-binutils \
-    riscv64-linux-gnu-gcc \
-    riscv64-linux-gnu-glibc \
-    # For test phase
-    qemu-system-riscv
-
-# Add compiler path
-ENV CROSS_COMPILE=riscv64-linux-gnu-
-
-RUN useradd --create-home user
-USER user
-WORKDIR /build
automation/gitlab-ci/build.yaml +48 -35

98 lines not on this page · raw message

automation/gitlab-ci/test.yaml +2 -2
index 70c946f87799..f7e1753ead34 100644
@@ -56,7 +56,7 @@
 .qemu-riscv64:
   extends: .test-jobs-common
   variables:
-    CONTAINER: archlinux:current-riscv64
+    CONTAINER: debian:12
     LOGFILE: qemu-smoke-riscv64.log
   artifacts:
     paths:
@@ -440,7 +440,7 @@ qemu-smoke-riscv64-gcc:
   script:
     - ./automation/scripts/qemu-smoke-riscv64.sh 2>&1 | tee ${LOGFILE}
   needs:
-    - archlinux-current-gcc-riscv64-debug
+    - debian-12-riscv64-gcc-debug
 
 qemu-smoke-ppc64le-powernv9-gcc:
   extends: .qemu-ppc64le
automation/scripts/containerize +2 -1
index 5c3f6782d9cd..6dbf55c8f89a 100755
@@ -27,12 +27,13 @@ case "_${CONTAINER}" in
     _alpine) CONTAINER="${BASE}/alpine:3.18" ;;
     _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.18-arm64v8" ;;
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
-    _riscv64) CONTAINER="${BASE}/archlinux:current-riscv64" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
+    _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
+    _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
Signature · 1 line
2.39.2
Reviewed-by on v2 11/12
#31 lore raw

Re: [PATCH 11/12] CI: Swap to debian for riscv64 build and test

Quoted 7 lines · Andrew Cooper

The containers are both much smaller, with stable toolchains over time, and this at least means we're not doing all testing with a single compiler.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and reposition the jobs to optimise starting the smoke test.

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#32 lore raw

Re: [PATCH 11/12] CI: Swap to debian for riscv64 build and test

In reply to Andrew Cooper
Quoted 10 lines · Andrew Cooper

The containers are both much smaller, with stable toolchains over time, and
this at least means we're not doing all testing with a single compiler.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH- *) and
reposition the jobs to optimise starting the smoke test.

LGTM: Oleksii Kurochko <[email protected]>

~ Oleksii

Quoted 198 lines

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
---
 .../archlinux/current-riscv64.dockerfile      | 22 ----- automation/gitlab-ci/build.yaml               | 83 +++++++++++------ --
 automation/gitlab-ci/test.yaml                |  4 +- automation/scripts/containerize               |  3 +-

 4 files changed, 52 insertions(+), 60 deletions(-)

 delete mode 100644 automation/build/archlinux/current- riscv64.dockerfile

automation/build/archlinux/current-riscv64.dockerfile deleted +0 -22
deleted file mode 100644
index f7770bf82a78..000000000000
@@ -1,22 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 archlinux
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-# Packages needed for the build
-RUN pacman --noconfirm --needed -Syu \
-    base-devel \
-    git \
-    inetutils \
-    riscv64-linux-gnu-binutils \
-    riscv64-linux-gnu-gcc \
-    riscv64-linux-gnu-glibc \
-    # For test phase
-    qemu-system-riscv
-
-# Add compiler path
-ENV CROSS_COMPILE=riscv64-linux-gnu-
-
-RUN useradd --create-home user
-USER user
-WORKDIR /build
automation/gitlab- renamed from automation/gitlab-ci/build.yaml +0 -0

ci/build.yaml
index e081664c4e95..4b9d80cc5632 100644

automation/gitlab-ci/build.yaml +0 -0
@@ -357,6 +357,13 @@ debian-12-ppc64le-gcc-debug:

     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y

automation/gitlab-ci/build.yaml +7 -0
+debian-12-riscv64-gcc-debug:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+

 # Arm32 cross-build

 debian-bookworm-gcc-arm32:

automation/gitlab-ci/build.yaml +0 -0
@@ -458,41 +465,6 @@ alpine-3.18-gcc-debug-arm64-earlyprintk:

       CONFIG_EARLY_UART_CHOICE_PL011=y
       CONFIG_EARLY_UART_BASE_ADDRESS=0x9000000

automation/gitlab-ci/build.yaml +0 -35
-# RISC-V 64 cross-build
-.riscv-fixed-randconfig:
-  variables: &riscv-fixed-randconfig
-    EXTRA_FIXED_RANDCONFIG: |
-      CONFIG_BOOT_TIME_CPUPOOLS=n
-      CONFIG_COVERAGE=n
-      CONFIG_EXPERT=y
-      CONFIG_GRANT_TABLE=n
-      CONFIG_MEM_ACCESS=n
-      CONFIG_PERF_COUNTERS=n
-      CONFIG_LIVEPATCH=n
-      CONFIG_XSM=n
-
-archlinux-current-gcc-riscv64:
-  extends: .gcc-riscv64-cross-build
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    HYPERVISOR_ONLY: y
-
-archlinux-current-gcc-riscv64-debug:
-  extends: .gcc-riscv64-cross-build-debug
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    HYPERVISOR_ONLY: y
-
-archlinux-current-gcc-riscv64-randconfig:
-  extends: .gcc-riscv64-cross-build
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    RANDCONFIG: y
-    <<: *riscv-fixed-randconfig
-

 # Yocto test jobs
 yocto-qemuarm64:
   extends: .yocto-test-arm64

automation/gitlab-ci/build.yaml +0 -0
@@ -739,3 +711,44 @@ debian-12-ppc64le-gcc:

     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y

automation/gitlab-ci/build.yaml +41 -0
+# RISC-V 64 cross-build
+debian-11-riscv64-gcc:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:11-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-11-riscv64-gcc-debug:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: debian:11-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-12-riscv64-gcc:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+.riscv-fixed-randconfig:
+  variables: &riscv-fixed-randconfig
+    EXTRA_FIXED_RANDCONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=n
+      CONFIG_COVERAGE=n
+      CONFIG_EXPERT=y
+      CONFIG_GRANT_TABLE=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_XSM=n
+
+debian-12-riscv64-gcc-randconfig:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    RANDCONFIG: y
+    <<: *riscv-fixed-randconfig
automation/gitlab- renamed from automation/gitlab-ci/test.yaml +0 -0

ci/test.yaml
index 70c946f87799..f7e1753ead34 100644

automation/gitlab-ci/test.yaml +0 -0
@@ -56,7 +56,7 @@

 .qemu-riscv64:
   extends: .test-jobs-common
   variables:
-    CONTAINER: archlinux:current-riscv64
+    CONTAINER: debian:12
     LOGFILE: qemu-smoke-riscv64.log
   artifacts:
     paths:

automation/gitlab-ci/test.yaml +0 -0
@@ -440,7 +440,7 @@ qemu-smoke-riscv64-gcc:

   script:
     - ./automation/scripts/qemu-smoke-riscv64.sh 2>&1 | tee ${LOGFILE}
   needs:
-    - archlinux-current-gcc-riscv64-debug
+    - debian-12-riscv64-gcc-debug

 qemu-smoke-ppc64le-powernv9-gcc:
   extends: .qemu-ppc64le

automation/scripts/containerize +2 -1
index 5c3f6782d9cd..6dbf55c8f89a 100755
@@ -27,12 +27,13 @@ case "_${CONTAINER}" in
      _alpine) CONTAINER="${BASE}/alpine:3.18" ;;
      _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.18-arm64v8" ;;
      _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
-    _riscv64) CONTAINER="${BASE}/archlinux:current-riscv64" ;;
      _centos7) CONTAINER="${BASE}/centos:7" ;;
      _fedora) CONTAINER="${BASE}/fedora:29";;
      _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
      _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
      _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
+    _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
+    _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
      _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
      _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
      _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
#33 lore raw

PATCH v2 12/12 CI: Refresh and upgrade the GCC-IBT container

In reply to Andrew Cooper

Upgrade from Debian buster to bookworm, GCC 11.3 to 11.4 and to be a non-root container.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

---
 ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} | 81 +++++++++++--------
 automation/gitlab-ci/build.yaml               |  4 +-
 automation/scripts/containerize               |  2 +-
 3 files changed, 49 insertions(+), 38 deletions(-)
 rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12-x86_64-gcc-ibt.dockerfile} (50%)
automation/build/debian/12-x86_64-gcc-ibt.dockerfile renamed from automation/build/debian/buster-gcc-ibt.dockerfile +46 -35
similarity index 50%
rename from automation/build/debian/buster-gcc-ibt.dockerfile
rename to automation/build/debian/12-x86_64-gcc-ibt.dockerfile
index ed9367aafbec..3ce60e7d9466 100644
@@ -1,28 +1,33 @@
 # syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:buster-slim AS builder
+FROM --platform=linux/amd64 debian:bookworm-slim AS builder
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
 
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        ca-certificates \
-        flex \
-        g++-multilib \
-        libc6-dev-i386 \
-        libgmp-dev \
-        libisl-dev \
-        libmpc-dev \
-        libmpfr-dev \
-        patch \
+RUN <<EOF
+#!/bin/bash
+    set -e
+    apt-get -y update
+
+    DEPS=(
+        bison
+        build-essential
+        ca-certificates
+        flex
+        g++-multilib
+        libc6-dev-i386
+        libgmp-dev
+        libisl-dev
+        libmpc-dev
+        libmpfr-dev
+        patch
         wget
+    )
 
-RUN mkdir /build
-WORKDIR /build
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
 
-RUN wget -q https://ftp.gnu.org/gnu/gcc/gcc-11.3.0/gcc-11.3.0.tar.xz -O - | tar xJ --strip=1
+RUN wget -q https://ftp.gnu.org/gnu/gcc/gcc-11.4.0/gcc-11.4.0.tar.xz -O - | tar xJ --strip=1
 RUN wget -q https://xenbits.xen.org/people/andrewcoop/gcc-11.2-Add-fcf-check-attribute-yes-no.patch -O - | patch -p1
 RUN ./configure \
         --prefix=/opt/gcc-11-ibt \
@@ -42,29 +47,35 @@ RUN ./configure \
 RUN make -j`nproc` && make -j`nproc` install
 
 
-FROM --platform=linux/amd64 debian:buster-slim
+FROM --platform=linux/amd64 debian:bookworm-slim
 COPY --from=builder /opt/gcc-11-ibt /opt/gcc-11-ibt
 
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="[email protected]"
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
 ENV PATH="/opt/gcc-11-ibt/bin:${PATH}"
 
-RUN mkdir /build
-WORKDIR /build
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
 
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        checkpolicy \
-        flex \
-        gawk \
-        make \
-        python3-minimal \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/gitlab-ci/build.yaml +2 -2
index 4b9d80cc5632..b4139414bc8e 100644
@@ -541,10 +541,10 @@ centos-7-gcc-debug:
   variables:
     CONTAINER: centos:7
 
-debian-buster-gcc-ibt:
+debian-12-x86_64-gcc-ibt:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: debian:buster-gcc-ibt
+    CONTAINER: debian:12-x86_64-gcc-ibt
     RANDCONFIG: y
     EXTRA_FIXED_RANDCONFIG: |
       CONFIG_XEN_IBT=y
automation/scripts/containerize +1 -1
index 6dbf55c8f89a..4d5669c5ad01 100755
@@ -34,7 +34,7 @@ case "_${CONTAINER}" in
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
     _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
-    _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
+    _bookworm-x86_64-gcc-ibt) CONTAINER="${BASE}/debian:12-x86_64-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
     _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
Signature · 1 line
2.39.2
Reviewed-by on v2 12/12
#34 lore raw

Re: [PATCH 12/12] CI: Refresh and upgrade the GCC-IBT container

Quoted 4 lines · Andrew Cooper

Upgrade from Debian buster to bookworm, GCC 11.3 to 11.4 and to be a non-root container.

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#35 lore raw
In reply to Andrew Cooper

~ Oleksii

Quoted 93 lines · Andrew Cooper

Patch 1 fixes a bug in the containerize script

Patches 2-5 remove useless/obsolete testing

Patch 6 fixes a bug with the archlinux testing configuration

Patches 7-9 adjust the PPC64 testing

Patches 10-11 adjust the RISCV64 testing

Patch 12 refreshes the custom GCC-IBT container

All changes here follow best-guidance for dockerfiles (non-root, heredocs for
improved legibility), and naming consistency improvements discussed previously
on the Committers call.  Breifly that's:

  $DISTRO-$VERSION(numeric)-$ARCH-*

which sort more nicely and don't require e.g. people to remember which order
buster/bullseye/bookworm are in terms of debain release.

For x86, this involves inserting an $ARCH of x86_64.  Right now, x86_64 is
implied by the absence, and all other variations have to state one.

Still to come:

 * OpenSUSE container fixes (resolves the final rolling distro vs allowed_failure problem)
 * Ubuntu/debian rename/trim
 * Add new jobs for latest releases (Ubuntu, Alpine, Fedora)

Andrew Cooper (12):
  CI: Fix CONTAINER_UID0=1 scripts/containerize

  CI: Remove useless/misleading randconfig jobs
  CI: Drop Debian Jessie dockerfiles
  CI: Drop Debian Stretch testing
  CI: Drop Ubuntu Trusty testing

  CI: Mark Archlinux/x86 as allowing failures

  CI: Introduce a debian:12-ppc64le container
  CI: Use debian:12-ppc64le for both build and test
  CI: Refresh bullseye-ppc64le as debian:11-ppc64le

  CI: Introduce debian:11/12-riscv64 containers
  CI: Swap to debian for riscv64 build and test

  CI: Refresh and upgrade the GCC-IBT container

 .../archlinux/current-riscv64.dockerfile      |  22 --

 automation/build/debian/11-ppc64le.dockerfile |  33 +++
 automation/build/debian/11-riscv64.dockerfile |  33 +++
 automation/build/debian/12-ppc64le.dockerfile |  36 +++
 automation/build/debian/12-riscv64.dockerfile |  36 +++
 ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} |  81 +++---

 .../build/debian/bullseye-ppc64le.dockerfile  |  32 --- .../build/debian/jessie-i386.dockerfile       |  55 ----- automation/build/debian/jessie.dockerfile     |  52 ---- .../build/debian/stretch-i386.dockerfile      |  57 ----- automation/build/debian/stretch.dockerfile    |  60 ----- automation/build/ubuntu/trusty.dockerfile     |  51 ---- automation/gitlab-ci/build.yaml               | 233 +++++++--------- --
 automation/gitlab-ci/test.yaml                |  13 +- automation/scripts/containerize               |  15 +- automation/scripts/qemu-smoke-ppc64le.sh      |   4 +- .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  |  38 ---

 17 files changed, 282 insertions(+), 569 deletions(-)

 delete mode 100644 automation/build/archlinux/current- riscv64.dockerfile
 create mode 100644 automation/build/debian/11-ppc64le.dockerfile create mode 100644 automation/build/debian/11-riscv64.dockerfile create mode 100644 automation/build/debian/12-ppc64le.dockerfile create mode 100644 automation/build/debian/12-riscv64.dockerfile rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12- x86_64-gcc-ibt.dockerfile} (50%)
 delete mode 100644 automation/build/debian/bullseye- ppc64le.dockerfile
 delete mode 100644 automation/build/debian/jessie-i386.dockerfile delete mode 100644 automation/build/debian/jessie.dockerfile delete mode 100644 automation/build/debian/stretch-i386.dockerfile delete mode 100644 automation/build/debian/stretch.dockerfile delete mode 100644 automation/build/ubuntu/trusty.dockerfile delete mode 100644 automation/tests-artifacts/qemu-system- ppc64/8.1.0-ppc64.dockerfile

#36 lore raw

[PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

In reply to Andrew Cooper

Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially. Testing docs isn't very interesting and saves a lot of space (pandoc pulls in all of Haskel). Other savings come from removing a huge pile of optional QEMU dependencies (QEMU just needs to build the Xen parts to be useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies, which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this total overhaul.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
CC: Olaf Hering <[email protected]>

This includes both of Olaf's recent changes, python3-packaging (now required for a rebuilt container), and findutils (not required yet; still gets pulled in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine. staging-4.16 does not, because of SeaBIOS which we've decided not to fix, hence it's safe to drop which.

---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile
automation/build/opensuse/tumbleweed-x86_64.dockerfile new file +76 -0
new file mode 100644
index 000000000000..8c5ff56cdc88
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica
+        # libxl
+        libuuid-devel
+        libyajl-devel
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        glib2-devel
+        libpixman-1-0-devel
+        meson
+        ninja
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
automation/build/suse/opensuse-tumbleweed.dockerfile deleted +0 -74

74 lines not on this page · raw message

automation/gitlab-ci/build.yaml +4 -4
index b4139414bc8e..fd6ca1270e4e 100644
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
automation/scripts/containerize +1 -1
index 4d5669c5ad01..c4f013702122 100755
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
Signature · 1 line
2.39.2
#37 lore raw

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

Quoted 79 lines · Andrew Cooper

Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially. Testing docs isn't very interesting and saves a lot of space (pandoc pulls in all of Haskel). Other savings come from removing a huge pile of optional QEMU dependencies (QEMU just needs to build the Xen parts to be useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies, which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this total overhaul.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
CC: Olaf Hering <[email protected]>

This includes both of Olaf's recent changes, python3-packaging (now required for a rebuilt container), and findutils (not required yet; still gets pulled in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17
Quoted 3 lines

I've tested this on staging-4.17 and staging, both of which build fine. staging-4.16 does not, because of SeaBIOS which we've decided not to fix, hence it's safe to drop which.

For good measure, runs with this container:

  4.17: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322567620
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493465

and the expected failure:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499885

~Andrew

#38 lore raw

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

In reply to Andrew Cooper

Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <[email protected]>:

Quoted 1 line

use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Names of rpm packages do change over time, configure already searches for "something" where it is available, instead of some library or header. If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:

libbz2-devel

    pkgconfig(bzip2) = 1.0.8
libzstd-devel
    pkgconfig(libzstd) = 1.5.6
lzo-devel
    pkgconfig(lzo2) = 2.10
xz-devel
    pkgconfig(liblzma) = 5.6.2
zlib-devel
    pkgconfig(zlib) = 1.3.1
libuuid-devel
    pkgconfig(uuid) = 2.40.1
libyajl-devel
    pkgconfig(yajl) = 2.1.0
ncurses-devel
    pkgconfig(form) = 6.5.20240629
    pkgconfig(formw) = 6.5.20240629
    pkgconfig(menu) = 6.5.20240629
    pkgconfig(menuw) = 6.5.20240629
    pkgconfig(ncurses) = 6.5.20240629
    pkgconfig(ncurses++) = 6.5.20240629
    pkgconfig(ncurses++w) = 6.5.20240629
    pkgconfig(ncursesw) = 6.5.20240629
    pkgconfig(panel) = 6.5.20240629
    pkgconfig(panelw) = 6.5.20240629
    pkgconfig(tic) = 6.5.20240629
    pkgconfig(tinfo) = 6.5.20240629
glib2-devel
    pkgconfig(gio-2.0) = 2.80.3
    pkgconfig(gio-unix-2.0) = 2.80.3
    pkgconfig(girepository-2.0) = 2.80.3
    pkgconfig(glib-2.0) = 2.80.3
    pkgconfig(gmodule-2.0) = 2.80.3
    pkgconfig(gmodule-export-2.0) = 2.80.3
    pkgconfig(gmodule-no-export-2.0) = 2.80.3
    pkgconfig(gobject-2.0) = 2.80.3
    pkgconfig(gthread-2.0) = 2.80.3
libpixman-1-0-devel
    pkgconfig(pixman-1) = 0.43.4

python3-devel is "special", can be used as is.

'git' could be just 'git-core'

Olaf

#39 lore raw

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

Quoted 4 lines · Olaf Hering

Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <[email protected]>:

Quoted 1 line

use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Names of rpm packages do change over time, configure already searches for "something" where it is available,

configure has to cope with distros taking different naming decisions, and while I appreciate that packages may change over time even within a distro ...

Quoted 34 lines

instead of some library or header.
If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:

libbz2-devel

    pkgconfig(bzip2) = 1.0.8
libzstd-devel
    pkgconfig(libzstd) = 1.5.6
lzo-devel
    pkgconfig(lzo2) = 2.10
xz-devel
    pkgconfig(liblzma) = 5.6.2
zlib-devel
    pkgconfig(zlib) = 1.3.1
libuuid-devel
    pkgconfig(uuid) = 2.40.1
libyajl-devel
    pkgconfig(yajl) = 2.1.0
ncurses-devel
    pkgconfig(form) = 6.5.20240629
    pkgconfig(formw) = 6.5.20240629
    pkgconfig(menu) = 6.5.20240629
    pkgconfig(menuw) = 6.5.20240629
    pkgconfig(ncurses) = 6.5.20240629
    pkgconfig(ncurses++) = 6.5.20240629
    pkgconfig(ncurses++w) = 6.5.20240629
    pkgconfig(ncursesw) = 6.5.20240629
    pkgconfig(panel) = 6.5.20240629
    pkgconfig(panelw) = 6.5.20240629
    pkgconfig(tic) = 6.5.20240629
    pkgconfig(tinfo) = 6.5.20240629
glib2-devel
    pkgconfig(gio-2.0) = 2.80.3
    pkgconfig(gio-unix-2.0) = 2.80.3
    pkgconfig(girepository-2.0) = 2.80.3
Quoted 8 lines
    pkgconfig(glib-2.0) = 2.80.3
    pkgconfig(gmodule-2.0) = 2.80.3
    pkgconfig(gmodule-export-2.0) = 2.80.3
    pkgconfig(gmodule-no-export-2.0) = 2.80.3
    pkgconfig(gobject-2.0) = 2.80.3
    pkgconfig(gthread-2.0) = 2.80.3
libpixman-1-0-devel
    pkgconfig(pixman-1) = 0.43.4

... this is just a differently-inconsistent list of names, with some very dubious aliases such as form and menu.

What makes pkgconfig() any more stable than the main package name?

Why do we want to be treating libraries different to other packages?

Quoted 1 line

python3-devel is "special", can be used as is.

As an aside, I looked for a python3-minimal or equivalent, and couldn't find one.  I take it there isn't one?

Quoted 1 line

'git' could be just 'git-core'

Thanks.  Will fix locally, for Tumbleweed and Leap.

~Andrew

#40 lore raw

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

Thu, 11 Jul 2024 23:39:42 +0100 Andrew Cooper <[email protected]>:

Quoted 3 lines

configure has to cope with distros taking different naming decisions, and while I appreciate that packages may change over time even within a distro ...

configure does "pkgconfig --exists something" to probe for some API. I'm sure noone will rename the pkgconfig files, "something.pc" in this case.

Quoted 2 lines

... this is just a differently-inconsistent list of names, with some very dubious aliases such as form and menu.

I should have truncated that list of names, 'pkgconfig(ncurses)' and 'pkgconfig(glib-2.0)' are relevant.

Quoted 1 line

What makes pkgconfig() any more stable than the main package name?

They are supposed to describe some API, and in some cases some functionality. It does not matter much which package provides the API.

Quoted 1 line

Why do we want to be treating libraries different to other packages?

Because there is no other namespace to describe functionality. It was implemented for pkg-config, it exists for some cmake and perl modules.

Quoted 3 lines
Quoted 1 line

python3-devel is "special", can be used as is.

As an aside, I looked for a python3-minimal or equivalent, and couldn't find one.  I take it there isn't one?

I think there are libraries compiled. There is indeed a 'pkgconfig(python3)', my search just failed to find it. Otherwise, if there are really just .py files in the tree, 'python3-base' is the string provided by 'python3N-base'.

Olaf

#41 lore raw

[PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

In reply to Andrew Cooper

Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially. Testing docs isn't very interesting and saves a lot of space. Other savings come from removing a huge pile of optional QEMU dependencies (QEMU just needs to build the Xen parts to be useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies, which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this total overhaul.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
CC: Olaf Hering <[email protected]>

This includes both of Olaf's recent changes, python3-packaging (now required for a rebuilt container), and findutils (not required yet; still gets pulled in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine. staging-4.16 does not, because of SeaBIOS which we've decided not to fix, hence it's safe to drop which.

v2:
* git -> git-core
* Use 'pkgconfig()' references

---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile
automation/build/opensuse/tumbleweed-x86_64.dockerfile new file +76 -0
new file mode 100644
index 000000000000..61e840fc67da
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
automation/build/suse/opensuse-tumbleweed.dockerfile deleted +0 -74

74 lines not on this page · raw message

automation/gitlab-ci/build.yaml +4 -4
index b4139414bc8e..fd6ca1270e4e 100644
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
automation/scripts/containerize +1 -1
index 4d5669c5ad01..c4f013702122 100755
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
Signature · 1 line
2.39.2
Reviewed-by
#42 lore raw

Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

Quoted 3 lines · Andrew Cooper

while removed are:

bc # ?

I don't know why, but this one is also installed by osstest. The commit adding it doesn't say why it was needed.

Also, looks like many QEMU containers are installing bc, but I can't find a single use. Maybe historic.

Quoted 1 line

libnl3-devel # Libxl COLO(?), option

Yes.

Quoted 1 line

libnuma-devel # ?

There's some mention of it in QEMU.

Quoted 1 line

libpng16-devel # Qemu ?

It's added to a few QEMU containers, but I can't find any use.

Quoted 1 line

libssh2-devel # ?

It seems like it was used by QEMU at some point, but they switch to 'libssh'. Probably optional, like many QEMU deps.

Quoted 1 line

libtasn1-devel # ?

QEMU, and it seems to be only used in tests.

Quoted 1 line

nasm # ?

'nasm' is used to build OVMF.

If we don't want to build OVMF in our GitLab CI, which I don't think would be useful to do anyway (no dependency on xen.git), then:

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#43 lore raw

Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

Quoted 9 lines · Anthony PERARD
Quoted 3 lines · Andrew Cooper

while removed are:

bc # ?

I don't know why, but this one is also installed by osstest. The commit adding it doesn't say why it was needed.

Also, looks like many QEMU containers are installing bc, but I can't find a single use. Maybe historic.

Part of the reason why our containers are so massive is because dependencies only ever accumulate...

But yes - this one is especially odd.

Quoted 6 lines
Quoted 1 line

nasm # ?

'nasm' is used to build OVMF.

If we don't want to build OVMF in our GitLab CI, which I don't think would be useful to do anyway (no dependency on xen.git)

Oh, interesting.

I'd assumed (by virtue of the jobs succeeding) that we were building OVMF by default, but we're clearly not.

./configure says:

  --enable-ovmf           Enable OVMF (default is DISABLED)

so it's really just a simple opt-in, which our build script does not.

In the longer term, we will want OVMF in Gitlab CI, but we'd want to pull it in as a separate artefact and use --with-system-ovmf=/path/to/artefact

So I agree we don't need to build OVMF inside this container.

Quoted 3 lines

, then:

Thanks.

~Andrew

#44 lore raw

[PATCH 14/12] CI: Refresh OpenSUSE Leap container

In reply to Andrew Cooper

See prior patch for most discussion.

Despite appearing to a fixed release (and therefore not marked as permitted failure), the dockerfile references the leap tag which is rolling in practice. Switch to 15.6 explicitly, for better test stability.

Vs tumbleweed, use zypper update rather than dist-upgrade, and retain the RomBIOS dependencies; bin86 and dev86.

In terms of size, this saves ~700M:

  registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
CC: Olaf Hering <[email protected]>

Runs with this new container:

4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493460
4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499881

---
 .../opensuse/leap-15.6-x86_64.dockerfile      | 79 +++++++++++++++++++
 .../build/suse/opensuse-leap.dockerfile       | 75 ------------------
 automation/gitlab-ci/build.yaml               | 16 ++--
 automation/scripts/containerize               |  2 +-
 4 files changed, 88 insertions(+), 84 deletions(-)
 create mode 100644 automation/build/opensuse/leap-15.6-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-leap.dockerfile
automation/build/opensuse/leap-15.6-x86_64.dockerfile new file +79 -0
new file mode 100644
index 000000000000..562006672a4e
@@ -0,0 +1,79 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/leap:15.6
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper update -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica
+        # libxl
+        libuuid-devel
+        libyajl-devel
+        # RomBIOS
+        bin86
+        dev86
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        glib2-devel
+        libpixman-1-0-devel
+        meson
+        ninja
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
automation/build/suse/opensuse-leap.dockerfile deleted +0 -75

75 lines not on this page · raw message

automation/gitlab-ci/build.yaml +8 -8
index fd6ca1270e4e..f9a9275ff293 100644
@@ -645,25 +645,25 @@ ubuntu-focal-clang-debug:
   variables:
     CONTAINER: ubuntu:focal
 
-opensuse-leap-clang:
+opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-clang-debug:
+opensuse-leap-15.6-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc:
+opensuse-leap-15.6-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc-debug:
+opensuse-leap-15.6-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
automation/scripts/containerize +1 -1
index c4f013702122..d8110b125333 100755
@@ -42,7 +42,7 @@ case "_${CONTAINER}" in
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
-    _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
+    _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
Signature · 1 line
2.39.2
#45 lore raw

[PATCH v2 14/12] CI: Refresh OpenSUSE Leap container

See prior patch for most discussion.

Despite appearing to be a fixed release (and therefore not marked as permitted failure), the dockerfile references the leap tag which is rolling in practice. Switch to 15.6 explicitly, for better test stability.

Vs tumbleweed, use zypper update rather than dist-upgrade, and retain the RomBIOS dependencies; bin86 and dev86.

In terms of size, this saves ~700M:

  registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>
CC: Olaf Hering <[email protected]>

Runs with this new container:

4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493460
4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499881

v2:
* git -> git-core
* Use 'pkgconfig()' references

---
 .../opensuse/leap-15.6-x86_64.dockerfile      | 79 +++++++++++++++++++
 .../build/suse/opensuse-leap.dockerfile       | 75 ------------------
 automation/gitlab-ci/build.yaml               | 16 ++--
 automation/scripts/containerize               |  2 +-
 4 files changed, 88 insertions(+), 84 deletions(-)
 create mode 100644 automation/build/opensuse/leap-15.6-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-leap.dockerfile
automation/build/opensuse/leap-15.6-x86_64.dockerfile new file +79 -0
new file mode 100644
index 000000000000..ea70dffe2afc
@@ -0,0 +1,79 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/leap:15.6
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper update -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # RomBIOS
+        bin86
+        dev86
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
automation/build/suse/opensuse-leap.dockerfile deleted +0 -75

75 lines not on this page · raw message

automation/gitlab-ci/build.yaml +8 -8
index fd6ca1270e4e..f9a9275ff293 100644
@@ -645,25 +645,25 @@ ubuntu-focal-clang-debug:
   variables:
     CONTAINER: ubuntu:focal
 
-opensuse-leap-clang:
+opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-clang-debug:
+opensuse-leap-15.6-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc:
+opensuse-leap-15.6-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc-debug:
+opensuse-leap-15.6-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
automation/scripts/containerize +1 -1
index c4f013702122..d8110b125333 100755
@@ -42,7 +42,7 @@ case "_${CONTAINER}" in
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
-    _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
+    _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
Signature · 1 line
2.39.2
Reviewed-by
#46 lore raw

Re: [PATCH v2 14/12] CI: Refresh OpenSUSE Leap container

Quoted 9 lines · Andrew Cooper

See prior patch for most discussion.

Despite appearing to be a fixed release (and therefore not marked as permitted failure), the dockerfile references the leap tag which is rolling in practice. Switch to 15.6 explicitly, for better test stability.

Vs tumbleweed, use zypper update rather than dist-upgrade, and retain the RomBIOS dependencies; bin86 and dev86.

Quoted 6 lines

In terms of size, this saves ~700M:

  registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#47 lore raw

[PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

In reply to Andrew Cooper

Rework the container to be non-root, use heredocs for legibility, and use use apt-get --no-install-recommends to keep the size down.

Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced by Xen 4.16. Therefore, drop the dependencies too.

This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
  registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

Runs with the new container:

4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805
4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805

---
 .../build/ubuntu/16.04-x86_64.dockerfile      | 65 +++++++++++++++++++
 automation/build/ubuntu/xenial.dockerfile     | 46 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 74 insertions(+), 55 deletions(-)
 create mode 100644 automation/build/ubuntu/16.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/xenial.dockerfile
automation/build/ubuntu/16.04-x86_64.dockerfile new file +65 -0
new file mode 100644
index 000000000000..11e878050b83
@@ -0,0 +1,65 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:16.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/ubuntu/xenial.dockerfile deleted +0 -46

46 lines not on this page · raw message

automation/gitlab-ci/build.yaml +8 -8
index 2216392a8414..a2e6815f30b3 100644
@@ -588,25 +588,25 @@ fedora-gcc-debug:
   variables:
     CONTAINER: fedora:29
 
-ubuntu-xenial-clang:
+ubuntu-16.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-clang-debug:
+ubuntu-16.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc:
+ubuntu-16.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc-debug:
+ubuntu-16.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
 ubuntu-bionic-clang:
   extends: .clang-x86-64-build
automation/scripts/containerize +1 -1
index 0ff53b6fe4e1..5fc41c36a27c 100755
@@ -52,9 +52,9 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
+    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
 
     *) guess_container ;;
 esac
Signature · 1 line
2.39.2
Reviewed-by
#48 lore raw

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

Quoted 6 lines · Andrew Cooper

Rework the container to be non-root, use heredocs for legibility, and use use apt-get --no-install-recommends to keep the size down.

Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced by Xen 4.16. Therefore, drop the dependencies too.

Quoted 6 lines

This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
  registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#49 lore raw

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

In reply to Andrew Cooper
Quoted 5 lines · Andrew Cooper
automation/scripts/containerize +0 -0
index 0ff53b6fe4e1..5fc41c36a27c 100755

automation/scripts/containerize case "_${CONTAINER}" in · line 59

Quoted 9 lines
automation/scripts/containerize +1 -1
@@ -52,9 +52,9 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
+    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
 
     *) guess_container ;;

I've been wondering why I couldn't apply the patch (and the next one), and there's these 2 lines of extra context ;-)

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#50 lore raw

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

Quoted 14 lines · Anthony PERARD
Quoted 13 lines · Andrew Cooper
automation/scripts/containerize +1 -1
index 0ff53b6fe4e1..5fc41c36a27c 100755
@@ -52,9 +52,9 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
+    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
 
Quoted 3 lines

automation/scripts/containerize case "_${CONTAINER}" in · line 59

Quoted 1 line
automation/scripts/containerize +0 -0
     *) guess_container ;;

I've been wondering why I couldn't apply the patch (and the next one), and there's these 2 lines of extra context ;-)

Yeah, sorry.  I found these when rebasing back into order.

https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=shortlog;h=refs/heads/auto-4.19
(and its simlarly-named variants) are the branches I'm doing all of this work on.

~Andrew

#51 lore raw

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

In reply to Andrew Cooper

automation/build/ubuntu/16.04-x86_64.dockerfile line 51

Quoted 3 lines · Andrew Cooper
automation/build/ubuntu/16.04-x86_64.dockerfile new file +1 -0
new file mode 100644
+        python3-setuptools

Another thing,

I've just notice that "python3-setuptools" was originally only introduced to focal container on purpose, by:
bbc72a7877d8 ("automation: Add python3's setuptools to some containers")

But now this patch series is introducing it to xenial and bionic. It might be worth mentioning, or removing the deps.

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#52 lore raw

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

Quoted 6 lines · Anthony PERARD

automation/build/ubuntu/16.04-x86_64.dockerfile line 51

Quoted 3 lines · Andrew Cooper
automation/build/ubuntu/16.04-x86_64.dockerfile new file +1 -0
new file mode 100644
+        python3-setuptools

Another thing,

Quoted 7 lines

I've just notice that "python3-setuptools" was originally only introduced to focal container on purpose, by:
bbc72a7877d8 ("automation: Add python3's setuptools to some containers")

But now this patch series is introducing it to xenial and bionic. It might be worth mentioning, or removing the deps.

Oh, good point.  I'll take this back out of Xenial/Bionic.

Stretch/Trusty have already been removed.

Thanks,

~Andrew

#53 lore raw

[PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

In reply to Andrew Cooper

As with 16.04 (Focal), except that ninja-build is available so QEMU can be built.

This halves the size of the container:

  registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
  registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

Runs with the new container:

4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548809
4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323516970

---
 .../build/ubuntu/18.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/build/ubuntu/bionic.dockerfile     | 48 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 57 deletions(-)
 create mode 100644 automation/build/ubuntu/18.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/bionic.dockerfile
automation/build/ubuntu/18.04-x86_64.dockerfile new file +72 -0
new file mode 100644
index 000000000000..24047e8f6ab2
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:18.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/ubuntu/bionic.dockerfile deleted +0 -48

48 lines not on this page · raw message

automation/gitlab-ci/build.yaml +8 -8
index a2e6815f30b3..c6f6a1ced7dc 100644
@@ -608,25 +608,25 @@ ubuntu-16.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-bionic-clang:
+ubuntu-18.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-clang-debug:
+ubuntu-18.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-gcc:
+ubuntu-18.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-gcc-debug:
+ubuntu-18.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
 ubuntu-focal-gcc:
   extends: .gcc-x86-64-build
automation/scripts/containerize +1 -1
index 5fc41c36a27c..d6132b4d22af 100755
@@ -51,10 +51,10 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
-    _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
+    _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
 
     *) guess_container ;;
 esac
Signature · 1 line
2.39.2
Reviewed-by
#54 lore raw

Re: [PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

Quoted 1 line · Andrew Cooper

As with 16.04 (Focal), except that ninja-build is available so QEMU can be

^ Xenial

These names all mixed up after a while :-)

Quoted 8 lines

built.

This halves the size of the container:

  registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
  registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#55 lore raw

Re: [PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

Quoted 5 lines · Anthony PERARD
Quoted 1 line · Andrew Cooper

As with 16.04 (Focal), except that ninja-build is available so QEMU can be

^ Xenial

These names all mixed up after a while :-)

Bah, yes.  This is one of the few cases where numbers are superior to names.  Will fix.

Quoted 10 lines
Quoted 9 lines

built.

This halves the size of the container:

  registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
  registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB

Thanks,

~Andrew

#56 lore raw

[PATCH 17/12] CI: Refresh Ubuntu Focal container as 20.04-x86_64

In reply to Andrew Cooper

Exactly as per 18.04 (Bionic). This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    20.04-x86_64           1.06GB
  registry.gitlab.com/xen-project/xen/ubuntu    focal                  1.57GB

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

Runs with the new container:

4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548811
4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323517116

---
 .../build/ubuntu/20.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/build/ubuntu/focal.dockerfile      | 49 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 58 deletions(-)
 create mode 100644 automation/build/ubuntu/20.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/focal.dockerfile
automation/build/ubuntu/20.04-x86_64.dockerfile new file +72 -0
new file mode 100644
index 000000000000..39fb297519ee
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:20.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/ubuntu/focal.dockerfile deleted +0 -49

49 lines not on this page · raw message

automation/gitlab-ci/build.yaml +8 -8
index c6f6a1ced7dc..6a2d796959a1 100644
@@ -628,25 +628,25 @@ ubuntu-18.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-focal-gcc:
+ubuntu-20.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-gcc-debug:
+ubuntu-20.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-clang:
+ubuntu-20.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-clang-debug:
+ubuntu-20.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
 opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
automation/scripts/containerize +1 -1
index d6132b4d22af..f27932dd4561 100755
@@ -40,7 +40,6 @@ case "_${CONTAINER}" in
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
-    _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
@@ -55,6 +54,7 @@ case "_${CONTAINER}" in
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
+    _focal)  CONTAINER="${BASE}/ubuntu:20.04-x86_64" ;;
 
     *) guess_container ;;
 esac
Signature · 1 line
2.39.2
Reviewed-by
#57 lore raw

Re: [PATCH 17/12] CI: Refresh Ubuntu Focal container as 20.04-x86_64

Quoted 6 lines · Andrew Cooper

Exactly as per 18.04 (Bionic). This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    20.04-x86_64           1.06GB
  registry.gitlab.com/xen-project/xen/ubuntu    focal                  1.57GB

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#58 lore raw

[PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

In reply to Andrew Cooper

The containers are exactly as per 20.04 (Focal). However, this now brings us to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.

The oldest and newest toolchains are the most likely to find problems with new code, so reduce the middle 3 releases (18/20/22) to just a single smoke test each.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>
CC: Shawn Anastasio <[email protected]>

Pipeline run showing all Ubuntu testing on staging:

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1371106799

There are 11 jobs in total.

---
 .../build/ubuntu/22.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 .../build/ubuntu/24.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/gitlab-ci/build.yaml               | 39 +++++-----
 automation/scripts/containerize               |  2 +
 4 files changed, 163 insertions(+), 22 deletions(-)
 create mode 100644 automation/build/ubuntu/22.04-x86_64.dockerfile
 create mode 100644 automation/build/ubuntu/24.04-x86_64.dockerfile
automation/build/ubuntu/22.04-x86_64.dockerfile new file +72 -0
new file mode 100644
index 000000000000..230903f624d9
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:22.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="[email protected]"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
automation/build/ubuntu/24.04-x86_64.dockerfile new file +72 -0

72 lines not on this page · raw message

automation/gitlab-ci/build.yaml +17 -22
index 6a2d796959a1..379cf17d65cc 100644
@@ -608,45 +608,40 @@ ubuntu-16.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-18.04-x86_64-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
-ubuntu-18.04-x86_64-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
 ubuntu-18.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-18.04-x86_64-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
 ubuntu-20.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-20.04-x86_64-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+ubuntu-22.04-x86_64-gcc:
+  extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:22.04-x86_64
 
-ubuntu-20.04-x86_64-clang:
+ubuntu-24.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:24.04-x86_64
 
-ubuntu-20.04-x86_64-clang-debug:
+ubuntu-24.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:24.04-x86_64
+
+ubuntu-24.04-x86_64-gcc:
+  extends: .gcc-x86-64-build
+  variables:
+    CONTAINER: ubuntu:24.04-x86_64
+
+ubuntu-24.04-x86_64-gcc-debug:
+  extends: .gcc-x86-64-build-debug
+  variables:
+    CONTAINER: ubuntu:24.04-x86_64
 
 opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
automation/scripts/containerize +2 -0
index f27932dd4561..9844133aff63 100755
@@ -55,6 +55,8 @@ case "_${CONTAINER}" in
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
     _focal)  CONTAINER="${BASE}/ubuntu:20.04-x86_64" ;;
+    _jammy)  CONTAINER="${BASE}/ubuntu:22.04-x86_64" ;;
+    _noble)  CONTAINER="${BASE}/ubuntu:24.04-x86_64" ;;
 
     *) guess_container ;;
 esac

base-commit: d46e4264c3fa68974fbb9b1bce514ea44af6d945
prerequisite-patch-id: b6875ad0bbd279067fdef31b9adf67e2a5abdf8a
prerequisite-patch-id: 5d4a01a02e7694939cdbf620a2e7509fee49dd92
prerequisite-patch-id: 65ecea93b1a320219deb8670264c3d464e00384e
prerequisite-patch-id: eb7042acce82d2bc324f8560fed3ffd57025d248
prerequisite-patch-id: 91c6663df2cfa139497e2d8b4dc17f1697f0c9f6
prerequisite-patch-id: 84c3345d580461986a3a9560735279baff21234f
prerequisite-patch-id: 5d732fc5f039fb25f7177279f63019f59acae4b1
prerequisite-patch-id: dd5006da276abc0e4da1ecb2c84d699c5bd98124
prerequisite-patch-id: 7e816655cd1066b42be5fa57a832c777c9867fc9
prerequisite-patch-id: fc25cefbee4cb69703df5c1f5b3eb317f03952af
prerequisite-patch-id: 531b3db898f54c05bff5fbcb69e0f7f62811c744
prerequisite-patch-id: ddb51b78aeaa2a6efe8035c39a128810c01aa822
prerequisite-patch-id: b4894b8120682429e1a74369c6b36bbbdc09d749
prerequisite-patch-id: c002530cb86a08b38980779a3d06cf491fdb9fae

Signature · 1 line
2.39.2
Reviewed-by
#59 lore raw

Re: [PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

Quoted 6 lines · Andrew Cooper

The containers are exactly as per 20.04 (Focal). However, this now brings us to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.

The oldest and newest toolchains are the most likely to find problems with new code, so reduce the middle 3 releases (18/20/22) to just a single smoke test each.

That would mean a bit less clang build-test, but I guess that would be mostly covered by FreeBSD testing on GitHub.

I tried to find out which version of clang were been used with this patch applied:

    Alpine clang version 16.0.6
    Debian clang version 14.0.6
    clang version 17.0.6
    clang version 18.1.8
    clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
    Ubuntu clang version 18.1.3 (1ubuntu1)
and the versions that won't be tested anymore:
    clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
    clang version 10.0.0-4ubuntu1
    Ubuntu clang version 14.0.0-1ubuntu1.1
Quoted 1 line

Anyway:

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#60 lore raw

Re: [PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

Quoted 23 lines · Anthony PERARD
Quoted 6 lines · Andrew Cooper

The containers are exactly as per 20.04 (Focal). However, this now brings us to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.

The oldest and newest toolchains are the most likely to find problems with new code, so reduce the middle 3 releases (18/20/22) to just a single smoke test each.

That would mean a bit less clang build-test, but I guess that would be mostly covered by FreeBSD testing on GitHub.

I tried to find out which version of clang were been used with this patch applied:

    Alpine clang version 16.0.6
    Debian clang version 14.0.6
    clang version 17.0.6
    clang version 18.1.8
    clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
    Ubuntu clang version 18.1.3 (1ubuntu1)
and the versions that won't be tested anymore:
    clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
    clang version 10.0.0-4ubuntu1
    Ubuntu clang version 14.0.0-1ubuntu1.1
Quoted 3 lines
Quoted 1 line

Anyway:

Thanks.

We could intermix Clang/GCC if we really want, but tbh (still!) the only thing you want to use Clang for is stricter/better diagnostics.  GCC is still a better optimising compiler for C, particularly for kernel activities.

Also, those clang versions are entirely obsolete.  If we wanted to do sensible testing, we'd need to be using the nightly builds and keeping within ~18mo of the release which I think is their support window.

Also also, this testing is CC=Clang, LD=ld, which is subtly different to the FreeBSD testing which is full-stack LLVM (so LD=lld).

There is definitely room for improvement in our Clang/LLVM testing.

~Andrew

#61 lore raw

[PATCH 20/12] CI: Swap from perl to perl-base in build containers

In reply to Andrew Cooper

We only need a basic perl environment, not the things that a primarily-perl project would want.

Discovered during the Fedora refresh where the difference is ~40M, but it's more modest for OpenSUSE and Ubuntu.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1372507213

---
 automation/build/opensuse/leap-15.6-x86_64.dockerfile  | 2 +-
 automation/build/opensuse/tumbleweed-x86_64.dockerfile | 2 +-
 automation/build/ubuntu/16.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/18.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/20.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/22.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/24.04-x86_64.dockerfile        | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)
automation/build/opensuse/leap-15.6-x86_64.dockerfile +1 -1
index ea70dffe2afc..d9ee34ff240f 100644
@@ -31,7 +31,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/opensuse/tumbleweed-x86_64.dockerfile +1 -1
index 61e840fc67da..cd90f7357a50 100644
@@ -31,7 +31,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/ubuntu/16.04-x86_64.dockerfile +1 -1
index d8617bfade5e..7dc2ad8c3ef5 100644
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/ubuntu/18.04-x86_64.dockerfile +1 -1
index 336697ad954b..12522cf48045 100644
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/ubuntu/20.04-x86_64.dockerfile +1 -1
index 39fb297519ee..d071949bfe5c 100644
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/ubuntu/22.04-x86_64.dockerfile +1 -1
index 230903f624d9..4f52b9d475ad 100644
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
automation/build/ubuntu/24.04-x86_64.dockerfile +1 -1
index 277f92facfd9..d125b9f739fa 100644
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
Signature · 1 line
2.39.2
#62 lore raw

Re: [PATCH 20/12] CI: Swap from perl to perl-base in build containers

Quoted 5 lines · Andrew Cooper

We only need a basic perl environment, not the things that a primarily-perl project would want.

Discovered during the Fedora refresh where the difference is ~40M, but it's more modest for OpenSUSE and Ubuntu.

"perl-base" on Ubuntu is quite different from "perl-base" on OpenSUSE Leap. I haven't check Tumbleweed but is probably the same as Leap. So, I think these would deserve to be in separated patches.

On Ubuntu, installing "perl" or "perl-base" or none of them makes no difference. We install "build-essentials" which pulls "dpkg-dev" which pulls "perl", so we don't test properly if "perl-base" is enough or if we need more.

I tried to find out which Perl module we would need, and I've got this list:

- get_maintainer

    Getopt::Long;
- add_maintainer
    Getopt::Long;
    File::Basename;
    List::MoreUtils;
    IO::Handle;
- stubdom/vtpmmgr:
    Digest::SHA qw(sha1);
    Math::BigInt only => 'GMP';
- kconfig
    Getopt::Long;
- ocaml/lib/xc/abi-check
    Data::Dumper;
- docs/gen-html-index
    Getopt::Long;
    IO::File;
    File::Basename;
- docs/xen-headers
    Getopt::Long;
    File::Find;
    IO::File;

(there's also tools/examples/xeninfo.pl, I dont't if that can still work or if we need to purge it)

Then I've got whatever module is in "perl-base" pkg.

For Leap's perl-base:
getopt:long
file:basename
io:handle
io:file
file:find
data:dumper
digest:sha
So, add_maintainer.pl and the script in stubdom/vtpmmgr won't work. So that's probably fine for Leap.

For Ubuntu 22.04 (not check others, but likely about the same) there's more packages:

    perl-base
        getopt::long
        file:basename
        io:handle
        io:file
    perl-modules-5.34
        file:find
        math:bigint
    libperl5.34
        data:dumper
        digest:sha
    liblist-moreutils-perl
        list:MoreUtils
    librpc-xml-perl
        rpc:xml

("perl" pkg pulls "perl-modules-5.34" and "libperl5.34")

So with just "perl-base", the ocaml's abi-check wouldn't work, as well as the docs's script (probably ok for "docs" if we don't use containers to build them). Also add_maintainer and stubdom/vtpmmgr like for Leap.

So I would suggest to not touch the Ubuntu containers, make the change to the OpenSUSE one and maybe add the limitation to the commit message.

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#63 lore raw

Re: [PATCH 20/12] CI: Swap from perl to perl-base in build containers

Quoted 8 lines · Anthony PERARD
Quoted 5 lines · Andrew Cooper

We only need a basic perl environment, not the things that a primarily-perl project would want.

Discovered during the Fedora refresh where the difference is ~40M, but it's more modest for OpenSUSE and Ubuntu.

"perl-base" on Ubuntu is quite different from "perl-base" on OpenSUSE Leap. I haven't check Tumbleweed but is probably the same as Leap.

/sigh - because that would have been too easy.

Quoted 6 lines

So, I think these would deserve to be in separated patches.

On Ubuntu, installing "perl" or "perl-base" or none of them makes no difference. We install "build-essentials" which pulls "dpkg-dev" which pulls "perl", so we don't test properly if "perl-base" is enough or if we need more.

Fine - lets leave the Ubuntu's alone.

Quoted 26 lines

I tried to find out which Perl module we would need, and I've got this list:

- get_maintainer

    Getopt::Long;
- add_maintainer
    Getopt::Long;
    File::Basename;
    List::MoreUtils;
    IO::Handle;
- stubdom/vtpmmgr:
    Digest::SHA qw(sha1);
    Math::BigInt only => 'GMP';
- kconfig
    Getopt::Long;
- ocaml/lib/xc/abi-check
    Data::Dumper;
- docs/gen-html-index
    Getopt::Long;
    IO::File;
    File::Basename;
- docs/xen-headers
    Getopt::Long;
    File::Find;
    IO::File;
Quoted 2 lines

(there's also tools/examples/xeninfo.pl, I dont't if that can still work or if we need to purge it)

Ewww.

my %xenhosts = ("192.0.2.10" => {"port" => "9363"}, ...

is enough to demonstrate it's a special purpose script.

That file isn't referenced by the Makefile, nor is anything in the vnc/ dir, so we should purge the lot.

Quoted 13 lines

Then I've got whatever module is in "perl-base" pkg.

For Leap's perl-base:
getopt:long
file:basename
io:handle
io:file
file:find
data:dumper
digest:sha
So, add_maintainer.pl and the script in stubdom/vtpmmgr won't work. So that's probably fine for Leap.

vtpmmgr isn't built by default, so gets 0 testing in CI.  I don't expect people to be using add_maintainer inside the containers either.

Quoted 18 lines

For Ubuntu 22.04 (not check others, but likely about the same) there's more packages:

    perl-base
        getopt::long
        file:basename
        io:handle
        io:file
    perl-modules-5.34
        file:find
        math:bigint
    libperl5.34
        data:dumper
        digest:sha
    liblist-moreutils-perl
        list:MoreUtils
    librpc-xml-perl
        rpc:xml
Quoted 5 lines

("perl" pkg pulls "perl-modules-5.34" and "libperl5.34")

So with just "perl-base", the ocaml's abi-check wouldn't work, as well as the docs's script (probably ok for "docs" if we don't use containers to build them).

Hmm yeah - we wouldn't want to break that.

Quoted 5 lines

Also add_maintainer and stubdom/vtpmmgr like for Leap.

So I would suggest to not touch the Ubuntu containers, make the change to the OpenSUSE one and maybe add the limitation to the commit message.

Ok - I'll do this.

~Andrew

#64 lore raw

[PATCH 19/12] docs: Fix install-man$(1)-pages if no manpages are generated

In reply to Andrew Cooper

All tools to build manpages are optional, and if none of them happen to be present, the intermediate working directory may not even be created.

Treat this as non-fatal, bringing the behaviour in line with install-html. Like the html side, it needs to be not-or to avoid Make thinking the rule has failed.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 docs/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docs/Makefile +1 -1
index 966a104490ac..b30cc619f8dd 100644
@@ -151,7 +151,7 @@ man$(1)-pages: $$(DOC_MAN$(1))
 .PHONY: install-man$(1)-pages
 install-man$(1)-pages: man$(1)-pages
 	$(INSTALL_DIR) $(DESTDIR)$(mandir)
-	cp -r man$(1) $(DESTDIR)$(mandir)
+	[ ! -d man$(1) ] || cp -r man$(1) $(DESTDIR)$(mandir)
 
 # Clean
 .PHONY: clean-man$(1)-pages
Signature · 1 line
2.39.2
Reviewed-by
#65 lore raw

Re: [PATCH 19/12] docs: Fix install-man$(1)-pages if no manpages are generated

Quoted 8 lines · Andrew Cooper

All tools to build manpages are optional, and if none of them happen to be present, the intermediate working directory may not even be created.

Treat this as non-fatal, bringing the behaviour in line with install-html. Like the html side, it needs to be not-or to avoid Make thinking the rule has failed.

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#66 lore raw

[PATCH 21/12] CI: Refresh and upgrade the Fedora container

In reply to Andrew Cooper

Fedora 29 is long out of date. Move forward 5 years to Fedora 39.

Inlcude all the usual improvements. Rework the container to be non-root, use heredocs for legibility, and switch to the new naming scheme.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>

I know Fedora 40 the latest release, but it suffers from the same stubdom failure as Archlinux currently does. Fedora 39 builds fine.

Fedora is (correctly) blocking, so adding 40 would be bad move at this point in the 4.19 release.

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1372042761 is a run with both 39 and 40, showing the failure.

---
 automation/build/fedora/29.dockerfile        | 46 ------------
 automation/build/fedora/39-x86_64.dockerfile | 74 ++++++++++++++++++++
 automation/gitlab-ci/build.yaml              |  8 +--
 3 files changed, 78 insertions(+), 50 deletions(-)
 delete mode 100644 automation/build/fedora/29.dockerfile
 create mode 100644 automation/build/fedora/39-x86_64.dockerfile
automation/build/fedora/29.dockerfile deleted +0 -46
deleted file mode 100644
index d5f017ac723d..000000000000
@@ -1,46 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 fedora:29
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="[email protected]"
-
-# install Xen depends
-RUN dnf -y install \
-        clang \
-        gcc \
-        gcc-c++ \
-        ncurses-devel \
-        zlib-devel \
-        python-devel \
-        python3-devel \
-        libuuid-devel \
-        pkgconfig \
-        flex \
-        bison \
-        libaio-devel \
-        glib2-devel \
-        yajl-devel \
-        pixman-devel \
-        glibc-devel \
-        make \
-        binutils \
-        git \
-        wget \
-        acpica-tools \
-        python-markdown \
-        patch \
-        checkpolicy \
-        dev86 \
-        xz-devel \
-        bzip2 \
-        nasm \
-        ocaml \
-        ocaml-findlib \
-        golang \
-        # QEMU
-        ninja-build \
-    && dnf clean all && \
-    rm -rf /var/cache/dnf
-
-RUN useradd --create-home user
-USER user
-WORKDIR /build
automation/build/fedora/39-x86_64.dockerfile new file +74 -0

74 lines not on this page · raw message

automation/gitlab-ci/build.yaml +4 -4
index 7ce88d38e7c7..3d35871d64f5 100644
@@ -575,15 +575,15 @@ debian-bookworm-32-gcc-debug:
   variables:
     CONTAINER: debian:bookworm-i386
 
-fedora-gcc:
+fedora-39-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: fedora:29
+    CONTAINER: fedora:39-x86_64
 
-fedora-gcc-debug:
+fedora-39-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: fedora:29
+    CONTAINER: fedora:39-x86_64
 
 ubuntu-16.04-x86_64-clang:
   extends: .clang-x86-64-build
Signature · 1 line
2.39.2
#67 lore raw

Re: [PATCH 21/12] CI: Refresh and upgrade the Fedora container

Quoted 4 lines · Andrew Cooper

Fedora 29 is long out of date. Move forward 5 years to Fedora 39.

Inlcude all the usual improvements. Rework the container to be non-root, use heredocs for legibility, and switch to the new naming scheme.

"clang" as been removed, but it's not mention in the commit message. The other changes are kind of "the usual improvement" yes, as been done to other containers (libaio, nasm, ...).

The patch is missing an update of the "containerize" script.

Quoted 5 lines
automation/build/fedora/39-x86_64.dockerfile new file +0 -0
new file mode 100644
index 000000000000..e2048a437581

...

automation/build/fedora/39-x86_64.dockerfile

Quoted 2 lines
automation/build/fedora/39-x86_64.dockerfile new file +2 -0
+
+    dnf -y install "${DEPS[@]}"

You might want to add --setopt=install_weak_deps=False to avoid installing "git" for example.
When running the original command by hand, dnf want to install those weak deps:
Installing weak dependencies:
apr-util-bdb
apr-util-openssl
cryptsetup-libs
diffutils
git
libbpf
libxkbcommon
mercurial
perl-NDBM_File
python3-fb-re2
python3-pip
qrencode-libs
subversion
systemd-networkd
systemd-resolved
And comparring the list of deps, these extra pkgs are installed because of weak deps:

    apr
    apr-util
    dbus
    dbus-broker
    dbus-common
    device-mapper
    device-mapper-libs
    git-core-doc
    kmod-libs
    libargon2
    libseccomp
    libserf
    perl-Error
    perl-File-Find
    perl-Git
    perl-TermReadKey
    perl-lib
    python3-zombie-imp
    re2
    subversion-libs
    systemd
    systemd-pam
    utf8proc
    xkeyboard-config

So, probably only "perl-File-Find" (can be written "perl(File::Find)" I think as well) might be needed, but only for docs/ I think, from the previous email.

It seems I go from 1.26GB to 1.18GB without those weak deps.

Cheers,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#68 lore raw

Re: [PATCH 21/12] CI: Refresh and upgrade the Fedora container

Quoted 6 lines · Anthony PERARD
Quoted 4 lines · Andrew Cooper

Fedora 29 is long out of date. Move forward 5 years to Fedora 39.

Inlcude all the usual improvements. Rework the container to be non-root, use heredocs for legibility, and switch to the new naming scheme.

"clang" as been removed, but it's not mention in the commit message.

Oops - I meant to mention that.  We don't do any clang builds with this container, so I though I'd keep it slimer until we change that.

Quoted 5 lines

The
other changes are kind of "the usual improvement" yes, as been done to other containers (libaio, nasm, ...).

The patch is missing an update of the "containerize" script.

Ah - I found the hunk in a different patch.  I'll fix.

Quoted 11 lines
Quoted 6 lines
automation/build/fedora/39-x86_64.dockerfile new file +0 -0
new file mode 100644
index 000000000000..e2048a437581

...

automation/build/fedora/39-x86_64.dockerfile

Quoted 2 lines
automation/build/fedora/39-x86_64.dockerfile new file +2 -0
+
+    dnf -y install "${DEPS[@]}"

You might want to add --setopt=install_weak_deps=False to avoid installing "git" for example.

Well, git-core is installed explicitly, but point taken.

Quoted 44 lines

When running the original command by hand, dnf want to install those weak deps:
Installing weak dependencies:
apr-util-bdb
apr-util-openssl
cryptsetup-libs
diffutils
git
libbpf
libxkbcommon
mercurial
perl-NDBM_File
python3-fb-re2
python3-pip
qrencode-libs
subversion
systemd-networkd
systemd-resolved
And comparring the list of deps, these extra pkgs are installed because of weak deps:

    apr
    apr-util
    dbus
    dbus-broker
    dbus-common
    device-mapper
    device-mapper-libs
    git-core-doc
    kmod-libs
    libargon2
    libseccomp
    libserf
    perl-Error
    perl-File-Find
    perl-Git
    perl-TermReadKey
    perl-lib
    python3-zombie-imp
    re2
    subversion-libs
    systemd
    systemd-pam
    utf8proc
    xkeyboard-config
Quoted 5 lines

So, probably only "perl-File-Find" (can be written "perl(File::Find)" I think as well) might be needed, but only for docs/ I think, from the previous email.

It seems I go from 1.26GB to 1.18GB without those weak deps.

Thanks.  I'll rework.

I have a specific plan for testing docs and it's going to be a dedicated container and job, as well as things being a bit more -Werror-like.

~Andrew

#69 lore raw

[PATCH 22/12] tools/examples: Remove more obsolete content

In reply to Andrew Cooper

xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script for collecting statistics from Xen hosts using the Xen-API") and has been touched exactly twice since to remove hardcoded IP addresses and paths.

The configuration files in vnc/* date from when we had a vendered version of Qemu living in the tree.

These have never (AFAICT) been wired into the make install rule.

---
CC: Anthony PERARD <[email protected]>
CC: Juergen Gross <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 tools/examples/vnc/Xservers |   5 -
 tools/examples/vnc/Xvnc-xen |  53 -------
 tools/examples/xeninfo.pl   | 284 ------------------------------------
 3 files changed, 342 deletions(-)
 delete mode 100644 tools/examples/vnc/Xservers
 delete mode 100755 tools/examples/vnc/Xvnc-xen
 delete mode 100644 tools/examples/xeninfo.pl
tools/examples/vnc/Xservers deleted +0 -5
deleted file mode 100644
index adc5748be6ba..000000000000
@@ -1,5 +0,0 @@
-# Configuration lines to go in /etc/X11/xdm/Xservers to
-# start Xvnc and connect back to a vncviewer in domain-0.
-# See 'man xdm' under 'LOCAL SERVER SPECIFICATION' for format details.
-
-:1 Xvnc local /usr/X11R6/bin/Xvnc-xen :1
\ No newline at end of file
tools/examples/vnc/Xvnc-xen deleted +0 -53

53 lines not on this page · raw message

tools/examples/xeninfo.pl deleted +0 -284

284 lines not on this page · raw message

Signature · 1 line
2.39.2
Reviewed-by
#70 lore raw

Re: [PATCH 22/12] tools/examples: Remove more obsolete content

Quoted 3 lines · Andrew Cooper

xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script for collecting statistics from Xen hosts using the Xen-API") and has been touched exactly twice since to remove hardcoded IP addresses and paths.

Xen-API, is that xapi?

Quoted 2 lines

The configuration files in vnc/* date from when we had a vendered version of Qemu living in the tree.

I guess QEMU at that time didn't have VNC support? Because looks like the vnc config was supposed to be put in a guest, and xm had support for it.

Quoted 3 lines

These have never (AFAICT) been wired into the make install rule.

Anyway, patch looks good:

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#71 lore raw

Re: [PATCH 22/12] tools/examples: Remove more obsolete content

Quoted 5 lines · Anthony PERARD
Quoted 3 lines · Andrew Cooper

xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script for collecting statistics from Xen hosts using the Xen-API") and has been touched exactly twice since to remove hardcoded IP addresses and paths.

Xen-API, is that xapi?

Yes.  xapi is a contraction of Xen-API, but even the docs intermix the two names.  https://xapi-project.github.io/xen-api/

Quoted 6 lines
Quoted 3 lines

The configuration files in vnc/* date from when we had a vendered version of Qemu living in the tree.

I guess QEMU at that time didn't have VNC support? Because looks like the vnc config was supposed to be put in a guest, and xm had support for it.

Thinking about it, this may have been for PV guests instead.  We used to have a mode in XenServer to inject vnc configuration into a PV guest in order to get a GUI installer rather than the TUI one.

Either way, xl doesn't have --vnc, although it does have -V which seems to have subsumed this functionality.

Quoted 6 lines
Quoted 4 lines

These have never (AFAICT) been wired into the make install rule.

Anyway, patch looks good:

Thanks,

~Andrew