Skip to contentSkip to conversation

automation: Fix build with new qemu-xen version

PATCH In review Anthony PERARD 3 participants 17 messages Jul 7, 2021 – Jul 8, 2021

lore
Share & tools
lore
#0 lore raw

cover automation: Fix build with new qemu-xen version

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.automation-add-ninja-v1

Adding ninja-build pkg when possible.

I'll push new containers soon.

Anthony PERARD (2):
automation: Adding ninja-build to some docker images
automation: Check if ninja is available before building QEMU

 automation/build/alpine/3.12.dockerfile        | 1 +
 automation/build/fedora/29.dockerfile          | 9 ++++++---
 automation/build/suse/opensuse-leap.dockerfile | 2 ++
 automation/build/ubuntu/bionic.dockerfile      | 2 ++
 automation/build/ubuntu/focal.dockerfile       | 2 ++
 automation/scripts/build                       | 4 +++-
 6 files changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
Anthony PERARD
#1 lore raw

PATCH 1/2 automation: Adding ninja-build to some docker images

This is to allow building the latest version of QEMU.

fedora/29:
In addition to adding "ninja", I've add to make some other
changes: some go build failed with mkdir /.cache no permission, so I've created a user.
(this was discovered while testing the new container with the script containerize.)

---
 automation/build/alpine/3.12.dockerfile        | 1 +
 automation/build/fedora/29.dockerfile          | 9 ++++++---
 automation/build/suse/opensuse-leap.dockerfile | 2 ++
 automation/build/ubuntu/bionic.dockerfile      | 2 ++
 automation/build/ubuntu/focal.dockerfile       | 2 ++
 5 files changed, 13 insertions(+), 3 deletions(-)
automation/build/alpine/3.12.dockerfile +1 -0
index bb8e5ece7414..4ee3ddc12e52 100644
@@ -47,6 +47,7 @@ RUN \
   apk add glib-dev && \
   apk add libattr && \
   apk add libcap-ng-dev && \
+  apk add ninja && \
   apk add pixman-dev && \
   \
   # cleanup
automation/build/fedora/29.dockerfile +6 -3
index 027b93ceaf4e..60a5d722668f 100644
@@ -2,9 +2,6 @@ FROM fedora:29
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="[email protected]"
 
-RUN mkdir /build
-WORKDIR /build
-
 # install Xen depends
 RUN dnf -y install \
         clang \
@@ -43,5 +40,11 @@ RUN dnf -y install \
         ocaml \
         ocaml-findlib \
         golang \
+        # QEMU
+        ninja-build \
     && dnf clean all && \
     rm -rf /var/cache/dnf
+
+RUN useradd --create-home user
+USER user
+WORKDIR /build
automation/build/suse/opensuse-leap.dockerfile +2 -0
index 573fbd8ae47e..03f8a7acad4f 100644
@@ -67,5 +67,7 @@ RUN zypper install -y --no-recommends \
         which \
         xz-devel \
         zlib-devel \
+        # QEMU
+        ninja \
         && \
         zypper clean -a
automation/build/ubuntu/bionic.dockerfile +2 -0
index 406a97494caf..1e5a27c70f15 100644
@@ -45,6 +45,8 @@ RUN apt-get update && \
         wget \
         git \
         nasm \
+        # QEMU
+        ninja-build \
         && \
         apt-get autoremove -y && \
         apt-get clean && \
automation/build/ubuntu/focal.dockerfile +2 -0
index 952a3172aa64..74655b9876d2 100644
@@ -44,6 +44,8 @@ RUN apt-get update && \
         wget \
         git \
         nasm \
+        # QEMU
+        ninja-build \
         && \
         apt-get autoremove -y && \
         apt-get clean && \
Signature · 1 line
Anthony PERARD
#2 lore raw

PATCH 2/2 automation: Check if ninja is available before building QEMU

In reply to Anthony PERARD

ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.

Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed

---
 automation/scripts/build | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
automation/scripts/build +3 -1
index 46b6903d2922..d8c77e244cda 100755
@@ -46,7 +46,9 @@ if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
 fi
 
 # Qemu requires Python 3.5 or later
-if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))"; then
+# QEMU requires ninja
+if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
+        || ! type ninja; then
     cfgargs+=("--with-system-qemu=/bin/false")
 fi
 
Signature · 1 line
Anthony PERARD
#3 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 4 lines · Anthony PERARD

ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.

Quoted 15 lines

Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed

Only the first failure seems to be related to ninja.

The second is a known bug which still needs addressing.  Our build system's logic to cope with checkpolicy being absent doesn't work.

Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.  It is quite possible that the iPXE failure is related.

The final one is probably because unstable has dropped clang-8.  The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.

~Andrew

#4 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 1 line

iPXE failure

it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Olaf

#5 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 5 lines · Olaf Hering

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 1 line

iPXE failure

it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for fixing build with GCC11" patch which has been sitting neglected on the list?

~Andrew

#6 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Am Wed, 7 Jul 2021 23:44:33 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 7 lines · Olaf Hering
Quoted 4 lines

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 1 line

iPXE failure

it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for fixing build with GCC11" patch which has been sitting neglected on the list?

Yes, but the build logs indicate gcc10 is used in this container. So it might be something else.

Olaf

#7 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

In reply to Andrew Cooper

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 2 lines

Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.

Something requests zlib-devel to be installed.

I suggest to provide all config.logs, not just the one from the top directory.

Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

Olaf

#8 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 6 lines · Olaf Hering

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 2 lines

Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.

Something requests zlib-devel to be installed.

Yeah - it shouldn't be hard to fix.

Quoted 3 lines

I suggest to provide all config.logs, not just the one from the top directory.

Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

We ought to yes, although I think at the moment we probably don't want to realise how stale all the containers are.

~Andrew

#9 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Am Wed, 7 Jul 2021 23:47:02 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 2 lines

We ought to yes, although I think at the moment we probably don't want to realise how stale all the containers are.

At least for Tumbleweed it is useful information.

opensuse-tumbleweed.dockerfile requests zlib-devel already.

tools/config.log is required to see what the failure actually is. The toplevel config.log is not helpful in this particular case.

Olaf

#10 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <[email protected]>:

Quoted 1 line

At least for Tumbleweed it is useful information.

CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"

Two years old for a rolling release.

Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Olaf

#11 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 9 lines · Olaf Hering

Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <[email protected]>:

Quoted 1 line

At least for Tumbleweed it is useful information.

CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"

Two years old for a rolling release.

Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Automatically rebuilding the containers is still a task on the CI working group's todo list.

That said, the tumbleweed containers were rebuilt far more recently than that, so something is wonky...

In a xen tree, see automation/build/Makefile for details, but tl;dr to rebuild:

$ make -C automation/build/ tumbleweed

and to push:

$ docker login registry.gitlab.com/xen-project/xen
$ make -C automation/build/ tumbleweed PUSH=1

Bear in mind that the final line will update the live CI system.  What I normally do to test new containers is to give it a test suffix, push to the live system, and push a branch to xen.git with the container names edited in automation/gitlab-ci/*.yaml.

~Andrew

#12 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Am Thu, 8 Jul 2021 13:51:49 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 2 lines

That said, the tumbleweed containers were rebuilt far more recently than that, so something is wonky...

After playing with 'automation/scripts/containerize', docker image ls shows:
registry.gitlab.com/xen-project/xen/suse opensuse-tumbleweed f9b84e2346f0 7 weeks ago 2.27GB

No idea how that created a container with broken content. zlib-devel.rpm exists, but libz.so is a broken symlink.

I will update the README to list the required steps for updating.

Olaf

#13 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

In reply to Andrew Cooper
Quoted 26 lines · Andrew Cooper
Quoted 4 lines · Anthony PERARD

ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.

Quoted 15 lines

Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed

Only the first failure seems to be related to ninja.

The second is a known bug which still needs addressing.  Our build system's logic to cope with checkpolicy being absent doesn't work.

Quoted 7 lines

Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.  It is quite possible that the iPXE failure is related.

The final one is probably because unstable has dropped clang-8.  The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.

So those containers are ones that I've noticed needed ninja (as found out by looking at a single pipeline), but one which trying to install ninja and thus rebuilding the container mean that the build was broken. I just didn't want to spend time trying to fix issues that should have been found out by a regular rebuild of the containers. So I've just listed the reason there and moved on.

Signature · 1 line
Anthony PERARD
#14 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 34 lines · Anthony PERARD
Quoted 25 lines · Andrew Cooper
Quoted 4 lines · Anthony PERARD

ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.

Quoted 15 lines

Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed

Only the first failure seems to be related to ninja.

The second is a known bug which still needs addressing.  Our build system's logic to cope with checkpolicy being absent doesn't work.

Quoted 7 lines

Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.  It is quite possible that the iPXE failure is related.

The final one is probably because unstable has dropped clang-8.  The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.

Quoted 5 lines

So those containers are ones that I've noticed needed ninja (as found out by looking at a single pipeline), but one which trying to install ninja and thus rebuilding the container mean that the build was broken. I just didn't want to spend time trying to fix issues that should have been found out by a regular rebuild of the containers.

That's fine, but as these containers weren't rebuilt, the details of any knock-on error aren't relevant here.

I'd go with simply:

"ninja is now required to build the latest version of QEMU, and not all distros have a suitable version.  Skip the QEMU build when ninja is not available".

Can fix on commit if you're happy with this.

~Andrew

#15 lore raw

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

Quoted 7 lines · Andrew Cooper

I'd go with simply:

"ninja is now required to build the latest version of QEMU, and not all distros have a suitable version.  Skip the QEMU build when ninja is not available".

Can fix on commit if you're happy with this.

Sounds good, thanks.

Signature · 1 line
Anthony PERARD
#16 lore raw
In reply to Anthony PERARD
Quoted 3 lines · Anthony PERARD

Adding ninja-build pkg when possible.

I'll push new containers soon.

I've pushed:
registry.gitlab.com/xen-project/xen/fedora:29 registry.gitlab.com/xen-project/xen/ubuntu:bionic registry.gitlab.com/xen-project/xen/ubuntu:focal registry.gitlab.com/xen-project/xen/alpine:3.12 registry.gitlab.com/xen-project/xen/suse:opensuse-leap

Quoted 12 lines

Anthony PERARD (2):
automation: Adding ninja-build to some docker images
automation: Check if ninja is available before building QEMU

 automation/build/alpine/3.12.dockerfile        | 1 +
 automation/build/fedora/29.dockerfile          | 9 ++++++---
 automation/build/suse/opensuse-leap.dockerfile | 2 ++
 automation/build/ubuntu/bionic.dockerfile      | 2 ++
 automation/build/ubuntu/focal.dockerfile       | 2 ++
 automation/scripts/build                       | 4 +++-
 6 files changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
Anthony PERARD