automation: Fix build with new qemu-xen version
cover automation: Fix build with new qemu-xen version
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.automation-add-ninja-v1
Adding ninja-build pkg when possible.
I'll push new containers soon.
Anthony PERARD (2):
automation: Adding ninja-build to some docker images
automation: Check if ninja is available before building QEMU
automation/build/alpine/3.12.dockerfile | 1 + automation/build/fedora/29.dockerfile | 9 ++++++--- automation/build/suse/opensuse-leap.dockerfile | 2 ++ automation/build/ubuntu/bionic.dockerfile | 2 ++ automation/build/ubuntu/focal.dockerfile | 2 ++ automation/scripts/build | 4 +++- 6 files changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
Anthony PERARD
PATCH 1/2 automation: Adding ninja-build to some docker images
This is to allow building the latest version of QEMU.
fedora/29:
In addition to adding "ninja", I've add to make some other
changes: some go build failed with mkdir /.cache no permission, so I've created a user.
(this was discovered while testing the new container with the script containerize.)
--- automation/build/alpine/3.12.dockerfile | 1 + automation/build/fedora/29.dockerfile | 9 ++++++--- automation/build/suse/opensuse-leap.dockerfile | 2 ++ automation/build/ubuntu/bionic.dockerfile | 2 ++ automation/build/ubuntu/focal.dockerfile | 2 ++ 5 files changed, 13 insertions(+), 3 deletions(-)
automation/build/alpine/3.12.dockerfile +1 -0
@@ -47,6 +47,7 @@ RUN \ apk add glib-dev && \ apk add libattr && \ apk add libcap-ng-dev && \ + apk add ninja && \ apk add pixman-dev && \ \ # cleanup
automation/build/fedora/29.dockerfile +6 -3
@@ -2,9 +2,6 @@ FROM fedora:29 LABEL maintainer.name="The Xen Project" \ maintainer.email="[email protected]" -RUN mkdir /build -WORKDIR /build - # install Xen depends RUN dnf -y install \ clang \
@@ -43,5 +40,11 @@ RUN dnf -y install \ ocaml \ ocaml-findlib \ golang \ + # QEMU + ninja-build \ && dnf clean all && \ rm -rf /var/cache/dnf + +RUN useradd --create-home user +USER user +WORKDIR /build
automation/build/suse/opensuse-leap.dockerfile +2 -0
@@ -67,5 +67,7 @@ RUN zypper install -y --no-recommends \ which \ xz-devel \ zlib-devel \ + # QEMU + ninja \ && \ zypper clean -a
automation/build/ubuntu/bionic.dockerfile +2 -0
@@ -45,6 +45,8 @@ RUN apt-get update && \ wget \ git \ nasm \ + # QEMU + ninja-build \ && \ apt-get autoremove -y && \ apt-get clean && \
automation/build/ubuntu/focal.dockerfile +2 -0
@@ -44,6 +44,8 @@ RUN apt-get update && \ wget \ git \ nasm \ + # QEMU + ninja-build \ && \ apt-get autoremove -y && \ apt-get clean && \
Signature · 1 line
Anthony PERARD
PATCH 2/2 automation: Check if ninja is available before building QEMU
In reply to Anthony PERARDninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.
Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed
--- automation/scripts/build | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
automation/scripts/build +3 -1
@@ -46,7 +46,9 @@ if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then fi # Qemu requires Python 3.5 or later -if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))"; then +# QEMU requires ninja +if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \ + || ! type ninja; then cfgargs+=("--with-system-qemu=/bin/false") fi
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 4 lines · Anthony PERARD
ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.
Quoted 15 lines
Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build fileninja_required_versionversion (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installed
Only the first failure seems to be related to ninja.
The second is a known bug which still needs addressing. Our build system's logic to cope with checkpolicy being absent doesn't work.
Tumbleweed is generally broken and fails at ./configure due to missing compression libraries. It is quite possible that the iPXE failure is related.
The final one is probably because unstable has dropped clang-8. The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.
~Andrew
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:
Quoted 1 line
iPXE failure
it just needs to be updated to ipxe.git#master to make it compatible with gcc11.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 5 lines · Olaf Hering
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:Quoted 1 line
iPXE failure
it just needs to be updated to ipxe.git#master to make it compatible with gcc11.
Ah great. I presume that is your "[PATCH v2] tools: ipxe: update for fixing build with GCC11" patch which has been sitting neglected on the list?
~Andrew
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Am Wed, 7 Jul 2021 23:44:33 +0100
schrieb Andrew Cooper <[email protected]>:
Quoted 7 lines · Olaf Hering
Quoted 4 lines
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:Quoted 1 line
iPXE failure
it just needs to be updated to ipxe.git#master to make it compatible with gcc11.
Ah great. I presume that is your "[PATCH v2] tools: ipxe: update for fixing build with GCC11" patch which has been sitting neglected on the list?
Yes, but the build logs indicate gcc10 is used in this container. So it might be something else.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
In reply to Andrew CooperAm Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:
Quoted 2 lines
Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.
Something requests zlib-devel to be installed.
I suggest to provide all config.logs, not just the one from the top directory.
Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 6 lines · Olaf Hering
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <[email protected]>:Quoted 2 lines
Tumbleweed is generally broken and fails at ./configure due to missing compression libraries.
Something requests zlib-devel to be installed.
Yeah - it shouldn't be hard to fix.
Quoted 3 lines
I suggest to provide all config.logs, not just the one from the top directory.
Also a "test -f
/etc/os-release&& cat/etc/os-release" should be added after "$CC --version", just to know how stale the container is.
We ought to yes, although I think at the moment we probably don't want to realise how stale all the containers are.
~Andrew
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Am Wed, 7 Jul 2021 23:47:02 +0100
schrieb Andrew Cooper <[email protected]>:
Quoted 2 lines
We ought to yes, although I think at the moment we probably don't want to realise how stale all the containers are.
At least for Tumbleweed it is useful information.
opensuse-tumbleweed.dockerfile requests zlib-devel already.
tools/config.log is required to see what the failure actually is. The toplevel config.log is not helpful in this particular case.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <[email protected]>:
Quoted 1 line
At least for Tumbleweed it is useful information.
CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"
Two years old for a rolling release.
Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 9 lines · Olaf Hering
Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <[email protected]>:Quoted 1 line
At least for Tumbleweed it is useful information.
CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"
Two years old for a rolling release.
Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.
Automatically rebuilding the containers is still a task on the CI working group's todo list.
That said, the tumbleweed containers were rebuilt far more recently than that, so something is wonky...
In a xen tree, see automation/build/Makefile for details, but tl;dr to rebuild:
$ make -C automation/build/ tumbleweed
and to push:
$ docker login registry.gitlab.com/xen-project/xen $ make -C automation/build/ tumbleweed PUSH=1
Bear in mind that the final line will update the live CI system. What I normally do to test new containers is to give it a test suffix, push to the live system, and push a branch to xen.git with the container names edited in automation/gitlab-ci/*.yaml.
~Andrew
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Am Thu, 8 Jul 2021 13:51:49 +0100
schrieb Andrew Cooper <[email protected]>:
Quoted 2 lines
That said, the tumbleweed containers were rebuilt far more recently than that, so something is wonky...
After playing with 'automation/scripts/containerize', docker image ls shows:
registry.gitlab.com/xen-project/xen/suse opensuse-tumbleweed f9b84e2346f0 7 weeks ago 2.27GB
No idea how that created a container with broken content. zlib-devel.rpm exists, but libz.so is a broken symlink.
I will update the README to list the required steps for updating.
Olaf
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
In reply to Andrew CooperQuoted 26 lines · Andrew Cooper
Quoted 4 lines · Anthony PERARD
ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.
Quoted 15 lines
Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build fileninja_required_versionversion (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installedOnly the first failure seems to be related to ninja.
The second is a known bug which still needs addressing. Our build system's logic to cope with
checkpolicybeing absent doesn't work.
Quoted 7 lines
Tumbleweed is generally broken and fails at ./configure due to missing compression libraries. It is quite possible that the iPXE failure is related.
The final one is probably because unstable has dropped clang-8. The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.
So those containers are ones that I've noticed needed ninja (as found out by looking at a single pipeline), but one which trying to install ninja and thus rebuilding the container mean that the build was broken. I just didn't want to spend time trying to fix issues that should have been found out by a regular rebuild of the containers. So I've just listed the reason there and moved on.
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 34 lines · Anthony PERARD
Quoted 25 lines · Andrew Cooper
Quoted 4 lines · Anthony PERARD
ninja is now required to build the latest version of QEMU, some container still don't have ninja and attempting to add it breaks the build for different reasons, so QEMU will be skip on those containers.
Quoted 15 lines
Failures:
- ubuntu/xenial:
fatal: ninja version (1.5.1) incompatible with build fileninja_required_versionversion (1.7.1).
- debian/unstable-i386: (debian-unstable-32) /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'. Stop.
- suse/opensuse-tumbleweed:
failed to build iPXE
- debian/unstable:
update of the container fails with:
The following packages have unmet dependencies:
clang-8 : Depends: libstdc++-8-dev but it is not installable
Depends: libgcc-8-dev but it is not installable
Depends: libobjc-8-dev but it is not installable
Recommends: llvm-8-dev but it is not going to be installed
Recommends: libomp-8-dev but it is not going to be installedOnly the first failure seems to be related to ninja.
The second is a known bug which still needs addressing. Our build system's logic to cope with
checkpolicybeing absent doesn't work.Quoted 7 lines
Tumbleweed is generally broken and fails at ./configure due to missing compression libraries. It is quite possible that the iPXE failure is related.
The final one is probably because unstable has dropped clang-8. The container hasn't been rebuilt in a while because we still haven't got auto-rebuild sorted.
Quoted 5 lines
So those containers are ones that I've noticed needed ninja (as found out by looking at a single pipeline), but one which trying to install ninja and thus rebuilding the container mean that the build was broken. I just didn't want to spend time trying to fix issues that should have been found out by a regular rebuild of the containers.
That's fine, but as these containers weren't rebuilt, the details of any knock-on error aren't relevant here.
I'd go with simply:
"ninja is now required to build the latest version of QEMU, and not all distros have a suitable version. Skip the QEMU build when ninja is not available".
Can fix on commit if you're happy with this.
~Andrew
Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU
Quoted 7 lines · Andrew Cooper
I'd go with simply:
"ninja is now required to build the latest version of QEMU, and not all distros have a suitable version. Skip the QEMU build when ninja is not available".
Can fix on commit if you're happy with this.
Sounds good, thanks.
Signature · 1 line
Anthony PERARD
Quoted 3 lines · Anthony PERARD
Adding ninja-build pkg when possible.
I'll push new containers soon.
I've pushed:
registry.gitlab.com/xen-project/xen/fedora:29 registry.gitlab.com/xen-project/xen/ubuntu:bionic registry.gitlab.com/xen-project/xen/ubuntu:focal registry.gitlab.com/xen-project/xen/alpine:3.12 registry.gitlab.com/xen-project/xen/suse:opensuse-leap
Quoted 12 lines
Anthony PERARD (2):
automation: Adding ninja-build to some docker images
automation: Check if ninja is available before building QEMUautomation/build/alpine/3.12.dockerfile | 1 + automation/build/fedora/29.dockerfile | 9 ++++++--- automation/build/suse/opensuse-leap.dockerfile | 2 ++ automation/build/ubuntu/bionic.dockerfile | 2 ++ automation/build/ubuntu/focal.dockerfile | 2 ++ automation/scripts/build | 4 +++- 6 files changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
Anthony PERARD