build: replace ninjatool with ninja
Subsystems (2)
cover build: replace ninjatool with ninja
This pull request is the last build system change for 5.2 from me, and it is simple: similar to how we are invoking
ROM or tests/tcg "make" from the main Makefile, we now invoke ninja to build QEMU. Unlike those cases, however, build.ninja targets are forwarded transparently.
The advantages cover various areas:
- maintainability: we drop scripts/ninjatool.py, which is a large and hairy piece of code, and generally remove one thing that can go wrong and one thing that is unique to QEMU;
- platform support: we remove the requirement for GNU make 3.82, which was annoying on Mac. We also avoid bugs on Windows due to meson emitting Windows rather than POSIX escapes (as expected by Ninja) and ninjatool ignoring the difference;
- speed: invoking "configure" does not have to generate 44k lines of rules, while invoking "Make" does not anymore have to parse 44k lines of rules.
- ease of use: Ninja tracks command lines, hence the problem of static library changing the objects they hold goes away
Paolo
Paolo Bonzini (7):
tests/Makefile.include: unbreak non-tcg builds
make: run shell with pipefail
tests: add missing generated sources to testqapi
configure: move QEMU_INCLUDES to meson
dockerfiles: enable Centos 8 PowerTools
add ninja to dockerfiles, CI configurations and test VMs
build: replace ninjatool with ninja
.cirrus.yml | 6 +- .travis.yml | 13 + Makefile | 42 +- configure | 29 +- docs/devel/build-system.rst | 6 +- meson.build | 34 +- scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 -------------------- tests/Makefile.include | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 4 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/include/meson.build | 8 +- tests/meson.build | 14 +- tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 27 files changed, 121 insertions(+), 1070 deletions(-) delete mode 100755 scripts/ninjatool.py
Signature · 1 line
2.26.2
PATCH 1/7 tests/Makefile.include: unbreak non-tcg builds
Remove from check-block the requirement that all TARGET_DIRS are built.
--- tests/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +1 -1
@@ -140,7 +140,7 @@ QEMU_IOTESTS_HELPERS-$(CONFIG_LINUX) = tests/qemu-iotests/socket_scm_helper$(EXE check: check-block check-block: $(SRC_PATH)/tests/check-block.sh qemu-img$(EXESUF) \ qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) \ - $(patsubst %-softmmu,qemu-system-%,$(filter %-softmmu,$(TARGET_DIRS))) + $(filter qemu-system-%, $(ninja-targets-c_LINKER) $(ninja-targets-cpp_LINKER)) @$< endif
Signature · 1 line
2.26.2
Re: [PATCH 1/7] tests/Makefile.include: unbreak non-tcg builds
Quoted 6 lines · Paolo Bonzini
Remove from check-block the requirement that all
TARGET_DIRSare built.Signed-off-by Paolo Bonzini <[email protected]>--- tests/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 2/7 make: run shell with pipefail
In reply to Paolo BonziniWithout pipefail, it is possible to miss failures if the recipes include pipes.
--- Makefile | 2 ++ 1 file changed, 2 insertions(+)
Makefile +2 -0
@@ -14,6 +14,8 @@ SRC_PATH=. # we have explicit rules for everything MAKEFLAGS += -rR +SHELL = /usr/bin/env bash -o pipefail + # Usage: $(call quiet-command,command and args,"NAME","args to print") # This will run "command and args", and either: # if V=1 just print the whole command and args
Signature · 1 line
2.26.2
Re: [PATCH 2/7] make: run shell with pipefail
Quoted 7 lines · Paolo Bonzini
Without pipefail, it is possible to miss failures if the recipes include pipes.
Signed-off-by Paolo Bonzini <[email protected]>--- Makefile | 2 ++ 1 file changed, 2 insertions(+)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 3/7 tests: add missing generated sources to testqapi
In reply to Paolo BonziniNinja notices them due to a different order in visiting the graph.
--- tests/include/meson.build | 8 ++++---- tests/meson.build | 14 ++++++++++++-- 2 files changed, 16 insertions(+), 6 deletions(-)
tests/include/meson.build +4 -4
@@ -10,7 +10,7 @@ test_qapi_outputs_extra = [ 'test-qapi-visit-sub-module.h', ] -test_qapi_outputs_extra = custom_target('QAPI test (include)', - output: test_qapi_outputs_extra, - input: test_qapi_files, - command: 'true') +test_qapi_files_extra = custom_target('QAPI test (include)', + output: test_qapi_outputs_extra, + input: test_qapi_files, + command: 'true')
tests/meson.build +12 -2
@@ -56,8 +56,18 @@ test_qapi_files = custom_target('Test QAPI files', # perhaps change qapi_gen to replace / with _, like Meson itself does? subdir('include') -libtestqapi = static_library('testqapi', sources: [test_qapi_files, genh, test_qapi_outputs_extra]) -testqapi = declare_dependency(link_with: libtestqapi) +test_qapi_sources = [] +test_qapi_headers = [] +i = 0 +foreach o: test_qapi_files.to_list() + test_qapi_files_extra.to_list() + if o.full_path().endswith('.h') + test_qapi_headers += o + endif + test_qapi_sources += o +endforeach + +libtestqapi = static_library('testqapi', sources: [genh, test_qapi_sources]) +testqapi = declare_dependency(link_with: libtestqapi, sources: [genh, test_qapi_headers]) testblock = declare_dependency(dependencies: [block], sources: 'iothread.c')
Signature · 1 line
2.26.2
Re: [PATCH 3/7] tests: add missing generated sources to testqapi
Quoted 7 lines · Paolo Bonzini
Ninja notices them due to a different order in visiting the graph.
Signed-off-by Paolo Bonzini <[email protected]>--- tests/include/meson.build | 8 ++++---- tests/meson.build | 14 ++++++++++++-- 2 files changed, 16 insertions(+), 6 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 4/7 configure: move QEMU_INCLUDES to meson
In reply to Paolo BonziniConfusingly, QEMU_INCLUDES is not used by configure tests. Moving it to meson.build ensures that Windows paths are specified instead of the msys paths like /c/Users/...
--- configure | 20 -------------------- meson.build | 30 ++++++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 22 deletions(-)
configure +0 -20
@@ -536,8 +536,6 @@ QEMU_CFLAGS="-fno-strict-aliasing -fno-common -fwrapv $QEMU_CFLAGS" QEMU_CFLAGS="-Wundef -Wwrite-strings -Wmissing-prototypes $QEMU_CFLAGS" QEMU_CFLAGS="-Wstrict-prototypes -Wredundant-decls $QEMU_CFLAGS" QEMU_CFLAGS="-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE $QEMU_CFLAGS" -QEMU_INCLUDES="-iquote . -iquote ${source_path} -iquote ${source_path}/accel/tcg -iquote ${source_path}/include" -QEMU_INCLUDES="$QEMU_INCLUDES -iquote ${source_path}/disas/libvixl" # Flags that are needed during configure but later taken care of by Meson CONFIGURE_CFLAGS="-std=gnu99 -Wall"
@@ -795,7 +793,6 @@ Linux) audio_possible_drivers="oss alsa sdl pa" linux="yes" linux_user="yes" - QEMU_INCLUDES="-isystem ${source_path}/linux-headers -Ilinux-headers $QEMU_INCLUDES" ;; esac
@@ -6918,22 +6915,6 @@ if test "$secret_keyring" = "yes" ; then echo "CONFIG_SECRET_KEYRING=y" >> $config_host_mak fi -if test "$tcg_interpreter" = "yes"; then - QEMU_INCLUDES="-iquote ${source_path}/tcg/tci $QEMU_INCLUDES" -elif test "$ARCH" = "sparc64" ; then - QEMU_INCLUDES="-iquote ${source_path}/tcg/sparc $QEMU_INCLUDES" -elif test "$ARCH" = "s390x" ; then - QEMU_INCLUDES="-iquote ${source_path}/tcg/s390 $QEMU_INCLUDES" -elif test "$ARCH" = "x86_64" || test "$ARCH" = "x32" ; then - QEMU_INCLUDES="-iquote ${source_path}/tcg/i386 $QEMU_INCLUDES" -elif test "$ARCH" = "ppc64" ; then - QEMU_INCLUDES="-iquote ${source_path}/tcg/ppc $QEMU_INCLUDES" -elif test "$ARCH" = "riscv32" || test "$ARCH" = "riscv64" ; then - QEMU_INCLUDES="-I${source_path}/tcg/riscv $QEMU_INCLUDES" -else - QEMU_INCLUDES="-iquote ${source_path}/tcg/${ARCH} $QEMU_INCLUDES" -fi - echo "ROMS=$roms" >> $config_host_mak echo "MAKE=$make" >> $config_host_mak echo "PYTHON=$python" >> $config_host_mak
@@ -6960,7 +6941,6 @@ echo "WINDRES=$windres" >> $config_host_mak echo "CFLAGS_NOPIE=$CFLAGS_NOPIE" >> $config_host_mak echo "QEMU_CFLAGS=$QEMU_CFLAGS" >> $config_host_mak echo "QEMU_CXXFLAGS=$QEMU_CXXFLAGS" >> $config_host_mak -echo "QEMU_INCLUDES=$QEMU_INCLUDES" >> $config_host_mak echo "GLIB_CFLAGS=$glib_cflags" >> $config_host_mak echo "GLIB_LIBS=$glib_libs" >> $config_host_mak echo "QEMU_LDFLAGS=$QEMU_LDFLAGS" >> $config_host_mak
meson.build +28 -2
@@ -93,9 +93,35 @@ add_project_arguments(config_host['QEMU_CXXFLAGS'].split(), native: false, language: 'cpp') add_project_link_arguments(config_host['QEMU_LDFLAGS'].split(), native: false, language: ['c', 'cpp', 'objc']) -add_project_arguments(config_host['QEMU_INCLUDES'].split(), - language: ['c', 'cpp', 'objc']) +if targetos == 'linux' + add_project_arguments('-isystem', meson.current_source_dir() / 'linux-headers', + '-isystem', 'linux-headers', + language: ['c', 'cpp']) +endif + +if 'CONFIG_TCG_INTERPRETER' in config_host + tcg_arch = 'tci' +elif config_host['ARCH'] == 'sparc64' + tcg_arch = 'sparc' +elif config_host['ARCH'] == 's390x' + tcg_arch = 's390' +elif config_host['ARCH'] in ['x86_64', 'x32'] + tcg_arch = 'i386' +elif config_host['ARCH'] == 'ppc64' + tcg_arch = 'ppc' +elif config_host['ARCH'] in ['riscv32', 'riscv64'] + tcg_arch = 'riscv' +else + tcg_arch = config_host['ARCH'] +endif +add_project_arguments('-iquote', meson.current_source_dir() / 'tcg' / tcg_arch, + '-iquote', '.', + '-iquote', meson.current_source_dir(), + '-iquote', meson.current_source_dir() / 'accel/tcg', + '-iquote', meson.current_source_dir() / 'include', + '-iquote', meson.current_source_dir() / 'disas/libvixl', + language: ['c', 'cpp', 'objc']) link_language = meson.get_external_property('link_language', 'cpp') if link_language == 'cpp'
Signature · 1 line
2.26.2
Re: [PATCH 4/7] configure: move QEMU_INCLUDES to meson
Quoted 9 lines · Paolo Bonzini
Confusingly,
QEMU_INCLUDESis not used by configure tests. Moving it to meson.build ensures that Windows paths are specified instead of the msys paths like /c/Users/...Signed-off-by Paolo Bonzini <[email protected]>--- configure | 20 -------------------- meson.build | 30 ++++++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 22 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 5/7 dockerfiles: enable Centos 8 PowerTools
In reply to Paolo Bonzinininja is included in the CentOS PowerTools repository.
--- tests/docker/dockerfiles/centos8.docker | 3 +++ 1 file changed, 3 insertions(+)
tests/docker/dockerfiles/centos8.docker +3 -0
@@ -1,6 +1,9 @@ FROM centos:8.1.1911 +RUN dnf -y install dnf-plugins-core +RUN dnf config-manager --set-enabled PowerTools RUN dnf -y update + ENV PACKAGES \ SDL-devel \ bzip2 \
Signature · 1 line
2.26.2
Re: [PATCH 5/7] dockerfiles: enable Centos 8 PowerTools
Quoted 12 lines · Paolo Bonzini
ninja is included in the CentOS PowerTools repository.
Signed-off-by Paolo Bonzini <[email protected]>--- tests/docker/dockerfiles/centos8.docker | 3 +++ 1 file changed, 3 insertions(+)tests/docker/dockerfiles/centos8.docker +0
-0
tests/docker/dockerfiles/centos8.docker line 5
Quoted 5 lines
tests/docker/dockerfiles/centos8.docker +2
-0@@ -1,6 +1,9 @@ FROM centos:8.1.1911 +RUN dnf -y install dnf-plugins-core +RUN dnf config-manager --set-enabled PowerTools RUN dnf -y update
This results in multiple image layers which is undesirable. Instead do
RUN dnf -y install dnf-plugins-core && \
dnf config-manager --set-enabled PowerTools && \
dnf -y update
tests/docker/dockerfiles/centos8.docker line 9
Quoted 4 lines
tests/docker/dockerfiles/centos8.docker +1
-0+ ENV PACKAGES \ SDL-devel \ bzip2 \
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 5/7] dockerfiles: enable Centos 8 PowerTools
In reply to Paolo BonziniQuoted 12 lines · Paolo Bonzini
ninja is included in the CentOS PowerTools repository.
Signed-off-by Paolo Bonzini <[email protected]>--- tests/docker/dockerfiles/centos8.docker | 3 +++ 1 file changed, 3 insertions(+)tests/docker/dockerfiles/centos8.docker +0
-0
tests/docker/dockerfiles/centos8.docker line 4
Quoted 4 lines
tests/docker/dockerfiles/centos8.docker +2
-0@@ -1,6 +1,9 @@ FROM centos:8.1.1911 +RUN dnf -y install dnf-plugins-core +RUN dnf config-manager --set-enabled PowerTools
Since config-manager won't be used besides here, I think it's safe and cheaper to simply do:
RUN sed -i /etc/yum.repos.d/CentOS-PowerTools.repo -e 's/^enabled=0/enabled=1/'
It's also a single command, reducing the number of layers.
- Cleber.
Quoted 10 lines
tests/docker/dockerfiles/centos8.docker +1
-0RUN dnf -y update + ENV PACKAGES \ SDL-devel \ bzip2 \Signature · 1 line
2.26.2
Re: [PATCH 5/7] dockerfiles: enable Centos 8 PowerTools
Quoted 21 lines · Cleber Rosa
Quoted 12 lines · Paolo Bonzini
ninja is included in the CentOS PowerTools repository.
Signed-off-by Paolo Bonzini <[email protected]>--- tests/docker/dockerfiles/centos8.docker | 3 +++ 1 file changed, 3 insertions(+)tests/docker/dockerfiles/centos8.docker +0
-0tests/docker/dockerfiles/centos8.docker line 4
Quoted 4 lines
tests/docker/dockerfiles/centos8.docker +2
-0@@ -1,6 +1,9 @@ FROM centos:8.1.1911 +RUN dnf -y install dnf-plugins-core +RUN dnf config-manager --set-enabled PowerToolsSince config-manager won't be used besides here, I think it's safe and cheaper to simply do:
Quoted 3 lines
RUN sed -i
/etc/yum.repos.d/CentOS-PowerTools.repo-e 's/^enabled=0/enabled=1/'It's also a single command, reducing the number of layers.
Yeah, I'm lazy. :) I'll use Dan's suggestion.
Paolo
PATCH 6/7 add ninja to dockerfiles, CI configurations and test VMs
In reply to Paolo Bonzini--- .cirrus.yml | 6 +++--- .travis.yml | 13 +++++++++++++ tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 1 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 18 files changed, 32 insertions(+), 9 deletions(-)
.cirrus.yml +3 -3
@@ -9,7 +9,7 @@ freebsd_12_task: install_script: - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ; - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed - nettle perl5 pixman pkgconf png usbredir + nettle perl5 pixman pkgconf png usbredir ninja script: - mkdir build - cd build
@@ -21,7 +21,7 @@ macos_task: osx_instance: image: catalina-base install_script: - - brew install pkg-config python gnu-sed glib pixman make sdl2 bash + - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja script: - mkdir build - cd build
@@ -36,7 +36,7 @@ macos_xcode_task: # this is an alias for the latest Xcode image: catalina-xcode install_script: - - brew install pkg-config gnu-sed glib pixman make sdl2 bash + - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja script: - mkdir build - cd build
.travis.yml +13 -0
@@ -49,6 +49,7 @@ addons: - libvdeplug-dev - libvte-2.91-dev - libzstd-dev + - ninja-build - sparse - uuid-dev - gcovr
@@ -177,6 +178,7 @@ jobs: addons: apt: packages: + - ninja-build - python3-sphinx - perl
@@ -211,6 +213,10 @@ jobs: # gprof/gcov are GCC features - name: "GCC gprof/gcov" dist: bionic + addons: + apt: + packages: + - ninja-build env: - CONFIG="--enable-gprof --enable-gcov --disable-libssh --target-list=${MAIN_SOFTMMU_TARGETS}"
@@ -281,6 +287,7 @@ jobs: - liburcu-dev - libusb-1.0-0-dev - libvte-2.91-dev + - ninja-build - sparse - uuid-dev language: generic
@@ -346,6 +353,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:
@@ -379,6 +387,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:
@@ -411,6 +420,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:
@@ -450,6 +460,7 @@ jobs: - libzstd-dev - nettle-dev - xfslibs-dev + - ninja-build # Tests dependencies - genisoimage env:
@@ -463,6 +474,7 @@ jobs: apt_packages: - libgcrypt20-dev - libgnutls28-dev + - ninja-build env: - CONFIG="--disable-containers --disable-system"
@@ -493,6 +505,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build env: - TEST_CMD="make check-unit" - CONFIG="--disable-containers --disable-tcg --enable-kvm
tests/docker/dockerfiles/centos7.docker +1 -0
@@ -27,6 +27,7 @@ ENV PACKAGES \ mesa-libEGL-devel \ mesa-libgbm-devel \ nettle-devel \ + ninja-build \ perl-Test-Harness \ pixman-devel \ python3 \
tests/docker/dockerfiles/centos8.docker +1 -0
@@ -22,6 +22,7 @@ ENV PACKAGES \ make \ mesa-libEGL-devel \ nettle-devel \ + ninja-build \ perl-Test-Harness \ pixman-devel \ python36 \
tests/docker/dockerfiles/debian10.docker +1 -0
@@ -26,6 +26,7 @@ RUN apt update && \ gettext \ git \ libncurses5-dev \ + ninja-build \ pkg-config \ psmisc \ python3 \
tests/docker/dockerfiles/fedora.docker +1 -0
@@ -75,6 +75,7 @@ ENV PACKAGES \ mingw64-SDL2 \ ncurses-devel \ nettle-devel \ + ninja-build \ nss-devel \ numactl-devel \ perl \
tests/docker/dockerfiles/travis.docker +1 -1
@@ -9,7 +9,7 @@ ENV LC_ALL en_US.UTF-8 RUN sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list RUN apt-get update RUN apt-get -y build-dep qemu -RUN apt-get -y install device-tree-compiler python3 python3-yaml dh-autoreconf gdb strace lsof net-tools gcovr +RUN apt-get -y install device-tree-compiler python3 python3-yaml dh-autoreconf gdb strace lsof net-tools gcovr ninja-build # Travis tools require PhantomJS / Neo4j / Maven accessible # in their PATH (QEMU build won't access them). ENV PATH /usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
tests/docker/dockerfiles/ubuntu.docker +1 -0
@@ -60,6 +60,7 @@ ENV PACKAGES \ libxen-dev \ libzstd-dev \ make \ + ninja-build \ python3-yaml \ python3-sphinx \ sparse \
tests/docker/dockerfiles/ubuntu1804.docker +1 -0
@@ -48,6 +48,7 @@ ENV PACKAGES \ make \ python3-yaml \ python3-sphinx \ + ninja-build \ sparse \ xfslibs-dev RUN apt-get update && \
tests/docker/dockerfiles/ubuntu2004.docker +1 -0
@@ -47,6 +47,7 @@ ENV PACKAGES flex bison \ libxen-dev \ libzstd-dev \ make \ + ninja-build \ python3-numpy \ python3-opencv \ python3-pil \
tests/vm/centos +1 -1
@@ -42,7 +42,7 @@ class CentosVM(basevm.BaseVM): self.wait_ssh() self.ssh_root_check("touch /etc/cloud/cloud-init.disabled") self.ssh_root_check("yum update -y") - self.ssh_root_check("yum install -y docker make git python3") + self.ssh_root_check("yum install -y docker make ninja-build git python3") self.ssh_root_check("systemctl enable docker") self.ssh_root("poweroff") self.wait()
tests/vm/centos.aarch64 +1 -1
@@ -23,7 +23,7 @@ import aarch64vm DEFAULT_CONFIG = { 'cpu' : "max", 'machine' : "virt,gic-version=max", - 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bison, "\ + 'install_cmds' : "yum install -y make ninja-build git python3 gcc gcc-c++ flex bison, "\ "yum install -y glib2-devel pixman-devel zlib-devel, "\ "yum install -y perl-Test-Harness, "\ "alternatives --set python /usr/bin/python3, "\
tests/vm/fedora +1 -1
@@ -32,7 +32,7 @@ class FedoraVM(basevm.BaseVM): pkgs = [ # tools 'git-core', - 'gcc', 'binutils', 'make', + 'gcc', 'binutils', 'make', 'ninja-build', # perl 'perl-Test-Harness',
tests/vm/freebsd +1 -0
@@ -34,6 +34,7 @@ class FreeBSDVM(basevm.BaseVM): "bzip2", "python37", "py37-setuptools", + "ninja", # gnu tools "bash",
tests/vm/netbsd +1 -0
@@ -32,6 +32,7 @@ class NetBSDVM(basevm.BaseVM): "xz", "python37", "py37-setuptools", + "ninja", # gnu tools "bash",
tests/vm/openbsd +1 -0
@@ -31,6 +31,7 @@ class OpenBSDVM(basevm.BaseVM): "pkgconf", "bzip2", "xz", "py3-setuptools", + "ninja", # gnu tools "bash",
tests/vm/ubuntu.aarch64 +1 -1
@@ -22,7 +22,7 @@ DEFAULT_CONFIG = { 'machine' : "virt,gic-version=3", 'install_cmds' : "apt-get update,"\ "apt-get build-dep -y --arch-only qemu,"\ - "apt-get install -y libfdt-dev pkg-config language-pack-en", + "apt-get install -y libfdt-dev pkg-config language-pack-en ninja-build", # We increase beyond the default time since during boot # it can take some time (many seconds) to log into the VM # especially using softmmu.
tests/vm/ubuntu.i386 +1 -1
@@ -18,7 +18,7 @@ import ubuntuvm DEFAULT_CONFIG = { 'install_cmds' : "apt-get update,"\ "apt-get build-dep -y qemu,"\ - "apt-get install -y libfdt-dev language-pack-en", + "apt-get install -y libfdt-dev language-pack-en ninja-build", } class UbuntuX86VM(ubuntuvm.UbuntuVM):
Signature · 1 line
2.26.2
Re: [PATCH 6/7] add ninja to dockerfiles, CI configurations and test VMs
Quoted 13 lines · Paolo Bonzini
Signed-off-by Paolo Bonzini <[email protected]>--- .cirrus.yml | 6 +++--- .travis.yml | 13 +++++++++++++ tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 1 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/vm/centos | 2 +-
Quoted 8 lines
tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 18 files changed, 32 insertions(+), 9 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 6/7] add ninja to dockerfiles, CI configurations and test VMs
In reply to Paolo BonziniQuoted 1 line · Paolo Bonzini
Signed-off-by Paolo Bonzini <[email protected]>
Quoted 337 lines
--- .cirrus.yml | 6 +++--- .travis.yml | 13 +++++++++++++ tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 1 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 18 files changed, 32 insertions(+), 9 deletions(-).cirrus.yml +3
-3@@ -9,7 +9,7 @@ freebsd_12_task: install_script: - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ; - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed - nettle perl5 pixman pkgconf png usbredir + nettle perl5 pixman pkgconf png usbredir ninja script: - mkdir build - cd build@@ -21,7 +21,7 @@ macos_task: osx_instance: image: catalina-base install_script: - - brew install pkg-config python gnu-sed glib pixman make sdl2 bash + - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja script: - mkdir build - cd build@@ -36,7 +36,7 @@ macos_xcode_task: # this is an alias for the latest Xcode image: catalina-xcode install_script: - - brew install pkg-config gnu-sed glib pixman make sdl2 bash + - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja script: - mkdir build - cd build.travis.yml +13
-0@@ -49,6 +49,7 @@ addons: - libvdeplug-dev - libvte-2.91-dev - libzstd-dev + - ninja-build - sparse - uuid-dev - gcovr@@ -177,6 +178,7 @@ jobs: addons: apt: packages: + - ninja-build - python3-sphinx - perl@@ -211,6 +213,10 @@ jobs: # gprof/gcov are GCC features - name: "GCC gprof/gcov" dist: bionic + addons: + apt: + packages: + - ninja-build env: - CONFIG="--enable-gprof --enable-gcov --disable-libssh --target-list=${MAIN_SOFTMMU_TARGETS}"@@ -281,6 +287,7 @@ jobs: - liburcu-dev - libusb-1.0-0-dev - libvte-2.91-dev + - ninja-build - sparse - uuid-dev language: generic@@ -346,6 +353,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:@@ -379,6 +387,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:@@ -411,6 +420,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build # Tests dependencies - genisoimage env:@@ -450,6 +460,7 @@ jobs: - libzstd-dev - nettle-dev - xfslibs-dev + - ninja-build # Tests dependencies - genisoimage env:@@ -463,6 +474,7 @@ jobs: apt_packages: - libgcrypt20-dev - libgnutls28-dev + - ninja-build env: - CONFIG="--disable-containers --disable-system"@@ -493,6 +505,7 @@ jobs: - libusb-1.0-0-dev - libvdeplug-dev - libvte-2.91-dev + - ninja-build env: - TEST_CMD="make check-unit" - CONFIG="--disable-containers --disable-tcg --enable-kvmtests/docker/dockerfiles/centos7.docker +1
-0@@ -27,6 +27,7 @@ ENV PACKAGES \ mesa-libEGL-devel \ mesa-libgbm-devel \ nettle-devel \ + ninja-build \ perl-Test-Harness \ pixman-devel \ python3 \tests/docker/dockerfiles/centos8.docker +1
-0@@ -22,6 +22,7 @@ ENV PACKAGES \ make \ mesa-libEGL-devel \ nettle-devel \ + ninja-build \ perl-Test-Harness \ pixman-devel \ python36 \tests/docker/dockerfiles/debian10.docker +1
-0@@ -26,6 +26,7 @@ RUN apt update && \ gettext \ git \ libncurses5-dev \ + ninja-build \ pkg-config \ psmisc \ python3 \tests/docker/dockerfiles/fedora.docker +1
-0@@ -75,6 +75,7 @@ ENV PACKAGES \ mingw64-SDL2 \ ncurses-devel \ nettle-devel \ + ninja-build \ nss-devel \ numactl-devel \ perl \tests/docker/dockerfiles/travis.docker +1
-1@@ -9,7 +9,7 @@ ENV LC_ALL en_US.UTF-8 RUN sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list RUN apt-get update RUN apt-get -y build-dep qemu -RUN apt-get -y install device-tree-compiler python3 python3-yaml dh-autoreconf gdb strace lsof net-tools gcovr +RUN apt-get -y install device-tree-compiler python3 python3-yaml dh-autoreconf gdb strace lsof net-tools gcovr ninja-build # Travis tools require PhantomJS / Neo4j / Maven accessible # in their PATH (QEMU build won't access them). ENV PATH /usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bintests/docker/dockerfiles/ubuntu.docker +1
-0@@ -60,6 +60,7 @@ ENV PACKAGES \ libxen-dev \ libzstd-dev \ make \ + ninja-build \ python3-yaml \ python3-sphinx \ sparse \tests/docker/dockerfiles/ubuntu1804.docker +1
-0@@ -48,6 +48,7 @@ ENV PACKAGES \ make \ python3-yaml \ python3-sphinx \ + ninja-build \ sparse \ xfslibs-dev RUN apt-get update && \tests/docker/dockerfiles/ubuntu2004.docker +1
-0@@ -47,6 +47,7 @@ ENV PACKAGES flex bison \ libxen-dev \ libzstd-dev \ make \ + ninja-build \ python3-numpy \ python3-opencv \ python3-pil \tests/vm/centos +1
-1@@ -42,7 +42,7 @@ class CentosVM(basevm.BaseVM): self.wait_ssh() self.ssh_root_check("touch /etc/cloud/cloud-init.disabled") self.ssh_root_check("yum update -y") - self.ssh_root_check("yum install -y docker make git python3") + self.ssh_root_check("yum install -y docker make ninja-build git python3") self.ssh_root_check("systemctl enable docker") self.ssh_root("poweroff") self.wait()tests/vm/centos.aarch64 +1
-1@@ -23,7 +23,7 @@ import aarch64vm DEFAULT_CONFIG = { 'cpu' : "max", 'machine' : "virt,gic-version=max", - 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bison, "\ + 'install_cmds' : "yum install -y make ninja-build git python3 gcc gcc-c++ flex bison, "\ "yum install -y glib2-devel pixman-devel zlib-devel, "\ "yum install -y perl-Test-Harness, "\ "alternatives --set python /usr/bin/python3, "\tests/vm/fedora +1
-1@@ -32,7 +32,7 @@ class FedoraVM(basevm.BaseVM): pkgs = [ # tools 'git-core', - 'gcc', 'binutils', 'make', + 'gcc', 'binutils', 'make', 'ninja-build', # perl 'perl-Test-Harness',tests/vm/freebsd +1
-0@@ -34,6 +34,7 @@ class FreeBSDVM(basevm.BaseVM): "bzip2", "python37", "py37-setuptools", + "ninja", # gnu tools "bash",tests/vm/netbsd +1
-0@@ -32,6 +32,7 @@ class NetBSDVM(basevm.BaseVM): "xz", "python37", "py37-setuptools", + "ninja", # gnu tools "bash",tests/vm/openbsd +1
-0@@ -31,6 +31,7 @@ class OpenBSDVM(basevm.BaseVM): "pkgconf", "bzip2", "xz", "py3-setuptools", + "ninja", # gnu tools "bash",tests/vm/ubuntu.aarch64 +1
-1@@ -22,7 +22,7 @@ DEFAULT_CONFIG = { 'machine' : "virt,gic-version=3", 'install_cmds' : "apt-get update,"\ "apt-get build-dep -y --arch-only qemu,"\ - "apt-get install -y libfdt-dev pkg-config language-pack-en", + "apt-get install -y libfdt-dev pkg-config language-pack-en ninja-build", # We increase beyond the default time since during boot # it can take some time (many seconds) to log into the VM # especially using softmmu.tests/vm/ubuntu.i386 +1
-1@@ -18,7 +18,7 @@ import ubuntuvm DEFAULT_CONFIG = { 'install_cmds' : "apt-get update,"\ "apt-get build-dep -y qemu,"\ - "apt-get install -y libfdt-dev language-pack-en", + "apt-get install -y libfdt-dev language-pack-en ninja-build", } class UbuntuX86VM(ubuntuvm.UbuntuVM):
Signature · 1 line
Alex Bennée
PATCH 7/7 build: replace ninjatool with ninja
In reply to Paolo BonziniNow that the build is done entirely by Meson, there is no need to keep the Makefile conversion. Instead, we can ask Ninja about the targets it exposes and forward them.
The main advantages are, from smallest to largest:
- reducing the possible namespace pollution within the Makefile
- removal of a relatively large Python program
- faster build because parsing Makefile.ninja is slower than parsing build.ninja; and faster build after Meson runs because we do not have to generate Makefile.ninja.
- tracking of command lines, which provides more accurate rebuilds
In addition the change removes the requirement for GNU make 3.82, which was annoying on Mac, and avoids bugs on Windows due to ninjatool not knowing how to convert Windows escapes to POSIX escapes.
--- Makefile | 40 +- configure | 9 +- docs/devel/build-system.rst | 6 +- meson.build | 4 - scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 ----------------------------------- tests/Makefile.include | 2 +- 7 files changed, 40 insertions(+), 1033 deletions(-) delete mode 100755 scripts/ninjatool.py
Makefile +28 -12
@@ -64,17 +64,34 @@ git-submodule-update: endif endif -export NINJA=./ninjatool +.PHONY: ninja-clean ninja-distclean +ninja-clean ninja-distclean:: +ifneq ($(NINJA),) +export NINJA +NINJAFLAGS = $(if $V,-v,) \ + $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \ + $(subst -k, -k0, $(filter -n -k,$(MAKEFLAGS))) -# Running meson regenerates both build.ninja and ninjatool, and that is -# enough to prime the rest of the build. -ninjatool: build.ninja - -Makefile.ninja: build.ninja ninjatool - ./ninjatool -t ninja2make --omit clean dist uninstall cscope TAGS ctags < $< > $@ -include Makefile.ninja +Makefile.ninja: build.ninja + { echo 'ninja-targets = \'; $(NINJA) -t targets all | sed 's/:.*//; $$!s/$$/ \\/'; } > $@ + +ninja-targets := $(filter-out build.ninja dist clean uninstall, $(ninja-targets)) +ninja-cmd-goals = $(or $(MAKECMDGOALS), all) +.PHONY: $(ninja-targets) run-ninja +$(ninja-targets): run-ninja -${ninja-targets-c_COMPILER} ${ninja-targets-cpp_COMPILER}: .var.command += -MP +# Use "| cat" to give Ninja a more "make-y" output. Use "+" to bypass the +# --output-sync line. +run-ninja: + +$(NINJA) $(NINJAFLAGS) $(sort $(filter $(ninja-targets), $(ninja-cmd-goals))) | cat + +ninja-clean:: + +$(quiet-@)$(NINJA) $(NINJAFLAGS) -t clean + +ninja-distclean:: + +$(quiet-@)$(NINJA) $(NINJAFLAGS) -t clean -g +endif # If MESON is empty, the rule will be re-evaluated after Makefiles are # reread (and MESON won't be empty anymore).
@@ -159,8 +176,8 @@ recurse-clean: $(addsuffix /clean, $(ROM_DIRS)) ###################################################################### -clean: recurse-clean ninja-clean clean-ctlist - if test -f ninjatool; then ./ninjatool $(if $(V),-v,) -t clean; fi +clean: recurse-clean ninja-clean + -test -f build.ninja && @$(NINJA) $(NINJAFLAGS) clean-ctlist || : # avoid old build problems by removing potentially incorrect old files rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h find . \( -name '*.so' -o -name '*.dll' -o -name '*.[oda]' \) -type f \
@@ -178,7 +195,6 @@ qemu-%.tar.bz2: $(SRC_PATH)/scripts/make-release "$(SRC_PATH)" "$(patsubst qemu-%.tar.bz2,%,$@)" distclean: clean ninja-distclean - -test -f ninjatool && ./ninjatool $(if $(V),-v,) -t clean -g rm -f config-host.mak config-host.h* rm -f tests/tcg/config-*.mak rm -f config-all-disas.mak config.status
@@ -187,7 +203,7 @@ distclean: clean ninja-distclean rm -f qemu-plugins-ld.symbols qemu-plugins-ld64.symbols rm -f *-config-target.h *-config-devices.mak *-config-devices.h rm -rf meson-private meson-logs meson-info compile_commands.json - rm -f Makefile.ninja ninjatool ninjatool.stamp Makefile.mtest + rm -f Makefile.ninja Makefile.mtest rm -f config.log rm -f linux-headers/asm rm -Rf .sdk
configure +6 -3
@@ -1904,7 +1904,7 @@ case "$meson" in *) meson=$(command -v "$meson") ;; esac -# Probe for ninja (used for compdb) +# Probe for ninja if test -z "$ninja"; then for c in ninja ninja-build samu; do
@@ -1913,6 +1913,9 @@ if test -z "$ninja"; then break fi done + if test -z "$ninja"; then + error_exit "Cannot find Ninja" + fi fi # Check that the C compiler works. Doing this here before testing
@@ -6921,6 +6924,7 @@ echo "PYTHON=$python" >> $config_host_mak echo "SPHINX_BUILD=$sphinx_build" >> $config_host_mak echo "GENISOIMAGE=$genisoimage" >> $config_host_mak echo "MESON=$meson" >> $config_host_mak +echo "NINJA=$ninja" >> $config_host_mak echo "CC=$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then echo "CONFIG_IASL=$iasl" >> $config_host_mak
@@ -7172,7 +7176,7 @@ fi mv $cross config-meson.cross rm -rf meson-private meson-info meson-logs -NINJA=${ninja:-$PWD/ninjatool} $meson setup \ +NINJA=$ninja $meson setup \ --prefix "$prefix" \ --libdir "$libdir" \ --libexecdir "$libexecdir" \
@@ -7204,7 +7208,6 @@ NINJA=${ninja:-$PWD/ninjatool} $meson setup \ if test "$?" -ne 0 ; then error_exit "meson setup failed" fi -touch ninjatool.stamp fi if test -n "${deprecated_features}"; then
docs/devel/build-system.rst +2 -4
@@ -404,10 +404,8 @@ Built by Meson: Built by Makefile: `Makefile.ninja` - A Makefile conversion of the build rules in build.ninja. The conversion - is straightforward and, were it necessary to debug the rules produced - by Meson, it should be enough to look at build.ninja. The conversion - is performed by scripts/ninjatool.py. + A Makefile include that bridges to ninja for the actual build. The + Makefile is just a list of targets included by Meson in build.ninja. `Makefile.mtest` The Makefile definitions that let "make check" run tests defined in
meson.build +0 -4
@@ -47,10 +47,6 @@ supported_cpus = ['ppc', 'ppc64', 's390x', 'sparc64', 'riscv32', 'riscv64', 'x86 cpu = host_machine.cpu_family() targetos = host_machine.system() -configure_file(input: files('scripts/ninjatool.py'), - output: 'ninjatool', - configuration: config_host) - if cpu in ['x86', 'x86_64'] kvm_targets = ['i386-softmmu', 'x86_64-softmmu'] elif cpu == 'aarch64'
scripts/mtest2make.py +3 -1
@@ -70,8 +70,9 @@ def process_tests(test, targets, suites): print('.test.driver.%d := %s' % (i, driver)) print('.test.env.%d := $(.test.env) %s' % (i, env)) print('.test.cmd.%d := %s' % (i, cmd)) + print('.test.deps.%d := %s' % (i, ' '.join(deps))) print('.PHONY: run-test-%d' % (i,)) - print('run-test-%d: %s' % (i, ' '.join(deps))) + print('run-test-%d: $(.test.deps.%d)' % (i,i)) print('\t@$(call .test.run,%d,$(.test.output-format))' % (i,)) test_suites = test['suite'] or ['default']
@@ -128,3 +129,4 @@ for name, suite in benchsuites.items(): emit_suite(name, suite, 'bench') print('run-tests: $(patsubst %, run-test-%, $(.tests))') +print('ninja-cmd-goals += $(foreach t, $(.tests), $(.test.deps.$t))')
scripts/ninjatool.py deleted +0 -1008
1,008 lines not on this page · raw message
tests/Makefile.include +1 -1
@@ -140,7 +140,7 @@ QEMU_IOTESTS_HELPERS-$(CONFIG_LINUX) = tests/qemu-iotests/socket_scm_helper$(EXE check: check-block check-block: $(SRC_PATH)/tests/check-block.sh qemu-img$(EXESUF) \ qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) \ - $(filter qemu-system-%, $(ninja-targets-c_LINKER) $(ninja-targets-cpp_LINKER)) + $(filter qemu-system-%, $(ninja-targets)) @$< endif
Signature · 1 line
2.26.2
Quoted 21 lines · Paolo Bonzini
Now that the build is done entirely by Meson, there is no need to keep the
Makefileconversion. Instead, we can ask Ninja about the targets it exposes and forward them.The main advantages are, from smallest to largest:
- reducing the possible namespace pollution within the
Makefile- removal of a relatively large Python program
- faster build because parsing Makefile.ninja is slower than parsing build.ninja; and faster build after Meson runs because we do not have to generate Makefile.ninja.
- tracking of command lines, which provides more accurate rebuilds
In addition the change removes the requirement for GNU make 3.82, which was annoying on Mac, and avoids bugs on Windows due to ninjatool not knowing how to convert Windows escapes to POSIX escapes.
Signed-off-by Paolo Bonzini <[email protected]>
Quoted 10 lines
--- Makefile | 40 +- configure | 9 +- docs/devel/build-system.rst | 6 +- meson.build | 4 - scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 ----------------------------------- tests/Makefile.include | 2 +- 7 files changed, 40 insertions(+), 1033 deletions(-) delete mode 100755 scripts/ninjatool.py
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 8 lines · Paolo Bonzini
This pull request is the last build system change for 5.2 from me, and it is simple: similar to how we are invoking
ROM or tests/tcg "make" from the mainMakefile, we now invoke ninja to build QEMU. Unlike those cases, however, build.ninja targets are forwarded transparently.27 files changed, 121 insertions(+), 1070 deletions(-)
The diffstat is certainly persuasive :-)
-- PMM
Quoted 17 lines · Paolo Bonzini
This pull request is the last build system change for 5.2 from me, and it is simple: similar to how we are invoking
ROM or tests/tcg "make" from the mainMakefile, we now invoke ninja to build QEMU. Unlike those cases, however, build.ninja targets are forwarded transparently.The advantages cover various areas:
- maintainability: we drop
scripts/ninjatool.py, which is a large and hairy piece of code, and generally remove one thing that can go wrong and one thing that is unique to QEMU;- platform support: we remove the requirement for GNU make 3.82, which was annoying on Mac. We also avoid bugs on Windows due to meson emitting Windows rather than POSIX escapes (as expected by Ninja) and ninjatool ignoring the difference;
Quoted 10 lines
- speed: invoking "configure" does not have to generate 44k lines of rules, while invoking "Make" does not anymore have to parse 44k lines of rules.
- ease of use: Ninja tracks command lines, hence the problem of static library changing the objects they hold goes away
Paolo
Hi,
I still have some issues when compiling on Windows with msys2/mingw64 after this patch set. Command line:
mkdir build && cd build && ../configure --cross-prefix=x86_64-w64-mingw32- --target-list=ppc-softmmu --enable-gtk --enable-sdl
Traceback (most recent call last):
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/mesonmain.py",
line 131, in run
return options.run_func(options)
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 245, in run
app.generate()
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 159, in generate
self._generate(env)
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 215, in _generate
intr.backend.generate()
File
"C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/backend/ninjabackend.py", line 483, in generate
ninja = environment.detect_ninja_command_and_version(log=True)
File
"C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/environment.py", line 177, in detect_ninja_command_and_version
name = os.path.basename(n)
File "C:/msys64/mingw64/lib/python3.8/ntpath.py", line 231, in basename
return split(p)[1]
File "C:/msys64/mingw64/lib/python3.8/ntpath.py", line 200, in split
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: meson setup failed
When I set --ninja=ninja explicitly, this error does not occur:
Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXE
However, compiling then runs into a problem:
$ make -j8
/mingw64/bin/python3 -B /home/hsp/src/qemu-master/meson/meson.py introspect
--targets --tests --benchmarks | /mingw64/bin/python3 -B
scripts/mtest2make.py > Makefile.mtest
{ echo 'ninja-targets = \'; ninja -t targets all | sed 's/:.*//; $!s/$/
\\/'; } > Makefile.ninja
ninja -j8 all | cat
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....The problem with optionrom was already present earlier and could be fixed by applying:
https://patchwork.ozlabs.org/project/qemu-devel/list/?series=202798
That patch set does by now not apply cleanly anymore.
Best,
Howard
Quoted 4 lines · Howard Spoelstra
ERROR: meson setup failed
When I set --ninja=ninja explicitly, this error does not occur:
Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXE
That is fixed by Meson 0.55.2, I'll add a submodule update. Thanks for the reminder!
Paolo
Quoted 12 lines · Paolo Bonzini
Quoted 4 lines · Howard Spoelstra
ERROR: meson setup failed
When I set --ninja=ninja explicitly, this error does not occur:
Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXEThat is fixed by Meson 0.55.2, I'll add a submodule update. Thanks for the reminder!
Paolo
Thanks Paolo,
Then only the issue regarding the pcbios/optionrom stuff remains ;-)
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....
Best,
Howard
Quoted 9 lines · Howard Spoelstra
Thanks Paolo,
Then only the issue regarding the pcbios/optionrom stuff remains ;-)
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....
It seems to work on cirrus CI: https://cirrus-ci.com/task/5383432727429120
Paolo
Quoted 4 lines
Thanks Paolo,
Quoted 9 lines
Then only the issue regarding the pcbios/optionrom stuff remains ;-)
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....Best,
Howard
Hi Howard,
one solution for this issue is to uncomment the following line in msys2_shell.cmd.
rem To activate windows native symlinks uncomment next line set MSYS=winsymlinks:nativestrict
Then tell Windows 10 it's okay to create symlinks without elevated rights. Here is a link with a description how to do this.
https://www.joshkel.com/2018/01/18/symlinks-in-windows/
I think since commit bf708f3c4a "optionrom: simplify Makefile" pc-bios/optionrom/Makefile in your build directory has to be a symbolic link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies the Makefile instead of creating a symbolic link.
With best regards,
Volker
Quoted 16 lines · Volker Rümelin
Quoted 11 lines
Thanks Paolo,
Then only the issue regarding the pcbios/optionrom stuff remains ;-)
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....Best,
HowardHi Howard,
one solution for this issue is to uncomment the following line in msys2_shell.cmd.
Quoted 7 lines
rem To activate windows native symlinks uncomment next line set MSYS=winsymlinks:nativestrict
Then tell Windows 10 it's okay to create symlinks without elevated rights. Here is a link with a description how to do this.
https://www.joshkel.com/2018/01/18/symlinks-in-windows/I think since commit
bf708f3c4a"optionrom: simplifyMakefile" pc-bios/optionrom/Makefile in your build directory has to be a symbolic link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies theMakefileinstead of creating a symbolic link.
Wow thanks for this. I had the same issue as Howard, and I can confirm that this fixes the problem for me. Having said that it was quite fiddly to get this working correctly - is there any reason why https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html still can't be merged?
ATB,
Mark.
Il gio 15 ott 2020, 20:49 Mark Cave-Ayland <[email protected]> ha scritto:
Quoted 5 lines
Is there any reason why
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html still can't be
merged?
Because it's not the right approach. There is no reason why building firmware cannot be done with cross compilers, so moving those directories to Meson (not because Meson can't handle them; more specifically, the issue is tying the firmware build to the QEMU build system) is going in the wrong direction.
The "Canadian cross" scenario, where you build on Linux a mingw GCC but the compiler is s390, is not even enough to describe the complexity in the case of QEMU, because there are multiple firmware for different machines.
However we already have all the infrastructure to do such builds, we just don't use it for the firmware. So, instead of the patch you recalled above, the tests/tcg machinery should be extended into something that can be reused for firmware. As an aside, orchestrating this multi-compiler part of the build is what the Makefiles will keep on handling for the foreseeable future. As an aside to the aside, tests/tcg is more than underdocumented and I forget everything about it 5 minutes after looking at it.
This is not something that I will be able to work on anytime soon. But still I don't think that going in the wrong direction is a good idea, even if temporarily.
Paolo
Quoted 6 lines
ATB,
Mark.
Quoted 8 lines · Paolo Bonzini
Il gio 15 ott 2020, 20:49 Mark Cave-Ayland <[email protected] <mailto:[email protected]>> ha scritto:
Is there any reason why https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html <https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html> still can't be merged?
Quoted 7 lines
Because it's not the right approach. There is no reason why building firmware cannot be done with cross compilers, so moving those directories to Meson (not because Meson can't handle them; more specifically, the issue is tying the firmware build to the QEMU build system) is going in the wrong direction.
The "Canadian cross" scenario, where you build on Linux a mingw GCC but the compiler is s390, is not even enough to describe the complexity in the case of QEMU, because there are multiple firmware for different machines.
However we already have all the infrastructure to do such builds, we just don't use it for the firmware. So, instead of the patch you recalled above, the tests/tcg machinery should be extended into something that can be reused for firmware. As an aside, orchestrating this multi-compiler part of the build is what the Makefiles will keep on handling for the foreseeable future. As an aside to the aside, tests/tcg is more than underdocumented and I forget everything about it 5 minutes after looking at it.
This is not something that I will be able to work on anytime soon. But still I don't think that going in the wrong direction is a good idea, even if temporarily.
That's a shame, although I do appreciate the huge amount of time and effort that you've put into this release in order to get the Meson build up and running, and so why taking on another large task is going to be lower down the list :)
At the moment OpenBIOS doesn't have a docker image capable of building the required binaries: I did experiment with trying to use the QEMU docker images for openbios-ppc but whilst the binary built successfully, it did not run compared to my hand-rolled compilers. So there's still some debugging to be done there...
ATB,
Mark.
Quoted 23 lines · Volker Rümelin
Quoted 7 lines
Thanks Paolo,
Then only the issue regarding the pcbios/optionrom stuff remains ;-)
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'.
Stop.
Quoted 6 lines
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....Best,
HowardHi Howard,
one solution for this issue is to uncomment the following line in msys2_shell.cmd.
rem To activate windows native symlinks uncomment next line set MSYS=winsymlinks:nativestrict
Quoted 10 lines
Then tell Windows 10 it's okay to create symlinks without elevated rights. Here is a link with a description how to do this.
https://www.joshkel.com/2018/01/18/symlinks-in-windows/I think since commit
bf708f3c4a"optionrom: simplifyMakefile" pc-bios/optionrom/Makefile in your build directory has to be a symbolic link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies theMakefileinstead of creating a symbolic link.
Thanks Volker!
I changed the Windows policy and setting in msys2_shell.cmd. However, I had to edit ming64.ini to uncomment the MSYS=winsymlinks:nativestrict there as well to get things going.
While it is great to have this fix, I can't say I'm really happy with the need to change the Windows policy and to have to fix msys2.
The patches Mark referred to and to which I also pointed earlier did fix this problem without changing Windows/Msys2/Mingw64 settings.
Best regards,
Howard
Quoted 66 lines · Paolo Bonzini
This pull request is the last build system change for 5.2 from me, and it is simple: similar to how we are invoking
ROM or tests/tcg "make" from the mainMakefile, we now invoke ninja to build QEMU. Unlike those cases, however, build.ninja targets are forwarded transparently.The advantages cover various areas:
- maintainability: we drop
scripts/ninjatool.py, which is a large and hairy piece of code, and generally remove one thing that can go wrong and one thing that is unique to QEMU;- platform support: we remove the requirement for GNU make 3.82, which was annoying on Mac. We also avoid bugs on Windows due to meson emitting Windows rather than POSIX escapes (as expected by Ninja) and ninjatool ignoring the difference;
- speed: invoking "configure" does not have to generate 44k lines of rules, while invoking "Make" does not anymore have to parse 44k lines of rules.
- ease of use: Ninja tracks command lines, hence the problem of static library changing the objects they hold goes away
Paolo
Paolo Bonzini (7):
tests/Makefile.include: unbreak non-tcg builds
make: run shell with pipefail
tests: add missing generated sources to testqapi
configure: moveQEMU_INCLUDESto meson
dockerfiles: enable Centos 8 PowerTools
add ninja to dockerfiles, CI configurations and test VMs
build: replace ninjatool with ninja.cirrus.yml | 6 +- .travis.yml | 13 + Makefile | 42 +- configure | 29 +- docs/devel/build-system.rst | 6 +- meson.build | 34 +- scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 -------------------- tests/Makefile.include | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 4 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/include/meson.build | 8 +- tests/meson.build | 14 +- tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 27 files changed, 121 insertions(+), 1070 deletions(-) delete mode 100755 scripts/ninjatool.py
Quoted 4 lines
--
2.26.2
That's good, it's broken my locally rST doc build
``` [226/1564] Generating input-keymap-qnum-to-qcode.c.inc with a custom command (wrapped by meson to capture output) [227/1564] Generating input-keymap-qcode-to-qnum.c.inc with a custom command (wrapped by meson to capture output) [228/1564] Generating shared QAPI source files with a custom command [229/1564] Generating input-keymap-qcode-to-linux.c.inc with a custom command (wrapped by meson to capture output) [230/1564] Generating input-keymap-win32-to-qcode.c.inc with a custom command (wrapped by meson to capture output) [231/1564] Generating QAPI rST doc with a custom command FAILED: tests/qapi-schema/doc-good.txt "/mingw64/bin/sphinx-build" "-Dversion=5.1.50" "-Drelease=" "-W" "-b" "text" "-E" "-c" "C:/work/xemu/qemu/docs" "-D" "master_doc=doc-good" "C:/work/xemu/qemu/tests/qapi-schema" "C:/work/xemu/qemu/build/tests/qapi-schema" CreateProcess failed: The system cannot find the file specified. [232/1564] Generating input-keymap-qcode-to-atset1.c.inc with a custom command (wrapped by meson to capture output) [233/1564] Generating generated-helpers-wrappers.h with a custom command (wrapped by meson to capture output) ninja: fatal: GetOverlappedResult: �����Ч�� make: *** [Makefile:87:run-ninja] 错误 1 /mingw64/bin/ninja -v -j1 tests/check-block-qdict.exe tests/check-qdict.exe tests/check-qjson.exe tests/check-qlist.exe tests/check-qlit.exe tests/check-qnull.exe tests/check-qnum.exe tests/check-qobject.exe tests/check-qom-interface.exe tests/check-qom-proplist.exe tests/check-qstring.exe tests/ptimer-test.exe tests/qapi-schema/doc-good.ref.nocr tests/qapi-schema/doc-good.txt.nocr tests/rcutorture.exe tests/test-aio-multithread.exe tests/test-aio.exe tests/test-authz-list.exe tests/test-authz-listfile.exe tests/test-authz-simple.exe tests/test-base64.exe tests/test-bdrv-drain.exe tests/test-bdrv-graph-mod.exe tests/test-bitcnt.exe tests/test-bitmap.exe tests/test-bitops.exe tests/test-block-backend.exe tests/test-block-iothread.exe tests/test-blockjob-txn.exe tests/test-blockjob.exe tests/test-bufferiszero.exe tests/test-char.exe tests/test-clone-visitor.exe tests/test-coroutine.exe tests/test-crypto-afsplit.exe tests/test-crypto-block.exe tests/test-crypto-cipher.exe tests/test-crypto-hash.exe tests/test-crypto-hmac.exe tests/test-crypto-ivgen.exe tests/test-crypto-pbkdf.exe tests/test-crypto-secret.exe tests/test-cutils.exe tests/test-hbitmap.exe tests/test-int128.exe tests/test-io-channel-buffer.exe tests/test-io-channel-command.exe tests/test-io-channel-file.exe tests/test-io-channel-socket.exe tests/test-io-task.exe tests/test-iov.exe tests/test-keyval.exe tests/test-logging.exe tests/test-mul64.exe tests/test-opts-visitor.exe tests/test-qapi-util.exe tests/test-qdev-global-props.exe tests/test-qdist.exe tests/test-qemu-opts.exe tests/test-qgraph.exe tests/test-qht.exe tests/test-qmp-cmds.exe tests/test-qmp-event.exe tests/test-qobject-input-visitor.exe tests/test-qobject-output-visitor.exe tests/test-rcu-list.exe tests/test-rcu-simpleq.exe tests/test-rcu-slist.exe tests/test-rcu-tailq.exe tests/test-replication.exe tests/test-shift128.exe tests/test-string-input-visitor.exe tests/test-string-output-visitor.exe tests/test-thread-pool.exe tests/test-throttle.exe tests/test-timed-average.exe tests/test-util-sockets.exe tests/test-uuid.exe tests/test-visitor-serialization.exe tests/test-vmstate.exe tests/test-write-threshold.exe tests/test-x86-cpuid.exe tests/test-xbzrle.exe | cat ```
That's because currently sphinx-build are found in configure not in meson, maybe it's worth to move the sphinx related thing into meson totally. --
此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo