tests: add wasmtime testsuite
v2 cover tests: add wasmtime testsuite
Hi,
I made some updates based on the feedback from Alex.
At the moment it mostly works for me on top of 6394578984da: aarch64, riscv64 and s390x are clean, but there are some failures on x86_64. With qemu-user vma rework it unfortunately fails in more places; I haven't analyzed these failures yet.
v1: https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00657.html
https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg02612.html
v1 -> v2:
* Drop gitlab jobs.
* Move to tests/third-party/.
* Use avocado. To be honest, I'm not sure how much value it brings here; I hoped that TAPRunner would at least print the number of passed tests, but it only checks whether at least one test failed).
* Add various check-* Makefile targets.
* Add json -> TAP 14 conversion.
* Add documentation.
* Move test binaries to the host system.
This prevents unnecessary full rebuilds of the Docker image.
* Add riscv64; bump Rust and Wasmtime versions.
* Do not use docker.py; unfortunately this leads to reimplementing some of its features: docker command detection based on $(ENGINE) and injecting the current user into the image.
* Disable core dumps.
Best regards,
Ilya
Ilya Leoshkevich (1):
tests: add wasmtime testsuite
Makefile | 1 + docs/devel/testing.rst | 9 +++ tests/Makefile.include | 6 ++ tests/third-party/Makefile.include | 50 ++++++++++++ tests/third-party/wasmtime/Dockerfile | 32 ++++++++ tests/third-party/wasmtime/Makefile.include | 49 ++++++++++++ tests/third-party/wasmtime/avocado-wrapper | 38 +++++++++ tests/third-party/wasmtime/avocado.cfg | 3 + tests/third-party/wasmtime/json2tap | 77 ++++++++++++++++++ tests/third-party/wasmtime/run-tests-wrapper | 82 ++++++++++++++++++++ tests/third-party/wasmtime/test | 48 ++++++++++++ 11 files changed, 395 insertions(+) create mode 100644 tests/third-party/Makefile.include create mode 100644 tests/third-party/wasmtime/Dockerfile create mode 100644 tests/third-party/wasmtime/Makefile.include create mode 100755 tests/third-party/wasmtime/avocado-wrapper create mode 100644 tests/third-party/wasmtime/avocado.cfg create mode 100755 tests/third-party/wasmtime/json2tap create mode 100755 tests/third-party/wasmtime/run-tests-wrapper create mode 100755 tests/third-party/wasmtime/test
Signature · 1 line
2.38.1
RFC PATCH v2 1/1 tests: add wasmtime testsuite
wasmtime is a WebAssembly runtime, which includes a large testsuite. This testsuite uses qemu-user (aarch64, riscv64, s390x and x86_64 are supported) in order to exercise foreign architectures. Over time it found several regressions in qemu itself, and it would be beneficial to catch the similar ones earlier.
To this end, this patch introduces Makefile targets that run stable wasmtime testsuite against qemu-{aarch64,riscv64,s390x,x86_64}.
--- Makefile | 1 + docs/devel/testing.rst | 9 +++ tests/Makefile.include | 6 ++ tests/third-party/Makefile.include | 50 ++++++++++++ tests/third-party/wasmtime/Dockerfile | 32 ++++++++ tests/third-party/wasmtime/Makefile.include | 49 ++++++++++++ tests/third-party/wasmtime/avocado-wrapper | 38 +++++++++ tests/third-party/wasmtime/avocado.cfg | 3 + tests/third-party/wasmtime/json2tap | 77 ++++++++++++++++++ tests/third-party/wasmtime/run-tests-wrapper | 82 ++++++++++++++++++++ tests/third-party/wasmtime/test | 48 ++++++++++++ 11 files changed, 395 insertions(+) create mode 100644 tests/third-party/Makefile.include create mode 100644 tests/third-party/wasmtime/Dockerfile create mode 100644 tests/third-party/wasmtime/Makefile.include create mode 100755 tests/third-party/wasmtime/avocado-wrapper create mode 100644 tests/third-party/wasmtime/avocado.cfg create mode 100755 tests/third-party/wasmtime/json2tap create mode 100755 tests/third-party/wasmtime/run-tests-wrapper create mode 100755 tests/third-party/wasmtime/test
Makefile +1 -0
@@ -287,6 +287,7 @@ export DESTDIR include $(SRC_PATH)/tests/lcitool/Makefile.include include $(SRC_PATH)/tests/docker/Makefile.include include $(SRC_PATH)/tests/vm/Makefile.include +include $(SRC_PATH)/tests/third-party/Makefile.include print-help-run = printf " %-30s - %s\\n" "$1" "$2" print-help = @$(call print-help-run,$1,$2)
docs/devel/testing.rst +9 -0
@@ -1475,3 +1475,12 @@ coverage-html`` which will create Further analysis can be conducted by running the ``gcov`` command directly on the various .gcda output files. Please read the ``gcov`` documentation for more information. + +Third-party testsuites +~~~~~~~~~~~~~~~~~~~~~~ + +Various third-party projects contain testsuites that exercise non-trivial +machine instructions or Linux kernel interfaces. While they may be too big for +the gating CI, developers can use the corresponding make targets to verify +their changes. The list of these targets can be found in the "Third-party +testing targets" section of ``make check-help`` output.
tests/Makefile.include +6 -0
@@ -22,6 +22,12 @@ endif @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests and related data" @echo + @echo "Third-party testing targets:" + @echo " $(MAKE) check-third-party Run third-party tests for all supported targets" + @echo " $(MAKE) check-third-party-TARGET Run third-party tests for given target" + @echo " $(MAKE) check-wasmtime Run wasmtime tests for all supported targets" + @echo " $(MAKE) check-wasmtime-TARGET Run wasmtime tests for given target" + @echo @echo "The following are useful for CI builds" @echo " $(MAKE) check-build Build most test binaries" @echo " $(MAKE) get-vm-images Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
tests/third-party/Makefile.include new file +50 -0
50 lines not on this page · raw message
tests/third-party/wasmtime/Dockerfile new file +32 -0
@@ -0,0 +1,32 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +# Image for building and running the wasmtime testsuite. + +ARG deb_arch +FROM registry.gitlab.com/qemu-project/qemu/qemu/debian-${deb_arch}-cross:latest + +RUN export DEBIAN_FRONTEND=noninteractive && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + curl && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y + +ARG group_id +ARG group_name +RUN if ! getent group ${group_id} >/dev/null 2>&1; then \ + groupdel ${group_name} >/dev/null 2>&1 ; \ + groupadd --gid=${group_id} ${group_name} ; \ + fi + +ARG user_id +ARG user_name +RUN if ! getent user ${user_id} >/dev/null 2>&1; then \ + userdel ${user_name} >/dev/null 2>&1 ; \ + useradd \ + --create-home \ + --gid=${group_id} \ + --uid=${user_id} \ + ${user_name} ; \ + fi
tests/third-party/wasmtime/Makefile.include new file +49 -0
49 lines not on this page · raw message
tests/third-party/wasmtime/avocado-wrapper new file +38 -0
@@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later + +# Run a single command with avocado. + +set -e -u -o pipefail + +# Script arguments. +if [ "$#" -lt 2 ]; then + cat <<HERE >&2 + +Usage: + + $0 TESTS_VENV_DIR COMMAND [ARGS... ] + +where TESTS_VENV_DIR should point to an existing virtualenv with avocado. + +HERE + exit 1 +fi +venv=$1 +shift + +# Avocado can run only individual binaries (no arguments, pipes, etc). +# So create a temporary trampoline script. +trampoline=$(mktemp ./trampoline.XXXXXXXXXX) +trap 'rm "$trampoline"' EXIT +basedir=$(cd "$(dirname "$0")" && pwd) +cat >"$trampoline" <<HERE +#!/bin/bash +set -e -u -o pipefail +${@@Q} | ${basedir@Q}/json2tap +HERE +chmod a+x "$trampoline" + +# Enter the virtualenv and run the trampoline script. +. "$venv"/bin/activate +avocado --config="$basedir/avocado.cfg" run "$trampoline"
tests/third-party/wasmtime/avocado.cfg new file +3 -0
@@ -0,0 +1,3 @@ +[plugins] +# So that avocado.plugins.resolvers:TapResolver is used instead. +disable = ['resolver.exec-test']
tests/third-party/wasmtime/json2tap new file +77 -0
77 lines not on this page · raw message
tests/third-party/wasmtime/run-tests-wrapper new file +82 -0
82 lines not on this page · raw message
tests/third-party/wasmtime/test new file +48 -0
48 lines not on this page · raw message
Signature · 1 line
2.38.1
PING: [PATCH RFC v2 0/1] tests: add wasmtime testsuite
In reply to Ilya LeoshkevichQuoted 51 lines · Ilya Leoshkevich
Hi,
I made some updates based on the feedback from Alex.
At the moment it mostly works for me on top of
6394578984da: aarch64, riscv64 and s390x are clean, but there are some failures onx86_64. With qemu-user vma rework it unfortunately fails in more places; I haven't analyzed these failures yet.v1:
https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00657.html
https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg02612.htmlv1 -> v2:
* Drop gitlab jobs.
* Move to tests/third-party/.
* Use avocado. To be honest, I'm not sure how much value it brings here; I hoped that TAPRunner would at least print the number of passed tests, but it only checks whether at least one test failed).
* Add various check-*Makefiletargets.
* Add json -> TAP 14 conversion.
* Add documentation.
* Move test binaries to the host system.
This prevents unnecessary full rebuilds of the Docker image.
* Add riscv64; bump Rust and Wasmtime versions.
* Do not usedocker.py; unfortunately this leads to reimplementing some
of its features: docker command detection based on $(ENGINE) and injecting the current user into the image.
* Disable core dumps.Best regards,
IlyaIlya Leoshkevich (1):
tests: add wasmtime testsuiteMakefile | 1 + docs/devel/testing.rst | 9 +++ tests/Makefile.include | 6 ++ tests/third-party/Makefile.include | 50 ++++++++++++ tests/third-party/wasmtime/Dockerfile | 32 ++++++++ tests/third-party/wasmtime/Makefile.include | 49 ++++++++++++ tests/third-party/wasmtime/avocado-wrapper | 38 +++++++++ tests/third-party/wasmtime/avocado.cfg | 3 + tests/third-party/wasmtime/json2tap | 77 ++++++++++++++++++ tests/third-party/wasmtime/run-tests-wrapper | 82
++++++++++++++++++++
tests/third-party/wasmtime/test | 48 ++++++++++++11 files changed, 395 insertions(+)
Quoted 9 lines
create mode 100644 tests/third-party/Makefile.include create mode 100644 tests/third-party/wasmtime/Dockerfile create mode 100644 tests/third-party/wasmtime/Makefile.include create mode 100755 tests/third-party/wasmtime/avocado-wrapper create mode 100644 tests/third-party/wasmtime/avocado.cfg create mode 100755 tests/third-party/wasmtime/json2tap create mode 100755 tests/third-party/wasmtime/run-tests-wrapper create mode 100755 tests/third-party/wasmtime/test
Hi,
I would like to ping this patch
(https://patchew.org/QEMU/[email protected]/)
Best regards,
Ilya