Skip to contentSkip to conversation

tests: add wasmtime testsuite

RFC PATCH In review Ilya Leoshkevich 1 participant 3 messages Dec 21, 2022 – Feb 13, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

v2 cover tests: add wasmtime testsuite

Hi,

I made some updates based on the feedback from Alex.

At the moment it mostly works for me on top of 6394578984da: aarch64, riscv64 and s390x are clean, but there are some failures on x86_64. With qemu-user vma rework it unfortunately fails in more places; I haven't analyzed these failures yet.

v1: https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00657.html

    https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg02612.html

v1 -> v2:
* Drop gitlab jobs.
* Move to tests/third-party/.
* Use avocado. To be honest, I'm not sure how much value it brings here; I hoped that TAPRunner would at least print the number of passed tests, but it only checks whether at least one test failed).
* Add various check-* Makefile targets.
* Add json -> TAP 14 conversion.
* Add documentation.
* Move test binaries to the host system.
This prevents unnecessary full rebuilds of the Docker image.
* Add riscv64; bump Rust and Wasmtime versions.
* Do not use docker.py; unfortunately this leads to reimplementing some of its features: docker command detection based on $(ENGINE) and injecting the current user into the image.
* Disable core dumps.

Best regards,
Ilya

Ilya Leoshkevich (1):
tests: add wasmtime testsuite

 Makefile                                     |  1 +
 docs/devel/testing.rst                       |  9 +++
 tests/Makefile.include                       |  6 ++
 tests/third-party/Makefile.include           | 50 ++++++++++++
 tests/third-party/wasmtime/Dockerfile        | 32 ++++++++
 tests/third-party/wasmtime/Makefile.include  | 49 ++++++++++++
 tests/third-party/wasmtime/avocado-wrapper   | 38 +++++++++
 tests/third-party/wasmtime/avocado.cfg       |  3 +
 tests/third-party/wasmtime/json2tap          | 77 ++++++++++++++++++
 tests/third-party/wasmtime/run-tests-wrapper | 82 ++++++++++++++++++++
 tests/third-party/wasmtime/test              | 48 ++++++++++++
 11 files changed, 395 insertions(+)
 create mode 100644 tests/third-party/Makefile.include
 create mode 100644 tests/third-party/wasmtime/Dockerfile
 create mode 100644 tests/third-party/wasmtime/Makefile.include
 create mode 100755 tests/third-party/wasmtime/avocado-wrapper
 create mode 100644 tests/third-party/wasmtime/avocado.cfg
 create mode 100755 tests/third-party/wasmtime/json2tap
 create mode 100755 tests/third-party/wasmtime/run-tests-wrapper
 create mode 100755 tests/third-party/wasmtime/test
Signature · 1 line
2.38.1
#1 lore raw

RFC PATCH v2 1/1 tests: add wasmtime testsuite

wasmtime is a WebAssembly runtime, which includes a large testsuite. This testsuite uses qemu-user (aarch64, riscv64, s390x and x86_64 are supported) in order to exercise foreign architectures. Over time it found several regressions in qemu itself, and it would be beneficial to catch the similar ones earlier.

To this end, this patch introduces Makefile targets that run stable wasmtime testsuite against qemu-{aarch64,riscv64,s390x,x86_64}.

---
 Makefile                                     |  1 +
 docs/devel/testing.rst                       |  9 +++
 tests/Makefile.include                       |  6 ++
 tests/third-party/Makefile.include           | 50 ++++++++++++
 tests/third-party/wasmtime/Dockerfile        | 32 ++++++++
 tests/third-party/wasmtime/Makefile.include  | 49 ++++++++++++
 tests/third-party/wasmtime/avocado-wrapper   | 38 +++++++++
 tests/third-party/wasmtime/avocado.cfg       |  3 +
 tests/third-party/wasmtime/json2tap          | 77 ++++++++++++++++++
 tests/third-party/wasmtime/run-tests-wrapper | 82 ++++++++++++++++++++
 tests/third-party/wasmtime/test              | 48 ++++++++++++
 11 files changed, 395 insertions(+)
 create mode 100644 tests/third-party/Makefile.include
 create mode 100644 tests/third-party/wasmtime/Dockerfile
 create mode 100644 tests/third-party/wasmtime/Makefile.include
 create mode 100755 tests/third-party/wasmtime/avocado-wrapper
 create mode 100644 tests/third-party/wasmtime/avocado.cfg
 create mode 100755 tests/third-party/wasmtime/json2tap
 create mode 100755 tests/third-party/wasmtime/run-tests-wrapper
 create mode 100755 tests/third-party/wasmtime/test
Makefile +1 -0
index a48103cc8a1..3a5250d3e46 100644
@@ -287,6 +287,7 @@ export DESTDIR
 include $(SRC_PATH)/tests/lcitool/Makefile.include
 include $(SRC_PATH)/tests/docker/Makefile.include
 include $(SRC_PATH)/tests/vm/Makefile.include
+include $(SRC_PATH)/tests/third-party/Makefile.include
 
 print-help-run = printf "  %-30s - %s\\n" "$1" "$2"
 print-help = @$(call print-help-run,$1,$2)
docs/devel/testing.rst +9 -0
index e10c47b5a7c..0cbbcae4bcd 100644
@@ -1475,3 +1475,12 @@ coverage-html`` which will create
 Further analysis can be conducted by running the ``gcov`` command
 directly on the various .gcda output files. Please read the ``gcov``
 documentation for more information.
+
+Third-party testsuites
+~~~~~~~~~~~~~~~~~~~~~~
+
+Various third-party projects contain testsuites that exercise non-trivial
+machine instructions or Linux kernel interfaces. While they may be too big for
+the gating CI, developers can use the corresponding make targets to verify
+their changes. The list of these targets can be found in the "Third-party
+testing targets" section of ``make check-help`` output.
tests/Makefile.include +6 -0
index 9422ddaece5..506fe5ab16a 100644
@@ -22,6 +22,12 @@ endif
 	@echo " $(MAKE) check-venv             Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean            Clean the tests and related data"
 	@echo
+	@echo "Third-party testing targets:"
+	@echo " $(MAKE) check-third-party         Run third-party tests for all supported targets"
+	@echo " $(MAKE) check-third-party-TARGET  Run third-party tests for given target"
+	@echo " $(MAKE) check-wasmtime            Run wasmtime tests for all supported targets"
+	@echo " $(MAKE) check-wasmtime-TARGET     Run wasmtime tests for given target"
+	@echo
 	@echo "The following are useful for CI builds"
 	@echo " $(MAKE) check-build            Build most test binaries"
 	@echo " $(MAKE) get-vm-images          Downloads all images used by avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
tests/third-party/Makefile.include new file +50 -0

50 lines not on this page · raw message

tests/third-party/wasmtime/Dockerfile new file +32 -0
new file mode 100644
index 00000000000..d1a20e3a2ae
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+# Image for building and running the wasmtime testsuite.
+
+ARG deb_arch
+FROM registry.gitlab.com/qemu-project/qemu/qemu/debian-${deb_arch}-cross:latest
+
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+                      curl && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y
+
+ARG group_id
+ARG group_name
+RUN if ! getent group ${group_id} >/dev/null 2>&1; then \
+        groupdel ${group_name} >/dev/null 2>&1 ; \
+        groupadd --gid=${group_id} ${group_name} ; \
+    fi
+
+ARG user_id
+ARG user_name
+RUN if ! getent user ${user_id} >/dev/null 2>&1; then \
+        userdel ${user_name} >/dev/null 2>&1 ; \
+        useradd \
+            --create-home \
+            --gid=${group_id} \
+            --uid=${user_id} \
+            ${user_name} ; \
+    fi
tests/third-party/wasmtime/Makefile.include new file +49 -0

49 lines not on this page · raw message

tests/third-party/wasmtime/avocado-wrapper new file +38 -0
new file mode 100755
index 00000000000..3d0f7388428
@@ -0,0 +1,38 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+# Run a single command with avocado.
+
+set -e -u -o pipefail
+
+# Script arguments.
+if [ "$#" -lt 2 ]; then
+    cat <<HERE >&2
+
+Usage:
+
+    $0 TESTS_VENV_DIR COMMAND [ARGS... ]
+
+where TESTS_VENV_DIR should point to an existing virtualenv with avocado.
+
+HERE
+    exit 1
+fi
+venv=$1
+shift
+
+# Avocado can run only individual binaries (no arguments, pipes, etc).
+# So create a temporary trampoline script.
+trampoline=$(mktemp ./trampoline.XXXXXXXXXX)
+trap 'rm "$trampoline"' EXIT
+basedir=$(cd "$(dirname "$0")" && pwd)
+cat >"$trampoline" <<HERE
+#!/bin/bash
+set -e -u -o pipefail
+${@@Q} | ${basedir@Q}/json2tap
+HERE
+chmod a+x "$trampoline"
+
+# Enter the virtualenv and run the trampoline script.
+. "$venv"/bin/activate
+avocado --config="$basedir/avocado.cfg" run "$trampoline"
tests/third-party/wasmtime/avocado.cfg new file +3 -0
new file mode 100644
index 00000000000..2c08150c9a3
@@ -0,0 +1,3 @@
+[plugins]
+# So that avocado.plugins.resolvers:TapResolver is used instead.
+disable = ['resolver.exec-test']
tests/third-party/wasmtime/json2tap new file +77 -0

77 lines not on this page · raw message

tests/third-party/wasmtime/run-tests-wrapper new file +82 -0

82 lines not on this page · raw message

tests/third-party/wasmtime/test new file +48 -0

48 lines not on this page · raw message

Signature · 1 line
2.38.1
#2 lore raw

PING: [PATCH RFC v2 0/1] tests: add wasmtime testsuite

In reply to Ilya Leoshkevich
Quoted 51 lines · Ilya Leoshkevich

Hi,

I made some updates based on the feedback from Alex.

At the moment it mostly works for me on top of 6394578984da: aarch64, riscv64 and s390x are clean, but there are some failures on x86_64. With qemu-user vma rework it unfortunately fails in more places; I haven't analyzed these failures yet.

v1:
https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00657.html

https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg02612.html

v1 -> v2:
* Drop gitlab jobs.
* Move to tests/third-party/.
* Use avocado. To be honest, I'm not sure how much value it brings here; I hoped that TAPRunner would at least print the number of passed tests, but it only checks whether at least one test failed).
* Add various check-* Makefile targets.
* Add json -> TAP 14 conversion.
* Add documentation.
* Move test binaries to the host system.
  This prevents unnecessary full rebuilds of the Docker image.
* Add riscv64; bump Rust and Wasmtime versions.
* Do not use docker.py; unfortunately this leads to reimplementing some
  of its features: docker command detection based on $(ENGINE) and injecting the current user into the image.
* Disable core dumps.

Best regards,
Ilya

Ilya Leoshkevich (1):
  tests: add wasmtime testsuite

 Makefile                                     |  1 + docs/devel/testing.rst                       |  9 +++ tests/Makefile.include                       |  6 ++ tests/third-party/Makefile.include           | 50 ++++++++++++ tests/third-party/wasmtime/Dockerfile        | 32 ++++++++ tests/third-party/wasmtime/Makefile.include  | 49 ++++++++++++ tests/third-party/wasmtime/avocado-wrapper   | 38 +++++++++ tests/third-party/wasmtime/avocado.cfg       |  3 + tests/third-party/wasmtime/json2tap          | 77 ++++++++++++++++++ tests/third-party/wasmtime/run-tests-wrapper | 82
++++++++++++++++++++
 tests/third-party/wasmtime/test              | 48 ++++++++++++

 11 files changed, 395 insertions(+)
Quoted 9 lines

 create mode 100644 tests/third-party/Makefile.include create mode 100644 tests/third-party/wasmtime/Dockerfile create mode 100644 tests/third-party/wasmtime/Makefile.include create mode 100755 tests/third-party/wasmtime/avocado-wrapper create mode 100644 tests/third-party/wasmtime/avocado.cfg create mode 100755 tests/third-party/wasmtime/json2tap create mode 100755 tests/third-party/wasmtime/run-tests-wrapper create mode 100755 tests/third-party/wasmtime/test

Hi,

I would like to ping this patch
(https://patchew.org/QEMU/[email protected]/)

Best regards,
Ilya