Skip to contentSkip to conversation

Convert remaining Avocado tests to functional

PATCH In review Thomas Huth 2 participants 17 messages Apr 14, 2025 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

v2 cover Convert remaining Avocado tests to functional

These patches convert the remaining Avocado tests bit by bit to the functional framework, or replace them with a test that has similar test coverage. This way we are finally able to get rid of the test/avocado folder in the end, everything is then integrated with the meson test runner. It should be possible now to run almost all tests in parallel by doing something like:

make -j$(nproc) check SPEED=thorough

v2:
- Use separate gitlab issue tickets for the flaky reverse-debugging tests
- Mark the aarch64 reverse debugging test as flaky, too
- Rename QEMU_CI_FUNCTIONAL_TESTING to QEMU_CI_FUNCTIONAL

Thomas Huth (15):
gitlab-ci: Remove the avocado tests from the CI pipelines tests/functional: Move the check for the parameters from avocado to
functional
tests/functional: Convert reverse_debugging tests to the functional
framework
tests/functional: Convert the i386 replay avocado test tests/avocado: Remove the LinuxKernelTest class tests/functional: Convert the 32-bit big endian Wheezy mips test tests/functional: Convert the 32-bit little endian Wheezy mips test tests/functional: Convert the 64-bit little endian Wheezy mips test tests/functional: Convert the 64-bit big endian Wheezy mips test tests/avocado: Remove the boot_linux.py tests tests/functional: Use the tuxrun kernel for the x86 replay test tests/functional: Use the tuxrun kernel for the aarch64 replay test tests/functional: Convert the SMMU test to the functional framework
gitlab-ci: Update QEMU_JOB_AVOCADO and QEMU_CI_AVOCADO_TESTING Remove the remainders of the Avocado tests

 MAINTAINERS                                   |  15 +-
 docs/about/build-platforms.rst                |  10 +-
 docs/devel/build-system.rst                   |  11 +-
 docs/devel/codebase.rst                       |   5 -
 docs/devel/testing/avocado.rst                | 581 ------------------
 docs/devel/testing/ci-jobs.rst.inc            |  19 +-
 docs/devel/testing/functional.rst             |   3 -
 docs/devel/testing/index.rst                  |   1 -
 docs/devel/testing/main.rst                   |  15 -
 tests/avocado/README.rst                      |  10 -
 configure                                     |   2 +-
 .gitlab-ci.d/base.yml                         |   8 +-
 .gitlab-ci.d/buildtest-template.yml           |  13 +-
 .gitlab-ci.d/buildtest.yml                    |  33 +-
 pythondeps.toml                               |   8 +-
 tests/Makefile.include                        |  60 +-
 tests/avocado/avocado_qemu/__init__.py        | 424 -------------
 tests/avocado/avocado_qemu/linuxtest.py       | 253 --------
 tests/avocado/boot_linux.py                   | 132 ----
 tests/avocado/boot_linux_console.py           |  96 ---
 tests/avocado/linux_ssh_mips_malta.py         | 205 ------
 tests/avocado/replay_kernel.py                | 110 ----
 tests/avocado/replay_linux.py                 | 206 -------
 tests/avocado/smmu.py                         | 139 -----
 tests/functional/meson.build                  |  13 +-
 tests/functional/qemu_test/tuxruntest.py      |   9 +-
 .../reverse_debugging.py                      | 114 +---
 tests/functional/test_aarch64_replay.py       |  37 +-
 .../functional/test_aarch64_reverse_debug.py  |  38 ++
 tests/functional/test_aarch64_smmu.py         | 205 ++++++
 tests/functional/test_i386_replay.py          |  28 +
 tests/functional/test_mips64_malta.py         |  35 ++
 tests/functional/test_mips64el_malta.py       |  22 +
 tests/functional/test_mips_malta.py           | 108 +++-
 tests/functional/test_mipsel_malta.py         |  22 +
 tests/functional/test_ppc64_reverse_debug.py  |  41 ++
 tests/functional/test_x86_64_replay.py        |  43 +-
 tests/functional/test_x86_64_reverse_debug.py |  36 ++
 38 files changed, 673 insertions(+), 2437 deletions(-)
 delete mode 100644 docs/devel/testing/avocado.rst
 delete mode 100644 tests/avocado/README.rst
 delete mode 100644 tests/avocado/avocado_qemu/__init__.py
 delete mode 100644 tests/avocado/avocado_qemu/linuxtest.py
 delete mode 100644 tests/avocado/boot_linux.py
 delete mode 100644 tests/avocado/boot_linux_console.py
 delete mode 100644 tests/avocado/linux_ssh_mips_malta.py
 delete mode 100644 tests/avocado/replay_kernel.py
 delete mode 100644 tests/avocado/replay_linux.py
 delete mode 100644 tests/avocado/smmu.py
 rename tests/{avocado => functional}/reverse_debugging.py (66%)
 create mode 100755 tests/functional/test_aarch64_reverse_debug.py
 create mode 100755 tests/functional/test_aarch64_smmu.py
 create mode 100755 tests/functional/test_i386_replay.py
 create mode 100755 tests/functional/test_mips64_malta.py
 create mode 100755 tests/functional/test_ppc64_reverse_debug.py
 create mode 100755 tests/functional/test_x86_64_reverse_debug.py
Signature · 1 line
2.49.0
#1 lore raw

PATCH v2 1/15 gitlab-ci: Remove the avocado tests from the CI pipelines

From: Thomas Huth <[email protected]>

We are going to move the remaining Avocado tests step by step into the functional test framework. Unfortunately, Avocado fails with an error if it cannot determine a test to run, so disable the tests here now to avoid failures in the Gitlab-CI during the next steps.

---
 .gitlab-ci.d/buildtest-template.yml | 11 ----------
 .gitlab-ci.d/buildtest.yml          | 33 +++++++++++------------------
 2 files changed, 12 insertions(+), 32 deletions(-)
.gitlab-ci.d/buildtest-template.yml +0 -11
index 39da7698b09..13fa4f4a4fc 100644
@@ -95,7 +95,6 @@
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
-      - ${CI_PROJECT_DIR}/avocado-cache
       - ${CI_PROJECT_DIR}/functional-cache
     policy: pull-push
   artifacts:
@@ -109,16 +108,6 @@
     reports:
       junit: build/tests/results/latest/results.xml
   before_script:
-    - mkdir -p ~/.config/avocado
-    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
-    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
-           >> ~/.config/avocado/avocado.conf
-    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
-           >> ~/.config/avocado/avocado.conf
-    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
-        du -chs ${CI_PROJECT_DIR}/*-cache ;
-      fi
-    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
     - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
     - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache
   after_script:
.gitlab-ci.d/buildtest.yml +12 -21
index 00f4bfcd9f3..431bc07d8fb 100644
@@ -29,8 +29,7 @@ functional-system-alpine:
       artifacts: true
   variables:
     IMAGE: alpine
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:avr arch:loongarch64 arch:mips64 arch:mipsel
+    MAKE_CHECK_ARGS: check-functional
 
 build-system-ubuntu:
   extends:
@@ -60,8 +59,7 @@ functional-system-ubuntu:
       artifacts: true
   variables:
     IMAGE: ubuntu2204
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:alpha arch:microblazeel arch:mips64el
+    MAKE_CHECK_ARGS: check-functional
 
 build-system-debian:
   extends:
@@ -92,8 +90,7 @@ functional-system-debian:
       artifacts: true
   variables:
     IMAGE: debian
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:arm arch:i386 arch:riscv64 arch:sh4 arch:sparc arch:xtensa
+    MAKE_CHECK_ARGS: check-functional
 
 crash-test-debian:
   extends: .native_test_job_template
@@ -155,9 +152,7 @@ functional-system-fedora:
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:microblaze arch:mips arch:xtensa arch:m68k
-      arch:riscv32 arch:ppc arch:sparc64
+    MAKE_CHECK_ARGS: check-functional
 
 crash-test-fedora:
   extends: .native_test_job_template
@@ -278,9 +273,7 @@ functional-system-centos:
       artifacts: true
   variables:
     IMAGE: centos9
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:ppc64 arch:or1k arch:s390x arch:x86_64 arch:rx
-      arch:sh4
+    MAKE_CHECK_ARGS: check-functional
 
 build-system-opensuse:
   extends:
@@ -309,8 +302,7 @@ functional-system-opensuse:
       artifacts: true
   variables:
     IMAGE: opensuse-leap
-    MAKE_CHECK_ARGS: check-avocado check-functional
-    AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64
+    MAKE_CHECK_ARGS: check-functional
 
 #
 # Flaky tests. We don't run these by default and they are allow fail
@@ -338,10 +330,9 @@ functional-system-flaky:
   allow_failure: true
   variables:
     IMAGE: debian
-    MAKE_CHECK_ARGS: check-avocado check-functional
+    MAKE_CHECK_ARGS: check-functional
     QEMU_JOB_OPTIONAL: 1
     QEMU_TEST_FLAKY_TESTS: 1
-    AVOCADO_TAGS: flaky
 
 # This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
 # the configure script. The container doesn't contain Xen headers so
@@ -482,8 +473,8 @@ clang-user:
 # Since slirp callbacks are used in QEMU Timers, we cannot use libslirp with
 # CFI builds, and thus have to disable it here.
 #
-# Split in three sets of build/check/avocado to limit the execution time of each
-# job
+# Split in three sets of build/check/functional to limit the execution time
+# of each job
 build-cfi-aarch64:
   extends:
     - .native_build_job_template
@@ -520,7 +511,7 @@ functional-cfi-aarch64:
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-avocado check-functional
+    MAKE_CHECK_ARGS: check-functional
 
 build-cfi-ppc64-s390x:
   extends:
@@ -558,7 +549,7 @@ functional-cfi-ppc64-s390x:
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-avocado check-functional
+    MAKE_CHECK_ARGS: check-functional
 
 build-cfi-x86_64:
   extends:
@@ -592,7 +583,7 @@ functional-cfi-x86_64:
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check-avocado check-functional
+    MAKE_CHECK_ARGS: check-functional
 
 tsan-build:
   extends: .native_build_job_template
Signature · 1 line
2.49.0
#2 lore raw

PATCH v2 2/15 tests/functional: Move the check for the parameters from avocado to functional

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

test_x86_64_pc in tests/avocado/boot_linux_console.py only checks whether the kernel parameters have correctly been passed to the kernel in the guest by looking for them in the console output of the guest. Let's move that to the functional test framework now, but instead of doing it in a separate test, let's do it for all tuxrun tests instead, so it is done automatically for all targets that have a tuxrun test.

---
 tests/avocado/boot_linux_console.py      | 34 ------------------------
 tests/functional/qemu_test/tuxruntest.py |  9 ++++---
 2 files changed, 5 insertions(+), 38 deletions(-)
tests/avocado/boot_linux_console.py +0 -34
index c15f39ae1f3..cbb1e2fb506 100644
@@ -9,17 +9,9 @@
 # later.  See the COPYING file in the top-level directory.
 
 import os
-import lzma
-import gzip
 import shutil
 
-from avocado import skip
-from avocado import skipUnless
-from avocado import skipUnless
 from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import interrupt_interactive_console_until_pattern
 from avocado_qemu import wait_for_console_pattern
 from avocado.utils import process
 from avocado.utils import archive
@@ -68,29 +60,3 @@ def extract_from_rpm(self, rpm, path):
         process.run("rpm2cpio %s | cpio -id %s" % (rpm, path), shell=True)
         os.chdir(cwd)
         return os.path.normpath(os.path.join(self.workdir, path))
-
-class BootLinuxConsole(LinuxKernelTest):
-    """
-    Boots a Linux kernel and checks that the console is operational and the
-    kernel command line is properly passed from QEMU to the kernel
-    """
-    timeout = 90
-
-    def test_x86_64_pc(self):
-        """
-        :avocado: tags=arch:x86_64
-        :avocado: tags=machine:pc
-        """
-        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
-                      '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
-                      '/vmlinuz')
-        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
-        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
-
-        self.vm.set_console()
-        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
-        self.vm.add_args('-kernel', kernel_path,
-                         '-append', kernel_command_line)
-        self.vm.launch()
-        console_pattern = 'Kernel command line: %s' % kernel_command_line
-        self.wait_for_console_pattern(console_pattern)
tests/functional/qemu_test/tuxruntest.py +5 -4
index ad74156f9c5..c2bd5baaae9 100644
@@ -77,12 +77,12 @@ def prepare_run(self, kernel, disk, drive, dtb=None, console_index=0):
         blockdev = "driver=raw,file.driver=file," \
             + f"file.filename={disk},node-name=hd0"
 
-        kcmd_line = self.KERNEL_COMMON_COMMAND_LINE
-        kcmd_line += f" root=/dev/{self.root}"
-        kcmd_line += f" console={self.console}"
+        self.kcmd_line = self.KERNEL_COMMON_COMMAND_LINE
+        self.kcmd_line += f" root=/dev/{self.root}"
+        self.kcmd_line += f" console={self.console}"
 
         self.vm.add_args('-kernel', kernel,
-                         '-append', kcmd_line,
+                         '-append', self.kcmd_line,
                          '-blockdev', blockdev)
 
         # Sometimes we need extra devices attached
@@ -103,6 +103,7 @@ def run_tuxtest_tests(self, haltmsg):
         wait to exit cleanly.
         """
         ps1='root@tuxtest:~#'
+        self.wait_for_console_pattern(self.kcmd_line)
         self.wait_for_console_pattern('tuxtest login:')
         exec_command_and_wait_for_pattern(self, 'root', ps1)
         exec_command_and_wait_for_pattern(self, 'cat /proc/interrupts', ps1)
Signature · 1 line
2.49.0
#3 lore raw

PATCH v2 3/15 tests/functional: Convert reverse_debugging tests to the functional framework

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

These tests are using the gdb-related library functions from the Avocado framework which we don't have in the functional framework yet. So for the time being, keep those imports and skip the test if the Avocado framework is not installed on the host.

---
 MAINTAINERS                                   |   2 +-
 tests/functional/meson.build                  |   4 +
 .../reverse_debugging.py                      | 114 +++---------------
 .../functional/test_aarch64_reverse_debug.py  |  38 ++++++
 tests/functional/test_ppc64_reverse_debug.py  |  41 +++++++
 tests/functional/test_x86_64_reverse_debug.py |  36 ++++++
 6 files changed, 139 insertions(+), 96 deletions(-)
 rename tests/{avocado => functional}/reverse_debugging.py (66%)
 create mode 100755 tests/functional/test_aarch64_reverse_debug.py
 create mode 100755 tests/functional/test_ppc64_reverse_debug.py
 create mode 100755 tests/functional/test_x86_64_reverse_debug.py
MAINTAINERS +1 -1
index d54b5578f88..af1d847f8ed 100644
@@ -3671,7 +3671,7 @@ F: docs/system/replay.rst
 F: stubs/replay.c
 F: tests/avocado/replay_kernel.py
 F: tests/avocado/replay_linux.py
-F: tests/avocado/reverse_debugging.py
+F: tests/functional/*reverse_debug*.py
 F: tests/functional/*replay*.py
 F: qapi/replay.json
 
tests/functional/meson.build +4 -0
index 0f8be30fe27..7b0f4ab0b1d 100644
@@ -13,6 +13,7 @@ endif
 test_timeouts = {
   'aarch64_aspeed' : 600,
   'aarch64_raspi4' : 480,
+  'aarch64_reverse_debug' : 180,
   'aarch64_rme_virt' : 1200,
   'aarch64_rme_sbsaref' : 1200,
   'aarch64_sbsaref_alpine' : 1200,
@@ -78,6 +79,7 @@ tests_aarch64_system_thorough = [
   'aarch64_raspi3',
   'aarch64_raspi4',
   'aarch64_replay',
+  'aarch64_reverse_debug',
   'aarch64_rme_virt',
   'aarch64_rme_sbsaref',
   'aarch64_sbsaref',
@@ -229,6 +231,7 @@ tests_ppc64_system_thorough = [
   'ppc64_powernv',
   'ppc64_pseries',
   'ppc64_replay',
+  'ppc64_reverse_debug',
   'ppc64_tuxrun',
   'ppc64_mac99',
 ]
@@ -311,6 +314,7 @@ tests_x86_64_system_thorough = [
   'x86_64_hotplug_cpu',
   'x86_64_kvm_xen',
   'x86_64_replay',
+  'x86_64_reverse_debug',
   'x86_64_tuxrun',
 ]
 
tests/functional/reverse_debugging.py renamed from tests/avocado/reverse_debugging.py +19 -95

158 lines not on this page · raw message

tests/functional/test_aarch64_reverse_debug.py new file +38 -0
new file mode 100755
index 00000000000..58d45328350
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Reverse debugging test
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from qemu_test import Asset, skipIfMissingImports, skipFlakyTest
+from reverse_debugging import ReverseDebugging
+
+
+@skipIfMissingImports('avocado.utils')
+class ReverseDebugging_AArch64(ReverseDebugging):
+
+    REG_PC = 32
+
+    KERNEL_ASSET = Asset(
+        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
+         'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
+        '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
+
+    @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2921")
+    def test_aarch64_virt(self):
+        self.set_machine('virt')
+        self.cpu = 'cortex-a53'
+        kernel_path = self.KERNEL_ASSET.fetch()
+        self.reverse_debugging(args=('-kernel', kernel_path))
+
+
+if __name__ == '__main__':
+    ReverseDebugging.main()
tests/functional/test_ppc64_reverse_debug.py new file +41 -0

41 lines not on this page · raw message

tests/functional/test_x86_64_reverse_debug.py new file +36 -0
new file mode 100755
index 00000000000..d713e91e144
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Reverse debugging test
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from qemu_test import skipIfMissingImports, skipFlakyTest
+from reverse_debugging import ReverseDebugging
+
+
+@skipIfMissingImports('avocado.utils')
+class ReverseDebugging_X86_64(ReverseDebugging):
+
+    REG_PC = 0x10
+    REG_CS = 0x12
+    def get_pc(self, g):
+        return self.get_reg_le(g, self.REG_PC) \
+            + self.get_reg_le(g, self.REG_CS) * 0x10
+
+    @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2922")
+    def test_x86_64_pc(self):
+        self.set_machine('pc')
+        # start with BIOS only
+        self.reverse_debugging()
+
+
+if __name__ == '__main__':
+    ReverseDebugging.main()
Signature · 1 line
2.49.0
Reviewed-by on v2 3/15
#4 lore raw

Re: [PATCH v2 03/15] tests/functional: Convert reverse_debugging tests to the functional framework

Quoted 8 lines · Thomas Huth

From: Thomas Huth <[email protected]>

These tests are using the gdb-related library functions from the Avocado framework which we don't have in the functional framework yet. So for the time being, keep those imports and skip the test if the Avocado framework is not installed on the host.

Quoted 12 lines
---
 MAINTAINERS                                   |   2 +-
 tests/functional/meson.build                  |   4 +
 .../reverse_debugging.py                      | 114 +++---------------
 .../functional/test_aarch64_reverse_debug.py  |  38 ++++++
 tests/functional/test_ppc64_reverse_debug.py  |  41 +++++++
 tests/functional/test_x86_64_reverse_debug.py |  36 ++++++
 6 files changed, 139 insertions(+), 96 deletions(-)
 rename tests/{avocado => functional}/reverse_debugging.py (66%)
 create mode 100755 tests/functional/test_aarch64_reverse_debug.py
 create mode 100755 tests/functional/test_ppc64_reverse_debug.py
 create mode 100755 tests/functional/test_x86_64_reverse_debug.py

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw

PATCH v2 4/15 tests/functional: Convert the i386 replay avocado test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Since this was the last test in tests/avocado/replay_kernel.py, we can remove that Avocado file now.

---
 MAINTAINERS                          |   1 -
 tests/avocado/replay_kernel.py       | 110 ---------------------------
 tests/functional/meson.build         |   1 +
 tests/functional/test_i386_replay.py |  28 +++++++
 4 files changed, 29 insertions(+), 111 deletions(-)
 delete mode 100644 tests/avocado/replay_kernel.py
 create mode 100755 tests/functional/test_i386_replay.py
MAINTAINERS +0 -1
index af1d847f8ed..42348df9d68 100644
@@ -3669,7 +3669,6 @@ F: include/system/replay.h
 F: docs/devel/replay.rst
 F: docs/system/replay.rst
 F: stubs/replay.c
-F: tests/avocado/replay_kernel.py
 F: tests/avocado/replay_linux.py
 F: tests/functional/*reverse_debug*.py
 F: tests/functional/*replay*.py
tests/avocado/replay_kernel.py deleted +0 -110

110 lines not on this page · raw message

tests/functional/meson.build +1 -0
index 7b0f4ab0b1d..4113b221de7 100644
@@ -151,6 +151,7 @@ tests_i386_system_quick = [
 ]
 
 tests_i386_system_thorough = [
+  'i386_replay',
   'i386_tuxrun',
 ]
 
tests/functional/test_i386_replay.py new file +28 -0
new file mode 100755
index 00000000000..7c4c2602da9
@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+#
+# Replay test that boots a Linux kernel on a i386 machine
+# and checks the console
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+from qemu_test import Asset
+from replay_kernel import ReplayKernelBase
+
+
+class I386Replay(ReplayKernelBase):
+
+    ASSET_KERNEL = Asset(
+         'https://storage.tuxboot.com/20230331/i386/bzImage',
+        'a3e5b32a354729e65910f5a1ffcda7c14a6c12a55e8213fb86e277f1b76ed956')
+
+    def test_pc(self):
+        self.set_machine('pc')
+        kernel_url = ()
+        kernel_path = self.ASSET_KERNEL.fetch()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        console_pattern = 'VFS: Cannot open root device'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)
+
+
+if __name__ == '__main__':
+    ReplayKernelBase.main()
Signature · 1 line
2.49.0
#6 lore raw

PATCH v2 5/15 tests/avocado: Remove the LinuxKernelTest class

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

All tests that used this class have been converted to the functional framework, so we can remove the boot_linux_console.py file now.

---
 tests/avocado/boot_linux_console.py | 62 -----------------------------
 1 file changed, 62 deletions(-)
 delete mode 100644 tests/avocado/boot_linux_console.py
tests/avocado/boot_linux_console.py deleted +0 -62
deleted file mode 100644
index cbb1e2fb506..00000000000
@@ -1,62 +0,0 @@
-# Functional test that boots a Linux kernel and checks the console
-#
-# Copyright (c) 2018 Red Hat, Inc.
-#
-# Author:
-#  Cleber Rosa <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-import os
-import shutil
-
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import process
-from avocado.utils import archive
-
-class LinuxKernelTest(QemuSystemTest):
-    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
-
-    def wait_for_console_pattern(self, success_message, vm=None):
-        wait_for_console_pattern(self, success_message,
-                                 failure_message='Kernel panic - not syncing',
-                                 vm=vm)
-
-    def extract_from_deb(self, deb, path):
-        """
-        Extracts a file from a deb package into the test workdir
-
-        :param deb: path to the deb archive
-        :param path: path within the deb archive of the file to be extracted
-        :returns: path of the extracted file
-        """
-        cwd = os.getcwd()
-        os.chdir(self.workdir)
-        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
-        process.run("ar x %s %s" % (deb, file_path))
-        archive.extract(file_path, self.workdir)
-        os.chdir(cwd)
-        # Return complete path to extracted file.  Because callers to
-        # extract_from_deb() specify 'path' with a leading slash, it is
-        # necessary to use os.path.relpath() as otherwise os.path.join()
-        # interprets it as an absolute path and drops the self.workdir part.
-        return os.path.normpath(os.path.join(self.workdir,
-                                             os.path.relpath(path, '/')))
-
-    def extract_from_rpm(self, rpm, path):
-        """
-        Extracts a file from an RPM package into the test workdir.
-
-        :param rpm: path to the rpm archive
-        :param path: path within the rpm archive of the file to be extracted
-                     needs to be a relative path (starting with './') because
-                     cpio(1), which is used to extract the file, expects that.
-        :returns: path of the extracted file
-        """
-        cwd = os.getcwd()
-        os.chdir(self.workdir)
-        process.run("rpm2cpio %s | cpio -id %s" % (rpm, path), shell=True)
-        os.chdir(cwd)
-        return os.path.normpath(os.path.join(self.workdir, path))
Signature · 1 line
2.49.0
#7 lore raw

PATCH v2 6/15 tests/functional: Convert the 32-bit big endian Wheezy mips test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

The test checks some entries in /proc and the output of some commands ... we put these checks into exportable functions now so that they can be reused more easily.

Additionally the linux_ssh_mips_malta.py uses SSH to test the networking of the guest. Since we don't have a SSH module in the functional framework yet, let's use the check_http_download() function here instead.

And while we're at it, also switch the NIC to e1000 now to get some more test coverage, since the "pcnet" device is already tested in the test test_mips_malta_cpio.

---
 tests/avocado/linux_ssh_mips_malta.py |   8 --
 tests/functional/meson.build          |   2 +-
 tests/functional/test_mips_malta.py   | 108 +++++++++++++++++++++++++-
 3 files changed, 107 insertions(+), 11 deletions(-)
tests/avocado/linux_ssh_mips_malta.py +0 -8
index d9bb525ad9c..73d294ad5cf 100644
@@ -172,14 +172,6 @@ def check_mips_malta(self, uname_m, endianess):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
-    def test_mips_malta32eb_kernel3_2_0(self):
-        """
-        :avocado: tags=arch:mips
-        :avocado: tags=endian:big
-        :avocado: tags=device:pcnet32
-        """
-        self.check_mips_malta('mips', 'be')
-
     def test_mips_malta32el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mipsel
tests/functional/meson.build +1 -1
index 4113b221de7..4dedfc7b9f7 100644
@@ -39,7 +39,7 @@ test_timeouts = {
   'arm_tuxrun' : 240,
   'arm_sx1' : 360,
   'intel_iommu': 300,
-  'mips_malta' : 120,
+  'mips_malta' : 480,
   'mipsel_replay' : 480,
   'mips64el_replay' : 180,
   'netdev_ethtool' : 180,
tests/functional/test_mips_malta.py +106 -2

129 lines not on this page · raw message

Signature · 1 line
2.49.0
#8 lore raw

PATCH v2 7/15 tests/functional: Convert the 32-bit little endian Wheezy mips test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Reuse the test function from the big endian test to easily convert the 32-bit little endian Wheezy mips test.

---
 tests/avocado/linux_ssh_mips_malta.py |  8 --------
 tests/functional/meson.build          |  1 +
 tests/functional/test_mipsel_malta.py | 22 ++++++++++++++++++++++
 3 files changed, 23 insertions(+), 8 deletions(-)
tests/avocado/linux_ssh_mips_malta.py +0 -8
index 73d294ad5cf..c1300aec961 100644
@@ -172,14 +172,6 @@ def check_mips_malta(self, uname_m, endianess):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
-    def test_mips_malta32el_kernel3_2_0(self):
-        """
-        :avocado: tags=arch:mipsel
-        :avocado: tags=endian:little
-        :avocado: tags=device:pcnet32
-        """
-        self.check_mips_malta('mips', 'le')
-
     def test_mips_malta64eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64
tests/functional/meson.build +1 -0
index 4dedfc7b9f7..cba0abf2920 100644
@@ -40,6 +40,7 @@ test_timeouts = {
   'arm_sx1' : 360,
   'intel_iommu': 300,
   'mips_malta' : 480,
+  'mipsel_malta' : 420,
   'mipsel_replay' : 480,
   'mips64el_replay' : 180,
   'netdev_ethtool' : 180,
tests/functional/test_mipsel_malta.py +22 -0
index fe9c3a172e6..9ee2884da8e 100755
@@ -13,6 +13,8 @@
 from qemu_test import interrupt_interactive_console_until_pattern
 from qemu_test import wait_for_console_pattern
 
+from test_mips_malta import mips_check_wheezy
+
 
 class MaltaMachineConsole(LinuxKernelTest):
 
@@ -57,6 +59,26 @@ def test_mips_malta32el_nanomips_16k_up(self):
     def test_mips_malta32el_nanomips_64k_dbg(self):
         self.do_test_mips_malta32el_nanomips(self.ASSET_KERNEL_64K)
 
+    ASSET_WHEEZY_KERNEL = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mipsel/'
+         'vmlinux-3.2.0-4-4kc-malta'),
+        'dc8a3648305b0201ca7a5cd135fe2890067a65d93c38728022bb0e656ad2bf9a')
+
+    ASSET_WHEEZY_DISK = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mipsel/'
+         'debian_wheezy_mipsel_standard.qcow2'),
+        '454f09ae39f7e6461c84727b927100d2c7813841f2a0a5dce328114887ecf914')
+
+    def test_wheezy(self):
+        kernel_path = self.ASSET_WHEEZY_KERNEL.fetch()
+        image_path = self.ASSET_WHEEZY_DISK.fetch()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 root=/dev/sda1')
+        mips_check_wheezy(self,
+            kernel_path, image_path, kernel_command_line,
+            dl_file='/boot/initrd.img-3.2.0-4-4kc-malta',
+            hsum='9fc9f250ed56a74e35e704ddfd5a1c5a5625adefc5c9da91f649288d3ca000f0')
+
 
 class MaltaMachineYAMON(QemuSystemTest):
 
Signature · 1 line
2.49.0
#9 lore raw

PATCH v2 8/15 tests/functional: Convert the 64-bit little endian Wheezy mips test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Reuse the test function from the 32-bit big endian test to easily convert the 64-bit little endian Wheezy mips test.

---
 tests/avocado/linux_ssh_mips_malta.py   |  8 --------
 tests/functional/meson.build            |  1 +
 tests/functional/test_mips64el_malta.py | 22 ++++++++++++++++++++++
 3 files changed, 23 insertions(+), 8 deletions(-)
tests/avocado/linux_ssh_mips_malta.py +0 -8
index c1300aec961..2fa5cf9a6c1 100644
@@ -179,11 +179,3 @@ def test_mips_malta64eb_kernel3_2_0(self):
         :avocado: tags=device:pcnet32
         """
         self.check_mips_malta('mips64', 'be')
-
-    def test_mips_malta64el_kernel3_2_0(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=endian:little
-        :avocado: tags=device:pcnet32
-        """
-        self.check_mips_malta('mips64', 'le')
tests/functional/meson.build +1 -0
index cba0abf2920..a4a317115b3 100644
@@ -42,6 +42,7 @@ test_timeouts = {
   'mips_malta' : 480,
   'mipsel_malta' : 420,
   'mipsel_replay' : 480,
+  'mips64el_malta' : 420,
   'mips64el_replay' : 180,
   'netdev_ethtool' : 180,
   'ppc_40p' : 240,
tests/functional/test_mips64el_malta.py +22 -0
index a8da15a26b3..dd37212f9df 100755
@@ -16,6 +16,8 @@
 from qemu_test import exec_command_and_wait_for_pattern
 from qemu_test import skipIfMissingImports, skipFlakyTest, skipUntrustedTest
 
+from test_mips_malta import mips_check_wheezy
+
 
 class MaltaMachineConsole(LinuxKernelTest):
 
@@ -90,6 +92,26 @@ def test_mips64el_malta_5KEc_cpio(self):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
+    ASSET_WHEEZY_KERNEL = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mipsel/'
+         'vmlinux-3.2.0-4-5kc-malta'),
+        '5e8b725244c59745bb8b64f5d8f49f25fecfa549f3395fb6d19a3b9e5065b85b')
+
+    ASSET_WHEEZY_DISK = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mipsel/'
+         'debian_wheezy_mipsel_standard.qcow2'),
+        '454f09ae39f7e6461c84727b927100d2c7813841f2a0a5dce328114887ecf914')
+
+    def test_wheezy(self):
+        kernel_path = self.ASSET_WHEEZY_KERNEL.fetch()
+        image_path = self.ASSET_WHEEZY_DISK.fetch()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 root=/dev/sda1')
+        mips_check_wheezy(self,
+            kernel_path, image_path, kernel_command_line, cpuinfo='MIPS 20Kc',
+            dl_file='/boot/initrd.img-3.2.0-4-5kc-malta',
+            hsum='7579f8b56c1187c7c04d0dc3c0c56c7a6314c5ddd3a9bf8803ecc7cf8a3be9f8')
+
 
 @skipIfMissingImports('numpy', 'cv2')
 class MaltaMachineFramebuffer(LinuxKernelTest):
Signature · 1 line
2.49.0
#10 lore raw

PATCH v2 9/15 tests/functional: Convert the 64-bit big endian Wheezy mips test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Reuse the test function from the 32-bit big endian test to easily convert the 64-bit big endian Wheezy mips test.

Since this was the last test in tests/avocado/linux_ssh_mips_malta.py, we can remove this avocado file now, too.

---
 MAINTAINERS                           |   1 -
 tests/avocado/linux_ssh_mips_malta.py | 181 --------------------------
 tests/functional/meson.build          |   2 +
 tests/functional/test_mips64_malta.py |  35 +++++
 4 files changed, 37 insertions(+), 182 deletions(-)
 delete mode 100644 tests/avocado/linux_ssh_mips_malta.py
 create mode 100755 tests/functional/test_mips64_malta.py
MAINTAINERS +0 -1
index 42348df9d68..72f6b208f5a 100644
@@ -1355,7 +1355,6 @@ F: hw/acpi/piix4.c
 F: hw/mips/malta.c
 F: hw/pci-host/gt64120.c
 F: include/hw/southbridge/piix.h
-F: tests/avocado/linux_ssh_mips_malta.py
 F: tests/functional/test_mips*_malta.py
 F: tests/functional/test_mips*_tuxrun.py
 
tests/avocado/linux_ssh_mips_malta.py deleted +0 -181

181 lines not on this page · raw message

tests/functional/meson.build +2 -0
index a4a317115b3..985ac5c27fe 100644
@@ -42,6 +42,7 @@ test_timeouts = {
   'mips_malta' : 480,
   'mipsel_malta' : 420,
   'mipsel_replay' : 480,
+  'mips64_malta' : 240,
   'mips64el_malta' : 420,
   'mips64el_replay' : 180,
   'netdev_ethtool' : 180,
@@ -191,6 +192,7 @@ tests_mipsel_system_thorough = [
 ]
 
 tests_mips64_system_thorough = [
+  'mips64_malta',
   'mips64_tuxrun',
 ]
 
tests/functional/test_mips64_malta.py new file +35 -0
new file mode 100755
index 00000000000..53c3e0c1221
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+#
+# Functional tests for the big-endian 64-bit MIPS Malta board
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+from qemu_test import LinuxKernelTest, Asset
+from test_mips_malta import mips_check_wheezy
+
+
+class MaltaMachineConsole(LinuxKernelTest):
+
+    ASSET_WHEEZY_KERNEL = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mips/'
+         'vmlinux-3.2.0-4-5kc-malta'),
+        '3e4ec154db080b3f1839f04dde83120654a33e5e1716863de576c47cb94f68f6')
+
+    ASSET_WHEEZY_DISK = Asset(
+        ('https://people.debian.org/~aurel32/qemu/mips/'
+         'debian_wheezy_mips_standard.qcow2'),
+        'de03599285b8382ad309309a6c4869f6c6c42a5cfc983342bab9ec0dfa7849a2')
+
+    def test_wheezy(self):
+        kernel_path = self.ASSET_WHEEZY_KERNEL.fetch()
+        image_path = self.ASSET_WHEEZY_DISK.fetch()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 root=/dev/sda1')
+        mips_check_wheezy(self,
+            kernel_path, image_path, kernel_command_line, cpuinfo='MIPS 20Kc',
+            dl_file='/boot/initrd.img-3.2.0-4-5kc-malta',
+            hsum='d98b953bb4a41c0fc0fd8d19bbc691c08989ac52568c1d3054d92dfd890d3f06')
+
+
+if __name__ == '__main__':
+    LinuxKernelTest.main()
Signature · 1 line
2.49.0
#11 lore raw

PATCH v2 10/15 tests/avocado: Remove the boot_linux.py tests

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

These tests are based on the cloudinit functions from Avocado. The cloudinit is very, very slow compared to our other tests, so most of these Avocado tests have either been disabled by default with a decorator, or have been marked to only run with KVM.

We won't include this sluggish cloudinit stuff in the functional framework, and we've already got plenty of other tests there that check pretty much the same things, so let's simply get rid of these old tests now.

---
 tests/avocado/boot_linux.py | 132 ------------------------------------
 1 file changed, 132 deletions(-)
 delete mode 100644 tests/avocado/boot_linux.py
tests/avocado/boot_linux.py deleted +0 -132
deleted file mode 100644
index a029ef4ad1e..00000000000
@@ -1,132 +0,0 @@
-# Functional test that boots a complete Linux system via a cloud image
-#
-# Copyright (c) 2018-2020 Red Hat, Inc.
-#
-# Author:
-#  Cleber Rosa <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2 or
-# later.  See the COPYING file in the top-level directory.
-
-import os
-
-from avocado_qemu.linuxtest import LinuxTest
-from avocado_qemu import BUILD_DIR
-
-from avocado import skipUnless
-
-
-class BootLinuxX8664(LinuxTest):
-    """
-    :avocado: tags=arch:x86_64
-    """
-    timeout = 480
-
-    def test_pc_i440fx_tcg(self):
-        """
-        :avocado: tags=machine:pc
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-    def test_pc_i440fx_kvm(self):
-        """
-        :avocado: tags=machine:pc
-        :avocado: tags=accel:kvm
-        """
-        self.require_accelerator("kvm")
-        self.vm.add_args("-accel", "kvm")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-    def test_pc_q35_tcg(self):
-        """
-        :avocado: tags=machine:q35
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-    def test_pc_q35_kvm(self):
-        """
-        :avocado: tags=machine:q35
-        :avocado: tags=accel:kvm
-        """
-        self.require_accelerator("kvm")
-        self.vm.add_args("-accel", "kvm")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-
-# For Aarch64 we only boot KVM tests in CI as booting the current
-# Fedora OS in TCG tests is very heavyweight. There are lighter weight
-# distros which we use in the machine_aarch64_virt.py tests.
-class BootLinuxAarch64(LinuxTest):
-    """
-    :avocado: tags=arch:aarch64
-    :avocado: tags=machine:virt
-    """
-    timeout = 720
-
-    def test_virt_kvm(self):
-        """
-        :avocado: tags=accel:kvm
-        :avocado: tags=cpu:host
-        """
-        self.require_accelerator("kvm")
-        self.vm.add_args("-accel", "kvm")
-        self.vm.add_args("-machine", "virt,gic-version=host")
-        self.vm.add_args('-bios',
-                         os.path.join(BUILD_DIR, 'pc-bios',
-                                      'edk2-aarch64-code.fd'))
-        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
-        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-
-# See the tux_baseline.py tests for almost the same coverage in a lot
-# less time.
-class BootLinuxPPC64(LinuxTest):
-    """
-    :avocado: tags=arch:ppc64
-    """
-
-    timeout = 360
-
-    @skipUnless(os.getenv('SPEED') == 'slow', 'runtime limited')
-    def test_pseries_tcg(self):
-        """
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-    def test_pseries_kvm(self):
-        """
-        :avocado: tags=machine:pseries
-        :avocado: tags=accel:kvm
-        """
-        self.require_accelerator("kvm")
-        self.vm.add_args("-accel", "kvm")
-        self.vm.add_args("-machine", "cap-ccf-assist=off")
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-class BootLinuxS390X(LinuxTest):
-    """
-    :avocado: tags=arch:s390x
-    """
-
-    timeout = 240
-
-    @skipUnless(os.getenv('SPEED') == 'slow', 'runtime limited')
-    def test_s390_ccw_virtio_tcg(self):
-        """
-        :avocado: tags=machine:s390-ccw-virtio
-        :avocado: tags=accel:tcg
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.launch_and_wait(set_up_ssh_connection=False)
Signature · 1 line
2.49.0
#12 lore raw

PATCH v2 11/15 tests/functional: Use the tuxrun kernel for the x86 replay test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

This way we can do a full boot in record-replay mode and should get a similar test coverage compared to the old replay test from tests/avocado/replay_linux.py. Thus remove the x86 avocado replay_linux test now.

---
 tests/avocado/replay_linux.py          | 46 --------------------------
 tests/functional/test_x86_64_replay.py | 43 ++++++++++++++++++------
 2 files changed, 33 insertions(+), 56 deletions(-)
tests/avocado/replay_linux.py +0 -46
index 59169224353..6ba283d3bf6 100644
@@ -118,52 +118,6 @@ def run_replay_dump(self, replay_path):
         except subprocess.CalledProcessError:
             self.fail('replay-dump.py failed')
 
-@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
-class ReplayLinuxX8664(ReplayLinux):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=accel:tcg
-    """
-
-    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
-
-    def test_pc_i440fx(self):
-        """
-        :avocado: tags=machine:pc
-        """
-        self.run_rr(shift=1)
-
-    def test_pc_q35(self):
-        """
-        :avocado: tags=machine:q35
-        """
-        self.run_rr(shift=3)
-
-@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
-class ReplayLinuxX8664Virtio(ReplayLinux):
-    """
-    :avocado: tags=arch:x86_64
-    :avocado: tags=virtio
-    :avocado: tags=accel:tcg
-    """
-
-    hdd = 'virtio-blk-pci'
-    cd = 'virtio-blk-pci'
-    bus = None
-
-    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
-
-    def test_pc_i440fx(self):
-        """
-        :avocado: tags=machine:pc
-        """
-        self.run_rr(shift=1)
-
-    def test_pc_q35(self):
-        """
-        :avocado: tags=machine:q35
-        """
-        self.run_rr(shift=3)
 
 @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
 class ReplayLinuxAarch64(ReplayLinux):
tests/functional/test_x86_64_replay.py +33 -10

63 lines not on this page · raw message

Signature · 1 line
2.49.0
#13 lore raw

PATCH v2 12/15 tests/functional: Use the tuxrun kernel for the aarch64 replay test

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

This way we can do a full boot in record-replay mode and should get a similar test coverage compared to the old replay test from tests/avocado/replay_linux.py.

Since the aarch64 test was the last avocado test in the tests/avocado/replay_linux.py file, we can remove this file now completely.

---
 MAINTAINERS                             |   1 -
 tests/avocado/replay_linux.py           | 160 ------------------------
 tests/functional/test_aarch64_replay.py |  37 ++++--
 3 files changed, 29 insertions(+), 169 deletions(-)
 delete mode 100644 tests/avocado/replay_linux.py
MAINTAINERS +0 -1
index 72f6b208f5a..e29910cbd77 100644
@@ -3668,7 +3668,6 @@ F: include/system/replay.h
 F: docs/devel/replay.rst
 F: docs/system/replay.rst
 F: stubs/replay.c
-F: tests/avocado/replay_linux.py
 F: tests/functional/*reverse_debug*.py
 F: tests/functional/*replay*.py
 F: qapi/replay.json
tests/avocado/replay_linux.py deleted +0 -160

160 lines not on this page · raw message

tests/functional/test_aarch64_replay.py +29 -8
index 029fef3cbf8..8afe95668e2 100755
@@ -5,27 +5,48 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 
-from qemu_test import Asset, skipIfOperatingSystem
+from subprocess import check_call, DEVNULL
+
+from qemu_test import Asset, skipIfOperatingSystem, get_qemu_img
 from replay_kernel import ReplayKernelBase
 
 
 class Aarch64Replay(ReplayKernelBase):
 
     ASSET_KERNEL = Asset(
-        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
-         'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
-        '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
+        'https://storage.tuxboot.com/buildroot/20241119/arm64/Image',
+        'b74743c5e89e1cea0f73368d24ae0ae85c5204ff84be3b5e9610417417d2f235')
+
+    ASSET_ROOTFS = Asset(
+        'https://storage.tuxboot.com/buildroot/20241119/arm64/rootfs.ext4.zst',
+        'a1acaaae2068df4648d04ff75f532aaa8c5edcd6b936122b6f0db4848a07b465')
 
     # Failing on Darwin: https://gitlab.com/qemu-project/qemu/-/issues/2907
     @skipIfOperatingSystem('Darwin')
     def test_aarch64_virt(self):
+        self.require_netdev('user')
         self.set_machine('virt')
-        self.cpu = 'cortex-a53'
+        self.cpu = 'cortex-a57'
         kernel_path = self.ASSET_KERNEL.fetch()
+
+        raw_disk = self.uncompress(self.ASSET_ROOTFS)
+        disk = self.scratch_file('scratch.qcow2')
+        qemu_img = get_qemu_img(self)
+        check_call([qemu_img, 'create', '-f', 'qcow2', '-b', raw_disk,
+                    '-F', 'raw', disk], stdout=DEVNULL, stderr=DEVNULL)
+
+        args = ('-drive', 'file=%s,snapshot=on,id=hd0,if=none' % disk,
+                '-drive', 'driver=blkreplay,id=hd0-rr,if=none,image=hd0',
+                '-device', 'virtio-blk-device,drive=hd0-rr',
+                '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22',
+                '-device', 'virtio-net,netdev=vnet',
+                '-object', 'filter-replay,id=replay,netdev=vnet')
+
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
-                               'console=ttyAMA0')
-        console_pattern = 'VFS: Cannot open root device'
-        self.run_rr(kernel_path, kernel_command_line, console_pattern)
+                               'console=ttyAMA0 root=/dev/vda')
+        console_pattern = 'Welcome to TuxTest'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+                    args=args)
 
 
 if __name__ == '__main__':
Signature · 1 line
2.49.0
#14 lore raw

PATCH v2 13/15 tests/functional: Convert the SMMU test to the functional framework

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

This test was using cloudinit and a "dnf install" command in the guest to exercise the NIC with SMMU enabled. Since we don't have the cloudinit stuff in the functional framework and we should not rely on having access to external networks (once our ASSETs have been cached), we rather boot into the initrd first, manually mount the root disk and then use the check_http_download() function from the functional framework here instead for testing whether the network works as expected.

Unfortunately, there seems to be a small race when using the files from Fedora 33: To enter the initrd shell, we have to send a "return" once. But it does not seem to work if we send it too early. Using a sleep(0.2) makes it work reliably for me, but to make it even more unlikely to trigger this situation, let's better limit the Fedora 33 tests to only run with KVM.

Finally, while we're at it, we also add some lines for testing writes to the hard disk, as we already do it in the test_intel_iommu test.

---
 MAINTAINERS                           |   2 +-
 tests/avocado/smmu.py                 | 139 -----------------
 tests/functional/meson.build          |   2 +
 tests/functional/test_aarch64_smmu.py | 205 ++++++++++++++++++++++++++
 4 files changed, 208 insertions(+), 140 deletions(-)
 delete mode 100644 tests/avocado/smmu.py
 create mode 100755 tests/functional/test_aarch64_smmu.py
MAINTAINERS +1 -1
index e29910cbd77..07f77c048e2 100644
@@ -211,7 +211,7 @@ L: [email protected]
 S: Maintained
 F: hw/arm/smmu*
 F: include/hw/arm/smmu*
-F: tests/avocado/smmu.py
+F: tests/functional/test_aarch64_smmu.py
 
 AVR TCG CPUs
 M: Michael Rolnik <[email protected]>
tests/avocado/smmu.py deleted +0 -139

139 lines not on this page · raw message

tests/functional/meson.build +2 -0
index 985ac5c27fe..b317ad42c5a 100644
@@ -18,6 +18,7 @@ test_timeouts = {
   'aarch64_rme_sbsaref' : 1200,
   'aarch64_sbsaref_alpine' : 1200,
   'aarch64_sbsaref_freebsd' : 720,
+  'aarch64_smmu' : 720,
   'aarch64_tuxrun' : 240,
   'aarch64_virt' : 360,
   'aarch64_virt_gpu' : 480,
@@ -88,6 +89,7 @@ tests_aarch64_system_thorough = [
   'aarch64_sbsaref',
   'aarch64_sbsaref_alpine',
   'aarch64_sbsaref_freebsd',
+  'aarch64_smmu',
   'aarch64_tcg_plugins',
   'aarch64_tuxrun',
   'aarch64_virt',
tests/functional/test_aarch64_smmu.py new file +205 -0

205 lines not on this page · raw message

Signature · 1 line
2.49.0
#15 lore raw

PATCH v2 14/15 gitlab-ci: Update QEMU_JOB_AVOCADO and QEMU_CI_AVOCADO_TESTING

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Since we don't run the Avocado jobs in the CI anymore, rename these variables to QEMU_JOB_FUNCTIONAL and QEMU_CI_FUNCTIONAL.

Also, there was a mismatch between the documentation and the implementation of QEMU_CI_AVOCADO_TESTING: While the documentation said that you had to "Set this variable to have the tests using the Avocado framework run automatically", you indeed needed to set it to make the pipelines appear in your dashboard - but they were never run automatically in forks and had to be triggered manually. Let's improve this now: No need to hide these pipelines from the users by default anymore (the functional tests should be stable enough nowadays), and rather allow the users to run the pipelines auto- matically with this switch now instead, as was documented.

---
 docs/devel/testing/ci-jobs.rst.inc  | 19 +++++++++----------
 .gitlab-ci.d/base.yml               |  8 ++------
 .gitlab-ci.d/buildtest-template.yml |  2 +-
 3 files changed, 12 insertions(+), 17 deletions(-)
docs/devel/testing/ci-jobs.rst.inc +9 -10
index 3756bbe3554..f1c541cc25b 100644
@@ -126,10 +126,10 @@ QEMU_JOB_PUBLISH
 The job is for publishing content after a branch has been
 merged into the upstream default branch.
 
-QEMU_JOB_AVOCADO
-~~~~~~~~~~~~~~~~
+QEMU_JOB_FUNCTIONAL
+~~~~~~~~~~~~~~~~~~~
 
-The job runs the Avocado integration test suite
+The job runs the functional test suite
 
 Contributor controlled runtime variables
 ----------------------------------------
@@ -149,13 +149,12 @@ the jobs to be manually started from the UI
 Set this variable to 2 to create the pipelines and run all
 the jobs immediately, as was the historical behaviour
 
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
+QEMU_CI_FUNCTIONAL
+~~~~~~~~~~~~~~~~~~
+By default, tests using the functional framework are not run automatically
+in the pipelines (because multiple artifacts have to be downloaded, which
+might cause a lot of network traffic). Set this variable to have the tests
+using the functional framework run automatically.
 
 Other misc variables
 --------------------
.gitlab-ci.d/base.yml +2 -6
index 25b88aaa06a..60a24a9d14d 100644
@@ -69,10 +69,6 @@ variables:
     - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: never
 
-    # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
-    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
-      when: never
-
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
@@ -101,8 +97,8 @@ variables:
       when: manual
       allow_failure: true
 
-    # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
-    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
+    # Functional jobs can be manually started in forks
+    - if: '$QEMU_JOB_FUNCTIONAL && $QEMU_CI_FUNCTIONAL != "1" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
       allow_failure: true
 
.gitlab-ci.d/buildtest-template.yml +1 -1
index 13fa4f4a4fc..d4f145fdb58 100644
@@ -114,4 +114,4 @@
     - cd build
     - du -chs ${CI_PROJECT_DIR}/*-cache
   variables:
-    QEMU_JOB_AVOCADO: 1
+    QEMU_JOB_FUNCTIONAL: 1
Signature · 1 line
2.49.0
#16 lore raw

PATCH v2 15/15 Remove the remainders of the Avocado tests

In reply to Thomas Huth

From: Thomas Huth <[email protected]>

Now that all Avocado tests have been converted to or been replaced by other functional tests, we can delete the remainders of the Avocado tests from the QEMU source tree.

---
 MAINTAINERS                             |   8 +-
 docs/about/build-platforms.rst          |  10 +-
 docs/devel/build-system.rst             |  11 +-
 docs/devel/codebase.rst                 |   5 -
 docs/devel/testing/avocado.rst          | 581 ------------------------
 docs/devel/testing/functional.rst       |   3 -
 docs/devel/testing/index.rst            |   1 -
 docs/devel/testing/main.rst             |  15 -
 tests/avocado/README.rst                |  10 -
 configure                               |   2 +-
 pythondeps.toml                         |   8 +-
 tests/Makefile.include                  |  60 +--
 tests/avocado/avocado_qemu/__init__.py  | 424 -----------------
 tests/avocado/avocado_qemu/linuxtest.py | 253 -----------
 14 files changed, 16 insertions(+), 1375 deletions(-)
 delete mode 100644 docs/devel/testing/avocado.rst
 delete mode 100644 tests/avocado/README.rst
 delete mode 100644 tests/avocado/avocado_qemu/__init__.py
 delete mode 100644 tests/avocado/avocado_qemu/linuxtest.py
MAINTAINERS +1 -7
index 07f77c048e2..5fd757c5ddb 100644
@@ -2073,7 +2073,7 @@ S: Supported
 F: hw/acpi/viot.c
 F: hw/acpi/viot.h
 
-ACPI/AVOCADO/BIOSBITS
+ACPI/FUNCTIONAL/BIOSBITS
 M: Ani Sinha <[email protected]>
 M: Michael S. Tsirkin <[email protected]>
 S: Supported
@@ -4246,12 +4246,6 @@ R: Philippe Mathieu-Daudé <[email protected]>
 S: Maintained
 F: tests/tcg/Makefile.target
 
-Integration Testing with the Avocado framework
-W: https://trello.com/b/6Qi1pxVn/avocado-qemu
-R: Cleber Rosa <[email protected]>
-S: Odd Fixes
-F: tests/avocado/
-
 GitLab custom runner (Works On Arm Sponsored)
 M: Alex Bennée <[email protected]>
 M: Philippe Mathieu-Daudé <[email protected]>
docs/about/build-platforms.rst +5 -5
index 1552b1a7044..52521552c8a 100644
@@ -123,11 +123,11 @@ Rust build dependencies
   to build QEMU.
 
 Optional build dependencies
-  Build components whose absence does not affect the ability to build
-  QEMU may not be available in distros, or may be too old for QEMU's
-  requirements.  Many of these, such as the Avocado testing framework
-  or various linters, are written in Python and therefore can also
-  be installed using ``pip``.  Cross compilers are another example
+  Build components whose absence does not affect the ability to build QEMU
+  may not be available in distros, or may be too old for our requirements.
+  Many of these, such as additional modules for the functional testing
+  framework or various linters, are written in Python and therefore can
+  also be installed using ``pip``.  Cross compilers are another example
   of optional build-time dependency; in this case it is possible to
   download them from repositories such as EPEL, to use container-based
   cross compilation using ``docker`` or ``podman``, or to use pre-built
docs/devel/build-system.rst +3 -8
index a759982f45c..258cfad3fe7 100644
@@ -134,7 +134,7 @@ in how the build process runs Python code.
 
 At this stage, ``configure`` also queries the chosen Python interpreter
 about QEMU's build dependencies.  Note that the build process does  *not*
-look for ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
+look for ``meson`` or ``sphinx-build`` binaries in the PATH;
 likewise, there are no options such as ``--meson`` or ``--sphinx-build``.
 This avoids a potential mismatch, where Meson and Sphinx binaries on the
 PATH might operate in a different Python environment than the one chosen
@@ -151,7 +151,7 @@ virtual environment with ``pip``, either from wheels in ``python/wheels``
 or by downloading the package with PyPI.  Downloading can be disabled with
 ``--disable-download``; and anyway, it only happens when a ``configure``
 option (currently, only ``--enable-docs``) is explicitly enabled but
-the dependencies are not present\ [#pip]_.
+the dependencies are not present.
 
 .. [#distlib] The scripts are created based on the package's metadata,
               specifically the ``console_script`` entry points.  This is the
@@ -164,10 +164,6 @@ the dependencies are not present\ [#pip]_.
               because the Python Packaging Authority provides a package
               ``distlib.scripts`` to perform this task.
 
-.. [#pip] ``pip`` might also be used when running ``make check-avocado``
-           if downloading is enabled, to ensure that Avocado is
-           available.
-
 The required versions of the packages are stored in a configuration file
 ``pythondeps.toml``.  The format is custom to QEMU, but it is documented
 at the top of the file itself and it should be easy to understand.  The
@@ -497,8 +493,7 @@ number of dynamically created files listed later.
   ``pyvenv/bin``, and calling ``pip`` to install dependencies.
 
 ``tests/Makefile.include``
-  Rules for external test harnesses. These include the TCG tests
-  and the Avocado-based integration tests.
+  Rules for external test harnesses like the TCG tests.
 
 ``tests/docker/Makefile.include``
   Rules for Docker tests. Like ``tests/Makefile.include``, this file is
docs/devel/codebase.rst +0 -5
index 1b09953197b..1da17f40494 100644
@@ -175,11 +175,6 @@ yet, so sometimes the source code is all you have.
 * `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_:
   QEMU `test <testing>` suite
 
-  - `avocado <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/avocado>`_:
-    Functional tests booting full VM using `Avocado framework <checkavocado-ref>`.
-    Those tests will be transformed and moved into
-    `tests/functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_
-    in the future.
   - `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_:
     Data for various tests.
   - `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_:
docs/devel/testing/avocado.rst deleted +0 -581

581 lines not on this page · raw message

docs/devel/testing/functional.rst +0 -3
index 9bc973392aa..8030cb4299f 100644
@@ -6,9 +6,6 @@ Functional testing with Python
 The ``tests/functional`` directory hosts functional tests written in
 Python. They are usually higher level tests, and may interact with
 external resources and with various guest operating systems.
-The functional tests have initially evolved from the Avocado tests, so there
-is a lot of similarity to those tests here (see :ref:`checkavocado-ref` for
-details about the Avocado tests).
 
 The tests should be written in the style of the Python `unittest`_ framework,
 using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
docs/devel/testing/index.rst +0 -1
index 1171f7db8f0..ccc2fc6cbc1 100644
@@ -10,7 +10,6 @@ testing infrastructure.
    main
    qtest
    functional
-   avocado
    acpi-bits
    ci
    fuzzing
docs/devel/testing/main.rst +0 -15
index 9869bcf0341..3c03d769539 100644
@@ -894,21 +894,6 @@ Python. You can run the functional tests simply by executing:
 
 See :ref:`checkfunctional-ref` for more details.
 
-Integration tests using the Avocado Framework
----------------------------------------------
-
-The ``tests/avocado`` directory hosts integration tests. They're usually
-higher level tests, and may interact with external resources and with
-various guest operating systems.
-
-You can run the avocado tests simply by executing:
-
-.. code::
-
-  make check-avocado
-
-See :ref:`checkavocado-ref` for more details.
-
 .. _checktcg-ref:
 
 Testing with "make check-tcg"
tests/avocado/README.rst deleted +0 -10
deleted file mode 100644
index 94488371bbe..00000000000
@@ -1,10 +0,0 @@
-=============================================
-Integration tests using the Avocado Framework
-=============================================
-
-This directory contains integration tests. They're usually higher
-level, and may interact with external resources and with various
-guest operating systems.
-
-For more information, please refer to ``docs/devel/testing.rst``,
-section "Integration tests using the Avocado Framework".
configure +1 -1
index 02f1dd2311f..000309cf610 100755
@@ -1685,7 +1685,7 @@ LINKS="$LINKS pc-bios/optionrom/Makefile"
 LINKS="$LINKS pc-bios/s390-ccw/Makefile"
 LINKS="$LINKS pc-bios/vof/Makefile"
 LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
-LINKS="$LINKS tests/avocado tests/data"
+LINKS="$LINKS tests/data"
 LINKS="$LINKS tests/qemu-iotests/check tests/qemu-iotests/Makefile"
 LINKS="$LINKS python"
 for f in $LINKS ; do
pythondeps.toml +2 -6
index c03c9df81b5..7eaaa0fed10 100644
@@ -27,9 +27,5 @@ pycotap = { accepted = ">=1.1.0", installed = "1.3.1" }
 sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
-[avocado]
-# Note that qemu.git/python/ is always implicitly installed.
-# Prefer an LTS version when updating the accepted versions of
-# avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
-pycdlib = { accepted = ">=1.11.0" }
+[testdeps]
+qemu.qmp = { accepted = ">=0.0.3", installed = "0.0.3" }
tests/Makefile.include +4 -56

104 lines not on this page · raw message

tests/avocado/avocado_qemu/__init__.py deleted +0 -424

424 lines not on this page · raw message

tests/avocado/avocado_qemu/linuxtest.py deleted +0 -253

253 lines not on this page · raw message

Signature · 1 line
2.49.0